sneak/quak
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Compare commits

base: sneak/quak:156fe871e89eac1ed8016bf3ab5b5e04d0829ffa
Branches Tags
sneak/quak:main sneak/quak:next sneak/quak:issue-217-files-one-collection
sneak/quak:v1.0.0-rc3 sneak/quak:v1.0.0-rc2 sneak/quak:v1.0.0-rc1
...
compare: sneak/quak:issue-217-files-one-collection
Branches Tags
sneak/quak:next sneak/quak:issue-217-files-one-collection sneak/quak:main
sneak/quak:v1.0.0-rc3 sneak/quak:v1.0.0-rc2 sneak/quak:v1.0.0-rc1
117 Commits
156fe871e8 ... issue-217-files-one-collection
Author SHA1 Message Date
sneak ec07fd1dc9 quak files COLLECTION fetches that collection's files alone (closes #217)
check / check (push) Canceled after 0s
Details 
`quak files COLLECTION` used to wait for a full refresh, which on an
empty cache fetched the files of every collection first. It now reads
the collection list, then calls the new `Library.refreshCollection()`.
That brings one collection up to date in the store as a refresh would.
Once the stored cursor is past 0, it only updates a collection the store
holds. For a new one, `quak files COLLECTION` waits for a refresh. A
refresh from a cursor of 0 replaces all the store holds, so collections
then sit in the server's order. `open()` awaits the first refresh while
the cursor is 0. The store lists what did not decrypt in a fixed order.

Model: opus-5-5
 2026-10-26 05:46:57 +00:00
clawbot 657266a60e A file's JSON holds keywords in place of its CLIP embedding (closes #218)
check / check (push) Canceled after 0s
Details 
`quak backup` and `quak backup-metadata` write each file's ML data without its
CLIP embedding, and `keywords` in its place: the titles of the Discover
categories of Ente's photos app that the file falls in, highest score first. A
file falls in a category when the cosine similarity of its CLIP embedding and
the text embedding of the category's query reaches the category's minimum
score; as in Ente's app, five categories leave out videos.

The 16 text embeddings ship in `src/clip-keyword-vectors.ts`, written by
`bin/clip-keyword-vectors` from Ente's list and CLIP text model. quak runs no
model and gains no dependency.

Disclosure: the text embeddings are committed model output, against
REPO_POLICIES' rule on generated files.

Model: opus-5-5
 2026-10-22 23:32:25 +02:00
sneak ae0f72971d Merge branch 'main' into next
check / check (push) Canceled after 0s
Details
2026-10-22 18:55:37 +02:00
clawbot 68d22b9927 -v raises the CLI's log level per repeat; -q prints only fatal errors (closes #212)
check / check (push) Canceled after 0s
Details 
Every line quak prints on stderr now goes through one log, in src/cli-log.ts, at one of six levels. The default, info, prints what quak printed before. -v adds each retry and the start and end of each refresh and ML data fetch; -vv adds each request with its result. -q prints only fatal lines, and is refused with -v; both are program options. Every non-zero exit logs one fatal line, so a run that exits 1 because files failed says how many and where to look. Every line reporting a failure is at error. --json changes stdout only, so backup and --json runs log their progress on stderr. withRetry and ApiClient report retries and requests through new callbacks.

Model: opus-5-5
 2026-10-18 21:46:59 +02:00
clawbot 92e0c6c791 quak files takes the collection ID as an optional argument (closes #211)
check / check (push) Canceled after 0s
Details 
Mandatory arguments are positional, so `quak files` takes the collection ID as an optional argument and the `--collection` option is removed. With a collection ID it lists that collection's files. Without one it lists every file in the account, collection by collection, each file once: the library keeps a record of a file for each collection that holds it, and only the first collection's record is listed, so that collection is its `collectionID` in `--json`. The `--help` text, the README and `TODO.md` say so.

Model: opus-5-5
 2026-10-18 20:17:10 +02:00
clawbot 66e3ac77e8 Names quak builds from server data fit in 255 bytes (closes #205)
check / check (push) Canceled after 0s
Details 
The default names of `quak get`, `quak get-thumb`, `downloadFile` and `downloadThumbnail`, and the album folders of `backup-metadata`, could be over 255 bytes, which a file system refuses. `defaultFileName` and `shortenKeepingExtension` in `src/filename.ts` now cut such a name short, keeping its extension when it fits; a live photo's image and video get their own extension before the cut. A cut name never starts with a dot and is never a Windows device name. `--out` and `outPath` are kept as given. A `backup-metadata` folder keeps its `<id>-`. `safeExtension` gives `.bin` for an extension of more than 16 letters and digits, so a save path's name always fits.

Model: opus-5-5
 2026-10-18 20:01:34 +02:00
clawbot 982b878d69 quak collections reads the collection list alone (closes #210)
check / check (push) Canceled after 0s
Details 
On an empty cache the read commands opened their library with a refresh that fetched and decrypted the files of every collection in turn, printing nothing and storing nothing until it had them all, so `quak collections` seemed to hang on a large account. The new `lib.freshCollections()` is the fresh read of the collection list alone: one request, and nothing stored, so a later refresh still fetches every collection's files. The read commands open their library with the new `refreshOnOpen: false` and make only the refresh they ask for. A refresh sends a "progress" event after each collection whose files it fetched; the commands that wait on one print it on stderr, except with `--json`.

Model: opus-5-5
 2026-10-18 19:01:34 +02:00
clawbot 7ec71bf7ec ML data that does not decrypt is recorded with its reason (closes #198)
check / check (push) Canceled after 0s
Details 
`fetchMLDataBatch` dropped an entry that did not decrypt, gunzip or parse,
so the file looked like one with no ML data and the run exited 0. It now
returns the reason beside the payloads, also for ML data that is not a
JSON object. The library gives the reasons from its last ML data fetch
that succeeded in `status().mlDataErrors` and the `fetchMLData` "done"
event; a fetch that fails leaves them as they were. `quak backup` writes
the reason as `mlDataError` and records the file in `failures.json`;
`backup-metadata` writes `mlDataError` and exits 1.

Judgement call: a file whose ML data is cached keeps it in its JSON when a
newer copy does not decode, as when the fetch fails.

Model: opus-5-5
 2026-10-16 07:23:16 +02:00
clawbot 8583863ec0 --verify keeps a mismatched original until a matching copy replaces it (closes #199)
check / check (push) Canceled after 0s
Details 
With --verify, an original whose hash does not match is no longer removed
before its replacement is downloaded. replaceOriginal, beside placeOriginal
in src/library/content.ts, has the content cache's new fetchOriginal write
the replacement to a temporary file beside the original, hashes it, and
renames it over the original only when it matches; a live photo's image and
video are both hashed before either is renamed. A failed or mismatching
download leaves the old bytes and album links, removes the temporary files,
and lands in failures.json. The replacement is always downloaded, never
copied from the cache. The leftover-file cleanup also takes a live photo's
replacement files, .quak-<pid>-<random>.<ext>.

Model: opus-5-5
 2026-10-16 06:56:19 +02:00
clawbot 960a7ce8a6 An album whose folder cannot be made no longer stops the backup (closes #197)
check / check (push) Canceled after 0s
Details 
Each album's folder, links and JSON are written inside one try. A failure
prints `FAILED album <name>: <reason>`, lists the album's ID in the new
`failedAlbums` of the result and of `--json`, and makes `quak backup` exit 1;
the other albums and `failures.json` are still written.

An album whose folder would have another album's JSON name, ignoring case,
gets its ID. A name over 255 bytes (250 for a folder, which leaves room for
`.json`) gets the ID and is cut short between two characters by
`shortenToBytes` in `src/filename.ts`. A link's ID goes right before its
extension.

Apart from its call to `uniqueNames`, the album loop is only indented into
the try.

Model: opus-5-5
 2026-10-16 05:23:27 +02:00
clawbot 20410f8f71 The binary make install builds retries a refused or unresolvable connection (closes #206)
check / check (push) Canceled after 0s
Details 
Bun's `fetch`, which `~/bin/quak` runs under, rejects with a plain `Error`
that has no `cause` and a code of bun's own. Running the built binary against
local listeners gave `ConnectionRefused` for a refused connection and a failed
TLS handshake, and `ConnectionRefused` or `FailedToOpenSocket` for a name that
does not resolve. Both join `TRANSPORT_CODES` in `src/retry.ts`, so they are
retried and called transient. They stay out of `CONNECT_CODES`, so a `POST` or
`PUT` is not replayed on them, because bun also gives them when it resends a
request on a new connection. `classify` in `src/backup.ts` records a
`TimeoutError` as transient, so a stalled download is no longer `unknown`.

Not changed: a certificate bun rejects is not retried.

Model: opus-5-5
 2026-10-16 04:56:20 +02:00
clawbot d016dbfaa7 A network failure is printed and recorded with its reason (closes #196)
check / check (push) Canceled after 0s
Details 
Node's `fetch` rejects every failure below HTTP as `fetch failed` and puts
the reason on `cause`. `describeError` in `src/errors.ts` gives the message
followed by the messages on the cause chain, and for an `AggregateError`,
which Node gives when every address of a host fails, the message of each
address. `run` prints it, and the backup logs and records it. `classify`
reads that text and calls a failure transient when `hasTransportCode`, now
exported by `src/retry.ts`, finds a transport errno on the chain. A failed
ML data fetch keeps its error as the cause. The bounded walk along the
chain moved from `src/retry.ts` to `src/errors.ts`.

A truncated download's message leaves the authentication failure to its
cause.

Model: opus-5-5
 2026-10-16 03:23:16 +02:00
clawbot 5a940d4e4e README says that make install needs nix (closes #200)
check / check (push) Canceled after 0s
Details 
`make install` runs `make build-bin`, which runs bun through
`nix-shell -p bun`, so on a machine without nix it stops with
`nix-shell: not found`, and the README named no such prerequisite.
Getting Started now says what its commands need and that `make install`
also needs nix, with a link to its download page. "Running the backup
from cron" says the same where it introduces `make install`, and that a
bun already installed does not help. The Makefile is unchanged, as the
plan on the issue decided. TODO.md's Next Step names the four bugs that
are open instead of saying none is.

Judgement call: "node 22 or later" is the `engines` entry in package.json.

Model: opus-5-5
 2026-10-16 01:40:19 +02:00
clawbot 3412e42af5 quak backup works on up to the content pool's limit of originals at once (closes #182)
check / check (push) Canceled after 0s
Details 
The backup awaited each original before starting the next, so it used
one of the content pool's five slots. lib.backup(), and so quak backup,
now runs as many workers as the content pool's limit (5 by default, or
the contentConcurrency of the pools the library was opened with), each
taking the next file from one shared list, so the limit covers the
--verify hashing too. Once a worker has thrown or the lock is lost, no
worker takes another file; the backup waits for the files under way and
checks its lock before the ML data fetch. runBackup rejects a
contentConcurrency that is not a positive integer before it takes the
lock. Failed files are listed in the order they failed.

Model: opus-5-5
 2026-10-14 09:06:59 +02:00
clawbot 22ee2db479 TODO.md's Next Step names the open issues, not a closed one (closes #194)
check / check (push) Canceled after 0s
Details 
Next Step said the cache design, issue 36, waits on sneak's review. That
issue closed on 2026-10-03. Next Step now names what is open: the backup
concurrency issue, built in pull request 192 and waiting on review, and
the live-account check that only sneak can run.

Model: opus-5-5
 2026-10-14 08:07:48 +02:00
clawbot e8e8e00887 A backup whose lock is lost stops with one line, not a stack trace (closes #184)
check / check (push) Canceled after 0s
Details 
proper-lockfile's default onCompromised throws from a timer, so a backup
whose backup.lock was removed or taken over by another backup died with
a stack trace. lockBackupDirectory now returns the lock, its release
function and why it was lost, and passes a handler that only records
the reason. The backup checks it after the refresh and before each
original, thumbnail, file's JSON and album, and before it writes
failures.json, and then fails with "lost the lock on DIR: REASON",
which quak backup prints as one line, exiting 1. lib.backup() takes a
caller's lock as lock, in place of lockHeld: true. Releasing a lost lock
leaves backup.lock alone, since it may be another backup's.

Model: opus-5-5
 2026-10-14 07:37:26 +02:00
clawbot bb3b5cd2da One album or file quak cannot decrypt no longer stops a refresh or a backup (closes #181)
check / check (push) Canceled after 0s
Details 
A file or album whose key, metadata or name does not decrypt used to
throw out of the refresh, so quak backup stopped before storing
anything, every run. Client.collectionsSince and filesSince now return
such a row in undecryptable with the reason, the refresh leaves it out,
and the store keeps the list in metadata.json until a later refresh
decrypts or deletes the record. lib.backup() records each such file in
failures.json unless a copy in another album decrypted, in which case it
stores the original from that copy. Each such album is listed in
BackupResult.undecryptableAlbums and printed as one FAILED album line.
quak backup exits 1 while either remains. An album or file that no
longer decrypts loses its album folder, JSON and links; its original
stays.

Model: opus-5-5
 2026-10-14 07:08:57 +02:00
clawbot 288825b11a quak backup no longer refreshes its library on a timer (closes #183)
check / check (push) Successful in 3m39s
Details 
The backup opened its library with the default 3-second refresh interval,
so for its whole run it asked the server for changes every few seconds,
and each refresh started another ML data pass. It now opens it with
refreshIntervalSeconds: Infinity. The library takes Infinity as no
background timer, since setTimeout would turn it into 1 ms. A refresh
interval of 3600, as the read commands use, would still refresh once an
hour in a backup that runs for hours.

Model: opus-5-5
 2026-10-14 04:10:14 +02:00
clawbot 54133cfa5f Test quak backup end to end against a fake Ente server (closes #186)
check / check (push) Successful in 3m19s
Details 
test/cli/backup-end-to-end.test.ts runs backupCommand through run, as
bin/quak.ts does, with the real loadSession reading a session saved after
logging in, and every request answered by a fake of the Ente HTTP API serving
data encrypted with libsodium. It covers the backup tree of an account with
two albums of its own and one shared with it, a second run that downloads
nothing, a download that succeeds after two 503s, a 404 recorded in
failures.json with exit 1, a refresh answered 401 with exit 3, and --verify
putting back an original changed on disk.

The fake server and the account it logs in moved from the backup-metadata
test into test/ente-server.ts, unchanged, so both tests use them.

Model: opus-5-5
 2026-10-14 03:38:38 +02:00
clawbot ab28d86aaa An ID on the command line is decimal digits only (closes #185)
check / check (push) Canceled after 0s
Details 
files --collection, get, get-thumb and helper fix-missing-thumbnails
--file read an ID with Number(), so 0x10 meant 16, 1e3 meant 1000 and
an empty string meant 0, and --file dropped an ID it could not read.
parseID in src/cli-commands.ts now takes an ID only when it is decimal
digits and a safe integer; anything else prints "Invalid file ID: ARG"
or "Invalid collection ID: ARG" and the command exits 1. The fixer
checks every --file ID before it opens the library, so one bad ID fails
the whole command before any request.

Model: opus-5-5
 2026-10-14 03:06:35 +02:00
clawbot 2402ec97f7 README: running quak backup from cron, and its exit codes (closes #170)
check / check (push) Successful in 3m59s
Details 
A new README section shows how to run quak backup unattended: log in once
as the job's user, a crontab with a backup every night and --verify on
Sundays, output appended to a log file, and the HOME and XDG_DATA_HOME the
job needs to find the saved session. A table gives each exit code as the
code returns it, and says which failures the next run retries. The
introduction now lists everything the backup keeps for each file. It, the
backup layout tree and the lib.backup() entry say EXIF and XMP are kept for
an image only, and dimensions for a JPEG only.

Judgement call: the --verify run takes Sunday's slot rather than a second
job that night, since an overlapping run would exit 2.

Model: opus-5-5
 2026-10-07 00:47:37 +02:00
clawbot ddf58af3cc quak backup --verify re-hashes stored originals and downloads again any that do not match (closes #168)
check / check (push) Failing after 1m22s
Details 
`--verify`, or `lib.backup({ verify: true })`, hashes each original already
at its save path as the download check does, streamed, a live photo as
`<imageHash>:<videoHash>`. A mismatch is logged, removed and put back in the
same run, and what is put back is hashed too, since a copy from the content
cache is not checked; one that still does not match, or a failed fetch, goes
into `failures.json`. A file with no recorded hash counts as unchecked. The
result, the summary and `--json` gain `verified`, `mismatched` and
`unchecked`.

Judgement call: a stored original that cannot be read for hashing is recorded as failed and left in place.
Judgement call: a copy put back that still does not match stays at its save path and is not counted as downloaded.
Judgement call: the summary prints the three counts only with `--verify`.

Model: opus-5-5
 2026-10-06 22:47:26 +02:00
clawbot bd77422965 quak backup refuses to run while another backup of the same directory runs, exit 2 (closes #169)
check / check (push) Successful in 3m10s
Details 
lib.backup() takes a lock, backup.lock in its download directory, made
with proper-lockfile, before its refresh, and removes it when it ends. A
second backup of the directory fails at once with an error naming it.
quak backup takes the lock itself before it opens its library and passes
lockHeld to the backup, so a refused run sends no request; it prints the
error as one line and exits 2. A lock untouched for 10 seconds, left by a
run that could not remove it, is taken over.

Deviation: yarn.lock was regenerated by yarn add in the pinned node image.
Judgement call: a run failing at its refresh leaves the directory, empty.
Judgement call: a lock removed mid-run stops that run with an uncaught
error, the library's default.

Model: opus-5-5
 2026-10-06 20:30:51 +02:00
clawbot 31b50a211d quak backup writes each original's EXIF, XMP and dimensions into its JSON (closes #167)
check / check (push) Successful in 3m18s
Details 
Each file's JSON gains imageMetadata, what extractImageMetadata finds in
the stored original (for a live photo, its image), or the reason the
read failed in imageMetadataError; a failed read fails neither the file
nor the run. A video is not read. An original is read when the run
stores it or when its JSON has neither field; otherwise the field is
carried over from that JSON, so a run does not read every original
again. The hand-built JPEG fixtures move to test/exif-jpeg.ts so the
backup tests can use them.

Judgement call: an original with nothing to record gets imageMetadata
{} instead of no field, so it is not read again on every run.

Model: opus-5-5
 2026-10-06 16:47:31 +02:00
clawbot f6317109bc An expired session exits 3 with one line saying to run quak login (closes #164)
check / check (push) Successful in 3m8s
Details 
A 401 from the server that ends a command reaches run in src/cli-run.ts as
an ApiError; run prints one line saying to run "quak login" and exits 3.
quak backup meets it on its first refresh, before it touches any file. For
the commands that load the saved session, a missing or corrupt session file
keeps its message and also exits 3.

Judgement call: quak logout is unchanged; it handles its own errors and
deletes the session file whatever the server answers.
Judgement call: quak backup still prints its two progress lines before the
error line.

Model: opus-5-5
 2026-10-06 14:01:48 +02:00
clawbot ad0e11407f quak backup retries failed requests for longer (closes #165)
check / check (push) Successful in 4m16s
Details 
src/retry.ts exports UNATTENDED_RETRY_OPTIONS beside the unchanged
default: 10 attempts, a 1 s base delay and a 60 s cap, so a request that
keeps failing waits at most 243 s before it gives up. bin/quak.ts loads
the backup's session with them, so its refresh, ML data and downloads
all use them; every other command keeps the default. What is retried
and the backoff formula are unchanged.

Model: opus-5-5
 2026-10-06 12:47:40 +02:00
clawbot fb4fff59fe Re-vendor the canonical files from sneak/prompts at dd4027b (closes #171)
check / check (push) Successful in 4m45s
Details 
.dockerignore, .gitignore, .prettierignore and REPO_POLICIES.md are
copied from sneak/prompts at dd4027b; .editorconfig, check.yml and
.prettierrc already matched. This repository's build artifacts follow
the copied content in .gitignore and, anchored, in .dockerignore. The
scripts follow the new policy: check runs fmt-check, cibuild bootstraps
and runs check before the image build, fmt and fmt-check find the
pinned yarn under nvm, and the image's last stage marks /app safe for
git. The build-context test expects the new pattern forms.

Judgement call: script/precommit runs lint and fmt-check without the tests, so the README's red-phase test commit can still land; the policy allows that form only where local testing is impossible.
Not applicable: the issue's .golangci.yml entries; this TypeScript repository has none.

Model: opus-5-5
 2026-10-06 10:30:29 +02:00
clawbot 2483c85321 quak backup writes the account and album records backup-metadata writes (closes #166)
check / check (push) Successful in 1m23s
Details 
quak backup now writes account.json with the account's email and user ID,
adds each album's ownerID, isShared, updationTime and, when present, its
magicMetadata, pubMagicMetadata and sharedMagicMetadata to the album's
JSON, and adds updationTime to each file's JSON. The fields and their order
match backup-metadata's account.json, _collection.json and per-file JSON.
The interface runBackup drives gains whoami, which the library passes
through from its client.

Model: opus-5-5
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
 2026-10-06 07:13:24 +02:00
clawbot ab4d5d2cc2 quak backup writes each file's ML data into its JSON (closes #163)
check / check (push) Successful in 1m37s
Details 
lib.backup() now waits for an ML data fetch, joining the one its refresh
started or starting one, before it writes the per-file JSON, and each
file's JSON carries the cached payload as mlData. When the fetch fails,
each file with no cached ML data gets mlDataError and an entry in
failures.json, so the result counts it as failed and quak backup exits 1;
the next run fetches again.

Judgement call: the wait comes after the originals are downloaded, so the fetch runs alongside the downloads.
Judgement call: a failed ML fetch is recorded per file in failures.json, which is how the exit code goes non-zero without changing src/cli-commands.ts.

Model: opus-5-5
 2026-10-06 05:13:27 +02:00
clawbot 14ac7d05ef Download-albums example test no longer depends on the disk's free space (closes #160)
check / check (push) Successful in 1m32s
Details 
The test left freeBelowBytes at its 50 GiB default, so on a disk with under
50 GiB free the content cache's limit fell to zero. Downloading another photo
then evicted photo 1's cached original, and the test fetched it a fourth time
instead of copying it. Both of its libraries now open with freeBelowBytes: 0.
Every other test that opens a library or content cache with the real free
space was read; none has a result that depends on it. The 50 GiB default is
unchanged.

Model: opus-5-5
 2026-10-03 15:30:53 +02:00
clawbot 9e94542a57 docker build . stamps the git tag or short commit, not dev (closes #154)
check / check (push) Failing after 46s
Details 
A plain `docker build .` now stamps the version from git rather than `dev`/`0.0.0`. After `tsc`, `script/build` writes into `dist/package.json` the version `script/version` decides: `VERSION` when given, otherwise `git describe --tags --always` (the tag; or tag, commits since and short commit; or the short commit), otherwise `package.json`'s. A checkout with `.git` that yields an empty, `dev` or `unknown` version fails the build. `.dockerignore` sends `.git` but not `.git/config`, so no remote URL or credential reaches the image. `ARG VERSION` has no default, and the host scripts' version still wins.

Not changed: `REPO_POLICIES.md` still says `ARG VERSION=dev` until the shared policy changes.

Model: opus-5-5
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
 2026-10-02 06:12:56 +02:00
clawbot e50d2a78c8 exif() returns every EXIF tag in the file (closes #156)
check / check (push) Failing after 59s
Details 
`photo.exif()` now returns every EXIF tag in the file, as the owner ruled, typed `ExifTags`: each tag keyed by name with `exifreader`'s `id`, `value`, `description` and `computed`. A tag with no name is keyed `undefined-` plus its number, and the embedded thumbnail's tags sit under `Thumbnail`. The thirteen typed methods stay, with the same names and types; each now picks its field from `exif()`'s tags. GPS latitude and longitude are worked out from the GPS tags and their reference tags, and a position with no reference tags gives neither. `backup-metadata --exif` output is unchanged.

Model: opus-5-5
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
 2026-10-02 05:30:05 +02:00
clawbot c5a588177c Thumbnail test on main: re-encode a small image so it cannot time out (closes #153)
check / check (push) Successful in 1m19s
Details 
The `main` half of the fix for the timed-out thumbnail test, carrying only the test change from `next` (d788c54). The test "re-encodes smaller until the thumbnail fits the recorded size" built a noisy 400x300 JPEG, which came close to vitest's 5 s limit and made `script/cibuild` fail on a busy host. It now uses a noisy 64x48 JPEG, keeps every assertion, and runs in about 1 s.

Model: opus-5-5
 2026-10-02 04:22:27 +02:00
clawbot 0c995a8c4f Remove the content cache's handling of an earlier version's live-photo ZIP (closes #151)
check / check (push) Failing after 44s
Details 
quak is pre-1.0 and keeps no handling of old data. The content cache no longer recognises or removes a live photo that an earlier quak version cached as one ZIP. A live-photo download no longer removes what was at its destination before renaming the image and video into place; the rename already replaces it. The README sentences and the tests about that old ZIP are gone. The two removed tests that also covered current behaviour are replaced by tests with no ZIP: the cache re-fetching a live photo recorded with no video, and the library telling the cache which files are live photos when it opens.

Model: opus-5-5
 2026-10-02 04:14:30 +02:00
clawbot d788c5457d Thumbnail test: re-encode a small image so it cannot time out (closes #153)
check / check (push) Failing after 45s
Details 
The test "re-encodes smaller until the thumbnail fits the recorded size" built a noisy 400x300 JPEG and encoded it several times. That came close to vitest's 5 s limit and timed out on a busy host, which made `script/cibuild` fail on `next` and `main`. It now uses a noisy 64x48 JPEG, still too big for the first, default-quality encoding to fit, and keeps every assertion. It runs in about 1 s.

Model: opus-5-5
 2026-10-02 03:49:23 +02:00
clawbot 10afa7a7f4 Example script: download every album's photos and metadata (closes #144)
check / check (push) Successful in 1m20s
Details 
`examples/download-albums.ts` logs in from `QUAK_EMAIL` and `QUAK_PASSWORD`, then walks every album from `await lib.fresh()`. For each photo it runs `photo.download()` to the photo's save path and writes `{savePath}.json`, which holds the photo's record and EXIF. For each album it writes `{dir}/albums/{collectionID}.json` with the album's name and its photos' save paths. A file is written only when its content changes, so a second run downloads and rewrites nothing.

The script opens the library with prefetching off, as `quak backup` does. The build type-checks `examples/`. The README says how to run it.

Model: opus-5-5
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
 2026-10-02 00:33:00 +02:00
clawbot d40f339c0b Photo: one async method per EXIF field (closes #148)
check / check (push) Successful in 1m37s
Details 
`Photo` gains thirteen async methods, one per `PhotoExif` field and named after it: `make()`, `model()`, `lensModel()`, `dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`, `iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()` and `gpsAltitude()`. Each calls `exif()` and returns its one field, or `undefined` when the file lacks it. `exif()` is unchanged.

`Photo` implements a type built from `PhotoExif`'s keys, so the type check fails when a field has no method. Each call reads the original again; a caller that wants several fields calls `exif()` once.

Model: opus-5-5
 2026-10-02 00:09:24 +02:00
clawbot 10e1a9ef39 Save path ./photos/YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.fileID.ext; download() from the cache first (closes #143)
check / check (push) Successful in 1m25s
Details 
Originals are saved at `{downloadDirectory}/YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.{fileID}{ext}`. The date is the photo's `takenAt` in local time, and `downloadDirectory` defaults to `./photos`, resolved when the library opens. The old `originals/` layout is gone.

`photo.download()` writes the original to `savePath`. It copies from the cache when the cache holds the original, and fetches otherwise. `lib.backup()` uses the same path and rule, and every album in `collections/` links to it. `isLocal` is true only when the original is at `savePath`.

For a file in several albums, one rule picks the copy everything uses: the most recently synced, with the lowest album ID breaking a tie.

Model: opus-5-5
 2026-10-01 23:20:58 +02:00
clawbot 67d554fb46 exif(): read HEIF/HEIC originals with exifreader (closes #145)
check / check (push) Successful in 2m10s
Details 
`photo.exif()` and `quak backup-metadata --exif` now read EXIF through `exifreader`. HEIC/HEIF originals get EXIF, including a live photo's image, as do the other formats `exifreader` reads. It replaces `exif-reader` and the hand-written JPEG scan. `PhotoExif` is unchanged.

The `backup-metadata` dump now holds `exifreader`'s tag output, with unnamed tags keyed `undefined-` plus their number. GPS altitude without a reference counts as above sea level. A latitude or longitude without its hemisphere tag, an unreadable text tag, and a date the parser rejects each give no field.

Licence: `exifreader` is MPL-2.0, used unmodified.

Model: opus-5-5
 2026-10-01 22:34:09 +02:00
clawbot 2b598d3622 Photo: save path, is-local, content bytes, metadata and EXIF getters (closes #141)
check / check (push) Successful in 1m24s
Details 
`Photo` gains:

- `savePath` and `isLocal`: synchronous, disk only. Where `lib.backup()` writes the original under the library's `downloadDirectory`, and whether all of it is there; a copy only in the cache does not count.
- `content()` and `exif()`: async, may download. `exif()` reads the common EXIF fields of a JPEG and returns `{}` for anything else.
- The getters `modifiedAt` and `hash`, also on `PhotoRecord`, and `year`.

For a live photo the backup has not stored yet, `savePath` carries the title's extension, and the image may be stored under a different one. The JPEG EXIF scan moved to `src/exif.ts`. The exported `PhotoContent` interface gains `savePath` and `isLocal`.

Judgement call: `iso` is read only when the file stores it as a single number.

Model: opus-5-5
 2026-10-01 17:58:23 +02:00
clawbot e6825abcdb TODO.md workflow: branch from and merge into next (closes #137)
check / check (push) Successful in 48s
Details 
The "Workflow" list said to branch from main and to merge to main or open a pull request. It now says to branch from next, open a pull request that targets next, that the repository manager squash-merges a reviewed pull request into next, and that only sneak merges next into main, as the README does. The other items are unchanged. No other text in the repo contradicted the policy.

Docs only; merged under the docs-only rule without an adversarial review.

Model: opus-5-5
Co-authored-by: clawbot <sneak+clawbot@sneak.cloud>
 2026-09-29 05:54:35 +02:00
clawbot c27e2cb629 README development workflow: branch from and merge into next (closes #135)
check / check (push) Successful in 1m0s
Details 
"Development workflow" and "For LLMs" said work branches off main and merges into main. They now say work branches from next, every pull request targets next, the repository manager squash-merges reviewed pull requests into next once make check is green, and only sneak merges next into main. The rest of both sections is unchanged.

Docs only; merged under the docs-only rule without an adversarial review.

Model: opus-5-5
 2026-09-29 05:38:36 +02:00
clawbot e6a9e929c2 Bring README and TODO.md in line with next after the milestone merge (closes #132)
check / check (push) Successful in 1m10s
Details 
README.md and TODO.md were read end to end against the code after the milestone merge, and every sentence the code contradicted was corrected. The corrections cover the login and key-derivation steps (the TOTP step, email OTP, the SRP library), retries and download staging, which CLI commands take which options, which of each file's metadata the backup keeps, how default and fresh reads behave, the cache options, and what the tests cover. TODO.md's next step now says no implementation work is open and the cache design waits on sneak.

Docs only.
Left as written: the development workflow's `main` base, a process question.
Merged under the docs-only rule after four reviews; the fix for the fourth review's one finding was not re-reviewed.

Model: opus-5-5
 2026-09-29 05:22:57 +02:00
sneak 9e6e038545 Merge pull request 'next → main: pre-1.0 milestone' (#31) from next into main
check / check (push) Successful in 52s
Details 
Reviewed-on: #31
 2026-09-29 03:12:31 +02:00
clawbot 1b7158fcd3 Test the live-photo writer's fsyncs (closes #130)
check / check (push) Successful in 43s
Details 
A test checks that the live-photo writer fsyncs the image's and the
video's temp files before renaming either into place, and fsyncs the
directory after both renames, as the writeAtomic test does for one file.

Model: opus-5-5
 2026-09-28 18:45:48 +02:00
clawbot 9e6deb21eb Store live photos as their image and their video (closes #107)
check / check (push) Successful in 41s
Details 
A live photo, which Ente stores as one ZIP, is unpacked as it downloads
into its image and its video, each `<fileID>.<ext>` with its extension
from the ZIP, beside `<fileID>.livephoto.json`, which names the two.
Both are checked against the recorded hash and renamed into place only
when both are complete. A ZIP with a second image or video, or whose
parts come to more than 20 times its size plus 16 MiB, is refused. The
backup and the content cache count a live photo as stored only with both
files, album folders link both, `quak get` writes both, and the content
result gives the video as `videoPath`. A ZIP an earlier version stored
is replaced.

Model: opus-5-5
 2026-09-28 18:06:01 +02:00
clawbot 04094a8cfb Mark the package private and declare engines and exports (closes #6)
check / check (push) Successful in 47s
Details 
quak is not published, so package.json is marked private and loses its
files field. engines.node is >=22, the major version script/bootstrap
and the Dockerfile use. The exports map makes "." and "./package.json"
the only importable paths, so the CLI-only modules stay internal.

Model: opus-5-5
 2026-09-23 14:17:53 +02:00
clawbot 6a7a10f489 Bring README and TODO.md in line with the tree on next2 (closes #111)
check / check (push) Successful in 57s
Details 
The README layout lists src/library/ and the other source files, the
backup layout names failures.json and the optional thumbnails/, "Opening
a library" says an empty cache opens with no data when the first refresh
fails, and the Testing section gives the 60-second hard cap and 20-second
target for make test, with the 90-second timeout in the Dockerfile's test
phase as the backstop that catches a hung test. The next step in both
files is storing live photos
(#107) instead of a v1.0.0 tag;
tagging and releases are sneak's call alone.

Model: opus-5-5
 2026-09-23 10:08:03 +02:00
clawbot 7740ebfd4d Test quak login and backup-metadata --exif (closes #110)
check / check (push) Successful in 42s
Details 
loginCommand now takes its login function and prompts from CliContext;
bin/quak.ts passes Client.login and the terminal prompts, so behaviour is
unchanged. Tests cover a login from QUAK_EMAIL/QUAK_PASSWORD with no
prompt, the TOTP prompt, a failed login, and the saved session's modes,
and show that --exif and --all each turn on EXIF extraction. Also rewraps
the header comment of src/cli-commands.ts.

Model: opus-5-5
 2026-09-23 09:27:40 +02:00
clawbot ae76eb3f74 Write each backed-up original once and skip the precache (closes #106)
check / check (push) Successful in 49s
Details 
An original fetched for a backup is now written by the download writer
straight into the backup's originals/, and the content cache records it
there instead of keeping its own copy. One the cache already held is
still copied. `quak backup` opens its library with the thumbnail and
originals precache off, as the one-shot commands do.

Model: opus-5-5
 2026-09-23 08:48:02 +02:00
clawbot f52c77f155 Refresh before backup-metadata and the thumbnail helpers answer (closes #100)
check / check (push) Successful in 1m0s
Details 
backup-metadata, helper list-missing-thumbnails and helper
fix-missing-thumbnails now await lib.fresh() before reading the library,
as collections, files, get and get-thumb already do. A file added since
the cache was written is included, and a failed refresh is thrown, so
the CLI prints one line and exits 1 instead of answering from a stale
or empty cache. The README lists them among the commands that refresh
first.

Model: opus-5-5
 2026-09-23 08:11:37 +02:00
clawbot 6757ddea94 Make backup wait for the server refresh and fail when it fails (closes #99)
check / check (push) Successful in 1m8s
Details 
lib.backup() refreshed through the background loop's refresh, which returns
at once when one is already running and swallows a failure, so a backup
could run on the previous file list, or on an empty cache, and exit 0. It
now uses the refresh fresh() uses: it joins a running refresh or starts
one, and rejects before touching any file when it fails. The CLI's error
wrapper prints that as one line and exits 1.

Model: opus-5-5
 2026-09-23 08:07:44 +02:00
clawbot 36642f4448 Print CLI errors as one line instead of a stack trace (closes #102)
check / check (push) Successful in 1m13s
Details 
An error a command throws now reaches the user as one `quak: MESSAGE`
line on stderr, and the CLI exits 1 once output has drained. The wrapper
moved from bin/quak.ts to src/cli-run.ts so it can be tested, and
bin/quak.ts awaits program.parseAsync() so async actions are awaited.

Model: opus-5-5
 2026-09-23 07:33:26 +02:00
clawbot fc396d1ecc Keep another process's downloads when opening a library (closes #105)
check / check (push) Successful in 1m18s
Details 
The download writer's temp files are now named .quak-<pid>-<random>.tmp.
removeLeftoverTempFiles moves from the backup into the download module and
deletes a .quak-*.tmp file only when the process ID in its name is no
longer running; the content cache calls it at open() instead of deleting
every temp file, so a download in progress in another process sharing the
cache survives. The README backup layout and TODO.md are updated.

Model: opus-5-5
 2026-09-23 07:25:28 +02:00
clawbot cb61582ae6 Make lint and test phases of the Dockerfile (closes #96)
check / check (push) Successful in 1m45s
Details 
Follows the template: Dockerfile.lint is gone; the Dockerfile has a lint
phase (eslint, prettier --check .) and a test phase (vitest, run as the
node user, which the not-writable-directory tests need), and its
last stage compiles and depends on both. script/lint and script/test
build one phase each with --no-cache; script/docker and script/cibuild
pass --no-cache, so CHECK_EPOCH and LINT_EPOCH are removed.
script/cibuild is the single image build, so CI runs lint and the tests
once each. The tests that checked the old layout are deleted,
REPO_POLICIES.md is re-copied and the README describes the new layout.

Model: opus-5-5
 2026-09-23 07:18:46 +02:00
clawbot cda57eebda Skip thumbnail repairs the server always refuses (closes #109)
check / check (push) Successful in 49s
Details 
The server accepts a new thumbnail only from the file's owner and only when
it is no larger than the thumbnail size it records. Both thumbnail helpers
now skip files another account owns without fetching them. The fixer skips
a file whose recorded thumbnail size is 0 or unknown before downloading it,
and otherwise tries smaller encodings until the encrypted thumbnail fits,
skipping the file if none does.

Model: opus-5-5
 2026-09-23 07:03:33 +02:00
clawbot c24c4dda4f Test the live-photo hash check's error paths (closes #117)
check / check (push) Successful in 16s
Details 
Three downloadFile tests cover a live photo whose ZIP names an unknown
compression method, one whose ZIP has no image entry and one with no
video entry. Each checks that nothing is stored and the error names the
file ID; the unreadable ZIP is also checked not to be retried. Each test
fails when its check in livePhotoHasher is removed.

Model: opus-5-5
 2026-09-23 06:56:15 +02:00
clawbot 4bb75ca323 quak logout ends the session on the server (closes #108)
check / check (push) Successful in 54s
Details 
logout now calls POST /users/logout with the saved token through the new
Client.logoutOnServer(), then deletes session.json even when that call
fails; in that case it says the server session could not be ended and
exits 1. It prints the account's cache directory and says it still holds
decrypted data. The default cache path moves into
defaultCacheDirectory(), shared with Library.open, so both name the same
directory.

Model: opus-5-5
 2026-09-23 06:51:12 +02:00
clawbot cd05a458dc Give backup album folders distinct names and remove stale entries (closes #103)
check / check (push) Successful in 32s
Details 
Within a collection's folder, files whose sanitized titles match (ignoring
case) each get their file ID added before the extension, and collections
whose sanitized names match get their ID added. This repeats until no name,
including one with an ID added, matches another, so no symlink or JSON
replaces another. Names are chosen across all collections, so a scoped run
names folders the same as a full one.

Each run first removes symlinks into originals/ that no longer belong to a
collection, and the folders quak wrote (a sibling JSON with an album ID) for
collections that are gone or renamed. Anything else is left alone; a folder
still holding user files keeps its JSON.

Model: opus-5-5
 2026-09-23 06:36:12 +02:00
clawbot c19943a520 Check downloaded originals against their recorded content hash (closes #68)
check / check (push) Successful in 58s
Details 
downloadFile, shared by quak get, the content cache and backup, hashes
the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and stores
nothing on a mismatch, failing with an error naming the file ID. A live
photo ZIP is unpacked as it streams with fflate's Unzip, in small
slices so memory stays bounded however far an entry expands, and its
image and video hashed separately as <imageHash>:<videoHash>.
decryptFile reads the older imageHash/videoHash fields for live
photos. A file with no recorded hash is stored unchecked.

Model: opus-5-5
 2026-09-23 06:16:57 +02:00
clawbot 390401af2c Start empty when the cache directory holds another account's cache (closes #104)
check / check (push) Successful in 35s
Details 
When metadata.json in the cache directory was written for a different,
non-zero user ID than the client's, Library.open deletes it and mldata/
and loads an empty store, so the first refresh enumerates from 0 and none
of the other account's collections, files, keys or ML results are served.
Only reachable with --cache-dir or an explicit cacheDirectory.

Model: opus-5-5
 2026-09-23 05:56:14 +02:00
clawbot bf3b20df2f backup-metadata: keep going when an ML data request fails (closes #101)
check / check (push) Successful in 27s
Details 
Each ML data request of up to 200 files is now tried on its own. A request
that still fails after its retries is logged, its files are written with the
reason in `mlDataError`, and the command exits 1 once the dump is complete.
`fetchMLData`, used only here, is removed in favour of a per-batch loop over
`fetchMLDataBatch`.

Model: opus-5-5
 2026-09-23 05:49:38 +02:00
clawbot d05b53d560 Single-source the version from package.json (closes #5)
check / check (push) Successful in 25s
Details 
src/index.ts imports package.json for VERSION and bin/quak.ts passes
VERSION to commander, so package.json is the only place the version is
written. tsc copies package.json to dist/package.json, so the import
resolves from the built output; script/build runs the built CLI with
--version to prove it. A test checks VERSION and quak --version against
package.json.

Model: opus-5-5
 2026-09-23 05:07:47 +02:00
clawbot 0ca8887f52 Test that close() waits for the originals precache (closes #93)
check / check (push) Successful in 36s
Details 
The close() test that holds a precache fetch open now runs twice: once
with only the thumbnail fill and once with only the originals fill, so
removing either wait from Precache.close() fails a test.

Model: opus-5-5
 2026-09-23 04:54:45 +02:00
clawbot f1836ced57 Fix two intermittently failing library tests (closes #90)
check / check (push) Successful in 27s
Details 
Library.close() now returns a promise that resolves once the work it
started has finished: an in-flight refresh with its cache write, the ML
data fetch, and running precache sweeps. The interval test could see a
refresh's new state, close, and remove the directory while the write was
still running. Every library test and CLI command now awaits close(), and
tests hold each of the three writes open to prove close() waits for it.

The precache test waited for its stub source to be called, but the cache
records a file only after checking it on disk, so status() could lag. It
now waits for both fills to report "done".

Model: opus-5-5
 2026-09-23 04:38:00 +02:00
clawbot c75c4f987c Pin three untested guards: download timer, URL fragment, short APP1 (closes #89)
check / check (push) Successful in 24s
Details 
The download idle deadline's timer is unref'd so it never holds the
process open, and a test checks no timer is left after a download
completes or fails. A test covers the rejection of "#" in a request
path. The EXIF scan accepts an APP1 segment only when its length is at
least 8, since a shorter one cannot hold the six-byte Exif header;
tests cover lengths 7 and 8.

Model: opus-5-5
 2026-09-23 03:44:45 +02:00
clawbot 28a2beeab8 Make the download deadline an idle deadline and cancel failed bodies (closes #24)
check / check (push) Successful in 17s
Details 
downloadTimeoutMs now aborts a file or thumbnail download only when no
bytes have arrived for that long (default 60 s, was a 600 s cap on the
whole transfer), so a slow download that keeps making progress completes.
The abort reason is still a TimeoutError, so retry classification is
unchanged. A download that fails before reading the whole response body
cancels it, including when the temp file cannot be opened or the header
is malformed, so a failed file no longer holds its connection.

Model: opus-5-5
 2026-09-23 03:14:42 +02:00
clawbot 2b410c3ed6 Build every ApiClient request URL in one place (closes #18)
check / check (push) Successful in 25s
Details 
getJSON built its URL with new URL and then overwrote the host and
path, which dropped a base path in a self-hosted apiOrigin; postJSON,
putJSON and the file and thumbnail download URLs joined strings. All of
them now go through one function next to ApiClient that accepts a path
with or without a leading slash and an origin with or without a
trailing slash or base path, and percent-encodes query parameters. A
path containing "?" or "#" is rejected.

Model: opus-5-5
 2026-09-23 02:57:42 +02:00
clawbot d07692897b Move CLI commands into testable functions and test them (closes #12)
check / check (push) Successful in 27s
Details 
The command bodies in bin/quak.ts become functions in
src/cli-commands.ts that take their options and a context (output
streams, session directory, cache directory, session loader) and return
an exit code. bin/quak.ts wires them to commander and exits with that
code once stdout and stderr have drained; nothing below it calls
process.exit. test/cli/commands.test.ts drives the commands with a fake
client and temp directories. Output is unchanged.

Model: opus-5-5
 2026-09-23 02:47:47 +02:00
clawbot ed535be1da Harden the backup tree's atomic copy (closes #22)
check / check (push) Successful in 27s
Details 
The backup copy now fsyncs its temp file before the rename and the
directory after it, using the download writer's new fsyncPath helper.
Each backup run deletes .quak-backup-*.tmp files whose process is no
longer running, leaving those of a concurrent backup alone. The rename
sites and the README backup layout state that a symlink at the
destination is replaced and the new file takes the temp file's
permissions, and the README names the temp files. Adds tests for a
missing and an unwritable destination directory for downloadFile and
downloadThumbnail.

Model: opus-5-5
 2026-09-23 02:44:46 +02:00
clawbot d545dcd8b1 Harden the retry classifier and pin per-attempt deadlines (closes #80)
check / check (push) Successful in 42s
Details 
A POST or PUT is replayed only when every errno in the cause chain is a
connect errno and the walk reached the end of the chain, and
postJSON/putJSON no longer follow redirects, so a redirect is an
ApiError that is not retried. getRetryOptions() returns a copy. New
tests pin every errno the classifier names, the cause-chain depth
limit, a chain deeper than the limit, a two-error cycle, and a fresh
deadline per attempt for every retrying entry point. The README
endpoint list is now the one place naming the requests the replay rule
covers; code comments point to it.

Model: opus-5-5
 2026-09-23 02:38:01 +02:00
clawbot 52f58f5d2b Harden the JPEG EXIF scan against malformed input (closes #11)
check / check (push) Successful in 29s
Details 
The segment scan behind `backup-metadata --exif` now checks every
segment length against the bytes that remain and stops on lengths
under 2, so a truncated or corrupt original can neither throw nor loop.
A malformed or unparseable EXIF segment is recorded as
`imageMetadata.exifError`, and a failure to read the original as
`imageMetadataError` in the per-file JSON, instead of the field being
silently left out. Tests use short hand-built byte arrays.

Model: opus-5-5
 2026-09-23 02:28:15 +02:00
clawbot b44c4ba6d7 Keep make test from collecting tests in nested checkouts (closes #25)
check / check (push) Successful in 33s
Details 
vitest does not read .gitignore when finding tests, so a checkout nested
under .claude/ had its whole test/ tree run as part of this suite.
vitest.config.ts adds .claude/** to vitest's default excludes. The new
packaging test plants a nested checkout in a temp directory and fails if
vitest, run with this config, would collect it.

Model: opus-5-5
 2026-09-23 02:18:31 +02:00
clawbot d50b296d3a Drop the deprecated @types/libsodium-wrappers-sumo stub (closes #27)
check / check (push) Successful in 46s
Details 
The package is an empty stub with no declarations; libsodium-wrappers-sumo
ships its own types. Removed with yarn remove, which regenerated yarn.lock.

Model: opus-5-5
 2026-09-23 02:15:08 +02:00
clawbot b7d6ab99f4 Validate session snapshots and wipe keys on logout (closes #10)
check / check (push) Successful in 41s
Details 
Client.fromJSON checks every snapshot field and each key's decoded length
and throws an error naming the bad field. toJSON reads the token through a
new ApiClient.getAuthToken and throws when there is none. logout zeroes the
key buffers in place; collectionsSince re-checks for logout after its
request so it never decrypts with zeroed keys. The CLI now reports a
corrupt session file separately from a missing one.

Model: opus-5-5
 2026-09-23 02:08:02 +02:00
clawbot 3871d6228e Sanitize file names taken from server metadata (closes #9)
check / check (push) Successful in 37s
Details 
A file title or album name decrypted from server data could name a path
outside the chosen directory (`../../.ssh/authorized_keys`). One module,
src/filename.ts, now makes such names safe for `quak get`/`get-thumb`
without `--out`, downloadFile/downloadThumbnail without outPath, and the
backup and metadata backup trees. Originals-cache extensions are limited to
letters and digits. A user-supplied path is still used as is. decryptFile
reads a missing or non-string title as "" and rejects metadata that is not
a JSON object.

Model: opus-5-5
 2026-09-23 02:04:31 +02:00
clawbot fe952d3e62 Rewrite the README API reference for the new library surface (closes #53)
check / check (push) Successful in 28s
Details 
Rewrites the README API reference to match the shipped library surface: Library.open options (XDG cache dir + userID, downloadDirectory, refreshIntervalSeconds, precache and cache-size options), default vs fresh reads (Library.fresh()), snapshot/subscribe, albums/photos/timeline, Photo.original/thumbnail, thumbnails.ensure, mldata search, backup, the request pools and on-disk layout, and the CLI (--cache-dir, fresh reads). Every documented signature verified against the code. Notes the deferred content-hash integrity check (#68). Also issue #13.

Model: opus-4-8
 2026-09-23 00:27:30 +02:00
clawbot d23d3f8f47 Port the quak CLI to the library API (closes #52)
check / check (push) Successful in 17s
Details 
Ports the CLI to the library API. collections/files/get/get-thumb use the fresh read variants (Library.fresh(), current server state); backup runs lib.backup; backup-metadata and the missing-thumbnail helpers enumerate via the library. files/get output is byte-identical to the pre-port CLI — raw metadata.title, microsecond creationTime, pre-port row order — exit codes unchanged. Adds --cache-dir; fixes the helper JPEG-only assumption (closes #17).

Model: opus-4-8
 2026-09-23 00:00:55 +02:00
clawbot aeccb489b5 Fresh read variants that await a server round-trip (closes #75)
check / check (push) Successful in 28s
Details 
Adds Library.fresh(): forces a refresh, awaits its completion and persist, then returns the albums/photos/timeline read namespaces now reflecting a completed server round trip — the caller awaits and is guaranteed current at resolve. Default reads and the background loop are unchanged (immediate-from-cache). Concurrent fresh reads coalesce to one in-flight refresh; a fresh read whose refresh fails rejects rather than answering stale. The CLI adopts fresh reads separately (#52).

Model: opus-4-8
 2026-09-22 23:29:15 +02:00
clawbot 2e00139d3c Precache all thumbnails and pinned originals inside open() (closes #48)
check / check (push) Successful in 26s
Details 
Precaches aggressively inside open(): every thumbnail (newest first, through the shared thumbnail pool, never evicted; visible/ahead requests preempt the background fill) and the pinned originals — the favorites album plus every file within precacheOriginalsDays (default 7) of the newest takenAt — through the content pool. The pinned set integrates with the #47 eviction hook; when the window moves or a favorite is removed, files become ordinary evictable originals. open() options precacheThumbnails/precacheOriginals/precacheOriginalsDays; progress via onProgress/status().

Model: opus-4-8
 2026-09-22 21:21:34 +02:00
clawbot e8575780e4 Content-similarity search surface over the CLIP index (closes #50)
check / check (push) Successful in 15s
Details 
Adds lib.mldata search over the CLIP index (#49): forFile returns a file's stored payload; similar ranks nearest files by cosine on the CLIP embedding; searchByEmbedding ranks the index against a caller-supplied query vector. All RAM-only, reusing the packed Float32Array index and id list. No text encoder is bundled — the caller provides the query embedding. (Redo of the reverted first attempt, now tsc-clean.)

Model: opus-4-8
 2026-09-22 20:45:59 +02:00
clawbot 17d1d74615 Rewrite the backup command on the library API with a durable failure ledger (closes #51)
check / check (push) Successful in 25s
Details 
Rewrites backup on the library API. backup() refreshes, fetches pending originals (and optionally thumbnails) through the pools reusing the content cache, then materialises the unchanged collections/ symlink views + per-collection JSON + sidecars from the model. A symlink failure no longer aborts the run (closes #8). failures.json reconciles against each run's attempted set — since-deleted/out-of-scope/resolved entries clear, still-failing retained, one attempt per file per run — and the exit-code contract is preserved.

Model: opus-4-8
 2026-09-22 20:21:49 +02:00
clawbot c05d63a2f0 Originals cache size limit with least-recently-used eviction (closes #47)
check / check (push) Successful in 25s
Details 
Bounds the originals cache and evicts least-recently-used. cacheOriginalsMaxBytes default 100 GiB; the effective limit adapts down via fs.statfs to keep freeBelowBytes (default 50 GiB) free. Over-limit writes evict unpinned originals oldest-mtime-first (mtime touched on read); pinned files (favorites + latest week) are skipped and an over-budget on-demand fetch proceeds over-limit. Every in-flight write is excluded from eviction, so concurrent fetches never delete each other's just-stored file. Only cacheDirectory/originals is evicted; downloadDirectory and thumbnails never.

Model: opus-4-8
 2026-09-22 20:13:22 +02:00
clawbot 5db59a6e2b On-disk content and thumbnail cache with per-photo fetch and prefetch (closes #46)
check / check (push) Successful in 33s
Details 
Adds the on-disk content and thumbnail cache keyed by fileID: originals/ and thumbnails/ under cacheDirectory, present-means-complete (streaming atomic rename), orphan temp reaping on open. Photo.original/thumbnail return a cached path with no network when present, else fetch through the shared request pool; thumbnails.ensure drives the thumbnail pool with priority, dedup and AbortSignal. One shared RequestPools set serves both the ML fetch and the content cache. Content-hash integrity is deferred (#68); authenticated streaming decrypt guarantees integrity now.

Model: opus-4-8
 2026-09-22 18:38:22 +02:00
clawbot 61dfec8d38 Revert ML search (#50) to restore a green next (build fix pending)
check / check (push) Successful in 18s
Details 
Reverts the #50 merge (224bd101): mlsearch.ts failed tsc in the CI build (make check does not run make build, so it slipped past review). next restored to green; #50 to be redone with make build in its gate.

Model: opus-4-8
 2026-09-22 18:21:54 +02:00
clawbot 224bd101ab Content-similarity search surface over the CLIP index (closes #50)
check / check (push) Failing after 23s
Details 
Adds lib.mldata search over the CLIP index (#49): forFile returns a file's stored payload; similar ranks nearest files by cosine on the CLIP embedding; searchByEmbedding ranks the index against a caller-supplied query vector. All RAM-only, reusing the packed Float32Array index and id list. No text encoder is bundled — the caller provides the query embedding.

Model: opus-4-8
 2026-09-22 18:14:13 +02:00
clawbot d7f415fe29 Fetch, store, and index per-file ML data (closes #49)
check / check (push) Successful in 14s
Details 
Adds the machine-learning (magic) data layer: fetches per-file ML payloads (face detections + CLIP embeddings) via the existing metadata-backup fetch through the metadata pool after each refresh, decrypts and gunzips them, and stores one mldata/<fileID>.json per file by rename (present-means-complete). A derived index (mldata/clip.f32 + clip.json) loads in one read and is rebuilt whenever it disagrees with the payloads on disk in either direction, so an interrupted backfill self-heals. Never in metadata.json; incremental on later refreshes; progress via onProgress/status.

Model: opus-4-8
 2026-09-22 17:54:40 +02:00
clawbot c5c1f387df In-process read surface: albums, photos, and timeline grouping (closes #44)
check / check (push) Successful in 25s
Details 
Adds the in-process read surface, served from RAM with args-object signatures: albums (list/byName/byID), photos (byID/records -> plain PhotoRecord[]), thin Album/Photo wrappers, and timeline.groups (day/week/month) with a PhotoFilter (albumID/text/fileTypes/hasLocation/includeArchived; hidden excluded). Week keys use ISO YYYY-Www; each file appears once per group, newest first. Built on the #43 snapshot projection; no network.

Model: opus-4-8
 2026-09-22 17:21:48 +02:00
clawbot 000d395c87 Three bounded request pools for metadata, content, and thumbnails (closes #45)
check / check (push) Successful in 55s
Details 
Adds three independent bounded request pools — metadata (10 in flight), content (5), thumbnails (25), each overridable — with on-demand-before-background priority and in-flight dedup (a shared key runs once); an idle pool never lends slots, and retries run inside a slot. Self-contained module; the content cache wires it into the library later.

Model: opus-4-8
 2026-09-22 15:31:21 +02:00
clawbot fbb8ae44a7 Plain-record library snapshot and change subscription (closes #43)
check / check (push) Successful in 30s
Details 
Adds a synchronous, key-free snapshot() returning LibrarySnapshot (one PhotoRecord per fileID, deduped, newest first) with edited-name/time precedence (pubMagicMetadata over basic metadata) in milliseconds, plus AlbumRecord (favorites identified by type). subscribe({onChange}) delivers LibraryChange (changed/removed albums and files, refreshedAt) only when a refresh changes something; unsubscribe stops delivery. Built on the existing refresh loop; served from RAM, safe to send over IPC.

Model: opus-4-8
 2026-09-22 15:22:52 +02:00
clawbot 57e0c69651 Accumulate streamDecrypt reads linearly, not quadratically (closes #21)
check / check (push) Successful in 14s
Details 
streamDecrypt no longer recopies the whole accumulation buffer on every network read. Reads are queued with a running byte count and a contiguous buffer is materialised only at each ENC_CHUNK_SIZE boundary, with a straddling read split via a subarray view, so each byte is copied once instead of O(n^2). TAG_FINAL truncation detection, final-chunk handling, retry, and the per-chunk progress hook are unchanged. A new test feeds a multi-chunk body through a ReadableStream that yields many small pieces, exercising the fragmented-read path.

Model: opus-4-8
 2026-09-22 15:13:09 +02:00
clawbot 7570055a5b Library.open with a transparent background refresh (closes #42)
check / check (push) Successful in 24s
Details 
Library.open loads metadata.json and serves reads from RAM: an empty cache awaits the first refresh, an existing cache returns at once and refreshes in the background so an unreachable server never stalls open(). A background timer refreshes every refreshIntervalSeconds (default 3), diffing only changed albums via the resumable cursor+tombstone enumerators and rewriting metadata.json only when something changed. A refresh failure is invisible to reads and surfaced via status()/onProgress; a failed save keeps status().lastError set and retries until one lands, so a stale disk is never masked. No sync()/refresh()/serverReachable surface; status() and close() included.

Model: opus-4-8
 2026-09-22 14:52:02 +02:00
clawbot 4b4f550f89 Stream decrypted downloads to disk with bounded memory (closes #40)
check / check (push) Successful in 32s
Details 
Originals no longer buffer the whole decrypted file in RAM: streamDecrypt hands each secretstream chunk to a sink and the download path writes it to the staged temp file, so peak memory is one chunk regardless of file size. The atomic write moved inside the retry loop; only a TAG_FINAL-authenticated attempt renames; truncation leaves no destination file. Does not close #21 (the streamDecrypt accumulation-buffer recopy stays open).

Model: opus-4-8
 2026-09-22 13:54:20 +02:00
clawbot 72ea8dcb01 On-disk JSON metadata store for the local cache (closes #41)
check / check (push) Successful in 23s
Details 
Adds the metadata.json store: loads whole into RAM with id-lookup Maps, rewrites whole through the exported fsync atomic writer (temp, fsync, rename, dir fsync); a missing, unparseable, or wrong-schema file loads as empty (it is a cache); directory 0700, file 0600. No lock file, no public sync().

Model: opus-4-8
 2026-09-22 12:52:50 +02:00
clawbot 42a6c17d49 Resumable, deletion-aware collection and file enumeration (closes #38)
check / check (push) Successful in 24s
Details 
Adds collectionsSince/filesSince taking a starting cursor, returning the resumable max-updationTime cursor and a separate list of tombstoned ids; filesSince throws instead of looping when the server reports hasMore without advancing (closes #7). listCollections/listFiles stay as thin, unaffected wrappers.

Model: opus-4-8
 2026-09-22 12:37:09 +02:00
clawbot 8f575550af Durable atomic writer with fsync and per-chunk download progress (closes #39)
check / check (push) Successful in 41s
Details 
The atomic writer fsyncs the staged temp file before rename and the directory after, and is exported for reuse. downloadFile/downloadThumbnail gain an optional per-chunk onProgress hook (non-decreasing, final equals bytesWritten; no-op when absent). Retry and TAG_FINAL checks unchanged.

Model: opus-4-8
 2026-09-22 12:01:06 +02:00
clawbot ead083c1d6 Carry file size, thumbnail size, and deletion flag through decryptFile (closes #37)
check / check (push) Successful in 25s
Details 
decryptFile now sets file.size/thumbnail.size from raw.info (undefined when absent) and carries an optional isDeleted on EnteFile. Plain EnteFile return, no caller changes; listFiles keeps filtering tombstones. Tests cover the three fields and the size-absent case.

Model: opus-4-8
 2026-09-22 12:01:03 +02:00
clawbot d1d6cdd4f0 Revert the 16 commits pushed to next on 2026-09-04 by an agent outside the managed fleet
check / check (push) Successful in 1m40s
Details 
sneak, 2026-09-05: "inference instance stopped. undo its rogue work." The
reverted commits stay in history; nothing else on next is touched.

Model: fable-5-1
 2026-09-05 09:33:23 +00:00
user 48db9b438a test(lint-once): read docker image/builder build as builds too
check / check (push) Successful in 1m12s
Details 
`docker image build` and `docker builder build` are management-command
spellings of the same build, take the same -f, and were emitting no edge at
all — the same shape as the buildx miss.
 2026-09-04 12:08:54 +00:00
user 81150f433c test(lint-once): disclose the build shapes read, correct the run: | claim
check / check (push) Successful in 13s
Details 
Header now states which docker invocation shapes are recognised as builds
(bare, buildx, and either through global flags) and which are not (compose),
and that the RUN keyword is read case-insensitively with any whitespace
separator. The workflow comment claimed a `run: |` block fails the count;
it does not — the count is unmoved by it and the pinned resolved list is
what turns it red.
 2026-09-04 12:06:44 +00:00
user f67a1c4d92 test(lint-once): match RUN case-insensitively, follow buildx builds
check / check (push) Successful in 1m8s
Details 
WIP round 9: items 1-3. Verification matrix still to re-run.
 2026-09-04 12:04:24 +00:00
user ffc817522e test(lint-once): pin the two-build and later-flag shapes
check / check (push) Successful in 15s
Details 
The false green: script/lint chaining a second docker build after the
lint image ran prettier twice and counted once. The misresolution: a
bare docker build followed by cp -f resolved to the cp's file. Pin
both, and each separator that bounds an invocation.
 2026-09-04 11:51:17 +00:00
user 88a5fcaa87 lint-once: bound the docker file flag to its own invocation
check / check (push) Failing after 42s
Details 
edgesOf used a single test/exec for docker build, so a line with two
builds produced one edge and the file flag was searched across the whole
line. Follow every occurrence with matchAll and slice each one to the
next shell separator before looking for its flag.

Also correct the README's claim that script/fmt-check is the one
formatting path left on the host.
 2026-09-04 11:49:38 +00:00
user 2fcb3e6ced test(lint-once): follow a short file flag with its value attached
check / check (push) Successful in 14s
Details 
`docker build -fDockerfile.lint .` is a plain `-f` naming a literal file —
the flag parser reads the attached value for any shorthand — but the
resolver's `[=\s]+` required a separator, so the invocation fell through to
the default `Dockerfile` edge. Mutating the `check` recipe to that spelling
left the suite 52/52 green while `-f Dockerfile.lint` was caught.

The header already promised this form was followed: it claimed the file
named by `-f` is resolved wherever the flag sits, and disclosed only a
bundled short cluster (`-qf <file>`) as unfollowed. Overclaiming is the
defect, so the resolver is taught the form rather than the claim narrowed.

The attached form is allowed only for the short flag, keeping `--force-rm`
out of it; `--file` still requires `=` or whitespace. The `-qf` cluster
limitation is untouched and still pinned.
 2026-09-04 11:37:08 +00:00
user 197296edba lint-once: follow every spelling of docker build's file flag
check / check (push) Successful in 1m17s
Details 
The edge resolver matched `-f <file>` only. `docker build --file=X` fell
through to the default `Dockerfile` edge, so a second prettier pass wired in
that way was followed into the wrong file, counted nothing, and left the suite
green -- the exact false green this test exists to prevent, reachable by
writing the flag the long way.

Mutation, adding one line to the `check` recipe, before this commit:

    @docker build -f Dockerfile.lint .        1 failed / 47  caught
    @docker build --file=Dockerfile.lint .    48 passed / 48  MISSED

After, all four spellings fail with `expected 2 to be 1`:
`-f X`, `-f=X`, `--file X`, `--file=X`.

Docker takes the value either way for both the short and long flag, so the
resolver now reads `(?:-f|--file)[=\s]+`, still searched anywhere in the
invocation rather than at a fixed position. A leading \s keeps a longer flag
ending in the same letters (`--force-rm`) from supplying the match.

The header claimed `docker build -f <file>` coverage without qualification,
which a reader could take to include the long form it did not follow; it now
names all four forms and the fallback. The one limitation it asserts -- a
bundled cluster like `-qf X` resolving to the default -- is pinned by a test,
since an unpinned limitation is how the header drifts back into overclaiming.
 2026-09-04 11:31:06 +00:00
user bd88eced84 README: state the container-lint claim as what is actually true
check / check (push) Successful in 1m14s
Details 
The Linting section this PR adds claimed "There is no host lint path" and
then, two paragraphs later, documented script/fmt-check as a host-side
formatting check. Both cannot be true, and the absolute one is the false one.

What is true is narrower: no lint path reachable from script/check or
script/precommit runs on the host, so every lint verdict those two produce
comes from the container. script/fmt-check stays as a standalone entrypoint,
now stated as the one host formatting path with nothing reaching it. The
"exactly one place" and "in the container only" phrasings elsewhere are
qualified the same way.
 2026-09-04 11:27:49 +00:00
user 956870889f Update the make variable header to match the ?= and prefix rules
check / check (push) Successful in 15s
Details
2026-09-04 11:19:45 +00:00
user 18b0f039b5 Pin the export/override prefix and ?= first-wins with tests
check / check (push) Failing after 7s
Details
2026-09-04 11:18:19 +00:00
user 18226d345b WIP: fix export/override prefix and ?= first-wins in make variable parser
check / check (push) Successful in 14s
Details 
Two defects in the lint-once resolver's make variable handling:

- The assignment pattern anchored at the start of the name, so
  `export FMT := script/fmt-check` was never collected and `@$(FMT)`
  went unresolved. An optional `export `/`override ` prefix is now
  allowed.
- `?=` assigns only when the name is unset, so the first assignment
  wins. The parser called .set() unconditionally, letting a later
  `FMT ?= script/build` overwrite an earlier `FMT := script/fmt-check`
  and resolve to a command make never runs.

Tests pinning both to follow.
 2026-09-04 11:17:32 +00:00
clawbot 075b1bb921 Expand single-line make variables in the lint-once resolver
check / check (push) Successful in 1m13s
Details 
A recipe of `@$(FMT)` with `FMT := script/fmt-check` gave `make check` two
prettier passes while the suite stayed green: the resolver read the line as
invoking nothing. The shipped Makefile already writes recipes that way
(`@$(YARN) tsc --watch`), so this was a gap in the repo's own house style.

Variables assigned a literal on one line (`:=`, `=`, `?=`) are collected in a
pass of their own and substituted into target and recipe lines. Values needing
evaluation -- another reference, a make function, a `define` body -- are left
verbatim, and the header's not-followed list now says so.
 2026-09-04 11:05:41 +00:00
clawbot ff4cc63c8b Name the define/endef hole in the lint-once header
check / check (push) Successful in 1m8s
Details 
A `define EXTRA ... endef` body pulled into a recipe as $(EXTRA) gives
`make check` a second prettier pass that the walk still scores as one.
The parser expands no variables, so this is a name it cannot resolve,
not a body it declines to read. The header's exclusion list says so
rather than claiming coverage the code does not have.
 2026-09-04 10:55:41 +00:00
clawbot f4ecef8820 Follow make conditionals and semicolon recipes in lint-once
check / check (push) Successful in 15s
Details 
makeRecipes() reset the current target on every non-tab line, so an
ifeq/endif block ended the recipe and every tab-indented line inside it
was discarded; and the target line's tail was read entirely as
prerequisites, so `check: ; @script/fmt-check` split to tokens that
named no target and vanished. Both gave `make check` a second prettier
pass with the suite green.

Conditional directives no longer end a recipe, and every branch is
treated as reachable rather than evaluating the condition. The target
line is split on the first `;`: what precedes it is the prerequisite
list, what follows is the first recipe line. Both pinned directly, and
the header's exclusion list now names what the parser actually skips.
 2026-09-04 10:54:24 +00:00
clawbot 9ed92e1231 Format the lint-once resolver changes
check / check (push) Successful in 34s
Details 
Reflow the heredoc RUN fixture array in the BuildKit heredoc test to
satisfy prettier --check, which rejected the single-line form. No
behavioural change to the tests or the resolver.
 2026-09-04 10:31:40 +00:00
clawbot 8bf5138582 WIP: follow Makefile prerequisites and $(MAKE) in lint-once
check / check (push) Failing after 37s
Details
2026-09-04 10:05:38 +00:00
sneak 2bfa11c10c Walk the path CI runs in lint-once, and pin the branch the container installs (closes #33)
check / check (push) Successful in 35s
Details 
The header of test/packaging/lint-once.test.ts claimed a duplicate prettier
pass is caught wherever it is added. It was not: the walk started at
`make check`, which never reads `Dockerfile`, so appending
`RUN yarn run prettier --check .` to the image that `script/cibuild` builds
left the suite green — two prettier passes on the one path where it matters
most. The walk now also starts at `.gitea/workflows/check.yml` and follows
its `run:` steps into `script/cibuild` and from there into both images, so
the graph under test is the one CI executes rather than the one it was
assumed to execute. Reaching `script/cibuild` and `Dockerfile` is asserted,
and the test and build image is asserted to invoke prettier zero times.

The lockfile assertion was a substring check against the whole of
`script/bootstrap`. That script has two install sites, and the containers
take the second, because the pinned node image ships yarn; changing that
site to a bare `yarn install` kept the suite green while the container's
install stopped being pinned. `install_js_deps` is now resolved out of the
script and split at its `missing yarn` guard, and every `yarn install`
occurrence in each branch is required to carry `--frozen-lockfile`. That the
container runs `script/bootstrap` at all is asserted too, so the lockfile
assertions cannot end up describing a script the image never executes.

Prettier is counted per occurrence instead of per line:
`prettier --check . && prettier --check src` was one invocation by the old
count. The `continue` that followed a counted line also dropped every
script, make, yarn and docker edge sharing that line, so a subtree could be
hidden behind a single `&&`; edges are now extracted from every line.

Undercounting is what would make this file worthless, so every way of
reaching nothing is a thrown error rather than a quiet zero: an unknown
Makefile target, an unknown package.json script, a missing script file, a
node that resolves to no commands, and an unknown node kind. All five are
tested, as is a walk that legitimately counts zero, and the cycle guard.

Every assertion in the file was mutation-tested: changed to assert something
else, run, and confirmed to fail for its own named reason. The two mutations
above were reproduced and both now turn the suite red.

test/packaging/entrypoints.test.ts said `make check` runs test, lint and
fmt-check. Formatting has been part of the lint container since the
duplicate host pass was removed, so the comment now says what it does.
 2026-08-10 13:41:12 +00:00
sneak a73f0abbe8 Check formatting once per make check, in the container (closes #29)
check / check (push) Successful in 59s
Details 
script/check ran script/test, script/lint and script/fmt-check. Since
linting moved into Docker, script/lint is a build of Dockerfile.lint,
which runs `prettier --check .` as a build step — so make check checked
formatting twice over the same tree: once in the container and once on
the host. script/precommit had the same pair.

Drop the script/fmt-check call from both. The container keeps the check,
because a successful Dockerfile.lint build is what CI treats as proof of
a clean tree, and it is the stronger of the two verdicts: its prettier is
digest-pinned and installed under --frozen-lockfile, while the host's is
whatever the working tree happens to have. The pre-commit hook is
unchanged in what it catches — script/lint still fails a badly formatted
tree, and therefore the commit.

script/fmt-check survives as a standalone entrypoint, as REPO_POLICIES.md
requires, for asking the formatting question by itself without docker.
Its verdict cannot drift from the container's: prettier is pinned to an
exact version, installed from yarn.lock in both places, and reads
.gitignore as its default ignore file, which is why .dockerignore keeps
.gitignore in the build context.

The count is asserted rather than promised. test/packaging/lint-once.test.ts
walks the invocation graph from each entrypoint — through the Makefile
shims, the script/ calls, the package.json scripts and the docker build
into Dockerfile.lint's RUN steps — and counts prettier invocations: one
per make check, one per script/precommit, and one each for make lint and
make fmt-check alone, so neither can become a no-op that satisfies the
count trivially. The walk also asserts which nodes it reached, so a
restructure that defeats the resolver fails the test instead of quietly
counting zero.

Observed: 2 prettier invocations per make check before, 1 after.
 2026-08-10 13:05:03 +00:00
sneak fed39d19cf Run all linting in Docker via Dockerfile.lint (closes #30)
check / check (push) Successful in 1m2s
Details 
Linting now happens in one place only: a new root Dockerfile.lint copies
the repo into the digest-pinned node image already used by Dockerfile and
runs eslint and prettier as build steps, so a successful build is a clean
lint. script/lint is reduced to building it, which also works where the
docker daemon is remote and bind mounts are impossible. No host lint path
survives: the "lint" script is gone from package.json, so there is no
second, unpinned way to get a lint verdict.

Caching is waived for lint, because a lint build over an unchanged tree
returns success in well under a second having linted nothing. LINT_EPOCH
is the cache buster and it fails closed exactly as CHECK_EPOCH does: an
unset ARG is the empty string, which is a perfectly stable cache key, so
the guard rejects it and a bare `docker build -f Dockerfile.lint .` errors
out instead of serving a green it did not earn. Both linters sit below the
guard, so a fresh epoch forces them to execute while the bootstrap and
dependency layers above stay cached.

That makes script/lint a docker build, which nothing inside a container
may call. script/check calls script/lint, so the Dockerfile image can no
longer run make check: the lint stage and its COPY --from=lint ordering
hack are deleted, and the remaining stage runs make test and make build
under the existing CHECK_EPOCH guard. script/cibuild is now the composite
gate and builds the lint image first, so a lint failure is reported before
the slower suite runs.

The .dockerignore exclusions are unchanged and still apply to the lint
build, including the .claude/ exclusion (eslint's flat config does not
ignore dot-directories, so a nested worktree in the context would be
linted) and the deliberate exception that keeps .gitignore in the context
for prettier. A new test asserts no per-Dockerfile ignore file shadows the
root one for either image, and test/packaging/lint-docker.test.ts asserts
the whole shape: the docker-only lint path, the digest pin, manifests
copied before sources, the fail-closed guard with both linters below it,
the absence of a lint stage or make check in Dockerfile, and the build
order in script/cibuild.
 2026-08-10 12:39:23 +00:00
108 changed files with 32249 additions and 2123 deletions
Expand all files Collapse all files
.dockerignore
+75 -39
View File
@@ -1,50 +1,86 @@
# Mirrors .gitignore, with one deliberate exception: .gitignore itself stays
# in the build context, because prettier 3 reads it as a default ignore file
# and dropping it would change what `make fmt-check` sees inside the image.
# .dockerignore does NOT use .gitignore semantics. Docker matches with
# moby/patternmatcher: filepath.Match plus `**`, so `*` does not cross
# `/` and an unprefixed pattern is anchored at the context root. Every
# depth-independent pattern therefore needs `**/`, or `config/.env` and
# `certs/server.key` still ship while this file reads as solved. Only
# genuinely root-anchored entries go unprefixed. Never transplant these
# into .gitignore, where `**/` is wrong.
#
# Matching is case-sensitive, so secrets use character ranges rather
# than an ALL-CAPS twin, which would still miss `Server.Key`.
#
# Extend with this repo's own host-built artifacts, written anchored:
# `/myapp`, never `**/myapp`, which also matches `cmd/myapp/` and
# deletes the package directory from the context.
# VCS
.git
# .git is sent without its config. Without a VERSION build argument the
# stage that compiles runs `git describe --tags --always` on .git, which
# does not need .git/config; that file can hold a credential, such as a
# password in a remote URL or the token the CI checkout step stores there.
# Each submodule keeps a config with the same exposure in its git directory
# under .git/modules/, nested again for a submodule's own submodules, or in
# its own .git directory when it keeps one.
# KNOWN GAP: a submodule whose name has a `config` segment (`config`,
# `deploy/config`, `config/lib`) loses its whole git directory, because
# `**/.git/modules/**/config` also matches that segment's directory
# under .git/modules/. Go's version stamping then fails the build;
# nothing leaks. Name such a submodule without that segment:
# `git submodule add --name`.
**/.git/config
**/.git/modules/**/config
# OS
.DS_Store
Thumbs.db
# Agent scratch: one full checkout of the repo per in-flight agent.
# Anchored because it occurs once where agents run at the repo root.
# KNOWN GAP: a repo running agents in subdirectories still ships
# `services/api/.claude/` and must add its own anchored entry.
.claude
# Editors
*.swp
*.swo
*~
*.bak
.idea/
.vscode/
*.sublime-*
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Re-include a committed template with a negation if the
# build needs one: `!docs/example.env`.
**/*.[eE][nN][vV]
**/.[eE][nN][vV].*
**/.[eE][nN][vV][rR][cC]
# Node
node_modules
# Private keys and the bundles carrying them. Public certificates
# (*.crt, *.cer) are deliberately absent: they are legitimate inputs.
**/*.[pP][eE][mM]
**/*.[kK][eE][yY]
**/*.[pP]12
**/*.[pP][fF][xX]
**/[iI][dD]_[rR][sS][aA]
**/[iI][dD]_[dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]
**/[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
**/[iI][dD]_[eE][dD]25519
**/[iI][dD]_[eE][dD]25519_[sS][kK]
# TypeScript / build artifacts
dist
build
*.tsbuildinfo
coverage
.nyc_output/
# Dependencies: restored inside the image, never copied in.
**/node_modules
# Vitest
.vitest-cache/
# OS metadata.
**/.DS_Store
**/Thumbs.db
# Environment / secrets
.env
.env.*
*.pem
*.key
# Editor state: never a build input, and it churns COPY.
**/*.swp
**/*.swo
**/*~
**/*.bak
**/.idea
**/.vscode
**/*.sublime-*
# TypeScript / build artifacts: the image compiles its own.
/dist
/build
/*.tsbuildinfo
/coverage
/.nyc_output
/.vitest-cache
# Compiled binary (built by make build-bin); around 100 MB
bin/quak
/bin/quak
# quak runtime data (in case anyone runs the CLI from inside the repo)
.quak/
# Local per-developer tool state, including agent worktrees. Correctness,
# not context size: a worktree copied in here has its own test/ tree, which
# vitest globs alongside the real one, so the containerised suite runs N+1
# times over and still reports success.
.claude/
/.quak
.gitignore
+32 -10
View File
@@ -11,9 +11,41 @@ Thumbs.db
.vscode/
*.sublime-*
# Agent scratch (worktrees of this repo, created and destroyed by
# in-flight tooling). Unanchored: .gitignore patterns already match at
# every depth, so no prefix is wanted here. This is not a .dockerignore
# entry and must not be given a `**/` prefix on the way into one.
.claude/
# Node
node_modules/
# Secrets. Unanchored like every entry above, so each matches at every
# depth. Matching is case-sensitive on Linux, so names use character
# ranges rather than a lowercase form that misses `Server.Key`.
# Environment files. `*.env` covers bare `.env` and the `prod.env`
# convention. Only the templates `example.env` and `sample.env` are
# re-included below. A repository that commits any other template adds
# its own negation after these lines, for example `!.env.example`.
*.[eE][nN][vV]
.[eE][nN][vV].*
.[eE][nN][vV][rR][cC]
!example.env
!sample.env
# Private keys and the bundles carrying them.
*.[pP][eE][mM]
*.[kK][eE][yY]
*.[pP]12
*.[pP][fF][xX]
[iI][dD]_[rR][sS][aA]
[iI][dD]_[dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]
[iI][dD]_[eE][cC][dD][sS][aA]_[sS][kK]
[iI][dD]_[eE][dD]25519
[iI][dD]_[eE][dD]25519_[sS][kK]
# TypeScript / build artifacts
dist/
build/
@@ -24,18 +56,8 @@ coverage/
# Vitest
.vitest-cache/
# Environment / secrets
.env
.env.*
*.pem
*.key
# Compiled binary (built by make build-bin)
bin/quak
# quak runtime data (in case anyone runs the CLI from inside the repo)
.quak/
# Local per-developer tool settings and scratch state, including the
# worktrees agents check out under this directory
.claude/
.prettierignore
-3
View File
@@ -1,5 +1,2 @@
node_modules/
yarn.lock
dist/
build/
coverage/
Dockerfile
+57 -20
View File
@@ -1,37 +1,74 @@
# Lint stage — fast feedback on formatting and lint issues
# Lint phase. The linters are invoked directly rather than through `make
# lint` or `script/lint`, which are themselves a docker build and would
# recurse into a daemon that does not exist in a build step.
#
# node 22.22.0 on Alpine 3.23.3 (node:22-alpine), 2026-08-09
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS lint
WORKDIR /app
COPY script/ script/
COPY package.json yarn.lock ./
RUN script/bootstrap
COPY . .
RUN make fmt-check
RUN make lint
# Check stage — the full suite and the build
COPY . .
RUN yarn run eslint .
RUN yarn run prettier --check .
# Test phase, same shape and for the same reason. The suite runs without
# verbose output first and is rerun verbosely only if it fails; the timeout
# catches a hung test.
#
# node 22.22.0 on Alpine 3.23.3 (node:22-alpine), 2026-08-09
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS check
WORKDIR /app
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34 AS test
# Force BuildKit to run the lint stage before proceeding. Without this the
# two stages run in parallel and a lint failure can lose the race.
COPY --from=lint /app/yarn.lock /dev/null
WORKDIR /app
COPY script/ script/
COPY package.json yarn.lock ./
RUN script/bootstrap
COPY . .
# CHECK_EPOCH is a cache buster: without it Docker serves `make check` from
# cache on an unchanged tree, the suite never executes, and the build still
# exits 0. The guard makes an absent argument a hard failure — an unset ARG
# is the empty string, which is a perfectly stable cache key, so a plain
# `docker build .` would otherwise still get the false green. Fail closed.
ARG CHECK_EPOCH
RUN [ -n "$CHECK_EPOCH" ] || exit 1
RUN make check
# Unlike the template, the suite runs as the image's non-root `node` user:
# root ignores directory permissions, so the tests of a destination that is
# not writable would otherwise fail. vitest writes into /app.
RUN chown -R node:node /app
USER node
RUN timeout 90 yarn run vitest run --reporter=dot || \
{ echo "--- Rerunning with verbose for details ---"; \
timeout 90 yarn run vitest run --reporter=verbose; exit 1; }
# Build stage, and the last stage: a plain `docker build .` names no target
# and so builds this one. Nothing is wanted from the two phases above; the
# copies are what make BuildKit build them first, so this image cannot be
# produced unless lint and test passed. A stage appended after this one
# would drop all three out of a plain build.
#
# node 22.22.0 on Alpine 3.23.3 (node:22-alpine), 2026-08-09
FROM node@sha256:e4bf2a82ad0a4037d28035ae71529873c069b13eb0455466ae0bc13363826e34
WORKDIR /app
COPY --from=lint /app/package.json /dev/null
COPY --from=test /app/package.json /dev/null
COPY script/ script/
COPY package.json yarn.lock ./
RUN script/bootstrap
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /app
COPY . .
# Version stamped into the build: the VERSION build arg when one is given,
# otherwise what script/version derives from the .git the build context
# carries (script/bootstrap installed git), so any `docker build .` of a
# clone stamps its commit. The label can only carry the build arg, and is
# empty without one.
ARG VERSION
LABEL org.opencontainers.image.version="${VERSION}"
ARG CHECK_EPOCH
RUN [ -n "$CHECK_EPOCH" ] || exit 1
RUN make build
Makefile
+4 -2
View File
@@ -26,8 +26,10 @@ check:
build:
@script/build
build-bin:
nix-shell -p bun --run "bun build bin/quak.ts --compile --outfile bin/quak"
# Bundles the built dist/, so the binary reports the version script/build
# stamped.
build-bin: build
nix-shell -p bun --run "bun build dist/bin/quak.js --compile --outfile bin/quak"
install: build-bin
mkdir -p ~/bin
README.md
+1219 -161
View File
File diff suppressed because it is too large Load Diff
REPO_POLICIES.md
+357 -86
View File
@@ -1,6 +1,6 @@
---
title: Repository Policies
last_modified: 2026-07-06
last_modified: 2026-10-04
---
This document covers repository structure, tooling, and workflow standards. Code
@@ -60,17 +60,28 @@ style conventions are in separate documents:
prerequisite since nvm requires bash. yarn is then pinned via
`corepack prepare yarn@<version> --activate`. Never install "latest" or "lts";
always exact versions. `script/cibuild` runs the CI build: it changes to the
repo root and runs `docker build .`; the Gitea workflow calls it. Four further
scripts are our own extensions to the standard: `script/check` runs
`script/test`, `script/lint`, and `script/fmt-check`; `script/precommit` is
what the git pre-commit hook runs, and it calls `script/check`;
`script/install-precommit` installs the git pre-commit hook (the `make hooks`
target shims to it); and `script/projectname` (literally that filename) simply
outputs the project's name. Scripts that need the name call
`script/projectname` — e.g. `script/docker` assembles its image tag from it —
so those scripts stay byte-identical across all repos. Repo-type-specific
pre-commit extras (e.g. `go mod tidy` verification in Go repos) belong in
`script/precommit`, not in the hook itself. Model scripts are at
repo root, runs `script/bootstrap`, runs `script/check`, and builds the image
with the version; the Gitea workflow calls it. **`script/cibuild` runs
`script/bootstrap` first**, because the workflow checks out the repo and runs
nothing else, while `script/fmt-check` runs the formatter on the host: on a
pristine checkout with nothing installed the run dies there, after the
containerised gates have passed. **The bootstrap alone is not enough**:
`script/bootstrap` installs node and yarn under nvm and leaves neither on the
`PATH` of the shell that called it, so a bare `yarn` still exits 127. The host
entrypoints that need yarn — `script/fmt` and `script/fmt-check` — therefore
source nvm for the pinned node version before invoking it, exactly as
`script/bootstrap`'s own install step does. A runner carrying nothing but
docker and git then gets through `script/check`. Four further scripts are our
own extensions to the standard: `script/check` runs `script/test`,
`script/lint` and `script/fmt-check`; `script/precommit` is what the git
pre-commit hook runs, and it calls `script/check`; `script/install-precommit`
installs the git pre-commit hook (the `make hooks` target shims to it); and
`script/projectname` (literally that filename) simply outputs the project's
name. Scripts that need the name call `script/projectname` — e.g.
`script/docker` assembles its image tag from it — so those scripts stay
byte-identical across all repos. Repo-type-specific pre-commit extras (e.g.
`go mod tidy` verification in Go repos) belong in `script/precommit`, not in
the hook itself. Model scripts are at
`https://git.eeqj.de/sneak/prompts/raw/branch/main/script/<name>`. The README
must document the provided scripts in an **Entrypoints** section (see the
README requirements below).
@@ -89,87 +100,198 @@ style conventions are in separate documents:
contributor should be able to understand the entire development workflow by
reading the Makefile.
- Every repo should have a `Dockerfile`. All Dockerfiles must run `make check`
as a build step so the build fails if the branch is not green. For non-server
repos, the Dockerfile should bring up a development environment and run
`make check`. For server repos, `make check` should run as an early build
stage before the final image is assembled. Dockerfiles install development
prerequisites by running `script/bootstrap` rather than duplicating installs
inline; COPY `script/` and the dependency manifests (`package.json` +
`yarn.lock`, `go.mod` + `go.sum`, etc.) before running it so the bootstrap
layer stays cached until dependencies change.
- Every repo should have a `Dockerfile`, and it carries the repo's gates: a
`lint` phase and a `test` phase, with the final stage depending on both so the
image cannot be built unless they pass. For non-server repos the final stage
brings up a development environment; for server repos it is the runtime image.
The gate phases and the build stage start from their pinned base images and
install what those images lack either inline, as the canonical Go `Dockerfile`
below does for `git`, or by running `script/bootstrap`, as the `prompts`
repo's own `Dockerfile` does for its yarn packages. The development
environment stage installs development prerequisites by running
`script/bootstrap` rather than duplicating its installs inline. A stage that
runs `script/bootstrap` COPYs `script/` and the dependency manifests
(`package.json` + `yarn.lock`, `go.mod` + `go.sum`, etc.) before running it.
- **Dockerfiles must use a separate lint stage for fail-fast feedback.** Go
repos use a multistage build where linting runs in an independent stage based
on the `golangci/golangci-lint` image (pinned by hash). This stage runs
`make fmt-check` and `make lint` before the full build begins. The build stage
then declares an explicit dependency on the lint stage via
`COPY --from=lint /src/go.sum /dev/null`, which forces BuildKit to complete
linting before proceeding to compilation and tests. This ensures lint failures
surface in seconds rather than minutes, without blocking on dependency
download or compilation in the build stage.
- **Linting and testing run in Docker, as phases of the `Dockerfile`.** There is
no separate lint file. `script/lint` and `script/test` each build one phase
and nothing else:
The standard pattern for a Go repo Dockerfile is:
```sh
docker build --no-cache --target lint -t "$(script/projectname)-lint" .
docker build --no-cache --target test -t "$(script/projectname)-test" .
```
**A stage that is not the last one in the file is built only when the final
stage's chain depends on it, or when `--target` names it.** That is why the
two gates are always invoked by name here, and why the final stage carries a
`COPY --from=` of a harmless file from each of them: without that edge a
plain `docker build .` builds the last stage alone and exits 0 having linted
and tested nothing.
**Every `docker build` in `script/` is tagged**, here and in
`script/cibuild` and `script/docker`. An untagged build leaves a dangling
image behind on every invocation, on every developer host and every CI
runner; a tagged one replaces the previous image.
Inside a phase the tool is invoked directly — `golangci-lint`, `go test`,
`eslint`, `prettier` — never through `make lint` or `script/test`, which are
themselves a `docker build` and would recurse into a daemon that does not
exist in a build step. Formatting is the exception and stays on the host:
`script/fmt` writes the working tree, and `script/fmt-check` is its
read-only twin.
**No lint verdict may come from a host invocation of the linter.** On a
shared host golangci-lint reads a result cache keyed on file content rather
than location, so a second checkout of the same content is served the first
one's findings, and a host-global lock in `$TMPDIR` makes concurrent runs
exit non-zero with `parallel golangci-lint is running` — a status a caller
cannot tell from real findings. Both have produced wrong verdicts in this
org, in both directions. A container has its own cache, its own `TMPDIR` and
a digest-pinned binary, so neither is reachable.
- **Any build that runs checks is built with `--no-cache`.** Docker invalidates
a `COPY` layer only when the copied content changes, so on an unchanged tree
the check `RUN` is served from cache, nothing executes, and the build still
exits 0. Every `docker build` in `script/` therefore passes `--no-cache`:
`script/lint`, `script/test`, `script/cibuild` and `script/docker` are the
four, and there is no fifth — `script/check` runs the two gate phases and
`script/fmt-check`, and builds no image of its own. A bare `docker build .` is
not evidence that anything ran: a sub-second build reporting success is a
cache hit, not a result. Never invalidate by pruning — `docker builder prune`
and friends destroy a build cache shared with every other build on the host.
When a check is added or changed, prove it works by planting a defect it must
catch and watching the run fail on it, then revert the defect. A green run
alone shows neither that the check ran nor that it covers what it should.
- **The gate phases are separate stages, and the build stage depends on both.**
The lint phase is based on the `golangci/golangci-lint` image (pinned by
hash), so lint failures surface in seconds rather than after a full compile,
and the test phase is based on the Debian Go image. The canonical Go repo
`Dockerfile`:
```dockerfile
# Lint stage — fast feedback on formatting and lint issues
# Lint phase
# golangci/golangci-lint:v2.x.x, YYYY-MM-DD
FROM golangci/golangci-lint@sha256:... AS lint
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make fmt-check
RUN make lint
RUN golangci-lint run --config .golangci.yml ./...
# Build stage
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
# Test phase. -race needs cgo and so a C compiler, which the Debian Go
# image ships and the alpine one does not.
# golang:1.x, YYYY-MM-DD
FROM golang@sha256:... AS test
WORKDIR /src
# Force BuildKit to run the lint stage before proceeding
COPY --from=lint /src/go.sum /dev/null
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN make test
RUN go test -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 90s -race -v ./...; exit 1; }
ARG VERSION=dev
RUN CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Build stage. Nothing is wanted from either phase above; the copies
# are what make BuildKit build them first, so this stage cannot run
# unless lint and test passed.
# golang:1.x-alpine, YYYY-MM-DD
FROM golang@sha256:... AS builder
COPY --from=lint /src/go.sum /dev/null
COPY --from=test /src/go.sum /dev/null
RUN apk add --no-cache git
# A tar-stream context keeps the sender's file owners, which git refuses.
RUN git config --system --add safe.directory /src
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
# Runtime stage
# The VERSION build arg when one is given, otherwise
# `git describe --tags --always` on the .git in the build context. With
# .git present, a version that is still empty, dev or unknown fails the
# build: git is missing or could not read the checkout.
ARG VERSION
RUN VERSION="${VERSION:-$(git describe --tags --always)}"; \
if [ -e .git ]; then \
case "$VERSION" in ""|dev|unknown) \
echo "version is '$VERSION' although .git is present" >&2; \
exit 1 ;; \
esac; \
fi; \
CGO_ENABLED=0 go build -trimpath \
-ldflags="-s -w -X main.Version=${VERSION}" \
-o /app ./cmd/app/
# Runtime stage, and the last one
FROM alpine@sha256:...
COPY --from=builder /app /usr/local/bin/app
ENTRYPOINT ["app"]
```
Key points:
- The lint stage uses the `golangci/golangci-lint` image directly (it
includes both Go and the linter), so there is no need to install the
linter separately.
- `COPY --from=lint /src/go.sum /dev/null` is a no-op file copy that creates
a stage dependency. BuildKit runs stages in parallel by default; without
this line, the build stage would not wait for lint to finish and a lint
failure might not fail the overall build.
- The lint phase uses the `golangci/golangci-lint` image directly (it has
both Go and the linter), so nothing needs installing.
- `COPY --from=<phase> /src/go.sum /dev/null` is a no-op copy whose only
purpose is the ordering edge. BuildKit runs stages in parallel by default,
and a stage nothing depends on is not built at all, so without these two
lines a red gate would not fail the build.
- Keep the runtime stage last, and if you add a stage after it, give it the
same two copies. A plain `docker build .` builds the last stage's chain
and nothing else.
- If the project uses `//go:embed` directives that reference build artifacts
(e.g. a web frontend compiled in a separate stage), the lint stage must
(e.g. a web frontend compiled in a separate stage), the lint phase must
create placeholder files so the embed directives resolve. Example:
`RUN mkdir -p web/dist && touch web/dist/index.html web/dist/style.css`.
The lint stage should not depend on the actual build output — it exists to
fail fast.
- If the project requires CGO or system libraries for linting (e.g.
`vips-dev`), install them in the lint stage with `apk add`.
- The build stage runs `make test` after compilation setup. Tests run in the
build stage, not the lint stage, because they may require compiled
artifacts or heavier dependencies.
- If the project requires CGO or system libraries for linting, install them
in the lint phase. The `golangci/golangci-lint` image is Debian-based and
has no `apk`, so install with `apt-get` under the Debian package name
(`libvips-dev`, where alpine says `vips-dev`), and delete the package
lists in the same `RUN`, so the layer does not keep them:
```dockerfile
RUN apt-get update \
&& apt-get install -y --no-install-recommends libvips-dev \
&& rm -rf /var/lib/apt/lists/*
```
- `.dockerignore` lets `.git` into the build context. It keeps out every git
`config` at any depth (`**/.git/config`, `**/.git/modules/**/config`): the
repository's own, each submodule's under `.git/modules/`, and that of a
submodule keeping its own `.git` directory. `git describe` does not need
them, and each can hold a credential: a password in a remote URL, or the
token the CI checkout step stores there. A submodule whose name has a
`config` segment (`config`, `deploy/config`, `config/lib`) loses its whole
git directory to `**/.git/modules/**/config`, and Go's version stamping
then fails the build: give it a name without that segment
(`git submodule add --name`). The stage that compiles has `git` (the
Debian Go image has it; an alpine one needs `apk add --no-cache git`) and
takes the version from the `VERSION` build argument when one is given,
otherwise from `git describe --tags --always`. That gives the tag on a
tagged commit; on a later commit, the tag, the number of commits since it
and the short commit (`v1.2.3-4-gabc1234`); and the short commit when no
tag is reachable. The stage that compiles also marks its working directory
safe for git (`git config --system --add safe.directory /src`): a context
sent as a tar stream keeps the sender's file owners, and git refuses a
checkout owned by another user, so the version would come out empty.
`ARG VERSION` has no default, and the build fails if the context carries
`.git` and the version still comes out empty, `dev` or `unknown`. A plain
`docker build .` with no build arguments must succeed; a Dockerfile that
refuses an empty build argument drops that refusal and keeps the argument.
- Every repo should have a Gitea Actions workflow (`.gitea/workflows/`) that
runs `script/cibuild` (which runs `docker build .`) on push. Since the
Dockerfile already runs `make check`, a successful build implies all checks
pass.
runs `script/cibuild` on push, and checks out the repo as its only other step.
That script bootstraps, runs the gate phases, and then builds the image, so a
successful run means every check passed; a bare `docker build .` does not
carry the same guarantee, because its gate phases may come from the cache. The
image build is uncached and so runs the gate phases a second time. That is the
price of the rule above, and it is worth paying: the image that ships is built
from a run of its own gates rather than from a cache entry. A separate
workflow limited to `main` by a `branches` list under `on: push` cannot be
checked by review: to try a change to it, add the feature branch to that list
and push, then remove the branch from the list again before merging. Keep any
job in it that publishes behind `if: github.ref_name == 'main'`, so the run
from the feature branch publishes nothing.
- Use platform-standard formatters: `black` for Python, `prettier` for
JS/CSS/Markdown/HTML, `go fmt` for Go. Always use default configuration with
@@ -189,14 +311,21 @@ style conventions are in separate documents:
module under test to verify it compiles/parses. There is no excuse for
`make test` to be a no-op.
- `make test` must complete in under 20 seconds. Add a 30-second timeout in the
Makefile.
- `make test` must complete in under 60 seconds. That is the hard cap, and a
suite that exceeds it fails. Under 20 seconds is the target. A suite between
20 and 60 seconds is still green, but the overage must be filed as an
improvement bug against that repo. Add a 90-second timeout to the test
invocation (`go test -timeout 90s`). The backstop deliberately sits above the
hard cap so that it catches a genuinely hung test rather than a merely slow
one.
- **`make test` should use the conditional verbose rerun pattern.** Run tests
without `-v` (verbose) first. If tests fail, automatically rerun with `-v` to
show full output. This keeps CI logs and `docker build` output clean on
success (just package/suite summaries) while providing full diagnostic detail
on failure (every test case, every assertion). The general shell pattern:
- **The test command should use the conditional verbose rerun pattern.** Run
tests without `-v` (verbose) first. If tests fail, automatically rerun with
`-v` to show full output. This keeps CI logs and `docker build` output clean
on success (just package/suite summaries) while providing full diagnostic
detail on failure (every test case, every assertion). The command lives in the
`test` phase of the `Dockerfile`, since `script/test` builds that phase; the
Makefile form below is the same pattern for any repo-local invocation:
```makefile
test:
@@ -209,11 +338,26 @@ style conventions are in separate documents:
```makefile
test:
@go test -timeout 30s -race -cover ./... || \
@go test -count=1 -timeout 90s -race -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \
go test -timeout 30s -race -v ./...; exit 1; }
go test -count=1 -timeout 90s -race -v ./...; exit 1; }
```
`-count=1` is required on both invocations: it defeats Go's test _result_
cache, so neither run can report a stored pass in place of running the
tests. It leaves the build cache alone, so it costs the runtime of the suite
and no recompilation.
That cache is Go's own, separate from Docker's layer cache. Go stores a
passing result in its cache directory (`GOCACHE`), and when the same tests
run again on unchanged code it prints that result, marked `(cached)`,
without running them. That matters on a developer's machine, where this
target runs and the directory lasts from one run to the next. The `test`
phase of the `Dockerfile` needs no `-count=1`: its base image holds no
result for this repo's tests and nothing before its `go test` step runs a
test, so there is nothing to replay. `--no-cache` (above) is what makes that
step run on an unchanged tree.
Python example:
```makefile
@@ -239,10 +383,84 @@ style conventions are in separate documents:
must be in `.gitignore`. No exceptions.
- `.gitignore` should be comprehensive from the start: OS files (`.DS_Store`),
editor files (`.swp`, `*~`), language build artifacts, and `node_modules/`.
Fetch the standard `.gitignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when setting up
a new repo.
editor files (`.swp`, `*~`), in-repo agent scratch directories (`.claude/`),
language build artifacts, and `node_modules/`. Fetch the standard `.gitignore`
from `https://git.eeqj.de/sneak/prompts/raw/branch/main/.gitignore` when
setting up a new repo. These patterns are written to `.gitignore`'s own
semantics, in which an unanchored pattern already matches at every depth; they
are not a `.dockerignore` and must not be transplanted into one unmodified.
- **`.dockerignore` does not use `.gitignore` semantics, and copying patterns
across unmodified leaves secrets in the build context.** Docker matches with
`moby/patternmatcher`: `filepath.Match` semantics plus a `**` extension, so
`*` does not cross `/` and a pattern without a leading `**/` is anchored at
the build-context root. A `.dockerignore` listing `.env`, `*.pem` and `*.key`
therefore excludes only the copies at the repository root, while `config/.env`
and `certs/server.key` still reach the context and can land in an image layer
— which is more dangerous than a short file with no secret patterns at all,
because it reads as solved and stops anyone looking. Give every
depth-independent pattern the `**/` prefix and leave only genuinely
root-anchored entries unprefixed: `.claude`, and the repo's own host-built
binary, written `/myapp` and never `**/myapp`, which would also match
`cmd/myapp/` and delete the package directory from the context. Matching is
case-sensitive, and an ALL-CAPS twin per pattern still misses `Server.Key`, so
secret names use character ranges — `**/*.[kK][eE][yY]`, `**/*.[pP][eE][mM]`,
and likewise for `.envrc` and the extensionless SSH keys. Where such a pattern
also catches something the build needs, re-include it with a negation
(`!docs/example.env`); deleting the pattern reopens the exposure for every
other file it covers. Fetch the standard `.dockerignore` from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.dockerignore` and extend
it with the repo's own artifacts.
- **In-repo agent scratch belongs in both files, written to each file's own
semantics.** `.claude/` holds one worktree per in-flight agent — an entire
additional checkout of the repo — so under `COPY . .` the build context
inflates by a multiple of the repo and another session's unreviewed work can
be copied into an image layer. In `.gitignore` the entry is `.claude/`,
unanchored. In `.dockerignore` it is `.claude`, anchored and with **no** `**/`
prefix, because the prefixed form would also delete any nested directory of
that name from the build. Anchoring carries a known gap that the canonical
`.dockerignore` states in its own comment, since consuming repos receive the
file and not the tracker: the directory is created in the agent's working
directory, so a repo running agents in subdirectories still ships
`services/api/.claude/` and must add its own anchored entry there.
- **A plain `docker build .` of a clone stamps the version that
`git describe --tags --always` gives**, derived from the `.git` in the build
context as the canonical `Dockerfile` above shows. Without its failure check,
a missing `git` or an unreadable checkout would leave `-X main.Version=` empty
and the build would still exit 0. `script/docker` and `script/cibuild` pass
the version they compute on the host; it takes precedence. They do this
byte-identically across repos:
```sh
# Own line: a failing command substitution inside an argument does not
# trip `set -e`, so the inline form degrades to an empty constant.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$(script/projectname)" .
```
`--always` makes an untagged repo yield an abbreviated commit hash rather
than failing, and the `[ -n "$version" ]` line is the single place the
fallback is applied — a live check that fires on a build from an export with
no `.git` and on a repository with no commits yet. Do not fold it into the
substitution as `|| echo unknown`, which makes the guard unreachable. The
Dockerfile's side is `ARG VERSION` in the stage that compiles, declared
there because `ARG` is stage-scoped; passing `VERSION` to a repo whose
Dockerfile declares no such `ARG` is ignored and costs nothing, which is why
the scripts stay byte-identical. One consequence for CI: the standard
checkout action clones shallow and fetches no tags, so a repo that embeds a
tag-derived version must set `fetch-depth: 0` on its checkout step.
- **Verify `.dockerignore` by enumerating the image, not by reading the
patterns.** Plant files at the root _and_ at least two directories deep, build
a probe image that does `COPY . .`, and list what actually landed
(`docker run --rm --entrypoint find IMAGE /app`). The `transferring context`
size is not a substitute: a nested secret is a few bytes, and BuildKit
transfers only the delta from the previous build.
- **No build artifacts in version control.** Code-derived data (compiled
bundles, minified output, generated assets) must never be committed to the
@@ -258,9 +476,56 @@ style conventions are in separate documents:
- Make all changes on a feature branch. You can do whatever you want on a
feature branch.
- `.golangci.yml` is standardized and must _NEVER_ be modified by an agent, only
manually by the user. Fetch from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml`.
- `.golangci.yml` is standardized. The vendored copy in a consuming repo must
_NEVER_ be modified by an agent: fetch it from
`https://git.eeqj.de/sneak/prompts/raw/branch/main/.golangci.yml` and keep it
byte-identical, so that no repo can quietly loosen its own linting. Linter
configuration changes are made to the canonical copy in the `prompts` repo and
reach consuming repos by re-vendoring; an agent may open a PR against
canonical, which only the user merges. One list is exempt from byte-identity,
because it cannot be written once for every repo: the `deny` list of the
`test-support` depguard rule, where a repo names its own test-support packages
by full import path. A repo adds entries there and changes nothing else, and a
re-vendor carries its entries forward. The canonical golangci-lint version is
v2.14.0 (released 2026-09-24), pinned as the digest of the lint phase's base
image
(`golangci/golangci-lint@sha256:ad862ba6b3798cbe0fd9fd7408d498fd74fbd2623a92406b2fd3898faf0bf98f`,
which reports `2.14.0 built with go1.27.0 from 114493f9`). A module's `go`
directive must not name a newer Go minor version than the one golangci-lint
was built with, or golangci-lint refuses to lint it: this release lints
`go 1.27.1` but not `go 1.28`. That digest is the only pin, since no repo
installs golangci-lint on the host. A repo sets the lint phase digest to the
one named here and re-vendors `.golangci.yml` in the same commit, whichever of
the two prompted the change: the canonical copy can name linters that an older
golangci-lint rejects, and a newer golangci-lint can add linters that
`default: all` switches on until the canonical copy disables them.
- **`script/bootstrap` installs a pinned tool by comparing versions, never by
testing presence.** An `if ! command -v <tool>; then install; fi` guard tests
`PATH` only, so on an already-provisioned machine the pin is inert and a
version bump is a silent no-op — while the Dockerfile, installing into a clean
image, gets the pinned version, so a local `make check` and `make docker` can
disagree about what the tool even is. The canonical form:
- compares the installed version against the pin over the **whole** version
token; a parser that stops at the first `-` reports `2.12.2` for a host
running `2.12.2-rc1` and skips the install;
- treats absent, non-zero, empty or unrecognised `--version` output as a
mismatch, so the failure direction is a redundant install and never a
skipped one;
- after installing, re-resolves the binary the way callers do — `hash -r`,
then through `PATH`, not through the directory the installer wrote to —
and fails naming the resolved path, since an install that a shadowing
binary hides succeeds while changing nothing any caller sees;
- is actually called, and prints the version on both success paths: a
function defined and never invoked has the same exit status and the same
empty output as one that worked.
Keep it POSIX sh: no arrays, no `[[`, no `grep -P`.
A Go tool a repo needs on the host is installed with `go install` pinned to
a commit hash (`go install <package>@<commit hash>`). It is never tracked as
a `go.mod` tool dependency or through a `tools.go` file, either of which
pulls the tool's own dependencies into the repo's `go.mod` and `go.sum`.
- When pinning images or packages by hash, add a comment above the reference
with the version and date (YYYY-MM-DD).
@@ -374,12 +639,14 @@ style conventions are in separate documents:
settings.
- Avoid putting files in the repo root unless necessary. Root should contain
only project-level config files (`README.md`, `Makefile`, `Dockerfile`,
`LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`, and
language-specific config). Everything else goes in a subdirectory. Canonical
subdirectory names:
only project-level config files (`README.md`, `AGENTS.md`, `Makefile`,
`Dockerfile`, `LICENSE`, `.gitignore`, `.editorconfig`, `REPO_POLICIES.md`,
and language-specific config). Everything else goes in a subdirectory.
Canonical subdirectory names:
- `bin/` — executable scripts and tools
- `cmd/` — Go command entrypoints
- `cmd/` — Go command entrypoints; thin only: one `main.go` per binary whose
body is a single call into `internal/` or `pkg/`, no project logic in
`cmd/`
- `configs/` — configuration templates and examples
- `deploy/` — deployment manifests (k8s, compose, terraform)
- `docs/` — documentation and markdown (README.md stays in root)
@@ -406,3 +673,7 @@ style conventions are in separate documents:
- Go: `go.mod`, `go.sum`, `.golangci.yml`
- JS: `package.json`, `yarn.lock`, `.prettierrc`, `.prettierignore`
- Python: `pyproject.toml`
- Guidance for coding agents lives in one `AGENTS.md` at the repository root. It
is never committed under a file or directory named after one agent tool, such
as `CLAUDE.md` or `.claude/`, and never split into separate memory files.
TODO.md
+680 -4
View File
@@ -1,12 +1,15 @@
# Workflow
- branch (from `main`)
- branch from `next`
- do the work in Next Step
- move Next Step to the top of Completed Steps
- move the top item of Future Steps into Next Step
- commit (`TODO.md` changes in the same commit as the work)
- merge to `main` if the branch is not protected, otherwise open a PR
- push
- open a pull request that targets `next`
- once the pull request has passed review, the repository manager squash-merges
it into `next`
- only sneak merges `next` into `main`
# Status
@@ -14,10 +17,684 @@ pre-1.0
# Next Step
Update the README API reference section to match the current implementation.
No bug is open, and no change sneak asked for is open. The check of quak against
a live Ente account, https://git.eeqj.de/sneak/quak/issues/187, needs an account
only sneak has, so it waits on him.
Tagging and releases are decided by sneak alone, and happen only when he
declares one.
# Completed Steps
- 2026-10-25: `quak files COLLECTION` fetches the files of that collection alone
(https://git.eeqj.de/sneak/quak/issues/217). It used to wait for a refresh,
which on an empty cache fetched the files of every collection first. It now
reads the collection list through `lib.freshCollections()`, then brings that
one collection up to date through the new `lib.refreshCollection()`. That
stores and saves its files as a refresh would, and leaves the refresh cursor
where it was. When the stored copy is current, it asks for none of the files.
It never runs at the same time as a refresh. Once a refresh has received the
account's collections, it stores no collection the cache does not hold, and
`quak files COLLECTION` waits for a refresh instead. A refresh from a cursor
of 0 replaces all the cache holds, so collections keep the server's order. A
library opened on a cache that only `lib.refreshCollection()` wrote awaits its
first refresh. `quak files` with no argument still waits for a refresh. The
store lists what did not decrypt by collection ID and file ID.
- 2026-10-22: The JSON `quak backup` and `quak backup-metadata` write for a file
holds `keywords` in place of the CLIP embedding in its ML data
(https://git.eeqj.de/sneak/quak/issues/218). The keywords are the titles of
the Discover categories of Ente's photos app that the file falls in, by the
app's rule: the cosine similarity of the file's CLIP embedding and the text
embedding of a category's query is at least the category's minimum score, and
videos stay out of the categories that leave them out. `src/clip-keywords.ts`
applies the rule. `src/clip-keyword-vectors.ts` holds the 16 categories with
their text embeddings. `bin/clip-keyword-vectors` writes it with Ente's CLIP
text model, and writes the CLIP embedding of a public-domain photo of a sunset
that the tests use.
- 2026-10-18: `-v`/`--verbose` raises the CLI's log level once for each repeat,
and `-q`/`--quiet` prints only fatal errors
(https://git.eeqj.de/sneak/quak/issues/212). The levels are fatal, error,
warning, info, debug and trace; the default is info, everything quak printed
on stderr before, `-v` gives debug and `-vv` trace. `src/cli-log.ts` holds the
level and the log the commands write stderr lines through. Debug adds each
retry, which `withRetry` reports through the new `onRetry`, and the start and
end of each refresh and ML data fetch; trace adds each request, which
`ApiClient` reports through the new `onRequest`. `--json` no longer keeps the
progress lines off stderr, so `backup` and `--json` runs log a refresh's
progress too. Every line that reports a failed file or album is at error,
which the progress callbacks of `backup`, `backup-metadata` and the thumbnail
helpers now say, and a command that exits 1 because files or albums failed
ends with one fatal line saying how many and where to look.
- 2026-10-18: `quak files` takes the collection ID as an optional argument
instead of the `--collection` option, which is gone
(https://git.eeqj.de/sneak/quak/issues/211). With no argument it lists every
file in the account once, collection by collection; a file in several
collections is listed with the first of them, which is its `collectionID` in
`--json`. The fresh read in `src/cli-read.ts` does this.
- 2026-10-18: Names quak builds from an album name or a title fit in a file name
(https://git.eeqj.de/sneak/quak/issues/205). `shortenKeepingExtension` in
`src/filename.ts` cuts a name short to a number of bytes, keeping its
extension when it fits and always its first character, and can put text right
before the extension; the backup's `withID` uses it for an ID.
`defaultFileName` there gives the name a file is saved under when the caller
gives no path: a prefix such as `thumb_` and the title with unsafe characters
replaced, cut short with it to 255 bytes, and, for a live photo's image or
video, with that part's own extension before the cut. `quak get`,
`quak get-thumb`, `downloadFile` and `downloadThumbnail` use it without
`--out` or `outPath`. `backup-metadata` cuts an album's name short so that its
folder, `<id>-<name>`, fits in 255 bytes. `safeExtension` gives `.bin` for an
extension of more than 16 letters and digits, so a save path's name always
fits.
- 2026-10-18: `quak collections` no longer waits for the files of every
collection (issue 210). On an empty cache it opened its library with a refresh
that fetched and decrypted the files of each collection in turn, printing
nothing and storing nothing until all were done, so on a large account it
seemed to hang. It now reads the collection list alone through the new
`lib.freshCollections()`, one request that stores nothing. The read commands
open their library with the new `refreshOnOpen: false`, so each makes only the
refresh it asks for. A refresh sends a `"progress"` event after each
collection whose files it fetched, and the commands that wait on one print it
on stderr, except with `--json`.
- 2026-10-16: ML data the server sends that does not decrypt, gunzip or parse is
no longer taken for no ML data (issue 198). `fetchMLDataBatch` returns why for
each such file beside the payloads; ML data that parses to something other
than a JSON object, such as `null`, is one of them. The library keeps the
reasons from its last ML data fetch that succeeded in memory, not in the
cache, and gives them in `status().mlDataErrors` and in the `fetchMLData`
event with status `"done"`; a fetch still running or one that fails does not
change them. `quak backup` writes the reason as `mlDataError` in the file's
JSON when no ML data is cached for it, and records the file in `failures.json`
with an error starting `ML data: `; `backup-metadata` writes `mlDataError`
too. Both exit 1. Tests serve ML data encrypted under a key that is not the
file's.
- 2026-10-16: With `--verify`, an original whose hash does not match is no
longer removed before its replacement is downloaded (issue 199).
`replaceOriginal` in `src/library/content.ts` downloads the replacement to a
temporary file beside it, a live photo's image and video each to its own,
hashes it and renames it over the original only when it matches. When the
download fails or does not match either, the original keeps its bytes and
links, and the file goes into `failures.json`. The replacement is always
downloaded, through the new `fetchOriginal` of the content cache, never copied
from the cache.
- 2026-10-16: The `~/bin/quak` that `make install` builds retries a refused
connection and a name that does not resolve, and the backup records either as
transient (issue 206). Bun's `fetch` reports them with codes of its own,
`ConnectionRefused` and `FailedToOpenSocket`, which `src/retry.ts` now counts
as transport failures. A `POST` or `PUT` is not replayed on them, because bun
also gives them when it resends a request on a new connection after a
connection it kept open closed, and the server may already have the request.
Bun already gave `ECONNRESET` for a reset and a `TimeoutError` for a deadline.
`classify` in `src/backup.ts` now records any `TimeoutError` as transient, so
a stalled download, whose error says only that it stalled, is no longer
`unknown`. The shapes were found by running the binary against local
listeners.
- 2026-10-15: An album whose folder or JSON `quak backup` cannot write no longer
ends the run before the other albums and `failures.json` are written (issue
197). The run prints `FAILED album <name>: <reason>`, lists the album's ID in
`failedAlbums` in the result and in `--json`, goes on, and exits 1. An album
whose folder would have the name of another album's JSON, as `x.json` beside
`x` would, gets its ID added. A folder or link name over 255 bytes is cut
short between two characters, by `shortenToBytes` in `src/filename.ts`, and
gets the ID.
- 2026-10-15: A network failure is printed and recorded with its reason, not as
just `fetch failed` (issue 196). `describeError` in `src/errors.ts` gives an
error's message followed by the message of each error on its `cause` chain,
such as `fetch failed: connect ECONNREFUSED 1.2.3.4:443`, and each address's
reason when a host has several and none could be reached. `run` prints that,
and the backup logs it and records it in `failures.json`, where `classify` now
calls the failure transient when the chain holds an errno that
`hasTransportCode` in `src/retry.ts` recognises, for a failed ML data fetch
too. The walk along the chain moved from `src/retry.ts` to `src/errors.ts`,
which both use. Tests refuse every connection of a backup, then only one
file's download, and fail the ML data fetch.
- 2026-10-15: The README says that `make install` needs nix (issue 200), in
Getting Started and in "Running the backup from cron": `make build-bin` runs
bun through `nix-shell -p bun`. The `Makefile` is unchanged.
- 2026-10-14: `lib.backup()`, and so `quak backup`, works on up to the content
pool's limit of originals at once, 5 by default, instead of one at a time
(issue 182). That many workers take files from one shared list, each doing all
the work for one original, its `--verify` hashing included, so no more
originals than that are open at once. `BackupLibrary` says the limit as
`contentConcurrency`. A test holds each download at the fake Ente server to
count those in flight. Once a worker has thrown, as one does when `onProgress`
throws, or the lock is lost, no worker takes another file; the backup fails
once the originals under way are in place, and a lost lock stops it before it
fetches ML data.
- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's
review, though that issue closed on 2026-10-03 (issue 194). It now names the
issues open on the tracker.
- 2026-10-14: A backup whose lock is lost no longer dies with a stack trace
(issue 184). `lockBackupDirectory` gives proper-lockfile an `onCompromised`
that records why the lock was lost instead of throwing from its timer, and
returns the lock, which `quak backup` passes to `lib.backup()` as `lock`. The
backup then stops before it starts another file with
`lost the lock on DIR: REASON`, which `quak backup` prints as one line before
it exits 1. Releasing a lost lock leaves `backup.lock` alone. Tests in
`test/cli/backup-end-to-end.test.ts` remove the lock, or put another backup's
in its place, while a download is held open.
- 2026-10-14: An album or file quak cannot decrypt, or whose metadata is not
JSON, no longer fails the refresh, so one bad record no longer stops
`quak backup` from storing anything (issue 181). The refresh leaves it out, an
album with its files, and the library keeps it with the reason in
`metadata.json` until the server sends a copy that decrypts or deletes it.
`quak backup` stores every other file, records each such file in
`failures.json` with an error starting `could not be decrypted:`, names each
such album's ID in a printed line, or with `--json` in `undecryptableAlbums`,
and exits 1 while any remain. A file whose copy in another album decrypts is
not recorded: its original is stored from that copy, and the album where it
does not decrypt neither links nor lists it. The read commands answer from
what decrypted.
- 2026-10-14: `quak backup` no longer asks the server for changes every 3
seconds while it runs (issue 183). It opens its library with
`refreshIntervalSeconds: Infinity`, which the library now takes as no
background timer, so the backup refreshes only as it starts and no timed
refresh starts another ML data pass during the run. A test holds a backup's
download open while two hours pass and counts its `/collections/v2` requests.
- 2026-10-14: `test/cli/backup-end-to-end.test.ts` runs `quak backup` as cron
does (issue 186): `backupCommand` and `run`, the real `loadSession` reading a
session saved after logging in, and every request answered by a fake of the
Ente HTTP API serving data encrypted with libsodium. It covers the backup tree
of an account with two albums of its own and one shared with it, a second run
that downloads nothing, a download that succeeds after two 503s, a 404 that
lands in `failures.json` with exit 1, a refresh answered 401 with exit 3, and
`--verify` putting back an original changed on disk. The fake server and the
account it logs in moved from the backup-metadata test into
`test/ente-server.ts`, unchanged.
- 2026-10-14: `files --collection`, `get`, `get-thumb` and
`helper fix-missing-thumbnails --file` take an ID only when it is written in
decimal digits and is a safe integer (issue 185). Before, `Number()` read
`0x10` as 16, `1e3` as 1000 and an empty string as 0, and `--file` dropped an
ID it could not read without a word. Anything else now prints
`Invalid file ID: <arg>` or `Invalid collection ID: <arg>` and exits 1 before
any request; one bad `--file` ID fails the whole command. `parseID` in
`src/cli-commands.ts` does the check for all four.
- 2026-10-06: The README says how to run `quak backup` from cron (issue 170):
log in once as the job's user, a crontab with a backup every night and
`--verify` on Sundays appending to a log file, and the `HOME` and
`XDG_DATA_HOME` the job needs to find the saved session. A table gives each
exit code: 0, the backup is complete; 1, files are in `failures.json` or
another error stopped the run; 2, another backup of the directory is running;
3, there is no usable session. The introduction lists everything the backup
keeps for each file.
- 2026-10-06: `quak backup --verify` and `lib.backup({ verify: true })` hash
each original already at its save path as the download check does, streamed, a
live photo as `<imageHash>:<videoHash>` (issue 168). One that does not match
the content hash its metadata records is logged, removed (both files of a live
photo) and put back in the same run, downloaded or copied from the cache, and
what is put back is hashed too. One that still does not match, or whose
download fails, goes into `failures.json`. One with no recorded hash is left
alone. The result, `--json` and the summary gain `verified`, `mismatched` and
`unchecked`. Without `--verify` nothing is hashed.
- 2026-10-06: Two backups of the same directory never run at once (issue 169).
`lib.backup()` takes a lock, `backup.lock` in its download directory, made
with `proper-lockfile`, before its refresh, and removes it when it ends,
whether it succeeds or fails. A second backup of the directory, from another
process or the same one, fails at once with an error naming the directory.
`quak backup` takes the lock before it opens its library, so a refused run
sends no request; it prints the error as one line and exits 2. A lock that has
gone 10 seconds untouched, left by a run that could not remove it, is taken
over by the next run.
- 2026-10-06: `quak backup` writes each original's EXIF, XMP and dimensions into
the file's JSON as `imageMetadata`, what `backup-metadata --exif` records
(issue 167): for a live photo from its image, for a video nothing, and `{}`
for an original with none of them. A failed read puts the reason in
`imageMetadataError` and fails neither the file nor the run. An original is
read when the run stores it or when its JSON has neither field; otherwise the
field is taken from that JSON. The hand-built JPEGs moved to
`test/exif-jpeg.ts`, beside the HEIC.
- 2026-10-06: When the server answers HTTP 401 and that ends a command that
loads the saved session, because the server no longer accepts its token, the
command prints one line,
`quak: the saved session is no longer valid; run "quak login"`, and exits 3
(issue 164). `quak backup` meets that 401 on the refresh it starts with,
before it touches any file. For the same commands, a missing or corrupt
session file keeps its message and also exits 3, so a cron job can tell that
the user must log in again. `quak logout` is unchanged. `run` in
`src/cli-run.ts` recognises the 401, which reaches it unchanged.
- 2026-10-06: `quak backup` retries a failed request for longer than the other
commands do (issue 165). `src/retry.ts` exports `UNATTENDED_RETRY_OPTIONS`
beside the unchanged default: 10 attempts, a 1 s base delay and a 60 s cap, so
a request that keeps failing waits at most 243 s before it gives up.
`bin/quak.ts` loads the backup's session with them, so its refresh, ML data
and downloads all use them. What is retried and the backoff formula are
unchanged.
- 2026-10-06: The files this repository copies from `sneak/prompts` are copied
again from its commit `dd4027b` (issue 171). `.gitignore` and `.dockerignore`
keep out more secret files, and this repository's build artifacts follow the
copied content. `script/check` runs `script/fmt-check` again, `script/cibuild`
runs `script/bootstrap` and `script/check` before the image build,
`script/fmt` and `script/fmt-check` find the pinned yarn under nvm, and the
image's last stage marks `/app` safe for git.
- 2026-10-06: `quak backup` writes the account and album records
`backup-metadata` writes (issue 166): `account.json` with the account's
`email` and `userID`, and in each album's JSON its `ownerID`, `isShared`,
`updationTime` and, when present, its three layers of magic metadata. Each
file's JSON gains its `updationTime`.
- 2026-10-05: `quak backup` writes each file's ML data (its faces and CLIP
embedding) into the file's JSON as `mlData`, the payload
`lib.mldata.forFile()` returns (issue 163). `lib.backup()` waits for an ML
data fetch before it writes the JSON files. When that fetch fails, each file
whose ML data is not cached gets the reason in `mlDataError` and counts as
failed, and the next run fetches it again.
- 2026-10-03: The download-albums example test no longer fails on a disk with
under 50 GiB free (issue 160). It opens its libraries with
`freeBelowBytes: 0`, so the free space of the disk it runs on cannot shrink
the cache, evict the original it cached and add a fetch to the count. No other
test's result depends on that free space. The 50 GiB default is unchanged.
- 2026-10-02: `docker build .` stamps the commit's tag or short commit, not
`dev` (issue 154). `script/build` writes the version `script/version` prints
into `dist/package.json`: the `VERSION` environment variable or build arg when
one is given, otherwise `git describe --tags --always`, otherwise the version
`package.json` declares. `.dockerignore` sends `.git`, and a checkout with
`.git` whose version comes out empty, `dev` or `unknown` fails the build.
`make build-bin` bundles the built `dist/`, so the single binary reports the
same version.
- 2026-10-02: `photo.exif()` returns every EXIF tag in the file as `ExifTags`,
keyed by tag name, each as exifreader decodes it, not only the thirteen common
fields (issue 156). The embedded thumbnail's tags are under `Thumbnail`,
without the thumbnail image. The thirteen typed methods stay, each picking its
field from the tags `exif()` returns, typed as in `PhotoExif`. The example
script's JSON files now carry every tag.
- 2026-10-01: The content cache no longer looks for a live photo that an earlier
version cached as one ZIP (issue 151). When the cache opens, a live photo's
file that no JSON file names is now always left alone.
- 2026-10-01: `examples/download-albums.ts` logs in, opens the library, and for
every album downloads each photo to its save path, writes the photo's record
and EXIF fields to a JSON file beside it, and writes the album's photos to
`albums/<collectionID>.json` (issue 144). The build compiles it to
`dist/examples/`; the README's "Examples" section says how to run it.
- 2026-10-01: A `Photo` has one async method for each field of `exif()`, named
and typed as in `PhotoExif`: `make()`, `model()`, `lensModel()`,
`dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`,
`iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()`
and `gpsAltitude()` (issue 148). Each calls `exif()` and returns its one
field, or undefined when the file lacks it. `Photo` implements a type with one
method per `PhotoExif` field, so the build's type check fails when a field has
no method. A test checks, on the JPEG and the HEIC, that each method gives the
same value as `exif()`.
- 2026-10-01: Each original's save path is
`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
download directory, which defaults to `photos` in the working directory (issue
143). The date is the photo's `takenAt` in the machine's time zone.
`photo.savePath` is always a string, with or without a content cache, and
`isLocal` is true only when the original is at its save path.
`photo.download()` puts the original there, copied from the cache when the
cache holds it and fetched otherwise. `lib.backup()` does the same for each
file, writes each file's JSON beside its original, and links `collections/` to
the save paths.
- 2026-10-01: `photo.exif()` and `backup-metadata --exif` read EXIF from
HEIC/HEIF originals, a live photo's HEIC image included, as well as JPEG and
the other image formats `exifreader` reads (issue 145). `exifreader` replaces
`exif-reader` and the JPEG segment scan; `PhotoExif` is unchanged. The `exif`
field of `backup-metadata --exif` is now exifreader's tag output, and
`exifRaw` holds the whole EXIF block it could not read. The tests use a real
HEIC, `test/exif.heic`.
- 2026-10-01: A `Photo` has `savePath`, `isLocal`, `content()`, `exif()`,
`modifiedAt`, `hash` and `year` (issue 141). `savePath` is where
`lib.backup()` writes the original under the library's download directory; for
a live photo not yet stored, it carries the title's extension, and the backup
may store the image under a different one. `isLocal` says whether the whole
original is there. Both look only at the disk. `content()` returns the
original's bytes and `exif()` the common EXIF fields of a JPEG; both may
download the original, and `exif()` downloads no video. `PhotoRecord` gains
`modifiedAt` and `hash`, and the JPEG EXIF scan moved to `src/exif.ts`.
- 2026-09-29: The "Workflow" list at the top of this file now says to branch
from `next` and open a pull request that targets `next`, that the repository
manager squash-merges a reviewed pull request into `next`, and that only sneak
merges `next` into `main`, as the README does (issue 137).
- 2026-09-29: The README's "Development workflow" and "For LLMs" sections now
say work branches from `next`, every pull request targets `next`, the
repository manager squash-merges reviewed pull requests into `next`, and only
sneak merges `next` into `main` (issue 135).
- 2026-09-29: Brought the README and this file in line with `next` after the
milestone merge (issue 132). The Next Step says no implementation work is open
and the cache design (issue 36) waits on sneak's review. README corrections:
the Getting Started comments say when the library refreshes; most, not all,
Makefile targets call a script; `script/lint` and `script/test` have no host
path, though `yarn test` does; the SRP handshake uses `fast-srp-hap`, outside
`crypto/`, and a thumbnail upload's MD5 uses `node:crypto`; the SRP password
is the first 16 bytes of a 32-byte subkey; the key attributes and token come
after SRP, after the TOTP code SRP may ask for, or after the email OTP that
replaces SRP when the account has email MFA on, and quak cannot answer a
passkey; the auth token is sent as URL-safe base64 with padding; every
`TypeError` is retried; each download attempt writes its own temporary files,
two for a live photo, and only the attempt that completes renames them into
place; `runMetadataBackup` records a failed download in the file's JSON; a
second `client.logout()` does not throw; `quak logout` with no session exits
0, and names the cache directory only when it knows it; `login`, `whoami` and
`logout` open no library; `--exif` records XMP and, for a JPEG, EXIF, and no
IPTC; which commands take `--json`; a failed ML data request may not have been
retried; the thumbnail fixer is not limited to baseline JPEG; `quak backup`
still fetches ML data; a backup's JSON holds the basic metadata fields quak
keeps and the private and public magic metadata, not every decrypted field,
and the README no longer lists what the magic metadata holds; the default
cache directory is the per-user one, not an XDG path on macOS; pinned
originals can exceed `cacheOriginalsMaxBytes`; which tests cover which
operations; a default read is only as current as the last refresh whose
requests all succeeded; `fresh()` and `lib.backup()` join a refresh already
running; a `Photo` has no `thumbnailPath` or `originalPath`; and `408` and
`429` are retried.
- 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks
that the image's and the video's temp files are fsynced before either is
renamed into place, and the directory after both renames, as the `writeAtomic`
test does for one file.
- 2026-09-23: Live photos are stored as their image and their video (issue 107).
A live photo, which Ente stores as one ZIP, is unpacked as it downloads into
`<fileID>.<ext>` for the image and for the video, each with its extension from
the ZIP, beside `<fileID>.livephoto.json`, which names the two. Both are
checked against the recorded hash and renamed into place only when both are
complete. A ZIP holding a second image or video, or whose two files come to
more than 20 times its size plus 16 MiB, is refused. The backup and the
content cache count the live photo as stored only with both files, the album
folder links both, `quak get` writes both, and the content result gives the
video as `videoPath`. A ZIP an earlier version stored is replaced on the next
backup run, and removed from the cache when the library opens it.
- 2026-09-23: Settled the package metadata (issue 6). quak is not published, so
`package.json` is marked `"private": true` and the `files` field is gone.
`engines.node` is `>=22`, the major version `script/bootstrap` and the
`Dockerfile` use. An `exports` map makes `.` and `./package.json` the only
importable paths; `runMetadataBackup`, the thumbnail helpers and their types
stay internal to the CLI.
- 2026-09-23: Brought the README and this file in line with the tree (issue
111). The layout lists `src/library/` and the other source files, the backup
layout names `failures.json` and the optional `thumbnails/`, "Opening a
library" says what happens when the first refresh fails, the Testing section
gives the 60-second hard cap and 20-second target for `make test` and names
the 90-second `timeout` in the `Dockerfile` as the backstop for a hung test,
and "Tag v1.0.0" is no longer listed as the next step.
- 2026-09-23: Tested `quak login` and `backup-metadata --exif` (issue 110).
`loginCommand` takes its login function and its prompts from `CliContext`, and
`bin/quak.ts` passes `Client.login` and the terminal prompts. Tests cover a
login from `QUAK_EMAIL` and `QUAK_PASSWORD` with no prompt, the TOTP prompt, a
failed login, and the saved session's modes, and show that `--exif` and
`--all` each turn on EXIF extraction and that it is off without them.
- 2026-09-23: `quak backup` writes each original once and no longer fills the
cache (issue 106). An original fetched for a backup is written by the download
writer straight into the backup's `originals/`, and the content cache records
it there instead of keeping its own copy; one the cache already held is still
copied. `quak backup` opens its library with the thumbnail and originals
precache off.
- 2026-09-23: `backup-metadata`, `helper list-missing-thumbnails` and
`helper fix-missing-thumbnails` refresh before they answer (issue 100). Each
awaits `lib.fresh()` before reading, so a file added since the cache was
written is included, and a failed refresh prints one line and exits 1 instead
of answering from a stale or empty cache. The README lists them among the
commands that refresh first.
- 2026-09-23: `quak backup` waits for the server refresh and fails when it fails
(issue 99). `lib.backup()` joins a refresh already running or starts one, as
`fresh()` does, and rejects before touching any file when it fails, leaving
`failures.json` as it was, so `quak backup` prints the error as one line and
exits 1 instead of backing up the previous run's file list, or nothing, and
exiting 0.
- 2026-09-23: CLI errors print a message instead of a stack trace (issue 102).
An error a command throws is printed as one `quak: MESSAGE` line on stderr and
the CLI exits 1 once output has drained. The wrapper that does this moved from
`bin/quak.ts` to `src/cli-run.ts`, and `bin/quak.ts` now awaits
`program.parseAsync()`.
- 2026-09-23: Opening a library no longer deletes another process's download in
progress (issue 105). The download writer's temp files are named
`.quak-<pid>-<random>.tmp`, and `removeLeftoverTempFiles`, moved from the
backup into the download module, deletes a `.quak-*.tmp` file only when the
process ID in its name is no longer running. The content cache calls it at
`open()` for `originals/` and `thumbnails/`, the backup as before.
- 2026-09-23: Re-vendored the lint and test setup from the template (issue 96).
Linting and testing are the `lint` and `test` phases of the `Dockerfile`;
`script/lint` and `script/test` each build one with `--no-cache`, and the last
stage compiles and depends on both, so `script/cibuild` is one build.
`Dockerfile.lint`, `CHECK_EPOCH`, `LINT_EPOCH` and the tests that checked them
are gone; `REPO_POLICIES.md` is re-copied.
- 2026-09-23: Stopped `helper fix-missing-thumbnails` retrying files the server
always refuses (issue 109). Both thumbnail helpers skip a file another account
owns without fetching it. The fixer skips a file whose recorded thumbnail size
is 0 or unknown before downloading it, and otherwise tries smaller encodings
(720 px quality 50 down to 160 px quality 20) until the encrypted thumbnail is
no larger than that size, skipping the file if none fits.
- 2026-09-23: Tested the live-photo hash check's error paths (issue 117). Tests
download a live photo whose ZIP names an unknown compression method, one whose
ZIP has no image entry and one with no video entry, and check that nothing is
stored and the error names the file ID; the unreadable one is not retried.
- 2026-09-23: `quak logout` ends the session on the server (issue 108). It calls
`POST /users/logout` through the new `Client.logoutOnServer()`, then deletes
`session.json` even when that call fails, says so and exits 1. It prints the
account's cache directory and says it still holds decrypted data. The default
cache path is now `defaultCacheDirectory()` in the library, shared with
`Library.open`.
- 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files
in one collection with the same title, and two collections with the same name,
each get their ID added to the name (`IMG_0001 (12345).JPG`, `Trip (10)/`), so
none replaces another's symlink or JSON. Each run removes symlinks into
`originals/` for files no longer in the collection, and the folders of deleted
or renamed collections, leaving anything else in `collections/` alone. The
README backup layout states the naming rule.
- 2026-09-23: Checked downloaded originals against their recorded content hash
(issue 68). `downloadFile`, which `quak get`, the content cache and backup all
use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and
stores nothing on a mismatch, failing with an error naming the file ID. A live
photo ZIP is unpacked as it streams with `fflate` and its image and video
hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older
clients' `imageHash` and `videoHash` fields for live photos. A file with no
recorded hash is stored unchecked.
- 2026-09-23: Kept one account's cache from mixing with another's (issue 104).
When `metadata.json` in the cache directory was written for a different,
non-zero user ID than the client's, `Library.open` deletes it and `mldata/`
and starts empty, so the first refresh enumerates from 0. This only happens
with `--cache-dir` or an explicit `cacheDirectory`; the default path already
includes the user ID. A test opens one account's cache as another account.
- 2026-09-23: `backup-metadata` no longer stops on one failed ML data request
(issue 101). Each request of up to 200 files is tried on its own; a failed one
is logged, its files are written with the reason in `mlDataError`, and the
command exits 1 once the dump is complete. `fetchMLData`, which only this
command used, is gone; the command calls `fetchMLDataBatch` per batch.
- 2026-09-23: Single-sourced the version string (issue 5). `package.json` is the
only place it is written: `src/index.ts` imports it for `VERSION` and
`bin/quak.ts` passes `VERSION` to commander. tsc copies `package.json` to
`dist/package.json`, so the import resolves from the built output too, and
`script/build` runs the built CLI with `--version` to prove it. A test checks
that `VERSION` and `quak --version` both equal the `package.json` version.
- 2026-09-23: Tested that `Library.close()` waits for the originals precache
(issue 93). The test that holds a precache fetch open while `close()` runs now
runs once with only the thumbnail fill and once with only the originals fill,
so dropping either wait from `Precache.close()` fails a test.
- 2026-09-23: Fixed two intermittently failing library tests (issue 90).
`Library.close()` now returns a promise that resolves once an in-flight
refresh (including its cache write), the ML data fetch and running precache
sweeps have finished; the library tests await it, so `afterEach` no longer
removes the cache directory while something is still writing into it. The
precache test waits for both fills to report "done" instead of for its stub
source to be called, which happened before the cache recorded the file.
- 2026-09-23: Pinned three guards reviewers found untested (issue 89). The
download idle deadline's timer is unref'd, so it can never keep the process
alive, and a test checks no timer is left after a download completes or fails.
A test covers the rejection of `#` in a request path. The EXIF scan compares
the `Exif` header only in an APP1 segment of length 8 or more, so it never
reads the next segment's bytes, with a test for a short one.
- 2026-09-23: Made the download deadline an idle deadline (issue 24).
`downloadTimeoutMs` now aborts a file or thumbnail download only after no
bytes have arrived for that long, default 60 seconds, instead of bounding the
whole transfer at 10 minutes, so a slow download that keeps making progress
completes. A download that fails before reading the whole body cancels it, so
a failed file no longer holds its connection.
- 2026-09-23: Every `ApiClient` request URL is now built by one function next to
the class (issue 18), so a self-hosted `apiOrigin` with a base path keeps it
on every request, a path works with or without a leading slash, and query
parameters are percent-encoded. A path containing `?` or `#` is rejected with
an error instead of being silently cut.
- 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies
moved from `bin/quak.ts` into `src/cli-commands.ts` as functions that take
their options and a context (output streams, session directory, cache
directory, session loader) and return an exit code; `bin/quak.ts` only wires
them to commander and exits with the code once stdout and stderr have drained,
so nothing below it calls `process.exit`. `test/cli/commands.test.ts` drives
them with a fake client: session file modes, logout, the missing and corrupt
session paths, and the output and exit code of `whoami`, `collections`,
`files`, `get`, `get-thumb`, `backup` and `helper list-missing-thumbnails`.
- 2026-09-23: Hardened the backup tree's atomic copy (issue 22). `copyAtomic`
fsyncs its temp file before the rename and the directory after it, through the
download writer's `fsyncPath`; each backup run deletes `.quak-backup-*.tmp`
files whose process is no longer running. The README backup layout names the
temp files and states that the rename replaces a symlink and takes the temp
file's permissions. Added tests for a missing and an unwritable destination
directory for `downloadFile` and `downloadThumbnail`.
- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue
11). Every segment length is checked against the remaining bytes and lengths
under 2 stop the scan, so a truncated or corrupt original can neither throw
nor loop. A malformed or unparseable EXIF segment is recorded as
`imageMetadata.exifError`, and a failure to read the original as
`imageMetadataError` in the per-file JSON, instead of the field being left
out.
- 2026-09-22: Hardened the retry classifier (issue 80). A `POST` or `PUT` is
replayed only when every errno in the cause chain is a connect errno, and it
no longer follows redirects. `getRetryOptions()` returns a copy. Tests pin
every errno the classifier names, the cause-chain depth limit, cycle
termination, and a fresh deadline per attempt for every retrying entry point.
The README's endpoint list is the one place that names the requests the replay
rule covers.
- 2026-09-22: Stopped `make test` collecting tests from checkouts nested under
`.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a
nested checkout ran the whole suite again; `vitest.config.ts` now adds
`.claude/**` to vitest's default excludes, and
`test/packaging/nested-checkout.test.ts` plants a nested checkout in a temp
directory and fails if vitest would collect it.
- 2026-09-22: Dropped the deprecated `@types/libsodium-wrappers-sumo` stub from
`devDependencies` (issue 27). It shipped no declarations; the types come from
`libsodium-wrappers-sumo` itself. `yarn.lock` regenerated by `yarn remove`.
- 2026-09-22: Hardened the client session lifecycle (issue 10).
`Client.fromJSON` checks every snapshot field and each key's decoded length
and names the bad field; `toJSON` reads the token through
`ApiClient.getAuthToken` and throws when there is none; `logout` zeroes the
key buffers, and `collectionsSince` re-checks for logout after its request so
it never decrypts with zeroed keys. The CLI reports a corrupt session file
separately from a missing one (`src/cli-session.ts`).
- 2026-09-22: Sanitized file names taken from server metadata (issue 9). A new
`src/filename.ts` holds the one sanitizer, used by `quak get`/`get-thumb`
without `--out`, `downloadFile`/`downloadThumbnail` without `outPath`, and the
backup and metadata backup trees; it removes separators, control characters,
leading dots and Windows device names, and falls back to a name built from the
ID for an empty title. Originals-cache extensions are letters and digits only,
else `.bin`. A user-supplied path is used as is. `decryptFile` reads a missing
or non-string title as "" and rejects metadata that is not a JSON object.
- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
snippets) to match the shipped cache/API library on `next` (issue 53, issue
13). Documented `Library.open` and its options, the default-read vs `fresh()`
distinction (and that the CLI's read commands are fresh), the record types and
`snapshot()`/`subscribe()`, the `albums`/`photos`/`timeline` read surface,
`Photo` content methods, `thumbnails.ensure`, the `mldata` search surface,
`backup()`, the three request pools (10/5/25), and the on-disk cache layout;
noted the deferred content-hash integrity check (issue 68). Docs-only; no code
changed.
- 2026-09-22: Added resumable, deletion-aware enumeration to `Client` (issue 38,
closes issue 7). `collectionsSince`/`filesSince` take a starting cursor,
decrypt live records, surface tombstoned ids in a separate `deleted` list (a
tombstone has nothing to decrypt, so it is a bare id, not a hollow record),
and return the max `updationTime` seen as the cursor to resume from.
`filesSince` refuses to loop when the diff reports `hasMore` without advancing
the cursor (issue 7). `listCollections`/`listFiles` are now thin wrappers that
enumerate from `sinceTime: 0` and drop deletions, so existing callers are
unaffected.
- 2026-09-22: Carried file size, thumbnail size, and the deletion flag through
`decryptFile` (issue 37, foundation for the cache/API design). Live files now
populate `file.size`/`thumbnail.size` from the server's `info` (left
`undefined` when the server omits it), and `isDeleted` is carried from the
diff row onto `EnteFile`. No caller change: `listFiles` still filters deleted
rows before decrypting. Surfacing a tombstone through decryption belongs to
the enumeration unit (issue 38).
- 2026-08-10: Made `lint-once.test.ts` enforce what its header claims. It walked
`make check` only, so it never read `Dockerfile` — the image CI builds through
`script/cibuild` — and a second `prettier --check .` could be added there with
the suite staying green. The walk now also starts at
`.gitea/workflows/check.yml` and follows its `run:` steps, so the graph under
test is the one CI executes rather than the one someone assumed it executes.
The lockfile assertion was a substring check against the whole of
`script/bootstrap`, which has two install sites and so reported the branch the
containers never take; the two branches are now resolved separately and every
`yarn install` in each is required to be `--frozen-lockfile`. Prettier is
counted per occurrence instead of per line, so two invocations chained with
`&&` no longer read as one, and edges are followed on counted lines instead of
being skipped. Every way for the walk to reach nothing — an unknown target, an
unknown script, a missing file, a node with no commands, an unknown node kind
— is a thrown error rather than a quiet zero. Every assertion in the file was
mutation-tested individually.
- 2026-08-10: Stopped `make check` running `prettier --check .` twice. Since
linting moved into Docker, the duplicate was one container pass and one host
pass of the same check: `script/lint` builds `Dockerfile.lint`, which runs
prettier as a build step, and `script/check` then called `script/fmt-check` as
well. The host call is gone from `script/check` and from `script/precommit`;
the container keeps checking formatting, because a successful
`Dockerfile.lint` build is what CI treats as proof of a clean tree, and it is
also what still fails the pre-commit hook on a badly formatted tree.
`script/fmt-check` survives as a standalone entrypoint, whose verdict cannot
drift from the container's. A test walks the invocation graph from each
entrypoint — through the Makefile shims, the `script/` calls and the
`docker build` — and asserts the prettier count, so the duplication cannot
come back unnoticed.
- 2026-08-10: Moved all linting into Docker. `script/lint` builds a new root
`Dockerfile.lint`, which copies the repo into the digest-pinned node image and
runs eslint and prettier as build steps, so a successful build is a clean
lint; no host lint path remains and `yarn lint` is gone from `package.json`. A
fail-closed `LINT_EPOCH` guard stops Docker serving the linter layers from
cache, which is how a lint build returns success in under a second having
linted nothing. The lint stage inside `Dockerfile` and its `COPY --from=lint`
ordering hack are gone: that image now runs `make test` and `make build` only,
because `script/check` calls `script/lint` and running it in a container would
mean docker inside docker. `script/cibuild` builds the lint image first, then
the test and build image.
- 2026-08-09: Made `make docker` green and policy-conformant. Multi-stage
Dockerfile: a lint stage runs `make fmt-check` and `make lint`, and the check
stage takes a `COPY --from=lint` dependency on it before running `make check`
@@ -66,7 +743,6 @@ Update the README API reference section to match the current implementation.
# Future Steps
- Tag v1.0.0.
- Future desktop client, separate repo:
- Electron app skeleton consuming this library.
- Local SQLite cache keyed on (collectionID, fileID, updationTime).
bin/clip-keyword-vectors
Executable
+31
View File
@@ -0,0 +1,31 @@
#!/bin/sh
# Writes src/clip-keyword-vectors.ts and test/clip-keywords-sunset.json by
# running bin/clip-keyword-vectors.py in a Python image pinned by digest, with
# its packages pinned by hash. Run it by hand from a checkout, then `make fmt`;
# no gate runs it. Besides the Python image, it downloads about 265 MB, most of
# it Ente's two CLIP models.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# python:3.13-slim (Python 3.13.17 on Debian 13.7), 2026-10-21. The wheels
# below are the ones for this image on linux/amd64.
IMAGE=python@sha256:57b29a7c050c4e679e4bea572e14b3965cf4b5839de2da49d55e49ffc4024f39
main() {
cd "$ROOT"
docker run --rm -i --platform linux/amd64 \
--user "$(id -u):$(id -g)" -e HOME=/tmp \
-v "$ROOT:/repo" -w /repo "$IMAGE" sh -c '
python -m venv /tmp/venv &&
/tmp/venv/bin/pip install --quiet --disable-pip-version-check \
--no-cache-dir --no-deps --require-hashes -r /dev/stdin &&
/tmp/venv/bin/python bin/clip-keyword-vectors.py' <<'EOF'
numpy==2.5.3 --hash=sha256:a5fa86b80fd24bcd1aff83ad23be44ea323de3f787be8f8b15d4a65621e25321
onnxruntime==1.30.0 --hash=sha256:86f940afc801ea9681a4da8af84fbe95e1d9ea7d80903952cc1bfad54faad38f
pillow==12.3.0 --hash=sha256:0847a763afefb695bc912d7c131e7e0632d4edc1d8698f58ddabec8e46b8b6d3
tokenizers==0.23.2 --hash=sha256:41c2f84d172449b4dadb9cdc508e3e364076613c35b16e76ecfe47a60d1e3305
EOF
}
main "$@"
bin/clip-keyword-vectors.py
+158
View File
@@ -0,0 +1,158 @@
"""Write src/clip-keyword-vectors.ts and test/clip-keywords-sunset.json.
bin/clip-keyword-vectors runs this in a pinned Python image. It downloads
Ente's Discover list, Ente's CLIP text and image models, the CLIP tokenizer
and a public-domain photo of a sunset, checks each against its sha256, and
writes:
- src/clip-keyword-vectors.ts: each Discover category of Ente's photos app,
with the text embedding Ente's CLIP text model gives its query;
- test/clip-keywords-sunset.json: the CLIP embedding Ente's image model gives
the photo, prepared as Ente's apps prepare a photo.
Both embeddings are scaled to length 1, as Ente's apps scale them.
"""
import hashlib
import io
import json
import urllib.request
import numpy as np
import onnxruntime
from PIL import Image, ImageOps
from tokenizers import Tokenizer
ENTE_COMMIT = "f5ad70a860ed25d0c8057b0efb4bd6eaefe4b0c5"
DISCOVER_PATH = "mobile/apps/photos/assets/discover.json"
DISCOVER_URL = (
f"https://raw.githubusercontent.com/ente-io/ente/{ENTE_COMMIT}/{DISCOVER_PATH}"
)
DISCOVER_SHA256 = "cf70ac7116b592f7921ca4cf67cdbb421a8e6b287fbe44b2be678b8d198e5c24"
TEXT_MODEL = "mobileclip_s2_text_opset18_quant.onnx"
TEXT_MODEL_SHA256 = "d92f33dfcff83077fc2e0d3414250710efbb51795dfd89767bdbefb5fdc47322"
IMAGE_MODEL = "mobileclip_s2_image_gelu_opset20.onnx"
IMAGE_MODEL_SHA256 = "205a430af825e501c5138e5bb9abea942482a7a4fd4a680e98e47cf0830dce7e"
# The CLIP tokenizer Hugging Face publishes for OpenAI's CLIP. Its vocabulary
# and merges are those of bpe_simple_vocab_16e6.txt, the vocabulary Ente's
# apps tokenize with.
TOKENIZER_URL = (
"https://huggingface.co/openai/clip-vit-base-patch32/resolve/"
"3d74acf9a28c67741b2f4f2ea7635f0aaf6f0268/tokenizer.json"
)
TOKENIZER_SHA256 = "b556ac8c99757ffb677208af34bc8c6721572114111a6e0aaf5fa69ff0b8d842"
# "Sunset off the Massachusetts coast", a NOAA photo in the public domain:
# https://commons.wikimedia.org/wiki/File:Sunset_off_the_Massachusetts_coast_-_NOAA.jpg
PHOTO_URL = (
"https://upload.wikimedia.org/wikipedia/commons/d/db/"
"Sunset_off_the_Massachusetts_coast_-_NOAA.jpg"
)
PHOTO_SHA256 = "be62f3289c9bf7b75af68cd6978021d54e8a531d5d4cc493f6e9fbf5e1ab74d3"
# Ente's text model takes 77 token ids, padded with 0; its image model takes
# a 256x256 RGB image.
TOKEN_COUNT = 77
IMAGE_SIZE = 256
def download(url, sha256):
# Wikimedia refuses requests without a User-Agent.
request = urllib.request.Request(url, headers={"User-Agent": "quak"})
with urllib.request.urlopen(request) as response:
data = response.read()
digest = hashlib.sha256(data).hexdigest()
if digest != sha256:
raise SystemExit(f"{url} has sha256 {digest}, not {sha256}")
return data
def ente_model(name, sha256):
return onnxruntime.InferenceSession(
download(f"https://models.ente.com/{name}", sha256),
providers=["CPUExecutionProvider"],
)
def run(session, tensor):
(output,) = session.run(None, {session.get_inputs()[0].name: tensor})
embedding = output[0]
return embedding / np.linalg.norm(embedding)
def text_embedding(session, tokenizer, query):
ids = tokenizer.encode(query).ids
ids += [0] * (TOKEN_COUNT - len(ids))
return run(session, np.array([ids], dtype=np.int32))
def image_embedding(session, photo):
image = ImageOps.exif_transpose(Image.open(io.BytesIO(photo))).convert("RGB")
# Scale to cover IMAGE_SIZE x IMAGE_SIZE and crop the centre.
square = ImageOps.fit(image, (IMAGE_SIZE, IMAGE_SIZE), Image.Resampling.BILINEAR)
pixels = np.asarray(square, dtype=np.float32) / 255
return run(session, pixels.transpose(2, 0, 1)[np.newaxis])
def numbers(embedding):
return ", ".join(str(value) for value in embedding)
def write_categories(prompts, embeddings):
lines = [
"// Written by bin/clip-keyword-vectors; do not edit by hand.",
"//",
"// The Discover categories of Ente's photos app, in their order, from",
f"// {DISCOVER_PATH} in https://github.com/ente-io/ente",
f"// at commit {ENTE_COMMIT}. Each has the text",
f"// embedding Ente's CLIP text model ({TEXT_MODEL})",
"// gives its query, scaled to length 1. `showVideo` is false for a category",
"// that leaves out videos.",
"",
"export const DISCOVER_CATEGORIES = [",
]
for prompt, embedding in zip(prompts, embeddings):
lines += [
" {",
f" title: {json.dumps(prompt['title'])},",
f" query: {json.dumps(prompt['query'])},",
f" minScore: {json.dumps(prompt['minScore'])},",
f" showVideo: {json.dumps(prompt.get('showVideo', True))},",
f" textEmbedding: [{numbers(embedding)}],",
" },",
]
lines.append("];")
with open("src/clip-keyword-vectors.ts", "w", encoding="utf-8") as file:
file.write("\n".join(lines) + "\n")
def write_fixture(embedding):
with open("test/clip-keywords-sunset.json", "w", encoding="utf-8") as file:
file.write(
"{\n"
f' "photo": {json.dumps(PHOTO_URL)},\n'
f' "sha256": {json.dumps(PHOTO_SHA256)},\n'
f' "embedding": [{numbers(embedding)}]\n'
"}\n"
)
def main():
prompts = json.loads(download(DISCOVER_URL, DISCOVER_SHA256))["prompts"]
tokenizer = Tokenizer.from_str(
download(TOKENIZER_URL, TOKENIZER_SHA256).decode("utf-8")
)
text_model = ente_model(TEXT_MODEL, TEXT_MODEL_SHA256)
embeddings = [
text_embedding(text_model, tokenizer, prompt["query"]) for prompt in prompts
]
write_categories(prompts, embeddings)
image_model = ente_model(IMAGE_MODEL, IMAGE_MODEL_SHA256)
write_fixture(image_embedding(image_model, download(PHOTO_URL, PHOTO_SHA256)))
if __name__ == "__main__":
main()
bin/quak.ts
+122 -345
View File
@@ -1,242 +1,131 @@
#!/usr/bin/env node
import { input, password as passwordPrompt } from "@inquirer/prompts";
import { stdout, stderr } from "node:process";
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { Command } from "commander";
import { input, password } from "@inquirer/prompts";
import { Command, Option } from "commander";
import envPaths from "env-paths";
import { Client, type ClientSnapshot } from "../src/client.js";
import { logLevel, logRequests, stderrLog, type Log } from "../src/cli-log.js";
import { init } from "../src/crypto/index.js";
import { runBackup } from "../src/backup.js";
import { runMetadataBackup } from "../src/metadata-backup.js";
import {
listMissingThumbnails,
fixMissingThumbnails,
} from "../src/thumbnails.js";
type CliContext,
loginCommand,
whoamiCommand,
logoutCommand,
collectionsCommand,
filesCommand,
getCommand,
getThumbCommand,
backupMetadataCommand,
backupCommand,
listMissingThumbnailsCommand,
fixMissingThumbnailsCommand,
} from "../src/cli-commands.js";
import { run as runCommand } from "../src/cli-run.js";
import { loadSession } from "../src/cli-session.js";
import { Client } from "../src/client.js";
import { VERSION } from "../src/index.js";
import { UNATTENDED_RETRY_OPTIONS } from "../src/retry.js";
const paths = envPaths("quak", { suffix: "" });
const sessionPath = join(paths.data, "session.json");
const loadSession = (): ClientSnapshot | null => {
if (!existsSync(sessionPath)) return null;
try {
return JSON.parse(readFileSync(sessionPath, "utf-8")) as ClientSnapshot;
} catch {
return null;
}
};
const saveSession = (snapshot: ClientSnapshot): void => {
mkdirSync(paths.data, { recursive: true, mode: 0o700 });
writeFileSync(sessionPath, JSON.stringify(snapshot, null, 2), {
mode: 0o600,
});
};
const requireSession = (): Client => {
const snapshot = loadSession();
if (!snapshot) {
stderr.write(
`Not logged in. Run "quak login" first.\nSession file: ${sessionPath}\n`,
);
process.exit(1);
}
return Client.fromJSON(snapshot);
};
const prompt = async (message: string): Promise<string> => input({ message });
const promptSecret = async (message: string): Promise<string> =>
passwordPrompt({ message, mask: true });
const program = new Command();
// Program options, so they are taken before or after any command's name.
// `-v` counts its repeats, the way commander's documentation counts a repeated
// option.
program
.name("quak")
.description("CLI for the Ente end-to-end encrypted photo service")
.version("0.0.0");
.version(VERSION)
.option(
"--cache-dir <path>",
"Directory for the local metadata/content cache " +
"(default: the per-user cache directory)",
)
.option(
"-v, --verbose",
"Log more on stderr: debug lines with -v, trace lines too with -vv",
(_value: string, previous: number) => previous + 1,
0,
)
.addOption(
new Option(
"-q, --quiet",
"Print nothing on stderr but the error a failing command exits with",
).conflicts("verbose"),
)
.configureHelp({ showGlobalOptions: true });
interface ProgramOptions {
cacheDir?: string;
verbose: number;
quiet?: boolean;
}
// The log for this run, at the level `-v` and `-q` ask for.
const log = (): Log => {
const { verbose, quiet } = program.opts<ProgramOptions>();
return stderrLog(stderr, logLevel(verbose, quiet ?? false));
};
const context = (): CliContext => ({
stdout,
log: log(),
sessionDir: paths.data,
cacheDir: program.opts<ProgramOptions>().cacheDir,
loadSession: (path) => loadSession(path, logRequests(log())),
login: (opts) => Client.login({ ...opts, apiOptions: logRequests(log()) }),
prompt: (message) => input({ message }),
promptSecret: (message) => password({ message, mask: true }),
});
const run = (command: Promise<number>): Promise<void> =>
runCommand(command, stdout, stderr, log(), (code) => process.exit(code));
program
.command("login")
.description("Log in to an Ente account and save the session")
.action(async () => {
await init();
const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
const password =
process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
stderr.write("Authenticating...\n");
try {
const client = await Client.login({
email,
password,
totp: async () => prompt("TOTP code: "),
emailOTP: async () => prompt("Email verification code: "),
});
saveSession(client.toJSON());
const info = client.whoami();
stderr.write(`Logged in as ${info.email} (user ${info.userID})\n`);
stderr.write(`Session saved to ${sessionPath}\n`);
} catch (err) {
stderr.write(
`Login failed: ${err instanceof Error ? err.message : err}\n`,
);
process.exit(1);
}
});
.action(() => run(loginCommand(context())));
program
.command("whoami")
.description("Print the logged-in account")
.action(() => {
const client = requireSession();
const info = client.whoami();
stdout.write(JSON.stringify(info) + "\n");
});
.action(() => run(whoamiCommand(context())));
program
.command("logout")
.description("Delete the saved session")
.action(async () => {
if (existsSync(sessionPath)) {
const { unlinkSync } = await import("node:fs");
unlinkSync(sessionPath);
stderr.write("Session deleted.\n");
} else {
stderr.write("No session found.\n");
}
});
.description("End the session on the server and delete the saved session")
.action(() => run(logoutCommand(context())));
program
.command("collections")
.description("List all collections (albums)")
.option("--json", "Output as JSON array")
.action(async (opts: { json?: boolean }) => {
await init();
const client = requireSession();
const collections = await client.listCollections();
if (opts.json) {
stdout.write(
JSON.stringify(
collections.map((c) => ({
id: c.id,
name: c.name,
type: c.type,
ownerID: c.ownerID,
isShared: c.isShared,
updationTime: c.updationTime,
})),
null,
2,
) + "\n",
);
} else {
for (const c of collections) {
stdout.write(
`${c.id}\t${c.type}\t${c.name}${c.isShared ? " (shared)" : ""}\n`,
);
}
}
});
.action((opts: { json?: boolean }) =>
run(collectionsCommand(context(), opts)),
);
program
.command("files")
.description("List files in a collection")
.requiredOption(
"--collection <id>",
"Collection ID (from `quak collections`)",
.description("List the files in a collection, or every file in the account")
.argument(
"[collectionID]",
"Collection ID (from `quak collections`); without it, every file " +
"is listed once",
)
.option("--json", "Output as JSON array")
.action(async (opts: { collection: string; json?: boolean }) => {
await init();
const client = requireSession();
const collectionID = Number(opts.collection);
if (!Number.isFinite(collectionID)) {
stderr.write("Invalid collection ID\n");
process.exit(1);
}
const collections = await client.listCollections();
const col = collections.find((c) => c.id === collectionID);
if (!col) {
stderr.write(`Collection ${collectionID} not found\n`);
process.exit(1);
}
const files = await client.listFiles(col.id, col.key);
if (opts.json) {
stdout.write(
JSON.stringify(
files.map((f) => ({
id: f.id,
title: f.metadata.title,
fileType: f.metadata.fileType,
creationTime: f.metadata.creationTime,
collectionID: f.collectionID,
})),
null,
2,
) + "\n",
);
} else {
for (const f of files) {
stdout.write(
`${f.id}\t${f.metadata.fileType}\t${f.metadata.title}\n`,
);
}
}
});
.action((collectionID: string | undefined, opts: { json?: boolean }) =>
run(filesCommand(context(), collectionID, opts)),
);
program
.command("get")
.description("Download and decrypt a single file")
.argument("<fileID>", "File ID (from `quak files`)")
.option("--out <path>", "Output file path")
.option(
"--collection <id>",
"Collection ID (required to look up the file key)",
)
.action(
async (
fileIDStr: string,
opts: { out?: string; collection?: string },
) => {
await init();
const client = requireSession();
const fileID = Number(fileIDStr);
if (!Number.isFinite(fileID)) {
stderr.write("Invalid file ID\n");
process.exit(1);
}
const collections = await client.listCollections();
let targetCol;
if (opts.collection) {
targetCol = collections.find(
(c) => c.id === Number(opts.collection),
);
}
// Search all collections (or the specified one) for the file
const searchCols = targetCol ? [targetCol] : collections;
for (const col of searchCols) {
const files = await client.listFiles(col.id, col.key);
const file = files.find((f) => f.id === fileID);
if (file) {
const result = await client.downloadFile(file, opts.out);
stderr.write(
`${result.bytesWritten} bytes -> ${result.path}\n`,
);
return;
}
}
stderr.write(`File ${fileID} not found\n`);
process.exit(1);
},
.option("--collection <id>", "Accepted for compatibility; ignored")
.action((fileID: string, opts: { out?: string }) =>
run(getCommand(context(), fileID, opts)),
);
program
@@ -244,49 +133,9 @@ program
.description("Download and decrypt a thumbnail")
.argument("<fileID>", "File ID (from `quak files`)")
.option("--out <path>", "Output file path")
.option(
"--collection <id>",
"Collection ID (required to look up the file key)",
)
.action(
async (
fileIDStr: string,
opts: { out?: string; collection?: string },
) => {
await init();
const client = requireSession();
const fileID = Number(fileIDStr);
if (!Number.isFinite(fileID)) {
stderr.write("Invalid file ID\n");
process.exit(1);
}
const collections = await client.listCollections();
let targetCol;
if (opts.collection) {
targetCol = collections.find(
(c) => c.id === Number(opts.collection),
);
}
const searchCols = targetCol ? [targetCol] : collections;
for (const col of searchCols) {
const files = await client.listFiles(col.id, col.key);
const file = files.find((f) => f.id === fileID);
if (file) {
const result = await client.downloadThumbnail(
file,
opts.out,
);
stderr.write(
`${result.bytesWritten} bytes -> ${result.path}\n`,
);
return;
}
}
stderr.write(`File ${fileID} not found\n`);
process.exit(1);
},
.option("--collection <id>", "Accepted for compatibility; ignored")
.action((fileID: string, opts: { out?: string }) =>
run(getThumbCommand(context(), fileID, opts)),
);
program
@@ -300,14 +149,9 @@ program
"Download each file and extract full EXIF/IPTC/XMP metadata (slow)",
)
.option("--all", "Alias for --exif")
.action(async (dir: string, opts: { exif?: boolean; all?: boolean }) => {
await init();
const client = requireSession();
await runMetadataBackup(client, dir, {
exif: opts.exif || opts.all,
onProgress: (msg) => stderr.write(msg + "\n"),
});
});
.action((dir: string, opts: { exif?: boolean; all?: boolean }) =>
run(backupMetadataCommand(context(), dir, opts)),
);
program
.command("backup")
@@ -316,35 +160,28 @@ program
)
.argument("<dir>", "Output directory")
.option("--json", "Print result as JSON instead of human-readable summary")
.action(async (dir: string, opts: { json?: boolean }) => {
await init();
const client = requireSession();
stderr.write("Starting backup...\n");
const result = await runBackup(client, dir, (msg) => {
if (!opts.json) stderr.write(msg + "\n");
});
if (opts.json) {
stdout.write(JSON.stringify(result, null, 2) + "\n");
} else {
stderr.write("\n--- Backup complete ---\n");
stderr.write(` Total files: ${result.totalFiles}\n`);
stderr.write(` Downloaded: ${result.downloaded}\n`);
stderr.write(` Skipped: ${result.skipped}\n`);
stderr.write(` Failed: ${result.failed}\n`);
if (result.errors.length > 0) {
stderr.write("\nFailed files:\n");
for (const e of result.errors) {
stderr.write(
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
);
}
}
}
process.exit(result.failed > 0 ? 1 : 0);
});
.option(
"--verify",
"Re-hash stored originals and download again any that do not match",
)
// A backup usually runs from cron with nobody watching, so every request
// it makes retries for longer than the other commands' requests do.
.action((dir: string, opts: { json?: boolean; verify?: boolean }) =>
run(
backupCommand(
{
...context(),
loadSession: (path) =>
loadSession(
path,
logRequests(log(), UNATTENDED_RETRY_OPTIONS),
),
},
dir,
opts,
),
),
);
const helper = program
.command("helper")
@@ -354,30 +191,9 @@ helper
.command("list-missing-thumbnails")
.description("List files whose thumbnails are missing or empty")
.option("--json", "Output as JSON array")
.action(async (opts: { json?: boolean }) => {
await init();
const client = requireSession();
const missing = await listMissingThumbnails(client, (msg) => {
if (!opts.json) stderr.write(msg + "\n");
});
if (opts.json) {
stdout.write(JSON.stringify(missing, null, 2) + "\n");
} else {
if (missing.length === 0) {
stderr.write("No missing thumbnails found.\n");
} else {
stderr.write(
`\n${missing.length} file(s) with missing thumbnails:\n`,
);
for (const m of missing) {
stdout.write(
`${m.fileID}\t${m.title}\t${m.collection}\t${m.reason}\n`,
);
}
}
}
});
.action((opts: { json?: boolean }) =>
run(listMissingThumbnailsCommand(context(), opts)),
);
helper
.command("fix-missing-thumbnails")
@@ -389,48 +205,9 @@ helper
"Specific file IDs to fix (default: fix all missing)",
)
.option("--json", "Output as JSON")
.action(async (opts: { file?: string[]; json?: boolean }) => {
await init();
const client = requireSession();
let fileIDs: number[];
if (opts.file && opts.file.length > 0) {
fileIDs = opts.file.map(Number).filter(Number.isFinite);
} else {
stderr.write("Scanning for missing thumbnails...\n");
const missing = await listMissingThumbnails(client, (msg) => {
if (!opts.json) stderr.write(msg + "\n");
});
fileIDs = missing.map((m) => m.fileID);
if (fileIDs.length === 0) {
stderr.write("No missing thumbnails found.\n");
return;
}
stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
}
const results = await fixMissingThumbnails(client, fileIDs, (msg) => {
if (!opts.json) stderr.write(msg + "\n");
});
if (opts.json) {
stdout.write(JSON.stringify(results, null, 2) + "\n");
} else {
const ok = results.filter((r) => r.success).length;
const fail = results.filter((r) => !r.success).length;
stderr.write(`\n--- Done ---\n`);
stderr.write(` Fixed: ${ok}\n`);
stderr.write(` Failed: ${fail}\n`);
if (fail > 0) {
stderr.write("\nFailed files:\n");
for (const r of results.filter((r) => !r.success)) {
stderr.write(` ${r.fileID}\t${r.title}\t${r.error}\n`);
}
}
}
process.exit(results.some((r) => !r.success) ? 1 : 0);
});
.action((opts: { file?: string[]; json?: boolean }) =>
run(fixMissingThumbnailsCommand(context(), opts)),
);
await init();
program.parse();
await program.parseAsync();
examples/download-albums.ts
+129
View File
@@ -0,0 +1,129 @@
// Download every album's photos to a directory, with each photo's metadata
// beside it, using only quak's public API. The README's "Examples" section
// describes the files it writes.
//
// yarn build
// QUAK_EMAIL=… QUAK_PASSWORD=… node dist/examples/download-albums.js [dir]
import { realpathSync } from "node:fs";
import { mkdir, readFile, writeFile } from "node:fs/promises";
import { join, relative } from "node:path";
import { stdin, stdout } from "node:process";
import { createInterface } from "node:readline/promises";
import { pathToFileURL } from "node:url";
import { Client, Library } from "../src/index.js";
// Write `text` to `path` unless the file already holds exactly that, so a
// second run rewrites nothing.
async function writeIfChanged(path: string, text: string): Promise<void> {
const current = await readFile(path, "utf-8").catch(() => undefined);
if (current !== text) await writeFile(path, text);
}
const pretty = (value: unknown): string =>
JSON.stringify(value, null, 2) + "\n";
// For every album in `lib`, download each photo to its save path, write the
// photo's metadata to `{savePath}.json`, and write the album's photos to
// `{dir}/albums/{collectionID}.json`. Returns how many photos it downloaded
// and how many were already at their save paths.
export async function downloadAlbums(
lib: Library,
dir: string,
): Promise<{ downloaded: number; alreadyLocal: number }> {
let downloaded = 0;
let alreadyLocal = 0;
// A photo in several albums is handled once.
const done = new Set<number>();
// fresh() waits for a refresh from the server and throws if it fails, so
// albums and photos added since the cache was last written are included.
const { albums } = await lib.fresh();
for (const album of albums.list()) {
const savePaths: string[] = [];
for (const photo of album.photos.list()) {
if (!done.has(photo.fileID)) {
done.add(photo.fileID);
if (photo.isLocal) alreadyLocal++;
else downloaded++;
await photo.download();
// The cache paths say where quak's cache keeps copies, not
// anything about the photo.
const record = { ...photo.record() };
delete record.thumbnailPath;
delete record.originalPath;
const exif = await photo.exif();
// savePath is read after download(): a live photo's names its
// image only once the image is stored.
await writeIfChanged(
`${photo.savePath}.json`,
pretty({ ...record, exif }),
);
}
savePaths.push(relative(dir, photo.savePath));
}
await mkdir(join(dir, "albums"), { recursive: true });
await writeIfChanged(
join(dir, "albums", `${album.collectionID}.json`),
pretty({
collectionID: album.collectionID,
name: album.name,
savePaths,
}),
);
}
return { downloaded, alreadyLocal };
}
// Ask for a login code on the terminal. Client.login calls this only when the
// account requires a code.
async function ask(question: string): Promise<string> {
const terminal = createInterface({ input: stdin, output: stdout });
try {
return await terminal.question(question);
} finally {
terminal.close();
}
}
async function main(): Promise<void> {
const email = process.env.QUAK_EMAIL;
const password = process.env.QUAK_PASSWORD;
if (!email || !password) {
console.error(
"Set QUAK_EMAIL and QUAK_PASSWORD to the account's email and password.",
);
process.exit(1);
}
const dir = process.argv[2] ?? "photos";
const client = await Client.login({
email,
password,
totp: () => ask("Two-factor code: "),
emailOTP: () => ask("Code sent to your email: "),
});
const lib = await Library.open({
client,
downloadDirectory: dir,
// As in `quak backup`: fetch only the originals this script saves,
// not every thumbnail and the recent originals into the cache too.
precacheThumbnails: false,
precacheOriginals: false,
});
try {
const { downloaded, alreadyLocal } = await downloadAlbums(lib, dir);
console.log(
`${downloaded} photos downloaded, ${alreadyLocal} already local, in ${dir}`,
);
} finally {
await lib.close();
}
}
// Run main() when node runs this file, not when a test imports it. argv[1] is
// the path as given, and import.meta.url has symlinks resolved.
const script = process.argv[1];
if (script && pathToFileURL(realpathSync(script)).href === import.meta.url) {
await main();
}
package.json
+16 -9
View File
@@ -9,29 +9,34 @@
"type": "git",
"url": "https://git.eeqj.de/sneak/quak.git"
},
"private": true,
"type": "module",
"engines": {
"node": ">=22"
},
"main": "./dist/src/index.js",
"types": "./dist/src/index.d.ts",
"exports": {
".": {
"types": "./dist/src/index.d.ts",
"import": "./dist/src/index.js"
},
"./package.json": "./package.json"
},
"bin": {
"quak": "./dist/bin/quak.js"
},
"files": [
"dist/",
"README.md",
"LICENSE"
],
"scripts": {
"build": "script/build",
"quak": "node ./dist/bin/quak.js",
"test": "vitest run",
"lint": "eslint .",
"fmt": "prettier --write .",
"fmt-check": "prettier --check ."
},
"devDependencies": {
"@eslint/js": "9.38.0",
"@types/libsodium-wrappers-sumo": "0.8.2",
"@types/node": "22.18.13",
"@types/proper-lockfile": "4.1.4",
"eslint": "9.38.0",
"prettier": "3.8.1",
"typescript": "5.9.3",
@@ -42,9 +47,11 @@
"@inquirer/prompts": "8.5.2",
"commander": "14.0.3",
"env-paths": "4.0.0",
"exif-reader": "2.0.3",
"exifreader": "4.46.0",
"fast-srp-hap": "2.0.4",
"fflate": "0.8.3",
"jpeg-js": "0.4.4",
"libsodium-wrappers-sumo": "0.8.4"
"libsodium-wrappers-sumo": "0.8.4",
"proper-lockfile": "4.1.2"
}
}
script/build
+31 -3
View File
@@ -1,7 +1,7 @@
#!/bin/sh
# script/build: compile the TypeScript sources into dist/, then verify that
# the artifacts package.json advertises are among the files the compiler
# actually wrote. tsc reports success by exit status alone and knows nothing
# script/build: compile the TypeScript sources into dist/, stamp the version
# script/version prints into it, then verify that the artifacts package.json
# advertises are among the files the compiler actually wrote. tsc reports success by exit status alone and knows nothing
# about the manifest, so without this step a green build can still ship a
# package whose main, types or bin resolve to nothing. Our own extension to
# scripts-to-rule-them-all.
@@ -45,10 +45,38 @@ for (const bin of bins) {
'
}
# src/index.ts imports ../package.json for the version, which tsc copies to
# dist/package.json. The version script/version prints is written into that
# copy only; the repo's own package.json is left as it is.
stamp_version() {
node -e '
const { readFileSync, writeFileSync } = require("node:fs");
const pkg = JSON.parse(readFileSync("dist/package.json", "utf-8"));
pkg.version = process.argv[1];
writeFileSync("dist/package.json", JSON.stringify(pkg, null, 4) + "\n");
' "$1"
}
# Running the built CLI proves the import resolves from dist/ and reports
# the stamped version.
verify_version() {
built="$(node dist/bin/quak.js --version)"
if [ "$built" != "$1" ]; then
echo "build: dist/bin/quak.js reports $built, the build stamped $1" >&2
exit 1
fi
echo "build: dist/bin/quak.js reports version $built"
}
main() {
cd "$ROOT"
# Own line, so that a failing script/version stops the build.
version="$("$ROOT/script/version")"
yarn run tsc
stamp_version "$version"
verify_entrypoints
verify_version "$version"
}
main "$@"
script/check
+3 -1
View File
@@ -1,6 +1,8 @@
#!/bin/sh
# script/check: run all checks (test, lint, fmt-check). Our own
# extension to scripts-to-rule-them-all. Must not modify any files.
# extension to scripts-to-rule-them-all. test and lint are Docker
# phases; fmt-check is native, because a formatter writes the working
# tree. Must not modify any files.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
script/cibuild
+19 -8
View File
@@ -1,17 +1,28 @@
#!/bin/sh
# script/cibuild: run the CI build. The Dockerfile runs script/check and
# script/build, and CHECK_EPOCH differs on every invocation, so those two
# layers cannot be served from Docker's cache: a green build here means
# the checks ran now, not that a previous run was remembered. The layers
# below the epoch (bootstrap, yarn install) are unaffected and stay
# cached. A build that omits the argument fails by design.
# script/cibuild: run the CI build. It bootstraps first: a CI runner
# checks out and runs this and nothing else, and script/fmt-check runs
# the formatter on the host, which a pristine checkout cannot do.
# --no-cache for the same reason as script/docker: the gate phases the
# final stage depends on are RUN steps, and a cached one is a check that
# did not run.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build --build-arg CHECK_EPOCH="$(date +%s)" .
"$SCRIPT_DIR/bootstrap"
"$SCRIPT_DIR/check"
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
}
main "$@"
script/docker
+10 -4
View File
@@ -1,9 +1,8 @@
#!/bin/sh
# script/docker: build the Docker image tagged with the project name.
# Identical in all repos; the tag comes from script/projectname.
# CHECK_EPOCH is passed for the same reason script/cibuild passes it: the
# Dockerfile refuses to build without it, so that no path to an image can
# quietly serve the check and build layers from cache.
# --no-cache because the gate phases the final stage depends on are RUN
# steps, and a cached one is a check that did not run.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
@@ -11,7 +10,14 @@ ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
docker build --build-arg CHECK_EPOCH="$(date +%s)" \
# Own line: a failing command substitution inside an argument does
# not trip `set -e`, so the inline form degrades silently to an
# empty constant. The VERSION build argument takes precedence over
# the version a build stage derives from the .git in the context.
version="$(git describe --tags --always --dirty 2>/dev/null || true)"
[ -n "$version" ] || version="unknown"
docker build --no-cache \
--build-arg VERSION="$version" \
-t "$("$SCRIPT_DIR/projectname")" .
}
script/fmt
+20 -1
View File
@@ -4,9 +4,28 @@ set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0"
# script/bootstrap installs node and yarn under nvm and leaves neither
# on the PATH of the shell that called it, so resolve the pinned
# toolchain here the way bootstrap's own install step does. nvm is a
# bash script, hence the subshell.
run_yarn() {
if command -v yarn >/dev/null 2>&1; then
exec yarn "$@"
fi
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "fmt: no yarn; run script/bootstrap first" >&2
exit 1
fi
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
}
main() {
cd "$ROOT"
yarn run prettier --write .
run_yarn run prettier --write .
}
main "$@"
script/fmt-check
+20 -1
View File
@@ -4,9 +4,28 @@ set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# Must match the pin in script/bootstrap.
NODE_VERSION="22.17.0"
# script/bootstrap installs node and yarn under nvm and leaves neither
# on the PATH of the shell that called it, so resolve the pinned
# toolchain here the way bootstrap's own install step does. nvm is a
# bash script, hence the subshell.
run_yarn() {
if command -v yarn >/dev/null 2>&1; then
exec yarn "$@"
fi
if [ ! -s "$HOME/.nvm/nvm.sh" ]; then
echo "fmt-check: no yarn; run script/bootstrap first" >&2
exit 1
fi
exec bash -c '. "$HOME/.nvm/nvm.sh" && nvm use "$1" >/dev/null &&
shift && exec yarn "$@"' bash "$NODE_VERSION" "$@"
}
main() {
cd "$ROOT"
yarn run prettier --check .
run_yarn run prettier --check .
}
main "$@"
script/lint
+14 -4
View File
@@ -1,13 +1,23 @@
#!/bin/sh
# script/lint: run the linter (eslint plus a prettier check).
# script/lint: run the linter. Linting is a phase of the Dockerfile and
# this builds that phase alone; the linter is never installed or run on
# a developer host, where a shared result cache and a host-global lock
# make its answer untrustworthy.
#
# The phase is not the last stage in the file, so it is built only when
# --target names it. --no-cache because a cached lint layer is a lint
# that did not run. The tag makes each build replace the previous image
# instead of leaving a dangling one behind.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
yarn run eslint .
yarn run prettier --check .
docker build --no-cache \
--target lint \
-t "$("$SCRIPT_DIR/projectname")-lint" .
}
main "$@"
script/precommit
+2 -2
View File
@@ -4,8 +4,8 @@
#
# Runs lint and fmt-check but deliberately NOT the tests, so the TDD
# red-phase commit (failing tests, no implementation yet) can land. CI
# runs make check via docker build, which catches any branch that
# ships red.
# runs script/cibuild, which runs the tests, and so catches any branch
# that ships red.
set -eu
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
script/test
+10 -16
View File
@@ -1,25 +1,19 @@
#!/bin/sh
# script/test: run the test suite. Uses `timeout` (GNU coreutils) when
# available so the run is hard-capped at 30s; on macOS without
# coreutils the cap is skipped.
# script/test: run the test suite. Testing is a phase of the Dockerfile
# and this builds that phase alone, on the same terms as script/lint:
# --target because a phase that is not the last stage is built only when
# named, --no-cache because a cached test layer is a test that did not
# run, and a tag so each build replaces the previous image.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
rerun_verbose() {
echo "--- Rerunning with verbose for details ---"
yarn run vitest run --reporter=verbose
exit 1
}
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd -P)"
ROOT="$(cd "$SCRIPT_DIR/.." && pwd -P)"
main() {
cd "$ROOT"
TIMEOUT="$(command -v timeout 2>/dev/null || command -v gtimeout 2>/dev/null || true)"
if [ -n "$TIMEOUT" ]; then
"$TIMEOUT" 30s yarn run vitest run --reporter=dot || rerun_verbose
else
yarn run vitest run --reporter=dot || rerun_verbose
fi
docker build --no-cache \
--target test \
-t "$("$SCRIPT_DIR/projectname")-test" .
}
main "$@"
script/version
Executable
+41
View File
@@ -0,0 +1,41 @@
#!/bin/sh
# script/version: print the version script/build stamps into the built
# package. Our own extension to scripts-to-rule-them-all.
#
# Order of precedence:
#
# 1. $VERSION, if set and not empty: an explicit value, such as the
# Dockerfile's VERSION build arg.
# 2. If this checkout has .git, `git describe --tags --always`: the tag
# on a tagged commit; the tag, the commits since it and the short
# commit on a later commit (v1.2.3-4-gabc1234); the short commit when
# no tag is reachable.
# 3. Otherwise, as in a source tarball, the version package.json declares.
#
# A checkout with .git whose version still comes out empty, dev or unknown
# fails: git is missing or could not read the checkout, and the build could
# not be traced back to its commit.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
main() {
cd "$ROOT"
version="${VERSION:-}"
if [ -e .git ]; then
if [ -z "$version" ]; then
version="$(git describe --tags --always || true)"
fi
case "$version" in
"" | dev | unknown)
echo "version: $ROOT has .git, but the version came out '$version'" >&2
exit 1
;;
esac
elif [ -z "$version" ]; then
version="$(node -p 'require("./package.json").version')"
fi
echo "$version"
}
main "$@"
src/api/client.ts
+165 -70
View File
@@ -19,14 +19,12 @@ const DEFAULT_FILES_ORIGIN = "https://files.ente.io";
const DEFAULT_THUMBS_ORIGIN = "https://thumbnails.ente.io";
const CLIENT_PACKAGE = "berlin.sneak.quak";
// Two deadlines rather than one, because a single number cannot serve both
// jobs. Thirty seconds is generous for a JSON call and short enough that a
// hung API connection cannot stall a backup for long. A file body is a
// different shape of problem: the deadline has to cover the whole transfer,
// which for a large video on a slow link is minutes, so a value sane for JSON
// would cancel legitimate downloads.
// Two deadlines of different kinds. `requestTimeoutMs` bounds a whole JSON
// call. `downloadTimeoutMs` is an idle deadline: a file or thumbnail download
// is aborted only when no bytes have arrived for that long, so a large video on
// a slow link that keeps making progress is never cut off.
export const DEFAULT_REQUEST_TIMEOUT_MS = 30_000;
export const DEFAULT_DOWNLOAD_TIMEOUT_MS = 600_000;
export const DEFAULT_DOWNLOAD_TIMEOUT_MS = 60_000;
export interface ApiClientOptions {
apiOrigin?: string;
@@ -38,6 +36,15 @@ export interface ApiClientOptions {
retry?: RetryOptions;
requestTimeoutMs?: number;
downloadTimeoutMs?: number;
// Called once for each request sent, with the status of its response or
// the error it failed with, for a caller that logs requests. A request
// that is retried is reported once for each attempt.
onRequest?: (request: {
method: string;
url: string;
status?: number;
error?: unknown;
}) => void;
}
export interface StreamOptions {
@@ -48,18 +55,44 @@ export interface StreamOptions {
retry?: boolean;
}
// Enforce a deadline over a response body, not merely over its headers.
// An abort signal that fires once `ms` pass without a call to `restart`. It
// aborts with a `TimeoutError`, the same reason `AbortSignal.timeout()` gives,
// so the retry classifier treats an idle download exactly as it treats any
// other deadline. `stop` must be called when the download ends. The timer is
// unref'd, so even one left running never keeps the process alive.
const idleDeadline = (ms: number) => {
const controller = new AbortController();
let timer: ReturnType<typeof setTimeout> | undefined;
const stop = (): void => clearTimeout(timer);
const restart = (): void => {
stop();
timer = setTimeout(() => {
controller.abort(
new DOMException(
`download stalled: no bytes received for ${ms} ms`,
"TimeoutError",
),
);
}, ms);
timer.unref();
};
restart();
return { signal: controller.signal, restart, stop };
};
// Enforce the idle deadline over a response body, not merely over its headers.
//
// `getFileStream` returns as soon as headers arrive; the bytes are pulled
// later, in the download layer. Whether the signal passed to `fetch` also
// tears down the body afterwards is up to the fetch implementation, so this
// wrapper makes it a property of quak instead: every read races the signal,
// and an abort errors the stream with the abort reason — which the retry
// classifier recognises.
// each chunk that arrives restarts the deadline, and an abort errors the
// stream with the abort reason — which the retry classifier recognises.
const deadlineStream = (
body: ReadableStream<Uint8Array>,
signal: AbortSignal,
deadline: ReturnType<typeof idleDeadline>,
): ReadableStream<Uint8Array> => {
const { signal } = deadline;
const reader = body.getReader();
let rejectOnAbort: (reason: unknown) => void = () => undefined;
const aborted = new Promise<never>((_resolve, reject) => {
@@ -73,7 +106,10 @@ const deadlineStream = (
const onAbort = (): void => rejectOnAbort(signal.reason);
if (signal.aborted) onAbort();
else signal.addEventListener("abort", onAbort, { once: true });
const release = (): void => signal.removeEventListener("abort", onAbort);
const release = (): void => {
deadline.stop();
signal.removeEventListener("abort", onAbort);
};
return new ReadableStream<Uint8Array>({
async pull(controller) {
@@ -84,6 +120,7 @@ const deadlineStream = (
controller.close();
return;
}
deadline.restart();
controller.enqueue(next.value);
} catch (err) {
release();
@@ -98,6 +135,30 @@ const deadlineStream = (
});
};
// The one place a request URL is built. `origin` may carry a base path (a
// self-hosted server behind a prefix) and may end in a slash; `path` may or
// may not start with one. Query parameters go only through `query`, which
// percent-encodes them: a `?` or `#` in `path` is an error, because
// `new URL` would otherwise quietly treat what follows as something else.
const buildURL = (
origin: string,
path: string,
query?: Record<string, string | number | undefined>,
): string => {
if (path.includes("?") || path.includes("#")) {
throw new Error(
`request path must not contain "?" or "#"; pass query parameters separately: ${path}`,
);
}
const url = new URL(
`${origin.replace(/\/+$/, "")}/${path.replace(/^\/+/, "")}`,
);
for (const [k, v] of Object.entries(query ?? {})) {
if (v !== undefined) url.searchParams.set(k, String(v));
}
return url.href;
};
export class ApiClient {
private readonly apiOrigin: string;
private readonly isCustomOrigin: boolean;
@@ -107,6 +168,7 @@ export class ApiClient {
private readonly retry: ResolvedRetryOptions;
private readonly requestTimeoutMs: number;
private readonly downloadTimeoutMs: number;
private readonly onRequest: NonNullable<ApiClientOptions["onRequest"]>;
private token: string | undefined;
constructor(opts?: ApiClientOptions) {
@@ -128,6 +190,7 @@ export class ApiClient {
opts?.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS;
this.downloadTimeoutMs =
opts?.downloadTimeoutMs ?? DEFAULT_DOWNLOAD_TIMEOUT_MS;
this.onRequest = opts?.onRequest ?? (() => {});
this.token = opts?.authToken;
}
@@ -139,11 +202,16 @@ export class ApiClient {
this.token = undefined;
}
getAuthToken(): string | undefined {
return this.token;
}
// The policy this client was configured with, so that a caller wrapping a
// whole operation in its own `withRetry` — the download layer — runs under
// the same settings rather than under the library defaults.
// A copy, so the caller cannot change this client's settings through it.
getRetryOptions(): ResolvedRetryOptions {
return this.retry;
return { ...this.retry };
}
private headers(extra?: Record<string, string>): Record<string, string> {
@@ -157,6 +225,33 @@ export class ApiClient {
return h;
}
// Send one request and report it to `onRequest`, with the status of its
// response or the error it failed with. `reportedURL` is the URL given to
// `onRequest`, when that must not be the one sent.
private async send(
url: string,
init: RequestInit & { method: string },
reportedURL = url,
): Promise<Response> {
let resp: Response;
try {
resp = await this._fetch(url, init);
} catch (err) {
this.onRequest({
method: init.method,
url: reportedURL,
error: err,
});
throw err;
}
this.onRequest({
method: init.method,
url: reportedURL,
status: resp.status,
});
return resp;
}
private async throwIfError(resp: Response): Promise<void> {
if (resp.ok) return;
const requestID = resp.headers.get("x-request-id") ?? undefined;
@@ -197,22 +292,10 @@ export class ApiClient {
path: string,
query?: Record<string, string | number | undefined>,
): Promise<T> {
const url = new URL(path, this.apiOrigin + "/");
// new URL with a base resolves relative paths; ensure we keep the
// origin from apiOrigin even when path starts with /
url.protocol = new URL(this.apiOrigin).protocol;
url.host = new URL(this.apiOrigin).host;
url.pathname = path;
if (query) {
for (const [k, v] of Object.entries(query)) {
if (v !== undefined) {
url.searchParams.set(k, String(v));
}
}
}
const url = buildURL(this.apiOrigin, path, query);
// A GET changes nothing, so it is retried under the full policy.
return withRetry(async () => {
const resp = await this._fetch(url.href, {
const resp = await this.send(url, {
method: "GET",
headers: this.headers(),
signal: AbortSignal.timeout(this.requestTimeoutMs),
@@ -223,24 +306,25 @@ export class ApiClient {
}
async postJSON<T>(path: string, body: unknown): Promise<T> {
const url = `${this.apiOrigin}${path}`;
// Idempotency: this reaches `/users/srp/create-session`,
// `/users/two-factor/verify` and `/users/ott`, all of which change
// server state — verifying a second factor consumes one of a small
// number of attempts. So a POST is replayed only on a failure that
// establishes no TCP connection to the server ever existed: DNS
// produced no address, or the peer refused the connection. A 5xx, a
// mid-flight reset, a routing errno (which Linux also delivers on an
// established socket) and a timeout are all left to the caller,
// because each of them can occur after the server has already acted.
const url = buildURL(this.apiOrigin, path);
// Not idempotent: a POST is replayed only when `isSafeToReplay`
// says no request byte can have reached the server. The endpoints
// this covers are listed in the README under "Endpoints used".
//
// Redirects are not followed. The origin has already received the
// request when it answers with one, so a connection refused by the
// redirect target would look replay-safe when it is not. The API has
// no legitimate redirect, so one surfaces as an `ApiError` with its
// 3xx status, which is not retried.
return withRetry(
async () => {
const resp = await this._fetch(url, {
const resp = await this.send(url, {
method: "POST",
headers: this.headers({
"Content-Type": "application/json",
}),
body: JSON.stringify(body),
redirect: "manual",
signal: AbortSignal.timeout(this.requestTimeoutMs),
});
await this.throwIfError(resp);
@@ -255,8 +339,8 @@ export class ApiClient {
opts?: StreamOptions,
): Promise<ReadableStream<Uint8Array>> {
const url = this.isCustomOrigin
? `${this.apiOrigin}/files/download/${fileID}`
: `${this.filesOrigin}/?fileID=${fileID}`;
? buildURL(this.apiOrigin, `/files/download/${fileID}`)
: buildURL(this.filesOrigin, "/", { fileID });
return this.streamRequest(url, opts);
}
@@ -275,16 +359,23 @@ export class ApiClient {
// object at one key in one request, so replaying it either overwrites
// the same bytes or lands them for the first time. There is no partial
// state to protect, hence the full policy rather than the POST rule.
// It is reported without its query, which holds the upload's
// signature: a log keeps what it is given.
const { origin, pathname } = new URL(presignedURL);
await withRetry(async () => {
const resp = await this._fetch(presignedURL, {
method: "PUT",
headers: {
"Content-Type": "application/octet-stream",
"Content-Length": String(data.length),
const resp = await this.send(
presignedURL,
{
method: "PUT",
headers: {
"Content-Type": "application/octet-stream",
"Content-Length": String(data.length),
},
body: data,
signal: AbortSignal.timeout(this.requestTimeoutMs),
},
body: data,
signal: AbortSignal.timeout(this.requestTimeoutMs),
});
origin + pathname,
);
if (!resp.ok) {
// An ApiError, not a bare Error: without the status on the
// error the upload path cannot be classified at all, and a
@@ -298,18 +389,17 @@ export class ApiClient {
}
async putJSON<T>(path: string, body: unknown): Promise<T> {
const url = `${this.apiOrigin}${path}`;
// Same idempotency rule as `postJSON`, for the same reason: this
// reaches `/files/thumbnail`, which registers an uploaded thumbnail
// against a file.
const url = buildURL(this.apiOrigin, path);
// Same replay and redirect rules as `postJSON`, for the same reasons.
return withRetry(
async () => {
const resp = await this._fetch(url, {
const resp = await this.send(url, {
method: "PUT",
headers: this.headers({
"Content-Type": "application/json",
}),
body: JSON.stringify(body),
redirect: "manual",
signal: AbortSignal.timeout(this.requestTimeoutMs),
});
await this.throwIfError(resp);
@@ -335,8 +425,8 @@ export class ApiClient {
opts?: StreamOptions,
): Promise<ReadableStream<Uint8Array>> {
const url = this.isCustomOrigin
? `${this.apiOrigin}/files/preview/${fileID}`
: `${this.thumbsOrigin}/?fileID=${fileID}`;
? buildURL(this.apiOrigin, `/files/preview/${fileID}`)
: buildURL(this.thumbsOrigin, "/", { fileID });
return this.streamRequest(url, opts);
}
@@ -345,22 +435,27 @@ export class ApiClient {
opts?: StreamOptions,
): Promise<ReadableStream<Uint8Array>> {
const once = async (): Promise<ReadableStream<Uint8Array>> => {
// A fresh deadline per attempt, so a retry gets the whole budget
// rather than the remainder of the one that just expired.
const signal = AbortSignal.timeout(this.downloadTimeoutMs);
const resp = await this._fetch(url, {
method: "GET",
headers: this.headers(),
signal,
});
await this.throwIfError(resp);
if (!resp.body) {
// Carries the status, and is not retryable: a response that
// arrived without a body is malformed, and asking again
// produces the same malformed response.
throw new ApiError("response body is null", resp.status);
// A fresh deadline per attempt. It also covers the wait for the
// headers, when no bytes have arrived either.
const deadline = idleDeadline(this.downloadTimeoutMs);
try {
const resp = await this.send(url, {
method: "GET",
headers: this.headers(),
signal: deadline.signal,
});
await this.throwIfError(resp);
if (!resp.body) {
// Carries the status, and is not retryable: a response
// that arrived without a body is malformed, and asking
// again produces the same malformed response.
throw new ApiError("response body is null", resp.status);
}
return deadlineStream(resp.body, deadline);
} catch (err) {
deadline.stop();
throw err;
}
return deadlineStream(resp.body, signal);
};
return opts?.retry === false ? once() : withRetry(once, this.retry);
}
src/backup.ts
+1015 -113
View File
File diff suppressed because it is too large Load Diff
src/cli-commands.ts
+630
View File
@@ -0,0 +1,630 @@
// The CLI's commands as plain functions.
//
// Each command takes its options and a `CliContext` and resolves to the exit
// code; a thrown error is left to the caller. Nothing here calls
// `process.exit`: `bin/quak.ts` wires these to the command line, and `run` in
// `cli-run.ts` prints a thrown error as one line and exits once output has
// drained. Output must stay byte-identical (see `cli-output.ts`). Each line on
// stderr goes through `ctx.log` at its level (see `cli-log.ts`). Every command
// that exits non-zero logs a fatal line saying why; one that exits 1 because
// files or albums failed logs it last, saying how many and where to look.
import {
copyFileSync,
existsSync,
mkdirSync,
statSync,
unlinkSync,
writeFileSync,
} from "node:fs";
import { extname, join } from "node:path";
import {
type Client,
type ClientSnapshot,
type LoginOptions,
} from "./client.js";
import { lockBackupDirectory, type BackupLock } from "./backup.js";
import { init } from "./crypto/index.js";
import {
defaultCacheDirectory,
Library,
type LibraryClient,
} from "./library/index.js";
import {
fileListRow,
fileListLine,
originalName,
thumbnailName,
} from "./cli-output.js";
import { type Log, logRefresh } from "./cli-log.js";
import { freshCollections, freshFiles, freshFile } from "./cli-read.js";
import { withExtension } from "./filename.js";
import { runMetadataBackup } from "./metadata-backup.js";
import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js";
export interface CliContext {
stdout: { write(text: string): unknown };
// Writes a line on stderr at a log level, if the run's level prints it.
log: Log;
// Directory holding `session.json`.
sessionDir: string;
// The `--cache-dir` global, or undefined to let the library pick its
// per-user default keyed by the account id.
cacheDir?: string;
// Reads the session file into a client, or null when there is none. The
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
loadSession: (path: string) => Client | null;
// Used by `login` only. The CLI passes `Client.login` and terminal
// prompts; tests pass fakes.
login: (opts: LoginOptions) => Promise<Client>;
prompt: (message: string) => Promise<string>;
// Like `prompt`, but the answer is masked as it is typed.
promptSecret: (message: string) => Promise<string>;
}
const sessionPath = (ctx: CliContext): string =>
join(ctx.sessionDir, "session.json");
// Write the session readable by its owner only, in a directory only its owner
// can enter.
export const saveSession = (
sessionDir: string,
snapshot: ClientSnapshot,
): void => {
mkdirSync(sessionDir, { recursive: true, mode: 0o700 });
writeFileSync(
join(sessionDir, "session.json"),
JSON.stringify(snapshot, null, 2),
{ mode: 0o600 },
);
};
// The saved client, or undefined after telling the user why there is none. The
// command then exits 3, the code for "log in again", as `run` in `cli-run.ts`
// does when the server no longer accepts the saved session.
const requireSession = (ctx: CliContext): Client | undefined => {
let client: Client | null;
try {
client = ctx.loadSession(sessionPath(ctx));
} catch (err) {
ctx.log("fatal", err instanceof Error ? err.message : String(err));
ctx.log(
"fatal",
`Run "quak logout" and then "quak login" to replace it.`,
);
return undefined;
}
if (!client) {
ctx.log("fatal", `Not logged in. Run "quak login" first.`);
ctx.log("fatal", `Session file: ${sessionPath(ctx)}`);
return undefined;
}
return client;
};
// `text` as an ID, or undefined after telling the user it is not one. An ID is
// written in decimal digits and is a safe integer; `Number()` alone would read
// `0x10` as 16, `1e3` as 1000 and "" as 0.
const parseID = (
ctx: CliContext,
kind: "file" | "collection",
text: string,
): number | undefined => {
const id = Number(text);
if (/^[0-9]+$/.test(text) && Number.isSafeInteger(id)) return id;
ctx.log("fatal", `Invalid ${kind} ID: ${text}`);
return undefined;
};
// A library client that omits `fetchMLData`, so the point commands below do not
// kick the library's background ML backfill: they read metadata, or fetch one
// file's content, and exit. `backup` and `backup-metadata` handle ML on their
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
// originals through the on-disk cache.
const readLibraryClient = (client: Client): LibraryClient => ({
whoami: () => client.whoami(),
collectionsSince: (args) => client.collectionsSince(args),
filesSince: (args) => client.filesSince(args),
contentSource: () => client.contentSource(),
});
// Open a library for a single point command. The aggressive background
// precache (issue #48) is off, because a one-shot `collections` or `get` must
// not start downloading the whole account. No refresh starts as it opens or on
// a timer. Each command asks for the refresh it needs. `collections` needs none
// (issue #210). `files COLLECTION` needs one only when `refreshCollection()`
// resolves to `false` (issue #217). While a refresh runs,
// each collection whose files it has fetched is logged, so a large account
// shows progress rather than nothing.
const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
Library.open({
client: readLibraryClient(client),
cacheDirectory: ctx.cacheDir,
refreshOnOpen: false,
refreshIntervalSeconds: Infinity,
precacheThumbnails: false,
precacheOriginals: false,
onProgress: logRefresh(ctx.log),
});
export const loginCommand = async (ctx: CliContext): Promise<number> => {
await init();
const email = process.env.QUAK_EMAIL ?? (await ctx.prompt("Email"));
const password =
process.env.QUAK_PASSWORD ?? (await ctx.promptSecret("Password"));
ctx.log("info", "Authenticating...");
try {
const client = await ctx.login({
email,
password,
totp: async () => ctx.prompt("TOTP code: "),
emailOTP: async () => ctx.prompt("Email verification code: "),
});
saveSession(ctx.sessionDir, client.toJSON());
const info = client.whoami();
ctx.log("info", `Logged in as ${info.email} (user ${info.userID})`);
ctx.log("info", `Session saved to ${sessionPath(ctx)}`);
} catch (err) {
ctx.log(
"fatal",
`Login failed: ${err instanceof Error ? err.message : err}`,
);
return 1;
}
return 0;
};
export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 3;
const info = client.whoami();
ctx.stdout.write(JSON.stringify(info) + "\n");
return 0;
};
// Ends the session on the server, then deletes the session file even when that
// failed, and exits 1 if it did. The cache is left in place; the user is told
// where it is.
export const logoutCommand = async (ctx: CliContext): Promise<number> => {
const path = sessionPath(ctx);
if (!existsSync(path)) {
ctx.log("info", "No session found.");
return 0;
}
await init();
let cacheDir = ctx.cacheDir;
let failure: string | undefined;
try {
const client = ctx.loadSession(path);
if (client) {
cacheDir ??= defaultCacheDirectory(client.whoami().userID);
await client.logoutOnServer();
client.logout();
}
} catch (err) {
failure = err instanceof Error ? err.message : String(err);
}
unlinkSync(path);
if (failure === undefined) {
ctx.log("info", "Session ended on the server.");
} else {
ctx.log("fatal", `Could not end the session on the server: ${failure}`);
}
ctx.log("info", "Session deleted.");
if (cacheDir !== undefined) {
ctx.log(
"warning",
`Cache directory ${cacheDir} still holds decrypted data; delete it to remove that data.`,
);
}
return failure === undefined ? 0 : 1;
};
export const collectionsCommand = async (
ctx: CliContext,
opts: { json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 3;
const lib = await openReadLibrary(ctx, client);
try {
// Ask the server for the collection list alone and list it in its
// order (issue #36 amendment, issues #52 and #210): the pre-library
// CLI printed current state in this order, not the albums
// projection's newest-first order.
const collections = await freshCollections(lib);
if (opts.json) {
ctx.stdout.write(
JSON.stringify(
collections.map((c) => ({
id: c.id,
name: c.name,
type: c.type,
ownerID: c.ownerID,
isShared: c.isShared,
updationTime: c.updationTime,
})),
null,
2,
) + "\n",
);
} else {
for (const c of collections) {
ctx.stdout.write(
`${c.id}\t${c.type}\t${c.name}${c.isShared ? " (shared)" : ""}\n`,
);
}
}
return 0;
} finally {
await lib.close();
}
};
// Lists the files of the collection `collectionIDStr` names, or without it every
// file in the account, each once (issue #211).
export const filesCommand = async (
ctx: CliContext,
collectionIDStr: string | undefined,
opts: { json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 3;
const collectionID =
collectionIDStr === undefined
? undefined
: parseID(ctx, "collection", collectionIDStr);
if (collectionIDStr !== undefined && collectionID === undefined) return 1;
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52). Each file prints from its own decrypted
// metadata (raw title, microsecond creationTime) via cli-output, and in
// the pre-library CLI's enumeration order, not the projection's
// newest-first order.
const files = await freshFiles(lib, collectionID);
if (!files) {
ctx.log("fatal", `Collection ${collectionID} not found`);
return 1;
}
if (opts.json) {
ctx.stdout.write(
JSON.stringify(files.map(fileListRow), null, 2) + "\n",
);
} else {
for (const file of files) {
ctx.stdout.write(fileListLine(file) + "\n");
}
}
return 0;
} finally {
await lib.close();
}
};
export const getCommand = async (
ctx: CliContext,
fileIDStr: string,
opts: { out?: string },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 3;
const fileID = parseID(ctx, "file", fileIDStr);
if (fileID === undefined) return 1;
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
ctx.log("fatal", `File ${fileID} not found`);
return 1;
}
const { photo, file } = resolved;
const result = await photo.original();
// Default name is the file's own title, as the pre-library CLI used
// (not the editedName-preferring projection title) (issue #52).
if (result.videoPath === undefined) {
const outPath = opts.out ?? originalName(file);
copyFileSync(result.path, outPath);
ctx.log("info", `${result.bytes} bytes -> ${outPath}`);
return 0;
}
// A live photo is written as its image and its video, each named after
// the title with its own extension, as Ente's clients name them. With
// --out, the image goes there and the video beside it.
const imageExt = extname(result.path);
const videoExt = extname(result.videoPath);
const imageOut = opts.out ?? originalName(file, imageExt);
const videoOut =
opts.out === undefined
? originalName(file, videoExt)
: withExtension(opts.out, videoExt);
if (imageOut.toLowerCase() === videoOut.toLowerCase()) {
ctx.log(
"fatal",
`File ${fileID} is a live photo, and its video would also be written to ${imageOut}`,
);
return 1;
}
copyFileSync(result.path, imageOut);
copyFileSync(result.videoPath, videoOut);
ctx.log("info", `${result.bytes} bytes -> ${imageOut}`);
ctx.log("info", `${statSync(videoOut).size} bytes -> ${videoOut}`);
return 0;
} finally {
await lib.close();
}
};
export const getThumbCommand = async (
ctx: CliContext,
fileIDStr: string,
opts: { out?: string },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 3;
const fileID = parseID(ctx, "file", fileIDStr);
if (fileID === undefined) return 1;
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
ctx.log("fatal", `File ${fileID} not found`);
return 1;
}
const { photo, file } = resolved;
const result = await photo.thumbnail();
// Default name is thumb_<file's own title>, as the pre-library CLI
// used (not the projection title) (issue #52).
const outPath = opts.out ?? thumbnailName(file);
copyFileSync(result.path, outPath);
ctx.log("info", `${result.bytes} bytes -> ${outPath}`);
return 0;
} finally {
await lib.close();
}
};
export const backupMetadataCommand = async (
ctx: CliContext,
dir: string,
opts: { exif?: boolean; all?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 3;
const lib = await openReadLibrary(ctx, client);
try {
// Refresh first so the dump holds current account state, not what the
// cache last held; a failed refresh throws.
await lib.fresh();
const result = await runMetadataBackup(lib, client, dir, {
exif: opts.exif || opts.all,
onProgress: (msg, level) => ctx.log(level ?? "info", msg),
});
if (result.filesWithMLDataError === 0) return 0;
ctx.log(
"fatal",
`quak: the ML data of ${result.filesWithMLDataError} file(s) could not be read; see mlDataError in their JSON in ${dir}`,
);
return 1;
} finally {
await lib.close();
}
};
export const backupCommand = async (
ctx: CliContext,
dir: string,
opts: { json?: boolean; verify?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 3;
ctx.log("info", "Starting backup...");
// The lock is taken before the library opens and starts its refresh, so a
// run refused while another backup of `dir` runs sends no request.
let lock: BackupLock;
try {
lock = await lockBackupDirectory(dir);
} catch (err) {
if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;
ctx.log("fatal", `quak: ${(err as Error).message}`);
return 2;
}
try {
// The precache is off: the backup fetches what it needs, and must not
// also fill the cache with every thumbnail and the recent originals.
// No refresh starts on a timer: the backup lists its files once, from
// the refresh it waits for, and can run for hours.
const lib = await Library.open({
client,
downloadDirectory: dir,
cacheDirectory: ctx.cacheDir,
refreshIntervalSeconds: Infinity,
precacheThumbnails: false,
precacheOriginals: false,
onProgress: logRefresh(ctx.log),
});
try {
const result = await lib.backup({
downloadDirectory: dir,
lock,
verify: opts.verify,
onProgress: (msg, level) => ctx.log(level ?? "info", msg),
});
if (opts.json) {
ctx.stdout.write(JSON.stringify(result, null, 2) + "\n");
} else {
ctx.log("info", "\n--- Backup complete ---");
ctx.log("info", ` Total files: ${result.totalFiles}`);
ctx.log("info", ` Downloaded: ${result.downloaded}`);
ctx.log("info", ` Skipped: ${result.skipped}`);
if (opts.verify) {
ctx.log("info", ` Verified: ${result.verified}`);
ctx.log("info", ` Mismatched: ${result.mismatched}`);
ctx.log("info", ` Unchecked: ${result.unchecked}`);
}
ctx.log("info", ` Failed: ${result.failed}`);
if (result.errors.length > 0) {
ctx.log("error", "\nFailed files:");
for (const e of result.errors) {
ctx.log(
"error",
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}`,
);
}
}
}
const albums =
result.undecryptableAlbums.length + result.failedAlbums.length;
if (result.failed === 0 && albums === 0) return 0;
ctx.log(
"fatal",
`quak: ${result.failed} file(s) and ${albums} album(s) failed; see ${join(dir, "failures.json")} and the FAILED lines`,
);
return 1;
} finally {
await lib.close();
}
} finally {
await lock.release();
}
};
export const listMissingThumbnailsCommand = async (
ctx: CliContext,
opts: { json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 3;
const lib = await openReadLibrary(ctx, client);
try {
// Refresh first so files added since the cache was written are
// checked; a failed refresh throws.
await lib.fresh();
const missing = await listMissingThumbnails(lib, client, (msg, level) =>
ctx.log(level ?? "info", msg),
);
if (opts.json) {
ctx.stdout.write(JSON.stringify(missing, null, 2) + "\n");
} else {
if (missing.length === 0) {
ctx.log("info", "No missing thumbnails found.");
} else {
ctx.log(
"info",
`\n${missing.length} file(s) with missing thumbnails:`,
);
for (const m of missing) {
ctx.stdout.write(
`${m.fileID}\t${m.title}\t${m.collection}\t${m.reason}\n`,
);
}
}
}
return 0;
} finally {
await lib.close();
}
};
export const fixMissingThumbnailsCommand = async (
ctx: CliContext,
opts: { file?: string[]; json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 3;
// One bad --file ID fails the whole command, before anything is fetched.
let fileIDs: number[] = [];
for (const text of opts.file ?? []) {
const fileID = parseID(ctx, "file", text);
if (fileID === undefined) return 1;
fileIDs.push(fileID);
}
const lib = await openReadLibrary(ctx, client);
try {
// Refresh first so files added since the cache was written are found;
// a failed refresh throws.
await lib.fresh();
if (fileIDs.length === 0) {
ctx.log("info", "Scanning for missing thumbnails...");
const missing = await listMissingThumbnails(
lib,
client,
(msg, level) => ctx.log(level ?? "info", msg),
);
fileIDs = missing.map((m) => m.fileID);
if (fileIDs.length === 0) {
ctx.log("info", "No missing thumbnails found.");
return 0;
}
ctx.log("info", `Found ${fileIDs.length} file(s) to fix.`);
}
const results = await fixMissingThumbnails(
lib,
client,
fileIDs,
(msg, level) => ctx.log(level ?? "info", msg),
);
const failed = results.filter((r) => r.status === "failed").length;
if (opts.json) {
ctx.stdout.write(JSON.stringify(results, null, 2) + "\n");
} else {
const fixed = results.filter((r) => r.status === "fixed").length;
const skipped = results.filter(
(r) => r.status === "skipped",
).length;
ctx.log("info", `\n--- Done ---`);
ctx.log("info", ` Fixed: ${fixed}`);
ctx.log("info", ` Skipped: ${skipped}`);
ctx.log("info", ` Failed: ${failed}`);
if (skipped > 0) {
ctx.log("info", "\nSkipped:");
for (const r of results.filter((r) => r.status === "skipped")) {
ctx.log("info", ` ${r.fileID}\t${r.title}\t${r.reason}`);
}
}
if (failed > 0) {
ctx.log("error", "\nFailed files:");
for (const r of results.filter((r) => r.status === "failed")) {
ctx.log("error", ` ${r.fileID}\t${r.title}\t${r.reason}`);
}
}
}
if (failed === 0) return 0;
ctx.log(
"fatal",
`quak: ${failed} file(s) failed; see the "Failed files:" list, or the JSON with --json`,
);
return 1;
} finally {
await lib.close();
}
};
src/cli-log.ts
+99
View File
@@ -0,0 +1,99 @@
// The CLI's log level, and the one function the commands write stderr lines
// through.
//
// Every line quak prints on stderr is a log line at one of six levels, from
// the quietest: fatal, error, warning, info, debug, trace. A run prints the
// lines at its own level and at every quieter one. Its level is info unless
// `-v` or `-q` asks for another (see "CLI surface" in the README). Every
// command that exits non-zero logs a fatal line saying why, so even `-q`, which
// prints only fatal lines, never leaves a non-zero exit unexplained. An error
// line reports something that failed, such as a file or an album, and a
// warning something that did not fail but needs the user's attention. stdout
// carries only a command's output and is not a log.
//
// The library itself writes no lines: its layers report through callbacks, and
// the CLI logs what they report. `logRequests` gives the API client options
// that log each request and each retry, and `logRefresh` the library's
// `onProgress` that logs what a refresh and an ML data fetch report.
import type { ApiClientOptions } from "./api/client.js";
import { describeError } from "./errors.js";
import type { RefreshEvent } from "./library/index.js";
import type { RetryOptions } from "./retry.js";
const LEVELS = ["fatal", "error", "warning", "info", "debug", "trace"] as const;
export type LogLevel = (typeof LEVELS)[number];
// Writes `line` on stderr when the run's level prints lines at `level`.
export type Log = (level: LogLevel, line: string) => void;
// The level asked for by `-v` given `verbose` times, or by `-q`: info without
// either, debug for `-v`, trace for `-vv` and more, fatal for `-q`.
// `bin/quak.ts` refuses `-q` and `-v` together.
export const logLevel = (verbose: number, quiet: boolean): LogLevel => {
if (quiet) return "fatal";
if (verbose === 0) return "info";
return verbose === 1 ? "debug" : "trace";
};
// The log that writes each line at `level` or quieter to `stderr`, ending it
// with a newline.
export const stderrLog =
(stderr: { write(text: string): unknown }, level: LogLevel): Log =>
(lineLevel, line) => {
if (LEVELS.indexOf(lineLevel) <= LEVELS.indexOf(level)) {
stderr.write(line + "\n");
}
};
// API client options that log each request at trace, with the status of its
// response or the error it failed with, and each retry at debug, with why and
// how long it waits. `retry` is the retry policy, the default when not given.
export const logRequests = (
log: Log,
retry?: RetryOptions,
): ApiClientOptions => ({
retry: {
...retry,
onRetry: (err, attempt, delayMs) =>
log(
"debug",
`Attempt ${attempt} failed, retrying in ${Math.round(delayMs)} ms: ${describeError(err)}`,
),
},
onRequest: ({ method, url, status, error }) =>
log(
"trace",
`${method} ${url}: ${error === undefined ? `HTTP ${status}` : describeError(error)}`,
),
});
// How a log line names each operation the library reports the start and end
// of.
const OPERATION_NAMES: Record<RefreshEvent["operation"], string> = {
refresh: "Refresh",
fetchMLData: "ML data fetch",
precacheThumbnails: "Thumbnail precache",
precacheOriginals: "Original precache",
};
// A library `onProgress` that logs, at info, each collection whose files a
// refresh has fetched, and at debug the start and end of each refresh and ML
// data fetch, with the error of one that failed.
export const logRefresh =
(log: Log) =>
(event: RefreshEvent): void => {
if (event.status === "progress") {
log(
"info",
`Fetched the files of ${event.fetched} of ${event.total} collections`,
);
return;
}
const error = event.error === undefined ? "" : `: ${event.error}`;
log(
"debug",
`${OPERATION_NAMES[event.operation]} ${event.status}${error}`,
);
};
src/cli-output.ts
+45
View File
@@ -0,0 +1,45 @@
// How the CLI presents a file's identity in `files`, `get`, and `get-thumb`.
//
// These read the file's own decrypted metadata — the raw title and the
// creationTime in microseconds — rather than the `PhotoRecord` projection the
// rest of the library exposes. The projection prefers `editedName`/`editedTime`
// and reports time in milliseconds, which is right for a photo browser but
// would change the CLI's externally-visible output. The pre-library CLI printed
// `metadata.title` and `metadata.creationTime` and named downloads after
// `metadata.title`, and issue #52 requires that output stay byte-identical, so
// the commands shape their output from the raw `EnteFile` through here.
import { defaultFileName } from "./filename.js";
import type { EnteFile, FileType, Microseconds } from "./model/types.js";
// One row of `quak files --json`.
export interface FileListRow {
id: number;
title: string;
fileType: FileType;
creationTime: Microseconds;
collectionID: number;
}
export const fileListRow = (file: EnteFile): FileListRow => ({
id: file.id,
title: file.metadata.title,
fileType: file.metadata.fileType,
creationTime: file.metadata.creationTime,
collectionID: file.collectionID,
});
// One line of `quak files` in its human, tab-separated form.
export const fileListLine = (file: EnteFile): string =>
`${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
// Default output path for `quak get` when `--out` is not given, made from the
// title (see `defaultFileName`), with `ext` in place of its extension for a
// live photo's image or video; `--out` is the user's and is used as is.
export const originalName = (file: EnteFile, ext?: string): string =>
defaultFileName(file.metadata.title, file.id, { ext });
// Default output path for `quak get-thumb` when `--out` is not given: `thumb_`
// and the title.
export const thumbnailName = (file: EnteFile): string =>
defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
src/cli-read.ts
+101
View File
@@ -0,0 +1,101 @@
// How the CLI's read commands obtain current data.
//
// `collections`, `files`, `get`, and `get-thumb` must answer for
// the account's state at the moment the command runs, not for whatever the
// local cache last happened to hold (owner amendment, issue #36). `files`
// without a collection ID, `get` and `get-thumb` therefore force a server
// round-trip through `Library.fresh()`, and only then read. A collection, file,
// or metadata change made elsewhere is then visible. `collections` asks the
// library for the collection list alone through `Library.freshCollections()`
// (issue #210). A refresh would also fetch the files of every collection, which
// `collections` does not print, and on a large account that takes long.
// `files COLLECTION` reads the collection list the same way, for the same
// reason. It then brings that one collection up to date through
// `Library.refreshCollection()` (issue #217), and fetches no other collection's
// files. Once a refresh has received the account's collections,
// `refreshCollection()` stores no collection the library does not hold.
// `files COLLECTION` then forces a refresh instead.
//
// `collections` lists in the server's order. `files` lists in the library's
// own enumeration order: `listCollections()`, then `listFiles()` for each.
// Those are the orders the pre-library CLI printed. The `albums`/`photos`
// projection would re-sort the rows newest first. After a refresh, the
// library holds collections in the order the server first sent them. The
// field values still come from each record's raw metadata via
// `cli-output.ts`. `files` without a collection ID lists every file in the
// account once (issue #211).
// The library keeps a record of a file for each collection that holds it, and
// only the first collection's record is listed.
import type { Collection, EnteFile } from "./model/types.js";
import type { Photo, PhotosAPI } from "./library/index.js";
// The slice of `Library` these helpers read. `Library` satisfies it
// structurally. A test can drive them with a stand-in that records the
// `fresh()`, `freshCollections()` and `refreshCollection()` calls. The
// stand-in serves records in a known enumeration order.
export interface FreshReadLibrary {
fresh(): Promise<unknown>;
freshCollections(): Promise<Collection[]>;
refreshCollection(collection: Collection): Promise<boolean>;
getCollection(id: number): Collection | undefined;
listCollections(): Collection[];
listFiles(collectionID: number): EnteFile[];
getFileByID(fileID: number): EnteFile | undefined;
photos: Pick<PhotosAPI, "byID">;
}
// Every live collection, current as of one collections request, in the
// server's order.
export const freshCollections = (
lib: FreshReadLibrary,
): Promise<Collection[]> => lib.freshCollections();
// The files of one collection, in enumeration order. `undefined` (not an empty
// list) when the collection does not exist, so the caller can tell "no such
// collection" from "an empty collection". They are current as of one
// collections request and the requests that bring that collection alone up to
// date. When `refreshCollection()` resolves to `false`, they are current as of
// a forced refresh instead. Without a collection ID, every file in the account,
// current as of a forced refresh. Each file is listed once, as held by the
// first collection that holds it.
export const freshFiles = async (
lib: FreshReadLibrary,
collectionID?: number,
): Promise<EnteFile[] | undefined> => {
if (collectionID !== undefined) {
const collections = await lib.freshCollections();
const collection = collections.find((c) => c.id === collectionID);
if (!collection) return undefined;
if (!(await lib.refreshCollection(collection))) {
await lib.fresh();
if (!lib.getCollection(collectionID)) return undefined;
}
return lib.listFiles(collectionID);
}
await lib.fresh();
const listed = new Set<number>();
const files: EnteFile[] = [];
for (const collection of lib.listCollections()) {
for (const file of lib.listFiles(collection.id)) {
if (listed.has(file.id)) continue;
listed.add(file.id);
files.push(file);
}
}
return files;
};
// One file, current as of a forced refresh, resolved to both its content
// handle (`Photo`, for fetching bytes) and its raw record (`EnteFile`, for the
// default output name and field values). `undefined` when the file is unknown.
export const freshFile = async (
lib: FreshReadLibrary,
fileID: number,
): Promise<{ photo: Photo; file: EnteFile } | undefined> => {
await lib.fresh();
const photo = lib.photos.byID({ fileID });
const file = lib.getFileByID(fileID);
if (!photo || !file) return undefined;
return { photo, file };
};
src/cli-run.ts
+50
View File
@@ -0,0 +1,50 @@
// Runs one CLI command for `bin/quak.ts` and exits with its code.
import type { Writable } from "node:stream";
import { ApiError } from "./api/client.js";
import type { Log } from "./cli-log.js";
import { describeError } from "./errors.js";
// Run a command and exit with its code once stdout/stderr have drained.
// Exiting before the drain can truncate piped output, and the library can keep
// the event loop alive after a command returns, so a plain return could hang.
// An error the command throws is logged as one fatal `quak: MESSAGE` line,
// without the stack trace, and exits 1. MESSAGE is the error as
// `describeError` gives it, so it carries the reason Node's `fetch` puts on
// `cause`. A 401 from the server means it no longer accepts the saved session:
// that logs one fatal line saying to log in again and exits 3, as a missing or
// corrupt session file does.
export const run = async (
command: Promise<number>,
stdout: Writable,
stderr: Writable,
log: Log,
exit: (code: number) => void,
): Promise<void> => {
let code: number;
try {
code = await command;
} catch (err) {
if (err instanceof ApiError && err.status === 401) {
log(
"fatal",
`quak: the saved session is no longer valid; run "quak login"`,
);
code = 3;
} else {
log("fatal", `quak: ${describeError(err)}`);
code = 1;
}
}
const pending = [stdout, stderr].filter((s) => s.writableLength > 0);
if (pending.length === 0) {
exit(code);
return;
}
let remaining = pending.length;
for (const s of pending) {
s.once("drain", () => {
if (--remaining === 0) exit(code);
});
}
};
src/cli-session.ts
+26
View File
@@ -0,0 +1,26 @@
// How the CLI reads its saved session file back into a `Client`.
//
// A missing file means "not logged in" and returns null. A file that exists but
// cannot be read back into a client (bad JSON, a missing field, a key of the
// wrong length) throws an error saying the session file is corrupt, so the CLI
// can tell the user which of the two it is. Needs `init()` first.
import { existsSync, readFileSync } from "node:fs";
import type { ApiClientOptions } from "./api/client.js";
import { Client } from "./client.js";
export const loadSession = (
path: string,
apiOptions?: ApiClientOptions,
): Client | null => {
if (!existsSync(path)) return null;
try {
return Client.fromJSON(
JSON.parse(readFileSync(path, "utf-8")),
apiOptions,
);
} catch (err) {
const reason = err instanceof Error ? err.message : String(err);
throw new Error(`Session file ${path} is corrupt: ${reason}`);
}
};
src/client.ts
+233 -43
View File
@@ -7,11 +7,16 @@ import {
} from "./auth/login.js";
import { unwrapAuth } from "./auth/unwrap.js";
import { init, fromBase64, toBase64 } from "./crypto/index.js";
import { fetchMLDataBatch, type MLData } from "./mldata-fetch.js";
import { decryptCollection, decryptFile } from "./model/index.js";
import {
downloadFile as dlFile,
downloadThumbnail as dlThumb,
} from "./download/index.js";
import {
makeDownloadContentSource,
type ContentSource,
} from "./library/content.js";
import type {
Collection,
EnteFile,
@@ -37,6 +42,30 @@ export interface ClientSnapshot {
publicKey: string;
}
// The result of a resumable enumeration. Live decrypted records and deleted
// ids are kept apart on purpose: a tombstone carries no key or metadata to
// decrypt, so it is a bare id rather than a hollowed-out record. A live row
// that cannot be decrypted, or whose metadata cannot be parsed, is in
// `undecryptable` with the reason instead, so one bad row does not fail the
// call; absent means there was none. `cursor` is the max `updationTime` seen,
// those rows included, to pass back into the next call.
export interface CollectionsPage {
collections: Collection[];
deleted: number[];
undecryptable?: { id: number; error: string }[];
cursor: number;
}
export interface FilesPage {
files: EnteFile[];
deleted: number[];
undecryptable?: { id: number; error: string }[];
cursor: number;
}
const errorMessage = (err: unknown): string =>
err instanceof Error ? err.message : String(err);
export class Client {
private readonly api: ApiClient;
private readonly email: string;
@@ -104,18 +133,57 @@ export class Client {
);
}
static fromJSON(
snapshot: ClientSnapshot,
apiOptions?: ApiClientOptions,
): Client {
const api = new ApiClient({ ...apiOptions, authToken: snapshot.token });
// Restore a client from a `toJSON()` snapshot. The snapshot usually comes
// straight from `JSON.parse` of a file on disk, so every field is checked
// before use; a bad one throws an error naming it. Needs `init()` first.
static fromJSON(snapshot: unknown, apiOptions?: ApiClientOptions): Client {
const invalid = (field: string, problem: string): Error =>
new Error(`Invalid session data: ${field} ${problem}`);
if (typeof snapshot !== "object" || snapshot === null) {
throw new Error("Invalid session data: not a JSON object");
}
const s = snapshot as Record<string, unknown>;
for (const field of ["email", "token"]) {
if (typeof s[field] !== "string" || s[field] === "") {
throw invalid(field, "must be a non-empty string");
}
}
if (!Number.isInteger(s.userID)) {
throw invalid("userID", "must be an integer");
}
const key = (field: string): Uint8Array => {
const value = s[field];
if (typeof value !== "string") {
throw invalid(field, "must be a base64 string");
}
let bytes: Uint8Array;
try {
bytes = fromBase64(value);
} catch {
throw invalid(field, "is not valid base64");
}
// The master key (secretbox) and the key pair (box) are all 32 bytes.
if (bytes.length !== 32) {
throw invalid(
field,
`must decode to 32 bytes, got ${bytes.length}`,
);
}
return bytes;
};
const api = new ApiClient({
...apiOptions,
authToken: s.token as string,
});
return new Client(
api,
snapshot.email,
snapshot.userID,
fromBase64(snapshot.masterKey),
fromBase64(snapshot.secretKey),
fromBase64(snapshot.publicKey),
s.email as string,
s.userID as number,
key("masterKey"),
key("secretKey"),
key("publicKey"),
);
}
@@ -124,6 +192,14 @@ export class Client {
return this.api;
}
// The content-cache byte source over this client's API: each fetch is the
// download layer's request + streaming decrypt + atomic write. `Library`
// calls this to enable the on-disk content cache.
contentSource(): ContentSource {
this.assertLoggedIn();
return makeDownloadContentSource(this.api);
}
private assertLoggedIn(): void {
if (this.loggedOut) throw new Error("Client has been logged out");
}
@@ -135,67 +211,181 @@ export class Client {
toJSON(): ClientSnapshot {
this.assertLoggedIn();
const token = this.api.getAuthToken();
if (!token) {
throw new Error("Cannot serialize client: it has no auth token");
}
return {
email: this.email,
userID: this.userID,
token: this.api["token"]!,
token,
masterKey: toBase64(this.masterKey),
secretKey: toBase64(this.secretKey),
publicKey: toBase64(this.publicKey),
};
}
// Ends this client's session on the server (`POST /users/logout`), so the
// token stops working everywhere, including in any saved copy of it. This
// client is left as it was; call `logout()` to clear it.
async logoutOnServer(): Promise<void> {
this.assertLoggedIn();
await this.api.postJSON("/users/logout", {});
}
// Zeroes the key buffers in place, so any copy of the reference held
// elsewhere is wiped too. Every method checks `assertLoggedIn` before
// touching the keys, so nothing decrypts with the zeroed keys.
logout(): void {
this.loggedOut = true;
this.api.clearAuthToken();
this.masterKey.fill(0);
this.secretKey.fill(0);
this.publicKey.fill(0);
}
async listCollections(): Promise<Collection[]> {
// Enumerate collections changed since `sinceTime`. Live collections are
// decrypted; tombstoned ones (isDeleted) are surfaced as bare ids, and
// ones that do not decrypt as ids with the reason. The returned cursor is
// the max `updationTime` seen — including tombstones, so the next sync
// resumes past them — falling back to `sinceTime` when the response is
// empty. `/collections/v2` returns the whole changed set in one response,
// so there is no pagination here.
async collectionsSince(args: {
sinceTime: number;
}): Promise<CollectionsPage> {
this.assertLoggedIn();
const { collections } = await this.api.getJSON<{
const { collections: raws } = await this.api.getJSON<{
collections: RawCollection[];
}>("/collections/v2", { sinceTime: 0 });
// The sync API keeps returning deleted collections as tombstones
// (isDeleted: true); their diff endpoint 404s, so drop them.
return collections
.filter((raw) => !raw.isDeleted)
.map((raw) =>
decryptCollection(
raw,
{
masterKey: this.masterKey,
publicKey: this.publicKey,
secretKey: this.secretKey,
},
this.userID,
),
);
}>("/collections/v2", { sinceTime: args.sinceTime });
// logout() may have zeroed the keys while the request was in flight.
this.assertLoggedIn();
const collections: Collection[] = [];
const deleted: number[] = [];
const undecryptable: { id: number; error: string }[] = [];
let cursor = args.sinceTime;
for (const raw of raws) {
if (raw.isDeleted) {
deleted.push(raw.id);
} else {
try {
collections.push(
decryptCollection(
raw,
{
masterKey: this.masterKey,
publicKey: this.publicKey,
secretKey: this.secretKey,
},
this.userID,
),
);
} catch (err) {
undecryptable.push({
id: raw.id,
error: errorMessage(err),
});
}
}
if (raw.updationTime > cursor) cursor = raw.updationTime;
}
return { collections, deleted, undecryptable, cursor };
}
async listFiles(
collectionID: number,
collectionKey: Uint8Array,
): Promise<EnteFile[]> {
// Enumerate a collection's files changed since `sinceTime`, paginating the
// diff from that cursor. Live rows are decrypted; tombstoned ones are
// surfaced as bare ids, and ones that do not decrypt or parse as ids with
// the reason. Returns the final cursor to resume from.
async filesSince(args: {
collectionID: number;
collectionKey: Uint8Array;
sinceTime: number;
}): Promise<FilesPage> {
this.assertLoggedIn();
const allFiles: EnteFile[] = [];
let sinceTime = 0;
const { collectionID, collectionKey } = args;
const files: EnteFile[] = [];
const deleted: number[] = [];
const undecryptable: { id: number; error: string }[] = [];
let cursor = args.sinceTime;
for (;;) {
const { diff, hasMore } = await this.api.getJSON<{
diff: RawEnteFile[];
hasMore: boolean;
}>("/collections/v2/diff", { collectionID, sinceTime });
}>("/collections/v2/diff", { collectionID, sinceTime: cursor });
let pageMax = cursor;
for (const raw of diff) {
if (!raw.isDeleted) {
allFiles.push(decryptFile(raw, collectionKey));
}
if (raw.updationTime > sinceTime) {
sinceTime = raw.updationTime;
if (raw.isDeleted) {
deleted.push(raw.id);
} else {
try {
files.push(decryptFile(raw, collectionKey));
} catch (err) {
undecryptable.push({
id: raw.id,
error: errorMessage(err),
});
}
}
if (raw.updationTime > pageMax) pageMax = raw.updationTime;
}
if (!hasMore) break;
if (!hasMore) {
cursor = pageMax;
break;
}
// The server says there is more, but this page did not advance the
// cursor: following hasMore would refetch the same page forever
// (#7). Stop with a clear error instead of looping.
if (pageMax <= cursor) {
throw new Error(
`/collections/v2/diff for collection ${collectionID} ` +
`returned hasMore with a cursor that did not advance ` +
`(stuck at ${cursor}); refusing to loop`,
);
}
cursor = pageMax;
}
return allFiles;
return { files, deleted, undecryptable, cursor };
}
// Whole-account listing: every live collection that decrypts, deletions
// hidden. A thin wrapper over `collectionsSince` from the beginning of
// time.
async listCollections(): Promise<Collection[]> {
const { collections } = await this.collectionsSince({ sinceTime: 0 });
return collections;
}
// Every live file in a collection that decrypts, deletions hidden. A thin
// wrapper over `filesSince` from the beginning of time.
async listFiles(
collectionID: number,
collectionKey: Uint8Array,
): Promise<EnteFile[]> {
const { files } = await this.filesSince({
collectionID,
collectionKey,
sinceTime: 0,
});
return files;
}
// Fetch machine-learning data (face detections + CLIP embeddings) for up
// to a batch of files, each decrypted with its own key, and for each file
// whose ML data did not decrypt, gunzip or parse, why not. One request;
// the library batches at `MLDATA_BATCH_SIZE` and schedules each batch
// through its metadata request pool.
async fetchMLData(args: {
fileIDs: number[];
fileKeys: Map<number, Uint8Array>;
}): Promise<{
payloads: Map<number, MLData>;
errors: Map<number, unknown>;
}> {
this.assertLoggedIn();
return fetchMLDataBatch(this.api, args.fileIDs, args.fileKeys);
}
async downloadFile(
src/clip-keyword-vectors.ts
+1836
View File
File diff suppressed because it is too large Load Diff
src/clip-keywords.ts
+57
View File
@@ -0,0 +1,57 @@
// Keywords for a file from its CLIP embedding.
//
// Ente stores no keywords. A file's ML data holds a CLIP embedding, the 512
// numbers Ente's image model gave the photo, and Ente's photos app sorts files
// into the categories of its Discover section with it: a file falls in a
// category when the cosine similarity of its CLIP embedding and the text
// embedding of the category's query is at least the category's `minScore`,
// unless the file is a video and the category leaves out videos. A file's
// keywords are the titles of the categories it falls in. The categories and
// their text embeddings are in `clip-keyword-vectors.ts`.
import { DISCOVER_CATEGORIES } from "./clip-keyword-vectors.js";
import { clipEmbedding } from "./library/mldata.js";
import type { MLData } from "./mldata-fetch.js";
// How many numbers a CLIP embedding from Ente's image model has.
const EMBEDDING_LENGTH = 512;
const norm = (vector: number[]): number => Math.hypot(...vector);
const cosineSimilarity = (a: number[], b: number[]): number => {
let dot = 0;
a.forEach((value, i) => {
dot += value * (b[i] ?? 0);
});
return dot / (norm(a) * norm(b));
};
// The titles of the Discover categories a file falls in, highest score first,
// or undefined when its ML data has no CLIP embedding of 512 finite numbers
// that are not all zero. Tests pass their own `categories`.
export const clipKeywords = (
mlData: MLData,
isVideo: boolean,
categories = DISCOVER_CATEGORIES,
): string[] | undefined => {
const embedding = clipEmbedding(mlData);
if (
!embedding ||
embedding.length !== EMBEDDING_LENGTH ||
norm(embedding) === 0
) {
return undefined;
}
const matches: { title: string; score: number }[] = [];
for (const category of categories) {
if (isVideo && !category.showVideo) continue;
const score = cosineSimilarity(embedding, category.textEmbedding);
if (score >= category.minScore) {
matches.push({ title: category.title, score });
}
}
return matches
.sort((a, b) => b.score - a.score)
.map((match) => match.title);
};
src/crypto/hash.ts
+22
View File
@@ -0,0 +1,22 @@
import sodium, { type StateAddress } from "libsodium-wrappers-sumo";
import { toBase64 } from "./encoding.js";
// The content hash an uploading client records in a file's metadata: unkeyed
// BLAKE2b with a 64-byte output over the original's bytes, fed in chunks, as
// standard base64 with padding. Named after the upstream client's functions.
// The output length is read at call time for the same reason as
// `streamTagFinal` in stream.ts: libsodium sets its constants only once ready.
export const chunkHashInit = (): StateAddress =>
sodium.crypto_generichash_init(null, sodium.crypto_generichash_BYTES_MAX);
export const chunkHashUpdate = (state: StateAddress, chunk: Uint8Array): void =>
sodium.crypto_generichash_update(state, chunk);
export const chunkHashFinal = (state: StateAddress): string =>
toBase64(
sodium.crypto_generichash_final(
state,
sodium.crypto_generichash_BYTES_MAX,
),
);
src/crypto/index.ts
+1
View File
@@ -7,6 +7,7 @@ export {
} from "./encoding.js";
export { deriveKEK, deriveLoginSubkey } from "./kdf.js";
export { decryptBox, decryptSealed } from "./box.js";
export { chunkHashFinal, chunkHashInit, chunkHashUpdate } from "./hash.js";
export {
decryptBlob,
encryptBlob,
src/download/index.ts
+529 -88
View File
@@ -1,7 +1,13 @@
import { randomUUID } from "node:crypto";
import { rename, rm, writeFile } from "node:fs/promises";
import { randomBytes } from "node:crypto";
import { readdirSync, rmSync } from "node:fs";
import { open, rename, rm } from "node:fs/promises";
import type { FileHandle } from "node:fs/promises";
import { dirname, join } from "node:path";
import { Unzip, UnzipInflate } from "fflate";
import {
chunkHashFinal,
chunkHashInit,
chunkHashUpdate,
fromBase64,
initStreamPull,
pullStreamChunk,
@@ -10,82 +16,151 @@ import {
streamTagFinal,
} from "../crypto/index.js";
import { TruncatedStreamError } from "../errors.js";
import { defaultFileName, safeExtension, withExtension } from "../filename.js";
import { withRetry } from "../retry.js";
import type { ApiClient } from "../api/client.js";
import type { EnteFile } from "../model/types.js";
export interface DownloadResult {
// Where the file was written. A live photo is written as two files, its
// image here and its video at `videoPath` (see `decryptLivePhoto`).
path: string;
// The decrypted length; for a live photo, that of the ZIP it arrives as.
bytesWritten: number;
videoPath?: string;
}
// Fired as decrypted plaintext accumulates, with the running total of
// plaintext bytes recovered so far. Within one download it is non-decreasing
// and its last value equals the final `bytesWritten`. A retry restarts the
// file from byte zero (see `fetchAndDecrypt`), so a fresh attempt begins its
// own count from zero.
export type ProgressCallback = (bytesDone: number) => void;
const ENC_CHUNK_SIZE = STREAM_CHUNK_SIZE + STREAM_CHUNK_OVERHEAD;
// Decrypt a secretstream body, handing each plaintext chunk to `sink` as it is
// produced rather than accumulating the whole file. Peak memory is one
// ciphertext chunk of network buffer plus one plaintext chunk — bounded by
// `STREAM_CHUNK_SIZE` regardless of the file's size — so a multi-gigabyte video
// no longer needs its size again in RAM. Returns the total plaintext length.
//
// The truncation contract is exactly the buffered version's, only the sink is
// new: a body cut short still decrypts and authenticates up to its last whole
// chunk, so the absence of TAG_FINAL is the sole evidence it was cut short, and
// this throws rather than let a caller keep a short file. The sink has already
// seen those chunks by then; the callers (`decryptToTemp`, `decryptLivePhoto`)
// stage them in temp files that are renamed into place only on a clean return,
// so a throw leaves nothing on disk.
const streamDecrypt = async (
stream: ReadableStream<Uint8Array>,
header: Uint8Array,
key: Uint8Array,
): Promise<Uint8Array> => {
sink: (plaintext: Uint8Array) => Promise<void>,
onProgress?: ProgressCallback,
): Promise<number> => {
const state = initStreamPull(header, key);
const reader = stream.getReader();
let buffer = new Uint8Array(0);
const plainChunks: Uint8Array[] = [];
// Incoming reads are held as-is and only stitched into a contiguous chunk
// at each `ENC_CHUNK_SIZE` boundary, so every received byte is copied once.
// Concatenating on each read instead — reallocating the whole accumulator
// per read — is O(n^2) in the bytes buffered, and for a 4 MiB chunk that
// memory churn dwarfs the libsodium decryption itself.
const pending: Uint8Array[] = [];
let pendingBytes = 0;
let totalPlain = 0;
let chunksPulled = 0;
let lastTag = -1;
for (;;) {
const { done, value } = await reader.read();
if (value) {
const merged = new Uint8Array(buffer.length + value.length);
merged.set(buffer);
merged.set(value, buffer.length);
buffer = merged;
}
while (buffer.length >= ENC_CHUNK_SIZE) {
const encChunk = buffer.slice(0, ENC_CHUNK_SIZE);
buffer = buffer.slice(ENC_CHUNK_SIZE);
const { plaintext, tag } = pullStreamChunk(state, encChunk);
plainChunks.push(plaintext);
totalPlain += plaintext.length;
chunksPulled++;
lastTag = tag;
}
if (done) {
if (buffer.length > 0) {
// Whatever is left over once every whole chunk has been
// consumed must be the stream's final chunk, and a final
// chunk that actually arrived in full authenticates. If it
// does not, the body stopped part-way through a chunk — the
// ordinary shape of a dropped connection. Poly1305 cannot
// tell a partial chunk from a corrupt one, so this is
// reported as the truncation it almost always is, with the
// authentication failure kept as the error's cause.
let pulled;
try {
pulled = pullStreamChunk(state, buffer);
} catch (err) {
throw new TruncatedStreamError(
`download: stream truncated: response body ended with ${buffer.length} trailing bytes that did not authenticate as a final chunk (transfer stopped mid-chunk, or the data is corrupt)`,
{ cause: err },
);
}
plainChunks.push(pulled.plaintext);
totalPlain += pulled.plaintext.length;
chunksPulled++;
lastTag = pulled.tag;
// Remove the first `size` bytes from `pending` as one contiguous buffer.
// A read that straddles the boundary is split with `subarray` (a view, no
// copy); its tail stays queued for the next chunk. `size` never exceeds
// `pendingBytes`, so the queue always holds enough.
const takeContiguous = (size: number): Uint8Array => {
const out = new Uint8Array(size);
let offset = 0;
while (offset < size) {
const piece = pending[0]!;
const need = size - offset;
if (piece.length <= need) {
out.set(piece, offset);
offset += piece.length;
pending.shift();
} else {
out.set(piece.subarray(0, need), offset);
pending[0] = piece.subarray(need);
offset += need;
}
break;
}
pendingBytes -= size;
return out;
};
const consume = async (
plaintext: Uint8Array,
tag: number,
): Promise<void> => {
await sink(plaintext);
totalPlain += plaintext.length;
chunksPulled++;
lastTag = tag;
onProgress?.(totalPlain);
};
try {
for (;;) {
const { done, value } = await reader.read();
if (value && value.length > 0) {
pending.push(value);
pendingBytes += value.length;
}
while (pendingBytes >= ENC_CHUNK_SIZE) {
const encChunk = takeContiguous(ENC_CHUNK_SIZE);
// A whole chunk that fails to authenticate while the stream
// carries on is corruption, not truncation; that error
// propagates unchanged.
const { plaintext, tag } = pullStreamChunk(state, encChunk);
await consume(plaintext, tag);
}
if (done) {
if (pendingBytes > 0) {
const buffer = takeContiguous(pendingBytes);
// Whatever is left over once every whole chunk has been
// consumed must be the stream's final chunk, and a final
// chunk that actually arrived in full authenticates. If
// it does not, the body stopped part-way through a chunk
// — the ordinary shape of a dropped connection. Poly1305
// cannot tell a partial chunk from a corrupt one, so this
// is reported as the truncation it almost always is, with
// the authentication failure kept as the error's cause.
// The message leaves that failure to the cause, which
// `describeError` appends to what `quak backup` prints and
// records. Only the pull is guarded: a sink failure on a
// chunk that did authenticate is a disk error, not a
// truncation.
let pulled;
try {
pulled = pullStreamChunk(state, buffer);
} catch (err) {
throw new TruncatedStreamError(
`download: stream truncated: response body ended with ${buffer.length} trailing bytes (transfer stopped mid-chunk, or the data is corrupt)`,
{ cause: err },
);
}
await consume(pulled.plaintext, pulled.tag);
}
break;
}
}
} finally {
reader.releaseLock();
}
// Only the last chunk of a secretstream carries TAG_FINAL. Everything a
// dropped connection did deliver still decrypts and authenticates, so the
// absence of TAG_FINAL is the only evidence that the body was cut short.
// Returning a short plaintext here would put a corrupt file on disk that
// later backup runs would treat as complete.
if (chunksPulled === 0) {
throw new TruncatedStreamError(
"download: stream truncated: response body contained no secretstream chunks",
@@ -97,31 +172,106 @@ const streamDecrypt = async (
`download: stream truncated: last chunk tag ${lastTag}, expected TAG_FINAL (${tagFinal})`,
);
}
const result = new Uint8Array(totalPlain);
let offset = 0;
for (const chunk of plainChunks) {
result.set(chunk, offset);
offset += chunk.length;
}
return result;
return totalPlain;
};
// Write `plaintext` to `destination` atomically: stage it in a temporary
// sibling file (same directory, so the rename cannot cross a filesystem
// boundary) and rename it into place. Callers therefore never observe a
// partially written destination, and a pre-existing file at that path is
// replaced only once the new contents are complete on disk.
const writeAtomic = async (
destination: string,
plaintext: Uint8Array,
): Promise<void> => {
// The random suffix keeps concurrent downloads of the same destination
// from stepping on each other's temporary file.
const tmpPath = join(dirname(destination), `.quak-${randomUUID()}.tmp`);
// Fsync a file or a directory, so its contents (for a directory, its entries)
// are on stable storage. Exported for the backup tree's copy, which needs the
// same durability as the writer below.
export const fsyncPath = async (path: string): Promise<void> => {
const handle = await open(path, "r");
try {
await writeFile(tmpPath, plaintext);
await handle.sync();
} finally {
await handle.close();
}
};
// A process-ID check: signal 0 delivers nothing and only reports whether the
// process exists. EPERM means it exists but belongs to another user.
const isRunning = (pid: number): boolean => {
try {
process.kill(pid, 0);
return true;
} catch (err) {
return (err as NodeJS.ErrnoException).code === "EPERM";
}
};
// Delete the temp files a killed process left in `dir`: the writer's
// `.quak-<pid>-<random>.tmp`, the backup copy's
// `.quak-backup-<name>-<pid>-<random>.tmp`, and the image and video of a live
// photo downloaded to replace a stored one, `.quak-<pid>-<random>.<ext>` (see
// `replaceOriginal`). Only files whose process is no longer running are
// removed, so another process writing into the same directory keeps its own. A
// reused process ID can only keep a leftover a while longer, never remove a
// live one.
export const removeLeftoverTempFiles = (dir: string): void => {
let names: string[];
try {
names = readdirSync(dir);
} catch {
return;
}
for (const name of names) {
const match = /^\.quak-(?:.*-)?(\d+)-[0-9a-z]*\.[0-9A-Za-z]+$/.exec(
name,
);
if (match && !isRunning(Number(match[1]))) {
rmSync(join(dir, name), { force: true });
}
}
};
// A new temp file name in `dir`. The random suffix keeps concurrent downloads
// of the same destination from stepping on each other's temporary file; the
// process ID lets `removeLeftoverTempFiles` tell a leftover from a write in
// progress.
const tempPathIn = (dir: string): string =>
join(dir, `.quak-${process.pid}-${randomBytes(16).toString("hex")}.tmp`);
// Stage a write to `destination` atomically and durably, then rename it into
// place. `fill` writes the contents into the open temp file handle — either the
// whole buffer at once (`writeAtomic`) or chunk by chunk as they decrypt
// (`decryptToTemp`). The temp file is a sibling of the destination (same
// directory, so the rename cannot cross a filesystem boundary), so callers
// never observe a partially written destination, and a pre-existing file is
// replaced only once the new contents are complete on disk.
//
// Durability against a power cut needs two fsyncs. Without them the write can
// return while the data or the rename is still only in the kernel's page
// cache, and a crash then resurrects an empty renamed file — exactly the
// corruption a later backup run treats as a complete download. So the temp
// file's contents are fsynced before the rename, and the containing directory
// is fsynced after it, so both the bytes and the new directory entry are on
// stable storage before this returns.
//
// On any failure — including a `fill` that throws because the stream was
// truncated — the temp file is removed, so the destination is untouched and no
// scratch file is left to fill the disk on repeated failures.
const stageAtomic = async (
destination: string,
fill: (handle: FileHandle) => Promise<void>,
): Promise<void> => {
const dir = dirname(destination);
const tmpPath = tempPathIn(dir);
try {
const handle = await open(tmpPath, "w");
try {
await fill(handle);
await handle.sync();
} finally {
await handle.close();
}
// `rename` replaces the destination's directory entry rather than
// writing through it: an existing symlink at `destination` is
// replaced, not followed, and the new file has the temp file's
// permissions, not those of the file it replaced.
await rename(tmpPath, destination);
// Fsync the directory so the rename itself survives a crash: renaming
// over a synced temp file still leaves the new directory entry in the
// page cache until the directory is synced.
await fsyncPath(dir);
} catch (err) {
// Best-effort cleanup. A failure to remove the temporary file must
// never replace the error that actually explains what went wrong.
@@ -130,7 +280,247 @@ const writeAtomic = async (
}
};
// Fetch a stream and decrypt it, retrying the whole sequence.
// Write `plaintext` to `destination` atomically and durably. Exported so the
// metadata store can reuse the same durable write for small whole-buffer
// payloads; originals go through `decryptToTemp` instead so they never buffer.
export const writeAtomic = async (
destination: string,
plaintext: Uint8Array,
): Promise<void> =>
stageAtomic(destination, (handle) => handle.writeFile(plaintext));
// Refuse an original whose bytes do not hash to what its uploader recorded.
// The error is not retried.
const checkHash = (file: EnteFile, actual: string): void => {
if (actual !== file.metadata.hash) {
throw new Error(
`download: file ${file.id}: content hash ${actual} does not match the hash its uploader recorded, ${file.metadata.hash}`,
);
}
};
// Decrypt `stream` straight to `destination`, one plaintext chunk at a time,
// under the atomic writer's temp-then-rename discipline. Memory stays bounded
// by the chunk size: each decrypted chunk is written to the temp file and
// dropped. The rename happens only after the stream authenticates as terminated
// on TAG_FINAL; a truncated stream throws and leaves the destination untouched.
// Returns the plaintext length written.
//
// `original` is the file whose original this is (none for a thumbnail, which
// has no recorded hash). When its metadata has a hash, the decrypted bytes are
// hashed as they stream and must match it, or nothing is stored.
const decryptToTemp = async (
destination: string,
stream: ReadableStream<Uint8Array>,
header: Uint8Array,
key: Uint8Array,
onProgress?: ProgressCallback,
original?: EnteFile,
): Promise<number> => {
const hash =
original?.metadata.hash === undefined ? undefined : chunkHashInit();
let bytesWritten = 0;
await stageAtomic(destination, async (handle) => {
bytesWritten = await streamDecrypt(
stream,
header,
key,
async (plaintext) => {
if (hash !== undefined) chunkHashUpdate(hash, plaintext);
await handle.write(plaintext);
},
onProgress,
);
if (original !== undefined && hash !== undefined) {
checkHash(original, chunkHashFinal(hash));
}
});
return bytesWritten;
};
// One of the two parts of a live photo being unpacked: the ZIP entry whose
// name starts with `kind`, written to its own temp file.
interface LivePhotoPart {
kind: "image" | "video";
tmpPath: string;
handle: FileHandle;
hash: ReturnType<typeof chunkHashInit>;
// Decompressed bytes not yet written.
pending: Uint8Array[];
// The entry's extension, set once all of the entry has been read.
ext?: string;
}
const openPart = async (
kind: "image" | "video",
dir: string,
): Promise<LivePhotoPart> => {
const tmpPath = tempPathIn(dir);
const handle = await open(tmpPath, "w");
return { kind, tmpPath, handle, hash: chunkHashInit(), pending: [] };
};
// A live photo arrives as a ZIP of its image and its video. Ente's clients
// name the entries `image.<ext>` and `video.<ext>`. This takes the entry whose
// name starts with `image` as the image and the one whose name starts with
// `video` as the video, and refuses a ZIP holding a second of either. It is
// written unpacked: each part is named `destination` with the extension
// replaced by its own entry's, and the two must differ ignoring case. When
// `defaultName` is set, `destination` is the name `defaultFileName` made from
// the title rather than a path the caller gave, and each part is named as
// `defaultFileName` names it with its own extension, so that the name is cut
// short to fit with that extension. When the file records a hash,
// `<imageHash>:<videoHash>` must match it, each over that part's own bytes.
// Only then are the image, then the video, renamed into place; on any failure
// neither is stored.
//
// The ZIP is chosen by its uploader and may expand enormously, so each part is
// written as it decompresses and never held, and the ZIP is refused once the
// two together come to more than 20 times its size plus 16 MiB, the limit
// Ente's mobile client and CLI set. The ZIP's size is known only at its end,
// so until then the limit is taken over the part of it decrypted so far.
// fflate's `Unzip` inflates each push in one piece before `push` returns, and
// deflate expands at most about 1000-fold, so the ZIP is pushed in 4 KiB
// slices, keeping each decompressed piece near 4 MiB, one plaintext chunk, and
// each piece is checked and written before the next slice is pushed. Every
// entry is started, even one that is not kept, because fflate keeps an
// unstarted entry's data in memory.
const decryptLivePhoto = async (
destination: string,
stream: ReadableStream<Uint8Array>,
header: Uint8Array,
key: Uint8Array,
onProgress: ProgressCallback | undefined,
file: EnteFile,
defaultName: boolean,
): Promise<DownloadResult> => {
const sliceSize = 4096;
const dir = dirname(destination);
const fail = (message: string, cause?: unknown): Error =>
new Error(`download: file ${file.id}: ${message}`, { cause });
const parts: LivePhotoPart[] = [];
try {
const image = await openPart("image", dir);
parts.push(image);
const video = await openPart("video", dir);
parts.push(video);
const claimed = new Set<LivePhotoPart>();
// Set to a part the ZIP holds a second entry for; the ZIP is then
// refused.
let repeated: LivePhotoPart | undefined;
const unzip = new Unzip((entry) => {
const part = parts.find((p) => entry.name.startsWith(p.kind));
if (part !== undefined) {
if (claimed.has(part)) repeated = part;
claimed.add(part);
}
entry.ondata = (err, data, final) => {
if (err) throw err;
if (part === undefined) return;
chunkHashUpdate(part.hash, data);
part.pending.push(data);
if (final) part.ext = safeExtension(entry.name);
};
entry.start();
});
unzip.register(UnzipInflate);
// The bytes of the ZIP pushed so far, and of the two parts they have
// decompressed to.
let zipBytes = 0;
let expanded = 0;
const push = async (
data: Uint8Array,
final: boolean,
): Promise<void> => {
zipBytes += data.length;
// fflate reports a bad ZIP by throwing, sometimes a TypeError,
// which the retry would take for a network failure. A bad ZIP is
// never retried, and nor are the refusals below.
try {
unzip.push(data, final);
} catch (err) {
throw fail("live photo is not a readable ZIP", err);
}
if (repeated !== undefined) {
throw fail(
`live photo ZIP holds more than one ${repeated.kind}`,
);
}
for (const part of parts) {
for (const piece of part.pending) expanded += piece.length;
}
if (expanded > 20 * zipBytes + 16 * 1024 * 1024) {
throw fail(
"live photo ZIP expands to more than 20 times its size plus 16 MiB",
);
}
for (const part of parts) {
for (const piece of part.pending)
await part.handle.write(piece);
part.pending = [];
}
};
const bytesWritten = await streamDecrypt(
stream,
header,
key,
async (plaintext) => {
for (let i = 0; i < plaintext.length; i += sliceSize) {
await push(plaintext.subarray(i, i + sliceSize), false);
}
},
onProgress,
);
await push(new Uint8Array(0), true);
if (image.ext === undefined || video.ext === undefined) {
throw fail(
"live photo ZIP does not hold both an image and a video",
);
}
if (file.metadata.hash !== undefined) {
checkHash(
file,
`${chunkHashFinal(image.hash)}:${chunkHashFinal(video.hash)}`,
);
}
if (image.ext.toLowerCase() === video.ext.toLowerCase()) {
throw fail(
`live photo's image and video have the same extension, ${video.ext}`,
);
}
const partPath = (ext: string): string =>
defaultName
? defaultFileName(file.metadata.title, file.id, { ext })
: withExtension(destination, ext);
const path = partPath(image.ext);
const videoPath = partPath(video.ext);
for (const part of parts) {
await part.handle.sync();
await part.handle.close();
}
await rename(image.tmpPath, path);
try {
await rename(video.tmpPath, videoPath);
} catch (err) {
await rm(path, { force: true }).catch(() => undefined);
throw err;
}
await fsyncPath(dir);
return { path, bytesWritten, videoPath };
} catch (err) {
// Best-effort cleanup, as in `stageAtomic`.
for (const part of parts) {
await part.handle.close().catch(() => undefined);
await rm(part.tmpPath, { force: true }).catch(() => undefined);
}
throw err;
}
};
// Fetch a stream and decrypt it to `destination`, retrying the whole sequence.
//
// The request is only the first third of a download. `getXStream` returns as
// soon as headers arrive, and the bytes are pulled here, so a socket reset
@@ -143,53 +533,104 @@ const writeAtomic = async (
// four attempts would mean sixteen requests for one file. The policy comes
// from the client so a caller that configured one gets it here too.
//
// A retry starts the file over from byte zero: the secretstream pull state is
// not resumable and there is no Range support on these endpoints.
// Because the plaintext is streamed to disk rather than buffered, the atomic
// write is part of the retried unit. A retry starts the file over from byte
// zero — the secretstream pull state is not resumable and there is no Range
// support — staging into a fresh temp file each time: a failed attempt writes
// and then removes its own temp file, and only the attempt that reaches
// TAG_FINAL renames one into place, so a download that needed three tries still
// performs exactly one rename over the destination.
//
// `defaultName` says `destination` is the name `defaultFileName` made, not a
// path the caller gave (see `decryptLivePhoto`).
const fetchAndDecrypt = async (
api: ApiClient,
openStream: () => Promise<ReadableStream<Uint8Array>>,
header: Uint8Array,
key: Uint8Array,
): Promise<Uint8Array> =>
destination: string,
onProgress?: ProgressCallback,
original?: EnteFile,
defaultName = false,
): Promise<DownloadResult> =>
withRetry(async () => {
const stream = await openStream();
return streamDecrypt(stream, header, key);
try {
if (original?.metadata.fileType === "livePhoto") {
return await decryptLivePhoto(
destination,
stream,
header,
key,
onProgress,
original,
defaultName,
);
}
const bytesWritten = await decryptToTemp(
destination,
stream,
header,
key,
onProgress,
original,
);
return { path: destination, bytesWritten };
} catch (err) {
// Cancel the body so its connection is closed now rather than held
// until the stream is garbage collected. A backup run carries on
// past a failed file, so without this every failure would hold a
// socket. This covers every failure, including a temp file that
// cannot be opened and a header that is rejected before the body
// is read.
await stream.cancel(err).catch(() => undefined);
throw err;
}
}, api.getRetryOptions());
// Write `file`'s original to `outPath`. A live photo is written as its image
// and its video beside `outPath` instead (see `decryptLivePhoto`).
export const downloadFile = async (
api: ApiClient,
file: EnteFile,
outPath?: string,
onProgress?: ProgressCallback,
): Promise<DownloadResult> => {
const resolvedPath = outPath ?? file.metadata.title;
// `outPath` is the caller's and is used as is. Without it, the file is
// written in the current directory under the name `defaultFileName` makes
// from the title, which is the server's.
const resolvedPath =
outPath ?? defaultFileName(file.metadata.title, file.id);
const header = fromBase64(file.file.decryptionHeader);
const plaintext = await fetchAndDecrypt(
return fetchAndDecrypt(
api,
() => api.getFileStream(file.id, { retry: false }),
header,
file.key,
resolvedPath,
onProgress,
file,
outPath === undefined,
);
// Outside the retry, deliberately: only the attempt that produced a
// complete, authenticated plaintext gets to stage a temporary file, so a
// download that needed three tries still performs exactly one write and
// one rename.
await writeAtomic(resolvedPath, plaintext);
return { path: resolvedPath, bytesWritten: plaintext.length };
};
export const downloadThumbnail = async (
api: ApiClient,
file: EnteFile,
outPath?: string,
onProgress?: ProgressCallback,
): Promise<DownloadResult> => {
const resolvedPath = outPath ?? `thumb_${file.metadata.title}`;
// As in `downloadFile`, with `thumb_` before the title.
const resolvedPath =
outPath ??
defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
const header = fromBase64(file.thumbnail.decryptionHeader);
const plaintext = await fetchAndDecrypt(
return fetchAndDecrypt(
api,
() => api.getThumbnailStream(file.id, { retry: false }),
header,
file.key,
resolvedPath,
onProgress,
);
await writeAtomic(resolvedPath, plaintext);
return { path: resolvedPath, bytesWritten: plaintext.length };
};
src/errors.ts
+71 -3
View File
@@ -1,10 +1,13 @@
// Error types that more than one layer of quak needs to recognise.
// Error types that more than one layer of quak needs to recognise, and how an
// error is described in what quak prints and in `failures.json`.
//
// They live here rather than beside the code that throws them so that the
// The types live here rather than beside the code that throws them so that the
// retry classifier can identify them without importing the HTTP client or the
// download layer — both of which import the classifier. `ApiError` is
// re-exported from `src/api/client.ts`, which is where callers have always
// imported it from and where it still belongs conceptually.
// imported it from and where it still belongs conceptually. The walk along an
// error's `cause` chain is here too, because the retry classifier reads the
// chain for errnos and `describeError` for messages.
export class ApiError extends Error {
readonly status: number;
@@ -43,3 +46,68 @@ export class TruncatedStreamError extends Error {
this.name = "TruncatedStreamError";
}
}
// `cause` is an arbitrary user-settable property and nothing prevents it from
// forming a cycle, so the walk is bounded. Hanging the process would be worse
// than misclassifying an error or describing only part of it.
const MAX_CAUSE_DEPTH = 8;
// The error and each error on its `cause` chain, top first, at most
// `MAX_CAUSE_DEPTH` of them. `complete` is false when the walk stopped at the
// depth limit with more of the chain still below it.
export const causeChain = (
err: unknown,
): { chain: object[]; complete: boolean } => {
const chain: object[] = [];
let current: unknown = err;
for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
if (current === null || typeof current !== "object") {
return { chain, complete: true };
}
chain.push(current);
const { cause } = current as { cause?: unknown };
if (cause === current) return { chain, complete: true };
current = cause;
}
return { chain, complete: current === null || typeof current !== "object" };
};
// An error as quak prints it and records it in `failures.json`: its message,
// then the message of each error on its `cause` chain, joined with ": ". Node's
// `fetch` rejects with just "fetch failed" and puts the reason on `cause`, so
// this gives "fetch failed: connect ECONNREFUSED 1.2.3.4:443". When the host
// has several addresses and every one fails, the cause is an `AggregateError`
// with an empty message and one error per address in `errors`, so an
// `AggregateError` is given as the messages of its errors, joined with ", ".
// A cause that is not an object, such as a thrown string, is given as
// `String(cause)`. Each message is trimmed of the white space around it, as
// OpenSSL ends its own with a newline. A message the text already holds is
// left out.
export const describeError = (err: unknown): string => {
if (!(err instanceof Error)) return String(err);
const { chain } = causeChain(err);
const messages = chain
.slice(1)
.map((cause) =>
cause instanceof AggregateError
? cause.errors
.map((e: unknown) =>
(e instanceof Error ? e.message : String(e)).trim(),
)
.join(", ")
: (cause as { message?: unknown }).message,
);
// `causeChain` stops at a cause that is not an object, so that one is read
// from the last error on the chain.
const last = chain[chain.length - 1] as { cause?: unknown };
if (last.cause !== undefined && typeof last.cause !== "object") {
messages.push(String(last.cause));
}
let text = err.message.trim();
for (const message of messages) {
if (typeof message !== "string") continue;
const trimmed = message.trim();
if (!text.includes(trimmed)) text += `: ${trimmed}`;
}
return text;
};
src/exif.ts
+164
View File
@@ -0,0 +1,164 @@
// EXIF in an original's bytes, read with exifreader, which reads it from JPEG,
// HEIC/HEIF, AVIF, PNG, WebP and the other image formats it supports.
// `backup-metadata --exif` records every EXIF tag it finds except the
// thumbnail's. `Photo.exif()` returns every tag, the thumbnail's included, and
// `Photo`'s typed methods return the common fields picked from them here.
import ExifReader, { type ExpandedTags } from "exifreader";
// The EXIF tags in `bytes` (`exif`), the embedded thumbnail's tags
// (`Thumbnail`), and where the EXIF block lies in `bytes` (`metadataRange`).
// Undefined when exifreader cannot read the file at all, such as a video. An
// EXIF block it finds but reads no tag from comes back as an empty `exif`. A
// tag exifreader has no name for is keyed `undefined-<tag number>`. Each tag's
// `computed` holds its value as a string or number, or as an array of them for
// a tag with several values, such as `GPSLatitude`'s `[40, 26, 46]`. A
// fraction with a zero denominator computes to null.
export const readExifTags = (bytes: Uint8Array): ExpandedTags | undefined => {
try {
return ExifReader.loadView(
new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength),
{
expanded: true,
computed: true,
includeOffsets: true,
includeUnknown: true,
includeTags: { exif: true, thumbnail: true },
},
);
} catch {
return undefined;
}
};
// Every EXIF tag of an original, keyed by name, each as exifreader decodes it
// (see `readExifTags`). The embedded thumbnail's own tags are under
// `Thumbnail`, so its `Orientation` or `ImageWidth` cannot hide the main
// image's.
export type ExifTags = Omit<NonNullable<ExpandedTags["exif"]>, "Thumbnail"> & {
Thumbnail?: Omit<
NonNullable<ExpandedTags["Thumbnail"]>,
"type" | "image" | "base64"
>;
};
// Every EXIF tag in `bytes`: `{}` when the file has no EXIF, exifreader cannot
// read its EXIF, or it is not an image exifreader reads.
export const readAllExifTags = (bytes: Uint8Array): ExifTags => {
const tags = readExifTags(bytes);
if (!tags?.Thumbnail) return tags?.exif ?? {};
// exifreader puts the thumbnail's JPEG image beside its tags, as `type`,
// `image` and `base64`. The image is not a tag, so it is left out.
const {
type: _type,
image: _image,
base64: _base64,
...thumbnail
} = tags.Thumbnail;
return { ...tags.exif, Thumbnail: thumbnail };
};
// The common EXIF fields of an original, one for each of `Photo`'s typed
// methods. Each is absent when the file lacks it.
export interface PhotoExif {
make?: string;
model?: string;
lensModel?: string;
// When the photo was taken, by the camera's clock. EXIF writes this as text
// with no time zone, and it is read as if it were UTC: the Date's UTC
// fields are the clock reading, which is the moment it was taken only when
// the clock was set to UTC.
dateTimeOriginal?: Date;
// The camera clock's offset from UTC, such as "+02:00".
offsetTimeOriginal?: string;
// Seconds.
exposureTime?: number;
fNumber?: number;
iso?: number;
// Millimetres.
focalLength?: number;
// The EXIF orientation code, 1 to 8.
orientation?: number;
// Decimal degrees, negative south of the equator and west of Greenwich.
gpsLatitude?: number;
gpsLongitude?: number;
// Metres, negative below sea level.
gpsAltitude?: number;
}
// exifreader gives "<faulty value>" for a tag whose value lies outside the
// file; that tag is left out like one the file lacks.
const asString = (v: unknown): string | undefined =>
typeof v === "string" && v.length > 0 && v !== "<faulty value>"
? v
: undefined;
const asNumber = (v: unknown): number | undefined =>
typeof v === "number" && Number.isFinite(v) ? v : undefined;
// EXIF writes a date and time as "2021:07:15 14:30:00". This is that reading
// in a Date's UTC fields.
const asDate = (v: unknown): Date | undefined => {
const m =
typeof v === "string"
? /^(\d{4}):(\d{2}):(\d{2}) (\d{2}:\d{2}:\d{2})$/.exec(v)
: null;
if (!m) return undefined;
const date = new Date(`${m[1]}-${m[2]}-${m[3]}T${m[4]}Z`);
return Number.isNaN(date.getTime()) ? undefined : date;
};
// GPSLatitude and GPSLongitude hold degrees, minutes and seconds, computed as
// three numbers. This is them in decimal degrees, negative when `ref`, the
// GPSLatitudeRef or GPSLongitudeRef tag, is `negativeRef` ("S" or "W").
// Without that tag the hemisphere is unknown, so it is undefined.
const asDegrees = (
dms: unknown,
ref: unknown,
negativeRef: string,
): number | undefined => {
if (!Array.isArray(dms) || ref === undefined) return undefined;
const [d, m, s] = dms.map(asNumber);
if (d === undefined || m === undefined || s === undefined) return undefined;
const degrees = d + m / 60 + s / 3600;
return ref === negativeRef ? -degrees : degrees;
};
// The common fields picked from an original's EXIF tags, `readAllExifTags`'s
// result: `{}` when there are none.
export const readPhotoExif = (tags: ExifTags): PhotoExif => {
const altitude = asNumber(tags.GPSAltitude?.computed);
const fields: PhotoExif = {
make: asString(tags.Make?.computed),
model: asString(tags.Model?.computed),
lensModel: asString(tags.LensModel?.computed),
dateTimeOriginal: asDate(tags.DateTimeOriginal?.computed),
offsetTimeOriginal: asString(tags.OffsetTimeOriginal?.computed),
exposureTime: asNumber(tags.ExposureTime?.computed),
fNumber: asNumber(tags.FNumber?.computed),
// Only when the tag holds a single number, as most cameras write it.
iso: asNumber(tags.ISOSpeedRatings?.computed),
focalLength: asNumber(tags.FocalLength?.computed),
orientation: asNumber(tags.Orientation?.computed),
gpsLatitude: asDegrees(
tags.GPSLatitude?.computed,
tags.GPSLatitudeRef?.computed,
"S",
),
gpsLongitude: asDegrees(
tags.GPSLongitude?.computed,
tags.GPSLongitudeRef?.computed,
"W",
),
// A GPSAltitudeRef of 1 means the altitude is below sea level.
gpsAltitude:
altitude !== undefined && tags.GPSAltitudeRef?.value === 1
? -altitude
: altitude,
};
// Leave out what the file lacks, so a missing field is absent rather than
// present and undefined.
return Object.fromEntries(
Object.entries(fields).filter(([, v]) => v !== undefined),
) as PhotoExif;
};
src/filename.ts
+111
View File
@@ -0,0 +1,111 @@
// File names built from server-supplied metadata.
//
// A file's title and a collection's name are decrypted from data the server
// hands us, and quak does not trust the server. Any name taken from them and
// used on disk goes through here, so it can only ever name one file inside the
// directory the caller chose: never a path, never `..`, never hidden, never a
// Windows device name. Such a name can still be too long for a file name, and
// `shortenToBytes` or `shortenKeepingExtension` cuts one short.
// `defaultFileName` does both for the name a file is saved under when the
// caller gives no path.
//
// A path the user typed (`--out`, `outPath`) is not passed through here: the
// caller is trusted, the server is not.
import { extname } from "node:path";
// Path separators, characters Windows forbids in file names, and control
// characters (NUL included).
// eslint-disable-next-line no-control-regex
const UNSAFE_CHARACTERS = /[/\\:*?"<>|\x00-\x1f\x7f]/g;
// Names Windows reserves for devices, with or without an extension.
const RESERVED_DEVICE_NAME = /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\.|$)/i;
// `name` made safe to use as a single file name. Each unsafe character becomes
// `_`, a leading run of dots becomes one `_`, and a device name gets a leading
// `_`. A name with none of these comes back unchanged. An empty name becomes
// `fallback`, which the caller derives from the record's ID.
export const sanitizeFileName = (name: string, fallback: string): string => {
if (name === "") return fallback;
const cleaned = name.replace(UNSAFE_CHARACTERS, "_").replace(/^\.+/, "_");
return RESERVED_DEVICE_NAME.test(cleaned) ? `_${cleaned}` : cleaned;
};
// The extension of `title` (".jpg"), or ".bin" when it has none, it holds
// anything but letters and digits, or it is more than 16 of them long.
export const safeExtension = (title: string): string => {
const ext = extname(title);
return /^\.[A-Za-z0-9]{1,16}$/.test(ext) ? ext : ".bin";
};
// `name` with its extension, if it has one, replaced by `ext` (".mov").
export const withExtension = (name: string, ext: string): string =>
name.slice(0, name.length - extname(name).length) + ext;
// The longest file name, in bytes, that ext4 and most other file systems allow.
export const MAX_NAME_BYTES = 255;
// `name` cut short to at most `maxBytes` bytes of UTF-8, the encoding a file
// name is written to disk in, between two characters, never inside one. A name
// that fits comes back unchanged.
export const shortenToBytes = (name: string, maxBytes: number): string => {
let bytes = 0;
let end = 0;
for (const character of name) {
bytes += Buffer.byteLength(character);
if (bytes > maxBytes) break;
end += character.length;
}
return name.slice(0, end);
};
// `name` cut short to fit in `maxBytes` bytes with `insert` put right before
// its extension (`IMG_0001 (12).jpg`), each cut made as `shortenToBytes`
// makes it. The part before the extension is cut first, so the extension is
// kept whenever it fits, but that part always keeps its first character:
// when the extension leaves no room for `insert` and that character, the
// extension is cut short too, and the part before it keeps the bytes that cut
// leaves over. So a name that does not start with a dot never comes back
// starting with one, which would hide it. `insert` is never cut. With no
// `insert`, a name that fits comes back unchanged.
export const shortenKeepingExtension = (
name: string,
maxBytes: number,
insert = "",
): string => {
const fullExt = extname(name);
const stem = name.slice(0, name.length - fullExt.length);
const first = Array.from(stem)[0] ?? "";
const ext = shortenToBytes(
fullExt,
maxBytes - Buffer.byteLength(insert + first),
);
const room = maxBytes - Buffer.byteLength(insert + ext);
return shortenToBytes(stem, room) + insert + ext;
};
// The name a file is saved under in the working directory when the caller
// gives no path (`quak get` without `--out`, `downloadFile` without
// `outPath`): `prefix` (`thumb_` for a thumbnail), then the file's title with
// unsafe characters replaced, or `file-<id>` when the title is empty, cut
// short so that the whole fits in a file name, keeping the title's extension
// whenever it fits. `ext` (".mov"), when given, takes the place of the
// title's extension before the name is cut short; a live photo's image and
// video are each named with their own. The cut can leave a device name
// (`CONSOLE.<long extension>` becomes `CON.<long extension>`); that name gets
// a leading `_`, as `sanitizeFileName` gives one, and is cut a byte shorter to
// leave room for it.
export const defaultFileName = (
title: string,
fileID: number,
opts: { prefix?: string; ext?: string } = {},
): string => {
const { prefix = "", ext } = opts;
const safe = sanitizeFileName(title, `file-${fileID}`);
const name = ext === undefined ? safe : withExtension(safe, ext);
const room = MAX_NAME_BYTES - Buffer.byteLength(prefix);
const cut = prefix + shortenKeepingExtension(name, room);
if (!RESERVED_DEVICE_NAME.test(cut)) return cut;
return `_${prefix}${shortenKeepingExtension(name, room - 1)}`;
};
src/index.ts
+66 -2
View File
@@ -1,6 +1,18 @@
export const VERSION = "0.0.0";
// A build reports the version script/build stamps into dist/package.json;
// package.json's own version is reported only when running from source. tsc
// copies package.json to dist/package.json, so this path resolves from source
// and from dist/src/.
import pkg from "../package.json" with { type: "json" };
export { Client, type LoginOptions, type ClientSnapshot } from "./client.js";
export const VERSION: string = pkg.version;
export {
Client,
type LoginOptions,
type ClientSnapshot,
type CollectionsPage,
type FilesPage,
} from "./client.js";
export {
ApiClient,
ApiError,
@@ -15,6 +27,7 @@ export {
isRetryable,
isSafeToReplay,
resolveRetryOptions,
UNATTENDED_RETRY_OPTIONS,
withRetry,
type ResolvedRetryOptions,
type RetryOptions,
@@ -27,6 +40,57 @@ export {
requestEmailOTP,
submitEmailOTP,
} from "./auth/login.js";
export {
Library,
DEFAULT_REFRESH_INTERVAL_SECONDS,
Album,
Photo,
type LibraryClient,
type LibraryOptions,
type LibraryStatus,
type RefreshEvent,
type RefreshProgressCallback,
type AlbumsAPI,
type PhotosAPI,
type TimelineAPI,
type PhotoFilter,
type TimelineGroup,
type GroupBy,
type SavePathLookup,
type ContentSource,
type ContentResult,
type ContentEvent,
type ContentOptions,
type PhotoContent,
type ThumbnailsAPI,
type ThumbnailPriority,
type EnsureOptions,
type EnsureResult,
type EnsureEvent,
lockBackupDirectory,
runBackup,
type BackupLock,
type BackupOptions,
type BackupResult,
type BackupError,
} from "./library/index.js";
export {
RequestPools,
BoundedPool,
DEFAULT_METADATA_CONCURRENCY,
DEFAULT_CONTENT_CONCURRENCY,
DEFAULT_THUMBNAIL_CONCURRENCY,
type RequestPoolsOptions,
type Priority,
type RunOptions,
} from "./library/pools.js";
export type {
AlbumRecord,
PhotoRecord,
LibrarySnapshot,
LibraryChange,
} from "./library/records.js";
export type { ExifTags, PhotoExif } from "./exif.js";
export { decryptCollection, decryptFile } from "./model/index.js";
export { downloadFile, downloadThumbnail } from "./download/index.js";
export type {
src/library/content.ts
+1088
View File
File diff suppressed because it is too large Load Diff
src/library/index.ts
+1159
View File
File diff suppressed because it is too large Load Diff
src/library/mldata.ts
+378
View File
@@ -0,0 +1,378 @@
// The on-disk cache of Ente's per-file machine-learning data and the CLIP
// index derived from it (issue #49).
//
// Under `<cacheDirectory>/mldata/` this keeps:
//
// - `<fileID>.json` — one decrypted, gunzipped payload per file, written by
// rename. Its presence means it is complete: a torn write never leaves a
// half-file, so the set of these files is the source of truth for what is
// cached. The full payload (face boxes, landmarks, embeddings) is read back
// from here on demand and never held in RAM.
//
// - `clip.f32` + `clip.json` — the derived index the content search runs on.
// `clip.json` lists the indexed fileIDs in order plus the embedding length;
// `clip.f32` is those CLIP embeddings packed as one `Float32Array`, so the
// index loads in a single read with no per-vector parse. The index is
// rebuilt from the payloads whenever it is missing or structurally
// disagrees with the files present, and appended to as new payloads arrive.
//
// - `fetched.json` — a small map of fileID to the `updationTime` it was
// fetched at. This is best-effort bookkeeping for refetch decisions (a file
// whose `updationTime` later advances is refetched); the payloads, not this
// file, remain the record of what is cached, so losing it only forgoes
// update-driven refetch until the next fetch rewrites it.
//
// In RAM this holds only the id list and the packed `Float32Array`.
import { mkdir, readFile, readdir } from "node:fs/promises";
import { join } from "node:path";
import { writeAtomic } from "../download/index.js";
import type { MLData } from "../mldata-fetch.js";
const CLIP_VECTORS = "clip.f32";
const CLIP_INDEX = "clip.json";
const FETCHED = "fetched.json";
// A payload file is named for its fileID alone; the derived files above are
// not, so this pattern picks out payloads and nothing else.
const PAYLOAD_RE = /^(\d+)\.json$/;
const BYTES_PER_FLOAT = 4;
// The on-disk form of `clip.json`.
interface ClipIndexFile {
fileIDs: number[];
embeddingLength: number;
}
// A file the model knows about, for deciding what to fetch.
export interface MLDataFile {
id: number;
updationTime: number;
}
// The RAM index the search reads: `fileIDs[i]` owns the `embeddingLength`
// floats of `embeddings` starting at `i * embeddingLength`.
export interface MLIndex {
fileIDs: number[];
embeddingLength: number;
embeddings: Float32Array;
}
// Pull the CLIP embedding out of a payload, or undefined when it is absent or
// misshapen. Kept strict so a bad payload is skipped rather than corrupting the
// packed index.
export const clipEmbedding = (payload: MLData): number[] | undefined => {
const clip = payload.clip;
if (typeof clip !== "object" || clip === null) return undefined;
const embedding = (clip as { embedding?: unknown }).embedding;
if (!Array.isArray(embedding)) return undefined;
if (embedding.some((v) => typeof v !== "number" || !Number.isFinite(v)))
return undefined;
return embedding as number[];
};
export class MLDataStore {
readonly dir: string;
// fileIDs whose payload JSON is present on disk (present means complete).
private readonly present = new Set<number>();
// fileID -> updationTime it was fetched at.
private readonly fetched = new Map<number, number>();
// The packed index and where each id sits in it.
private ids: number[] = [];
private embeddingLength = 0;
private embeddings = new Float32Array(0);
private readonly pos = new Map<number, number>();
private constructor(dir: string) {
this.dir = dir;
}
// Open (creating the directory) and load the id list and packed index into
// RAM, rebuilding the index from the payloads when it is missing or does
// not match the files present.
static async open(dir: string): Promise<MLDataStore> {
const store = new MLDataStore(dir);
await mkdir(dir, { recursive: true });
await store.loadPresent();
await store.loadFetched();
if (!(await store.tryLoadIndex())) await store.rebuildIndex();
return store;
}
// The fileIDs among `files` that must be fetched: every file with no
// payload yet (first run, then new files), plus any whose `updationTime`
// has advanced past the one its cached payload was fetched at. Returned
// sorted and unique.
neededFor(files: MLDataFile[]): number[] {
const latest = new Map<number, number>();
for (const f of files) {
const seen = latest.get(f.id);
if (seen === undefined || f.updationTime > seen)
latest.set(f.id, f.updationTime);
}
const needed: number[] = [];
for (const [id, updationTime] of latest) {
if (!this.present.has(id)) {
needed.push(id);
continue;
}
const at = this.fetched.get(id);
if (at !== undefined && updationTime > at) needed.push(id);
}
return needed.sort((a, b) => a - b);
}
// Store a batch of fetched payloads: write one file per id, fold their CLIP
// embeddings into the packed index (in place for a refetch, appended for a
// new file), and persist the derived files. Returns how many payloads were
// stored and how many ids the index now holds.
async storeFetched(
payloads: Map<number, MLData>,
updation: Map<number, number>,
): Promise<{ stored: number; indexed: number }> {
if (payloads.size === 0) return { stored: 0, indexed: this.ids.length };
for (const [id, payload] of payloads) {
await this.writePayload(id, payload);
this.present.add(id);
const at = updation.get(id);
if (at !== undefined) this.fetched.set(id, at);
}
const updates: { at: number; vector: number[] }[] = [];
const appends: { id: number; vector: number[] }[] = [];
for (const [id, payload] of payloads) {
const vector = clipEmbedding(payload);
if (!vector) continue;
if (this.embeddingLength === 0 && this.ids.length === 0)
this.embeddingLength = vector.length;
// The index is fixed-width; a vector of another length (never seen
// from Ente's CLIP model) is stored but left out of the index.
if (vector.length !== this.embeddingLength) continue;
const at = this.pos.get(id);
if (at !== undefined) updates.push({ at, vector });
else appends.push({ id, vector });
}
for (const { at, vector } of updates)
this.embeddings.set(vector, at * this.embeddingLength);
if (appends.length > 0) {
const length = this.embeddingLength;
const grown = new Float32Array(
this.embeddings.length + appends.length * length,
);
grown.set(this.embeddings);
let offset = this.embeddings.length;
for (const { id, vector } of appends) {
grown.set(vector, offset);
this.pos.set(id, this.ids.length);
this.ids.push(id);
offset += length;
}
this.embeddings = grown;
}
await this.persistIndex();
await this.persistFetched();
return { stored: payloads.size, indexed: this.ids.length };
}
// The packed index the search runs on. The id list is copied so callers
// cannot disturb the store's own order; the embeddings are the live buffer.
getIndex(): MLIndex {
return {
fileIDs: [...this.ids],
embeddingLength: this.embeddingLength,
embeddings: this.embeddings,
};
}
// The full payload for a file, read from disk, or undefined when it is not
// cached or does not parse.
async readPayload(fileID: number): Promise<MLData | undefined> {
if (!this.present.has(fileID)) return undefined;
let raw: string;
try {
raw = await readFile(this.payloadPath(fileID), "utf-8");
} catch {
return undefined;
}
try {
return JSON.parse(raw) as MLData;
} catch {
return undefined;
}
}
stats(): { stored: number; indexed: number } {
return { stored: this.present.size, indexed: this.ids.length };
}
private payloadPath(id: number): string {
return join(this.dir, `${id}.json`);
}
private async writePayload(id: number, payload: MLData): Promise<void> {
await writeAtomic(
this.payloadPath(id),
new TextEncoder().encode(JSON.stringify(payload)),
);
}
private async loadPresent(): Promise<void> {
let names: string[];
try {
names = await readdir(this.dir);
} catch {
return;
}
for (const name of names) {
const match = PAYLOAD_RE.exec(name);
if (match) this.present.add(Number(match[1]));
}
}
private async loadFetched(): Promise<void> {
let raw: string;
try {
raw = await readFile(join(this.dir, FETCHED), "utf-8");
} catch {
return;
}
try {
const parsed = JSON.parse(raw) as Record<string, unknown>;
for (const [key, value] of Object.entries(parsed)) {
const id = Number(key);
if (
Number.isInteger(id) &&
typeof value === "number" &&
this.present.has(id)
)
this.fetched.set(id, value);
}
} catch {
// Corrupt bookkeeping degrades refetch decisions, never fails open.
}
}
// Load the packed index if it is present and agrees with the payloads in
// both directions: every id it names must still be present, its vector file
// must be exactly the size the id count and embedding length imply, and no
// embedding-bearing payload on disk may be missing from it. Returns whether
// it loaded.
private async tryLoadIndex(): Promise<boolean> {
let metaRaw: string;
try {
metaRaw = await readFile(join(this.dir, CLIP_INDEX), "utf-8");
} catch {
return false;
}
let meta: ClipIndexFile;
try {
meta = JSON.parse(metaRaw) as ClipIndexFile;
} catch {
return false;
}
if (
!Array.isArray(meta.fileIDs) ||
typeof meta.embeddingLength !== "number"
)
return false;
if (meta.fileIDs.some((id) => !this.present.has(id))) return false;
// The reverse must hold too. A payload carrying an embedding but absent
// from the index means the index is stale — realistically the process
// died after storeFetched renamed the payloads into place but before it
// rewrote clip.json/clip.f32. Loading such an index as "consistent"
// would drop those embeddings for good (neededFor sees the payloads
// present and never refetches), so treat it as a disagreement and
// rebuild. Only present ids the index omits are read; a payload
// legitimately without an embedding stays out and forces no rebuild.
const indexed = new Set(meta.fileIDs);
for (const id of this.present) {
if (indexed.has(id)) continue;
const payload = await this.readPayload(id);
if (payload && clipEmbedding(payload)) return false;
}
let bytes: Buffer;
try {
bytes = await readFile(join(this.dir, CLIP_VECTORS));
} catch {
return false;
}
const expected =
meta.fileIDs.length * meta.embeddingLength * BYTES_PER_FLOAT;
if (bytes.byteLength !== expected) return false;
// One read, no parse: copy into an aligned buffer and view it as
// floats. The copy is needed because a Buffer from the pool can start
// at an offset a Float32Array cannot be laid over.
const aligned = new Uint8Array(bytes.byteLength);
aligned.set(bytes);
this.embeddings = new Float32Array(aligned.buffer);
this.embeddingLength = meta.embeddingLength;
this.ids = [...meta.fileIDs];
this.pos.clear();
this.ids.forEach((id, i) => this.pos.set(id, i));
return true;
}
// Rebuild the packed index by reading every payload present, then persist
// it. Payloads without a CLIP embedding (or of an unexpected length) are
// simply not indexed.
private async rebuildIndex(): Promise<void> {
this.ids = [];
this.pos.clear();
this.embeddingLength = 0;
const vectors: number[][] = [];
for (const id of [...this.present].sort((a, b) => a - b)) {
const payload = await this.readPayload(id);
if (!payload) continue;
const vector = clipEmbedding(payload);
if (!vector) continue;
if (this.embeddingLength === 0)
this.embeddingLength = vector.length;
if (vector.length !== this.embeddingLength) continue;
this.pos.set(id, this.ids.length);
this.ids.push(id);
vectors.push(vector);
}
const length = this.embeddingLength;
const packed = new Float32Array(this.ids.length * length);
vectors.forEach((vector, i) => packed.set(vector, i * length));
this.embeddings = packed;
await this.persistIndex();
}
private async persistIndex(): Promise<void> {
const meta: ClipIndexFile = {
fileIDs: this.ids,
embeddingLength: this.embeddingLength,
};
await writeAtomic(
join(this.dir, CLIP_INDEX),
new TextEncoder().encode(JSON.stringify(meta)),
);
await writeAtomic(
join(this.dir, CLIP_VECTORS),
new Uint8Array(
this.embeddings.buffer,
this.embeddings.byteOffset,
this.embeddings.byteLength,
),
);
}
private async persistFetched(): Promise<void> {
const record: Record<string, number> = {};
for (const [id, at] of this.fetched) record[id] = at;
await writeAtomic(
join(this.dir, FETCHED),
new TextEncoder().encode(JSON.stringify(record)),
);
}
}
src/library/mlsearch.ts
+129
View File
@@ -0,0 +1,129 @@
// The content-similarity search surface over the CLIP index (issue #50).
//
// This is `lib.mldata`. It answers three questions against the ML-data cache
// (#49) without touching the network:
//
// - `forFile` returns the whole stored payload (face boxes, landmarks,
// embedding) for a file, read from disk on demand — the only method here
// that touches the disk, and the only one that is async.
// - `similar` and `searchByEmbedding` rank fileIDs by cosine similarity over
// the packed `Float32Array` index alone. That index (~50k×512) already
// lives in RAM, so each query is a plain loop over it and nothing else.
//
// quak bundles no text encoder (owner-deferred), so `searchByEmbedding` takes
// the query vector the caller has produced elsewhere; `similar` uses the
// query file's own indexed embedding.
import type { MLData } from "../mldata-fetch.js";
import type { MLDataStore, MLIndex } from "./mldata.js";
// How many nearest files a query returns when the caller names no limit.
const DEFAULT_LIMIT = 20;
// One ranked result: a fileID and its cosine similarity to the query, in
// [-1, 1]. Callers wanting only the ids read `.fileID`.
export interface SimilarResult {
fileID: number;
score: number;
}
export interface MLDataAPI {
// The whole stored ML payload for a file, or undefined when it is not
// cached. Reads the payload from disk, so it is async.
forFile(args: { fileID: number }): Promise<MLData | undefined>;
// The files nearest the given file by cosine over their CLIP embeddings,
// most similar first, excluding the file itself. Empty when the file has
// no indexed embedding.
similar(args: { fileID: number; limit?: number }): SimilarResult[];
// The files nearest a caller-supplied query embedding by cosine, most
// similar first. Empty when the query is the wrong length for the index,
// has zero magnitude, or the index is empty.
searchByEmbedding(args: {
embedding: ArrayLike<number>;
limit?: number;
}): SimilarResult[];
}
// Rank the packed index by cosine similarity to `query`, most similar first,
// and return the top `limit`. `skip` (a query file's own id) is left out. Both
// each row's magnitude and the query's are computed here rather than cached:
// the index mutates as ML data is fetched, and one plain pass over ~50k×512
// floats is fast enough that a norm cache would only add a staleness bug. A
// zero-magnitude vector has no direction, so it is dropped rather than divided
// by zero.
const topByCosine = (
index: MLIndex,
query: ArrayLike<number>,
limit: number,
skip?: number,
): SimilarResult[] => {
const { fileIDs, embeddingLength, embeddings } = index;
if (embeddingLength === 0 || query.length !== embeddingLength) return [];
// Every indexed read below is in range: the inner loops run to
// `embeddingLength`, the query is exactly that long (checked above), and
// the packed buffer holds `fileIDs.length * embeddingLength` floats.
// `noUncheckedIndexedAccess` still widens each read to `number | undefined`,
// so they are asserted non-null rather than paying a per-element guard in
// this hot ~50k×512 loop.
let queryNorm = 0;
for (let k = 0; k < embeddingLength; k++) {
const q = query[k]!;
queryNorm += q * q;
}
queryNorm = Math.sqrt(queryNorm);
if (queryNorm === 0) return [];
const results: SimilarResult[] = [];
for (let i = 0; i < fileIDs.length; i++) {
const id = fileIDs[i]!;
if (id === skip) continue;
const base = i * embeddingLength;
let dot = 0;
let norm = 0;
for (let k = 0; k < embeddingLength; k++) {
const v = embeddings[base + k]!;
dot += query[k]! * v;
norm += v * v;
}
if (norm === 0) continue;
results.push({
fileID: id,
score: dot / (queryNorm * Math.sqrt(norm)),
});
}
// Descending score, ties broken by ascending fileID for a stable order.
results.sort((a, b) => b.score - a.score || a.fileID - b.fileID);
return results.slice(0, Math.max(0, Math.trunc(limit)));
};
// Build the search surface over a store the library supplies lazily (the store
// is absent when the client cannot fetch ML data). Reading it per call keeps
// the surface current as the index grows.
export const makeMLDataAPI = (
store: () => MLDataStore | undefined,
): MLDataAPI => ({
forFile: ({ fileID }): Promise<MLData | undefined> => {
const s = store();
return s ? s.readPayload(fileID) : Promise.resolve(undefined);
},
similar: ({ fileID, limit }): SimilarResult[] => {
const s = store();
if (!s) return [];
const index = s.getIndex();
const pos = index.fileIDs.indexOf(fileID);
if (pos < 0) return [];
const base = pos * index.embeddingLength;
const query = index.embeddings.subarray(
base,
base + index.embeddingLength,
);
return topByCosine(index, query, limit ?? DEFAULT_LIMIT, fileID);
},
searchByEmbedding: ({ embedding, limit }): SimilarResult[] => {
const s = store();
if (!s) return [];
return topByCosine(s.getIndex(), embedding, limit ?? DEFAULT_LIMIT);
},
});
src/library/pools.ts
+181
View File
@@ -0,0 +1,181 @@
// Three bounded request pools for metadata, content, and thumbnails (issue
// #45).
//
// Ente meters differently by traffic class, so quak keeps three independent
// pools instead of one global limit: metadata is cheap and chatty, original
// content is heavy, thumbnails are small but numerous. Each pool is a
// `BoundedPool` — a plain concurrency limiter — and the three run at the
// design's caps (10 / 5 / 25) unless the caller overrides them.
//
// Two behaviours beyond a bare limiter, both per pool:
//
// - Priority: work waiting for a slot is ordered on-demand before
// background/precache, so a slot that frees up serves the request a user is
// waiting on ahead of speculative prefetch. Within one priority the order
// is first-come-first-served.
//
// - In-flight dedup: a task submitted under a `key` that a still-pending task
// already carries is not run a second time; both callers await the one
// result. The key is released the moment that task settles — success or
// failure — so a later request for the same key runs afresh. Callers key by
// the id whose fetch must not be duplicated (a fileID, say).
//
// The pool holds a task's slot for that task's entire lifetime. A task that
// retries internally is doing so inside its slot: the slot is not freed between
// attempts, which is what keeps a retrying request counted against the cap. The
// pool knows nothing of the retry policy; it only holds the slot until the
// task's promise settles.
//
// This module is self-contained infrastructure. Routing a given `Client` call
// to the right pool belongs to the unit that wires the pools into the cache;
// here there is only the machinery.
export const DEFAULT_METADATA_CONCURRENCY = 10;
export const DEFAULT_CONTENT_CONCURRENCY = 5;
export const DEFAULT_THUMBNAIL_CONCURRENCY = 25;
// On-demand work is served before background/precache work waiting in the same
// pool.
export type Priority = "on-demand" | "background";
export interface RunOptions {
// Defaults to "background": an unmarked request yields to on-demand work.
priority?: Priority;
// When set, a task already pending under this key is shared instead of run
// again. Omit for work that must always execute.
key?: string | number;
}
// One queued submission awaiting a slot. `start` runs the task and holds the
// slot until it settles.
interface Waiter {
priority: Priority;
// Submission order, used to break ties within a priority (FIFO).
seq: number;
start: () => void;
}
export class BoundedPool {
readonly concurrency: number;
private active = 0;
private nextSeq = 0;
private readonly waiting: Waiter[] = [];
// Keyed by a caller-supplied dedup key; holds the shared promise for as
// long as that task is pending, cleared when it settles.
private readonly pending = new Map<string | number, Promise<unknown>>();
constructor(concurrency: number) {
if (!Number.isInteger(concurrency) || concurrency < 1) {
throw new RangeError(
`concurrency must be a positive integer, got ${concurrency}`,
);
}
this.concurrency = concurrency;
}
// Submit `task` to the pool. It runs once a slot is free, subject to
// priority; the returned promise settles with the task's result. With a
// `key`, a still-pending submission under the same key is returned instead
// of running `task` again.
run<T>(task: () => Promise<T>, opts: RunOptions = {}): Promise<T> {
const { key } = opts;
if (key !== undefined) {
const shared = this.pending.get(key);
if (shared !== undefined) return shared as Promise<T>;
}
const promise = this.enqueue(task, opts.priority ?? "background");
if (key !== undefined) {
this.pending.set(key, promise);
const release = (): void => {
// Only clear our own entry: a fresh submission under the same
// key after this one settled must not be evicted here.
if (this.pending.get(key) === promise) this.pending.delete(key);
};
promise.then(release, release);
}
return promise;
}
private enqueue<T>(task: () => Promise<T>, priority: Priority): Promise<T> {
return new Promise<T>((resolve, reject) => {
const start = (): void => {
this.active++;
// Hold the slot until the task fully settles — every internal
// retry included — then admit the next waiter.
void (async () => {
try {
resolve(await task());
} catch (err) {
reject(err);
} finally {
this.active--;
this.pump();
}
})();
};
this.waiting.push({ priority, seq: this.nextSeq++, start });
this.pump();
});
}
// Admit waiters until the pool is full or the queue is empty.
private pump(): void {
while (this.active < this.concurrency) {
const next = this.takeNext();
if (next === undefined) return;
next.start();
}
}
// Remove and return the highest-priority waiter: on-demand before
// background, earliest submission first within a priority.
private takeNext(): Waiter | undefined {
let bestIndex = -1;
let best: Waiter | undefined;
for (let i = 0; i < this.waiting.length; i++) {
const w = this.waiting[i];
if (w === undefined) continue;
if (best === undefined || this.precedes(w, best)) {
best = w;
bestIndex = i;
}
}
if (best === undefined) return undefined;
this.waiting.splice(bestIndex, 1);
return best;
}
private precedes(a: Waiter, b: Waiter): boolean {
if (a.priority !== b.priority) return a.priority === "on-demand";
return a.seq < b.seq;
}
}
export interface RequestPoolsOptions {
metadataConcurrency?: number;
contentConcurrency?: number;
thumbnailConcurrency?: number;
}
// The three pools the design calls for, each independent: an idle pool never
// lends its slots to a busy one.
export class RequestPools {
readonly metadata: BoundedPool;
readonly content: BoundedPool;
readonly thumbnails: BoundedPool;
constructor(opts: RequestPoolsOptions = {}) {
this.metadata = new BoundedPool(
opts.metadataConcurrency ?? DEFAULT_METADATA_CONCURRENCY,
);
this.content = new BoundedPool(
opts.contentConcurrency ?? DEFAULT_CONTENT_CONCURRENCY,
);
this.thumbnails = new BoundedPool(
opts.thumbnailConcurrency ?? DEFAULT_THUMBNAIL_CONCURRENCY,
);
}
}
src/library/precache.ts
+269
View File
@@ -0,0 +1,269 @@
// The aggressive local precache (issue #48), started from `Library.open` with
// no caller input.
//
// Two background fills run concurrently through the shared request pools (#45):
//
// - Thumbnails: every file in the account, newest first, through the
// thumbnail pool until all are on disk. Never evicted. The pool is the same
// one `thumbnails.ensure` uses, so a visible or ahead request always jumps
// ahead of this background fill and a fileID both want is fetched once.
//
// - Originals (the pinned set): through the content pool, the favorites album
// first, then every file whose `takenAt` falls in the latest
// `originalsDays` window — the days ending at the newest file in the
// account. The pinned set is the eviction predicate (#47): a pinned
// original is never evicted, and a file that leaves the set (a favorite
// removed, or the window moving past it on a later refresh) becomes an
// ordinary, evictable original with its bytes left in place.
//
// Both fills yield to on-demand work: every fetch goes to its pool at
// background priority, which the pool serves only after on-demand requests. A
// file already cached costs one map lookup (`pathsFor`) and no fetch. Each
// sweep is driven in bounded chunks so the pool's waiting queue never grows to
// the whole account, keeping on-demand preemption and per-admit cost cheap on a
// large library. Failures are not fatal: an uncached file is retried on the
// next sweep, which `Library` re-kicks after every refresh.
import type { EnsureResult } from "./content.js";
import type { DerivedRecords } from "./records.js";
const ONE_DAY_MS = 24 * 60 * 60 * 1000;
export const DEFAULT_PRECACHE_ORIGINALS_DAYS = 7;
// How many files a sweep submits to a pool before awaiting them. Bounds the
// pool's waiting queue so on-demand work is never stuck behind the whole
// account; the values track each pool's concurrency (#45).
const THUMBNAIL_CHUNK = 25;
const ORIGINAL_CHUNK = 5;
// The precache metrics `Library.status()` surfaces.
export interface PrecacheStatus {
thumbnailsCached: number;
thumbnailsTotal: number;
originalsCached: number;
originalsPinned: number;
}
// A background-fill progress event. Each active sweep fires "started" before
// its fetches and "done" (with the count newly on disk) after; a sweep with
// nothing left to fetch is silent.
export interface PrecacheEvent {
operation: "precacheThumbnails" | "precacheOriginals";
status: "started" | "done";
fetched?: number;
}
// The slice of the content cache the precache drives. The real `ContentCache`
// satisfies it; tests inject a fake.
export interface PrecacheCache {
pathsFor(fileID: number): { originalPath?: string; thumbnailPath?: string };
ensureThumbnails(args: {
fileIDs: number[];
priority: "background";
signal?: AbortSignal;
}): Promise<EnsureResult[]>;
ensureOriginals(args: {
fileIDs: number[];
signal?: AbortSignal;
}): Promise<EnsureResult[]>;
}
export interface PrecacheOptions {
// Default true; false disables the fill and, for originals, the pinning.
thumbnails?: boolean;
originals?: boolean;
// The latest-week window length in days; default 7.
originalsDays?: number;
onEvent?: (event: PrecacheEvent) => void;
}
export class Precache {
private readonly doThumbnails: boolean;
private readonly doOriginals: boolean;
private readonly originalsDays: number;
private readonly onEvent?: (event: PrecacheEvent) => void;
private cache?: PrecacheCache;
// Every file in the account, newest first (the thumbnail fill order).
private thumbOrder: number[] = [];
// The pinned originals in fetch order: favorites first, then the window.
private originalsOrder: number[] = [];
private pinned = new Set<number>();
// A sweep runs at most once per fill at a time; a re-kick while one runs is
// a no-op, and the next refresh re-kicks after it finishes. Each holds the
// running sweep, so `close()` can wait for it.
private thumbSweep?: Promise<void>;
private originalsSweep?: Promise<void>;
private readonly aborter = new AbortController();
private closed = false;
constructor(opts: PrecacheOptions = {}) {
this.doThumbnails = opts.thumbnails ?? true;
this.doOriginals = opts.originals ?? true;
this.originalsDays =
opts.originalsDays ?? DEFAULT_PRECACHE_ORIGINALS_DAYS;
this.onEvent = opts.onEvent;
}
// Attach the cache the fills fetch through. `isPinned` works before this is
// called, so the cache can be constructed with `isPinned` wired in and then
// bound here.
bind(cache: PrecacheCache): void {
this.cache = cache;
}
// Whether an original is pinned (favorites + the latest-week window), the
// eviction predicate (#47). False for every file when originals precaching
// is disabled.
isPinned(fileID: number): boolean {
return this.pinned.has(fileID);
}
// Recompute the fill orders and the pinned set from the current projection.
// Called at open and after every change to the store.
update(records: DerivedRecords): void {
const photos = [...records.photos.values()].sort(
(a, b) => b.takenAt - a.takenAt || b.fileID - a.fileID,
);
this.thumbOrder = this.doThumbnails ? photos.map((p) => p.fileID) : [];
const pinned = new Set<number>();
const order: number[] = [];
if (this.doOriginals) {
// Favorites first, in the album's own newest-first order.
for (const album of records.albums.values()) {
if (album.type !== "favorites") continue;
for (const id of album.fileIDs) {
if (!pinned.has(id)) {
pinned.add(id);
order.push(id);
}
}
}
// Then the latest-week window, ending at the newest file. Photos
// are newest first, so stop once one falls before the window start.
const newest = photos[0];
if (newest !== undefined) {
const windowStart =
newest.takenAt - this.originalsDays * ONE_DAY_MS;
for (const p of photos) {
if (p.takenAt < windowStart) break;
if (!pinned.has(p.fileID)) {
pinned.add(p.fileID);
order.push(p.fileID);
}
}
}
}
this.pinned = pinned;
this.originalsOrder = order;
}
status(): PrecacheStatus {
const cache = this.cache;
let thumbnailsCached = 0;
let originalsCached = 0;
if (cache) {
for (const id of this.thumbOrder)
if (cache.pathsFor(id).thumbnailPath !== undefined)
thumbnailsCached++;
for (const id of this.originalsOrder)
if (cache.pathsFor(id).originalPath !== undefined)
originalsCached++;
}
return {
thumbnailsCached,
thumbnailsTotal: this.thumbOrder.length,
originalsCached,
originalsPinned: this.originalsOrder.length,
};
}
// Kick both fills. Idempotent: a fill already sweeping is left alone. Safe
// to call after every refresh; a finished fill starts a fresh sweep that
// picks up new files and retries any that failed before.
start(): void {
if (this.closed || !this.cache) return;
if (this.doThumbnails) this.kickThumbnails();
if (this.doOriginals) this.kickOriginals();
}
// Stop the fills. In-flight fetches are left to settle; queued ones drop.
// Resolves once both sweeps have finished, so nothing is still writing.
async close(): Promise<void> {
this.closed = true;
this.aborter.abort();
await Promise.all([
this.thumbSweep?.catch(() => {}),
this.originalsSweep?.catch(() => {}),
]);
}
private kickThumbnails(): void {
if (this.thumbSweep) return;
this.thumbSweep = this.sweep(
"precacheThumbnails",
() => this.thumbOrder,
(id) => this.cache!.pathsFor(id).thumbnailPath !== undefined,
THUMBNAIL_CHUNK,
(chunk) =>
this.cache!.ensureThumbnails({
fileIDs: chunk,
priority: "background",
signal: this.aborter.signal,
}),
).finally(() => {
this.thumbSweep = undefined;
});
}
private kickOriginals(): void {
if (this.originalsSweep) return;
this.originalsSweep = this.sweep(
"precacheOriginals",
() => this.originalsOrder,
(id) => this.cache!.pathsFor(id).originalPath !== undefined,
ORIGINAL_CHUNK,
(chunk) =>
this.cache!.ensureOriginals({
fileIDs: chunk,
signal: this.aborter.signal,
}),
).finally(() => {
this.originalsSweep = undefined;
});
}
// One fill sweep: skip files already on disk (one lookup each), fetch the
// rest in bounded chunks, and report progress only when there was work.
private async sweep(
operation: PrecacheEvent["operation"],
order: () => number[],
present: (fileID: number) => boolean,
chunkSize: number,
fetch: (chunk: number[]) => Promise<EnsureResult[]>,
): Promise<void> {
const todo = order().filter((id) => !present(id));
if (todo.length === 0) return;
this.emit({ operation, status: "started" });
let fetched = 0;
for (let i = 0; i < todo.length && !this.closed; i += chunkSize) {
const results = await fetch(todo.slice(i, i + chunkSize));
for (const r of results) if (r.path !== undefined) fetched++;
}
this.emit({ operation, status: "done", fetched });
}
private emit(event: PrecacheEvent): void {
if (!this.onEvent) return;
// A misbehaving callback must not break the fill loop.
try {
this.onEvent(event);
} catch {
// ignore
}
}
}
src/library/read.ts
+537
View File
@@ -0,0 +1,537 @@
// The in-process read surface over the local cache (issue #44).
//
// A CLI or an in-process script reads albums, photos, and a grouped timeline
// through `lib.albums`, `lib.photos`, and `lib.timeline`. Every call is
// answered synchronously from the same plain-record projection the GUI reads
// (`deriveRecords`, issue #43); nothing here touches the network. Every method
// takes a single named-argument object.
//
// The `Album` and `Photo` classes are thin, in-process-only wrappers over
// those records: a caller that holds an object reference gets typed field
// access and, for an album, its photos. They are not sent across IPC — the
// plain records are the serializable surface, and `record()` returns one.
//
// A `Photo` also fetches its own bytes: `original()`, `thumbnail()`,
// `download()`, `content()`, `exif()` and the methods that each return one
// EXIF field go through the on-disk content cache (issue #46), and are
// the one place in this module that may touch the network. A library opened
// without a content source leaves that cache absent, and those methods then
// throw. `savePath` and `isLocal` look only at the disk and need no cache.
import { readFile } from "node:fs/promises";
import {
readAllExifTags,
readPhotoExif,
type ExifTags,
type PhotoExif,
} from "../exif.js";
import type { CollectionType, EnteFile, FileType } from "../model/types.js";
import type { ContentOptions, ContentResult, PhotoContent } from "./content.js";
import type { AlbumRecord, PhotoRecord, DerivedRecords } from "./records.js";
// Where a photo's original is saved, and whether all of it is there. The
// library answers both from the disk, with or without a content cache.
export interface SavePathLookup {
savePath(file: EnteFile): string;
isLocal(file: EnteFile): boolean;
}
// Newest first, with fileID as a stable tiebreak so equal-timed files order
// deterministically — the same order the record projection uses.
const byNewest = (a: PhotoRecord, b: PhotoRecord): number =>
b.takenAt - a.takenAt || b.fileID - a.fileID;
// Albums newest updated first, collection id breaking ties. This is the order
// `albums.list` returns and the order `byName` resolves a name collision in.
const byNewestAlbum = (a: AlbumRecord, b: AlbumRecord): number =>
b.updationTime - a.updationTime || b.collectionID - a.collectionID;
// A method for each `PhotoExif` field, named after it, taking the options
// `exif()` takes and giving that field. `Photo` implements it, so the build's
// type check fails when `PhotoExif` has a field `Photo` has no method for.
type PhotoExifMethods = {
[K in keyof PhotoExif]-?: (opts?: ContentOptions) => Promise<PhotoExif[K]>;
};
// A single photo. Field access mirrors `PhotoRecord`; `record()` returns the
// underlying plain record for callers that need the IPC-safe value. `file` is
// the membership the record is read from, so the save path carries the date of
// `takenAt` and stays known after a refresh removes the file from the library.
export class Photo implements PhotoExifMethods {
constructor(
private readonly rec: PhotoRecord,
private readonly file: EnteFile,
private readonly saves: SavePathLookup,
private readonly cache?: PhotoContent,
) {}
get fileID(): number {
return this.rec.fileID;
}
get albumIDs(): number[] {
return this.rec.albumIDs;
}
get title(): string {
return this.rec.title;
}
get takenAt(): number {
return this.rec.takenAt;
}
get modifiedAt(): number {
return this.rec.modifiedAt;
}
// The local-time year of `takenAt`.
get year(): number {
return new Date(this.rec.takenAt).getFullYear();
}
get fileType(): FileType {
return this.rec.fileType;
}
get caption(): string | undefined {
return this.rec.caption;
}
get width(): number | undefined {
return this.rec.width;
}
get height(): number | undefined {
return this.rec.height;
}
get latitude(): number | undefined {
return this.rec.latitude;
}
get longitude(): number | undefined {
return this.rec.longitude;
}
get hash(): string | undefined {
return this.rec.hash;
}
get isArchived(): boolean {
return this.rec.isArchived;
}
get isHidden(): boolean {
return this.rec.isHidden;
}
// Where `download()` and `lib.backup()` put the original under the
// library's download directory, whether or not it is there yet:
// `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID><ext>`. For a live photo
// already stored, its image. For a live photo not yet stored, it carries
// the title's extension, and the image may be stored under a different
// one.
get savePath(): string {
return this.saves.savePath(this.file);
}
// Whether the whole original is at `savePath`. A copy only in the cache
// does not count.
get isLocal(): boolean {
return this.saves.isLocal(this.file);
}
record(): PhotoRecord {
return this.rec;
}
// Fetch and cache the full-resolution original, returning its on-disk path
// and byte length; for a live photo, its image's, and its video's path as
// `videoPath`. Served from the cache (or the save path) when already
// present, otherwise fetched through the content pool.
async original(opts?: ContentOptions): Promise<ContentResult> {
return this.cacheOrThrow().original(this.rec.fileID, opts);
}
// Put the original at `savePath` and return it there, as `original()`
// does. When it is already there, nothing is written. When the cache holds
// it, it is copied from there; otherwise it is fetched straight to
// `savePath`.
async download(): Promise<ContentResult> {
return this.cacheOrThrow().download(this.file);
}
// As `original`, for the thumbnail, through the thumbnail pool.
async thumbnail(opts?: ContentOptions): Promise<ContentResult> {
return this.cacheOrThrow().thumbnail(this.rec.fileID, opts);
}
// The original's bytes, read from where `original()` puts it. For a live
// photo, its image's.
async content(opts?: ContentOptions): Promise<Uint8Array> {
const { path } = await this.original(opts);
return readFile(path);
}
// Every EXIF tag of the original, keyed by name (see `ExifTags`), read
// from `content()`, so this may download it. EXIF is read from any image
// format exifreader reads, JPEG and HEIC/HEIF among them; any other file
// gives `{}`, and a video gives it without fetching anything. Like the
// other content methods, it throws when there is no content cache, video
// or not.
async exif(opts?: ContentOptions): Promise<ExifTags> {
this.cacheOrThrow();
if (this.rec.fileType === "video") return {};
return readAllExifTags(await this.content(opts));
}
// One field each, named and typed as in `PhotoExif`, picked from the tags
// `exif()` returns, and undefined when the file lacks it. Each call runs
// `exif()`, which reads the original again.
async make(opts?: ContentOptions): Promise<PhotoExif["make"]> {
return readPhotoExif(await this.exif(opts)).make;
}
async model(opts?: ContentOptions): Promise<PhotoExif["model"]> {
return readPhotoExif(await this.exif(opts)).model;
}
async lensModel(opts?: ContentOptions): Promise<PhotoExif["lensModel"]> {
return readPhotoExif(await this.exif(opts)).lensModel;
}
async dateTimeOriginal(
opts?: ContentOptions,
): Promise<PhotoExif["dateTimeOriginal"]> {
return readPhotoExif(await this.exif(opts)).dateTimeOriginal;
}
async offsetTimeOriginal(
opts?: ContentOptions,
): Promise<PhotoExif["offsetTimeOriginal"]> {
return readPhotoExif(await this.exif(opts)).offsetTimeOriginal;
}
async exposureTime(
opts?: ContentOptions,
): Promise<PhotoExif["exposureTime"]> {
return readPhotoExif(await this.exif(opts)).exposureTime;
}
async fNumber(opts?: ContentOptions): Promise<PhotoExif["fNumber"]> {
return readPhotoExif(await this.exif(opts)).fNumber;
}
async iso(opts?: ContentOptions): Promise<PhotoExif["iso"]> {
return readPhotoExif(await this.exif(opts)).iso;
}
async focalLength(
opts?: ContentOptions,
): Promise<PhotoExif["focalLength"]> {
return readPhotoExif(await this.exif(opts)).focalLength;
}
async orientation(
opts?: ContentOptions,
): Promise<PhotoExif["orientation"]> {
return readPhotoExif(await this.exif(opts)).orientation;
}
async gpsLatitude(
opts?: ContentOptions,
): Promise<PhotoExif["gpsLatitude"]> {
return readPhotoExif(await this.exif(opts)).gpsLatitude;
}
async gpsLongitude(
opts?: ContentOptions,
): Promise<PhotoExif["gpsLongitude"]> {
return readPhotoExif(await this.exif(opts)).gpsLongitude;
}
async gpsAltitude(
opts?: ContentOptions,
): Promise<PhotoExif["gpsAltitude"]> {
return readPhotoExif(await this.exif(opts)).gpsAltitude;
}
private cacheOrThrow(): PhotoContent {
if (!this.cache) {
throw new Error(
"Photo content requires a library opened with a content cache",
);
}
return this.cache;
}
}
// A single album. `photos.list()` returns the album's photos as wrappers,
// newest first (the record already stores `fileIDs` in that order).
export class Album {
constructor(
private readonly rec: AlbumRecord,
private readonly records: DerivedRecords,
private readonly saves: SavePathLookup,
private readonly content?: PhotoContent,
) {}
get collectionID(): number {
return this.rec.collectionID;
}
get name(): string {
return this.rec.name;
}
get type(): CollectionType {
return this.rec.type;
}
get isShared(): boolean {
return this.rec.isShared;
}
get updationTime(): number {
return this.rec.updationTime;
}
get fileIDs(): number[] {
return this.rec.fileIDs;
}
get photos(): { list: () => Photo[] } {
return { list: (): Photo[] => this.listPhotos() };
}
record(): AlbumRecord {
return this.rec;
}
private listPhotos(): Photo[] {
const out: Photo[] = [];
for (const id of this.rec.fileIDs) {
const p = this.records.photos.get(id);
const file = this.records.files.get(id);
if (p && file) {
out.push(new Photo(p, file, this.saves, this.content));
}
}
return out;
}
}
export interface AlbumsAPI {
list(): Album[];
byName(args: { albumName: string }): Album | undefined;
byID(args: { collectionID: number }): Album | undefined;
}
export interface PhotosAPI {
byID(args: { fileID: number }): Photo | undefined;
// Plain records for the requested ids, in the order requested, each id at
// most once, unknown ids dropped.
records(args: { fileIDs: number[] }): PhotoRecord[];
}
export type GroupBy = "day" | "week" | "month";
// A filter over the timeline. All fields are optional and combine with AND.
// Hidden photos are never included, regardless of this filter.
export interface PhotoFilter {
// Keep only photos that belong to this album.
albumID?: number;
// Case-insensitive substring of the title, caption, or any album name the
// photo belongs to.
text?: string;
// Keep only photos of one of these types.
fileTypes?: FileType[];
// `true` keeps only geotagged photos; `false` keeps only those without a
// location; omitted places no constraint.
hasLocation?: boolean;
// Archived photos are excluded unless this is `true`. Defaults to `false`.
includeArchived?: boolean;
}
export interface TimelineGroup {
// The period's identity: `YYYY-MM-DD` for day, `YYYY-Www` (ISO 8601 week,
// e.g. `2025-W32`) for week, and `YYYY-MM` for month.
key: string;
// Local-time milliseconds at the start of the period.
startsAt: number;
// The period's files, newest first, each file once.
fileIDs: number[];
}
export interface TimelineAPI {
groups(args: { groupBy: GroupBy; filter?: PhotoFilter }): TimelineGroup[];
}
// The surface `Library.fresh()` resolves to (issue #75). It is the same three
// read namespaces as the default `albums`/`photos`/`timeline`, handed back only
// after a forced refresh has brought the local copy current.
export interface FreshReads {
albums: AlbumsAPI;
photos: PhotosAPI;
timeline: TimelineAPI;
}
export const makeAlbumsAPI = (
derive: () => DerivedRecords,
saves: SavePathLookup,
content?: PhotoContent,
): AlbumsAPI => ({
list: (): Album[] => {
const records = derive();
return [...records.albums.values()]
.sort(byNewestAlbum)
.map((rec) => new Album(rec, records, saves, content));
},
byID: ({ collectionID }): Album | undefined => {
const records = derive();
const rec = records.albums.get(collectionID);
return rec ? new Album(rec, records, saves, content) : undefined;
},
byName: ({ albumName }): Album | undefined => {
const records = derive();
// Names are not unique in Ente; resolve a collision deterministically
// to the newest-updated album, matching `list` order.
const match = [...records.albums.values()]
.sort(byNewestAlbum)
.find((rec) => rec.name === albumName);
return match ? new Album(match, records, saves, content) : undefined;
},
});
export const makePhotosAPI = (
derive: () => DerivedRecords,
saves: SavePathLookup,
content?: PhotoContent,
): PhotosAPI => ({
byID: ({ fileID }): Photo | undefined => {
const records = derive();
const rec = records.photos.get(fileID);
const file = records.files.get(fileID);
return rec && file ? new Photo(rec, file, saves, content) : undefined;
},
records: ({ fileIDs }): PhotoRecord[] => {
const { photos } = derive();
const seen = new Set<number>();
const out: PhotoRecord[] = [];
for (const id of fileIDs) {
if (seen.has(id)) continue;
const rec = photos.get(id);
if (rec) {
out.push(rec);
seen.add(id);
}
}
return out;
},
});
export const makeTimelineAPI = (derive: () => DerivedRecords): TimelineAPI => ({
groups: ({ groupBy, filter }): TimelineGroup[] => {
const records = derive();
return groupPhotos(filterPhotos(records, filter), groupBy);
},
});
// Apply a `PhotoFilter` to the projection. Hidden photos are always dropped;
// archived photos are dropped unless `includeArchived` asks for them.
const filterPhotos = (
records: DerivedRecords,
filter?: PhotoFilter,
): PhotoRecord[] => {
const f = filter ?? {};
const includeArchived = f.includeArchived ?? false;
const needle = f.text?.toLowerCase();
const out: PhotoRecord[] = [];
for (const rec of records.photos.values()) {
if (rec.isHidden) continue;
if (rec.isArchived && !includeArchived) continue;
if (f.albumID !== undefined && !rec.albumIDs.includes(f.albumID))
continue;
if (f.fileTypes !== undefined && !f.fileTypes.includes(rec.fileType))
continue;
if (f.hasLocation !== undefined) {
const has =
rec.latitude !== undefined && rec.longitude !== undefined;
if (has !== f.hasLocation) continue;
}
if (needle !== undefined && !matchesText(rec, needle, records))
continue;
out.push(rec);
}
return out;
};
const matchesText = (
rec: PhotoRecord,
needle: string,
records: DerivedRecords,
): boolean => {
if (rec.title.toLowerCase().includes(needle)) return true;
if (rec.caption !== undefined && rec.caption.toLowerCase().includes(needle))
return true;
for (const id of rec.albumIDs) {
const album = records.albums.get(id);
if (album && album.name.toLowerCase().includes(needle)) return true;
}
return false;
};
// Bucket photos into periods, groups newest first, members newest first.
const groupPhotos = (
photos: PhotoRecord[],
groupBy: GroupBy,
): TimelineGroup[] => {
const buckets = new Map<
string,
{ startsAt: number; recs: PhotoRecord[] }
>();
for (const rec of photos) {
const { key, startsAt } = periodOf(rec.takenAt, groupBy);
const bucket = buckets.get(key);
if (bucket) bucket.recs.push(rec);
else buckets.set(key, { startsAt, recs: [rec] });
}
const groups: TimelineGroup[] = [];
for (const [key, bucket] of buckets) {
bucket.recs.sort(byNewest);
groups.push({
key,
startsAt: bucket.startsAt,
fileIDs: bucket.recs.map((r) => r.fileID),
});
}
groups.sort((a, b) => b.startsAt - a.startsAt);
return groups;
};
const pad = (n: number): string => String(n).padStart(2, "0");
const dateKey = (d: Date): string =>
`${d.getFullYear()}-${pad(d.getMonth() + 1)}-${pad(d.getDate())}`;
const WEEK_MS = 7 * 24 * 60 * 60 * 1000;
// The ISO 8601 week key `YYYY-Www` for the week starting at the given Monday.
// The week-year is the year of that week's Thursday, so it can differ from the
// calendar year at the January/December boundary (e.g. 2024-12-30 is 2025-W01).
const isoWeekKey = (monday: Date): string => {
const thursday = new Date(
monday.getFullYear(),
monday.getMonth(),
monday.getDate() + 3,
);
const isoYear = thursday.getFullYear();
// Thursday of ISO week 1 is the Thursday of the week containing January 4.
const jan4 = new Date(isoYear, 0, 4);
const week1Thursday = new Date(
isoYear,
0,
4 + 3 - ((jan4.getDay() + 6) % 7),
);
const week =
1 +
Math.round((thursday.getTime() - week1Thursday.getTime()) / WEEK_MS);
return `${isoYear}-W${pad(week)}`;
};
// The period a millisecond instant falls in, in local time. Weeks start on
// Monday. `Date` normalizes out-of-range day arguments, so the week's Monday
// is correct across month and year boundaries.
const periodOf = (
takenAt: number,
groupBy: GroupBy,
): { key: string; startsAt: number } => {
const d = new Date(takenAt);
const year = d.getFullYear();
const month = d.getMonth();
const day = d.getDate();
if (groupBy === "month") {
const start = new Date(year, month, 1);
return { key: `${year}-${pad(month + 1)}`, startsAt: start.getTime() };
}
if (groupBy === "week") {
// getDay(): 0=Sunday..6=Saturday; shift so Monday is the week start.
const fromMonday = (d.getDay() + 6) % 7;
const start = new Date(year, month, day - fromMonday);
return { key: isoWeekKey(start), startsAt: start.getTime() };
}
const start = new Date(year, month, day);
return { key: dateKey(start), startsAt: start.getTime() };
};
src/library/records.ts
+298
View File
@@ -0,0 +1,298 @@
// Plain records projected from the decrypted store, and the diff between two
// projections. These are the library's GUI-facing surface: they hold no key
// material and no binary, so they survive `structuredClone`/JSON across the
// Electron IPC boundary where methods and file keys cannot go (design #36,
// owner ruling 5). The decrypted `Collection`/`EnteFile` objects stay in RAM in
// the main process; the window only ever sees these records.
//
// Ente holds edited/basic times in microseconds; records expose `takenAt` and
// `modifiedAt` in milliseconds. The magic-metadata field names below are the
// ones the Ente clients write, confirmed against the repo's own fixtures:
// `w`/`h` in test/cli/metadata-backup.test.ts, `visibility` in
// test/library/store.test.ts.
import type {
Collection,
CollectionType,
EnteFile,
FileType,
} from "../model/types.js";
// Ente private-magic-metadata visibility values.
const VISIBILITY_ARCHIVED = 1;
const VISIBILITY_HIDDEN = 2;
// A single photo, deduplicated across the collections it belongs to. No key,
// no binary: safe to send to a window.
export interface PhotoRecord {
fileID: number;
// Every collection this file is a member of, ascending.
albumIDs: number[];
// `pubMagicMetadata.editedName` when the user renamed the file, else the
// basic-metadata title.
title: string;
// Milliseconds. `pubMagicMetadata.editedTime` when the user edited the
// date, else basic-metadata `creationTime`.
takenAt: number;
// Milliseconds. Basic-metadata `modificationTime`.
modifiedAt: number;
fileType: FileType;
caption?: string;
width?: number;
height?: number;
latitude?: number;
longitude?: number;
// The content hash the uploader recorded (`FileMetadata.hash`); files from
// very old clients have none.
hash?: string;
isArchived: boolean;
isHidden: boolean;
// Local cache paths, set once a later phase caches the bytes; unset here.
// A live photo's `originalPath` is its image.
thumbnailPath?: string;
originalPath?: string;
}
export interface AlbumRecord {
collectionID: number;
name: string;
// `favorites` identifies the account's favorites album.
type: CollectionType;
isShared: boolean;
updationTime: number;
// The album's files, newest first.
fileIDs: number[];
}
export interface LibrarySnapshot {
albums: AlbumRecord[];
photos: PhotoRecord[];
// Wall-clock milliseconds when the snapshot was taken.
takenAt: number;
}
// What one refresh or `refreshCollection()` changed in the records.
export interface LibraryChange {
// Full records for albums/photos added or changed.
albumsChanged: AlbumRecord[];
photosChanged: PhotoRecord[];
fileIDsRemoved: number[];
albumIDsRemoved: number[];
// Wall-clock milliseconds when the store changed.
refreshedAt: number;
}
// The by-id projection of the store at one moment; the source for both
// `snapshotFrom` (sorted arrays for the GUI) and `diffRecords` (change sets).
export interface DerivedRecords {
albums: Map<number, AlbumRecord>;
photos: Map<number, PhotoRecord>;
// The membership each photo's record is read from, for its `Photo`'s save
// path. It holds the file's key, so it stays in this process: no snapshot
// or change carries it.
files: Map<number, EnteFile>;
}
const asString = (v: unknown): string | undefined =>
typeof v === "string" && v.length > 0 ? v : undefined;
const asNumber = (v: unknown): number | undefined =>
typeof v === "number" && Number.isFinite(v) ? v : undefined;
const microsToMillis = (micros: number): number => Math.floor(micros / 1000);
// Newest first, with fileID as a stable tiebreak so equal-timed files order
// deterministically.
const byNewestPhoto = (a: PhotoRecord, b: PhotoRecord): number =>
b.takenAt - a.takenAt || b.fileID - a.fileID;
// A photo's `takenAt` in milliseconds: `pubMagicMetadata.editedTime` when the
// user edited the date, else basic-metadata `creationTime`.
export const takenAtOf = (file: EnteFile): number =>
microsToMillis(
asNumber(file.pubMagicMetadata?.editedTime) ??
file.metadata.creationTime,
);
// The membership a file's record is read from: the most recently synced, lowest
// collection id to break ties. Whatever dates a file's save path takes this
// membership too, so the path always carries the record's `takenAt`.
export const representative = (memberships: EnteFile[]): EnteFile =>
memberships.reduce((best, m) =>
m.updationTime > best.updationTime ||
(m.updationTime === best.updationTime &&
m.collectionID < best.collectionID)
? m
: best,
);
// Build one PhotoRecord from every membership of a file. The memberships share
// the same underlying file, so metadata is read from a single representative;
// `albumIDs` gathers them all.
const toPhotoRecord = (
fileID: number,
memberships: EnteFile[],
): PhotoRecord => {
const albumIDs = memberships
.map((m) => m.collectionID)
.sort((a, b) => a - b);
const rep = representative(memberships);
const pub = rep.pubMagicMetadata ?? {};
const priv = rep.magicMetadata ?? {};
const visibility = asNumber(priv.visibility);
const record: PhotoRecord = {
fileID,
albumIDs,
title: asString(pub.editedName) ?? rep.metadata.title,
takenAt: takenAtOf(rep),
modifiedAt: microsToMillis(rep.metadata.modificationTime),
fileType: rep.metadata.fileType,
isArchived: visibility === VISIBILITY_ARCHIVED,
isHidden: visibility === VISIBILITY_HIDDEN,
};
const caption = asString(pub.caption);
if (caption !== undefined) record.caption = caption;
const width = asNumber(pub.w);
if (width !== undefined) record.width = width;
const height = asNumber(pub.h);
if (height !== undefined) record.height = height;
if (rep.metadata.latitude !== undefined)
record.latitude = rep.metadata.latitude;
if (rep.metadata.longitude !== undefined)
record.longitude = rep.metadata.longitude;
if (rep.metadata.hash !== undefined) record.hash = rep.metadata.hash;
return record;
};
const toAlbumRecord = (
collection: Collection,
files: EnteFile[],
takenAtByFile: Map<number, number>,
): AlbumRecord => {
const fileIDs = files
.filter((f) => f.collectionID === collection.id)
.map((f) => f.id)
.sort(
(a, b) =>
(takenAtByFile.get(b) ?? 0) - (takenAtByFile.get(a) ?? 0) ||
b - a,
);
return {
collectionID: collection.id,
name: collection.name,
type: collection.type,
isShared: collection.isShared,
updationTime: collection.updationTime,
fileIDs,
};
};
// The cache paths known for a file, so the projection can expose them on the
// record without the read layer reaching into the content cache itself.
export type CachedPathLookup = (fileID: number) => {
originalPath?: string;
thumbnailPath?: string;
};
// Project the decrypted collections and file memberships into by-id records.
// `files` is every membership (a file appears once per collection it is in).
// `cachedPaths`, when given, fills each record's cache paths.
export const deriveRecords = (
collections: Collection[],
files: EnteFile[],
cachedPaths?: CachedPathLookup,
): DerivedRecords => {
const byFileID = new Map<number, EnteFile[]>();
for (const f of files) {
const arr = byFileID.get(f.id);
if (arr) arr.push(f);
else byFileID.set(f.id, [f]);
}
const photos = new Map<number, PhotoRecord>();
const photoFiles = new Map<number, EnteFile>();
const takenAtByFile = new Map<number, number>();
for (const [fileID, memberships] of byFileID) {
const record = toPhotoRecord(fileID, memberships);
if (cachedPaths) {
const paths = cachedPaths(fileID);
if (paths.originalPath !== undefined)
record.originalPath = paths.originalPath;
if (paths.thumbnailPath !== undefined)
record.thumbnailPath = paths.thumbnailPath;
}
photos.set(fileID, record);
photoFiles.set(fileID, representative(memberships));
takenAtByFile.set(fileID, record.takenAt);
}
const albums = new Map<number, AlbumRecord>();
for (const c of collections) {
albums.set(c.id, toAlbumRecord(c, files, takenAtByFile));
}
return { albums, photos, files: photoFiles };
};
// Sorted, GUI-ready arrays: albums newest updated first, photos newest first.
export const snapshotFrom = (
records: DerivedRecords,
takenAt: number,
): LibrarySnapshot => ({
albums: [...records.albums.values()].sort(
(a, b) =>
b.updationTime - a.updationTime || b.collectionID - a.collectionID,
),
photos: [...records.photos.values()].sort(byNewestPhoto),
takenAt,
});
// Records compare by value; they are plain and built with a fixed key order, so
// a serialized form is a sound equality key.
const same = (a: unknown, b: unknown): boolean =>
JSON.stringify(a) === JSON.stringify(b);
const diffMap = <T>(
prev: Map<number, T>,
next: Map<number, T>,
): { changed: T[]; removed: number[] } => {
const changed: T[] = [];
for (const [id, record] of next) {
const before = prev.get(id);
if (before === undefined || !same(before, record)) changed.push(record);
}
const removed: number[] = [];
for (const id of prev.keys()) if (!next.has(id)) removed.push(id);
removed.sort((a, b) => a - b);
return { changed, removed };
};
// The change between two projections, or undefined when nothing changed.
export const diffRecords = (
prev: DerivedRecords,
next: DerivedRecords,
refreshedAt: number,
): LibraryChange | undefined => {
const albums = diffMap(prev.albums, next.albums);
const photos = diffMap(prev.photos, next.photos);
if (
albums.changed.length === 0 &&
albums.removed.length === 0 &&
photos.changed.length === 0 &&
photos.removed.length === 0
) {
return undefined;
}
return {
albumsChanged: albums.changed,
photosChanged: photos.changed,
fileIDsRemoved: photos.removed,
albumIDsRemoved: albums.removed,
refreshedAt,
};
};
src/library/store.ts
+254
View File
@@ -0,0 +1,254 @@
// On-disk JSON metadata store for the local cache.
//
// The store keeps one `metadata.json` file holding the account's server
// state: the user id, a schema version, the cursor for the incremental
// collections listing, the decrypted collection and file records, and the
// collections and files that did not decrypt. The whole file is read
// into RAM on load and rewritten as a whole on save; there is no partial
// update and no lock file. A separate refresh unit populates the store from
// the server — this module only stores what it is given.
//
// The file is a cache, so it is never trusted to exist or to be intact: a
// missing or unreadable file loads as an empty store rather than an error, and
// the refresh unit then repopulates it.
import { mkdir, chmod, readFile } from "node:fs/promises";
import { dirname } from "node:path";
import { writeAtomic } from "../download/index.js";
import type { Collection, EnteFile, Microseconds } from "../model/types.js";
import { representative } from "./records.js";
// Bumped only when the on-disk shape changes incompatibly. A file written
// under a different version is discarded on load (see `load`): re-fetching
// from the server is always safe and cheaper than migrating a cache.
export const METADATA_SCHEMA_VERSION = 1;
// Directory and file modes match `session.json`: the records hold decrypted
// key material, so on a shared machine only the owner may read them.
const DIR_MODE = 0o700;
const FILE_MODE = 0o600;
// On-disk shapes. They mirror the in-memory model exactly except for the
// binary `key`, which JSON cannot hold and which is stored as base64.
type StoredCollection = Omit<Collection, "key"> & { key: string };
type StoredFile = Omit<EnteFile, "key"> & { key: string };
// A collection, or with `fileID` a file in it, that did not decrypt or parse,
// and why. A refresh or `refreshCollection()` records it on fetching such a
// copy. A later fetch drops it on getting a copy that decrypts, or on learning
// that the server deleted it. Each fetch asks only for what changed, so the
// store keeps the record between fetches.
export interface Undecryptable {
collectionID: number;
fileID?: number;
error: string;
}
interface StoredMetadata {
schemaVersion: number;
userID: number;
collectionsSinceTime: Microseconds;
collections: StoredCollection[];
files: StoredFile[];
undecryptable: Undecryptable[];
}
const encodeKey = (key: Uint8Array): string =>
Buffer.from(key).toString("base64");
const decodeKey = (encoded: string): Uint8Array =>
new Uint8Array(Buffer.from(encoded, "base64"));
// A file membership is identified by the pair (collectionID, fileID): the same
// underlying file can belong to several collections, each a distinct record
// with its own key.
const fileKey = (collectionID: number, fileID: number): string =>
`${collectionID}:${fileID}`;
export class MetadataStore {
readonly path: string;
readonly schemaVersion = METADATA_SCHEMA_VERSION;
userID = 0;
collectionsSinceTime: Microseconds = 0;
private readonly collections = new Map<number, Collection>();
private readonly files = new Map<string, EnteFile>();
private undecryptable: Undecryptable[] = [];
private constructor(path: string) {
this.path = path;
}
// Load the store at `path`. A missing file, an unreadable one, unparseable
// contents, or a mismatched schema version all yield an empty store bound
// to that path — never a thrown error, because the file is only a cache.
static async load(path: string): Promise<MetadataStore> {
const store = new MetadataStore(path);
let raw: string;
try {
raw = await readFile(path, "utf8");
} catch {
return store;
}
try {
const parsed = JSON.parse(raw) as StoredMetadata;
if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) {
return store;
}
store.userID = parsed.userID ?? 0;
store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0;
for (const stored of parsed.collections ?? []) {
const collection: Collection = {
...stored,
key: decodeKey(stored.key),
};
store.collections.set(collection.id, collection);
}
for (const stored of parsed.files ?? []) {
const file: EnteFile = {
...stored,
key: decodeKey(stored.key),
};
store.files.set(fileKey(file.collectionID, file.id), file);
}
for (const record of parsed.undecryptable) {
if (typeof record.collectionID !== "number") {
throw new Error("damaged record of what did not decrypt");
}
store.undecryptable.push(record);
}
} catch {
// Any corruption discards the partial result: a half-read cache is
// worse than an empty one, since the refresh unit will rebuild it.
return new MetadataStore(path);
}
return store;
}
// Rewrite the whole file. The directory is created 0700 and the file left
// 0600; the write itself is the download layer's durable atomic writer
// (temp file, fsync, rename, dir fsync), so a reader never sees a partial
// file and a crash cannot leave a truncated one. There is no lock file and
// no `sync()` beyond the writer's own fsyncs.
async save(): Promise<void> {
const model: StoredMetadata = {
schemaVersion: METADATA_SCHEMA_VERSION,
userID: this.userID,
collectionsSinceTime: this.collectionsSinceTime,
collections: [...this.collections.values()].map((c) => ({
...c,
key: encodeKey(c.key),
})),
files: [...this.files.values()].map((f) => ({
...f,
key: encodeKey(f.key),
})),
undecryptable: this.undecryptable,
};
const dir = dirname(this.path);
// chmod after mkdir so the mode is 0700 even when the directory
// already existed with a looser mode; mkdir alone would not tighten
// an existing directory.
await mkdir(dir, { recursive: true, mode: DIR_MODE });
await chmod(dir, DIR_MODE);
const payload = new TextEncoder().encode(
JSON.stringify(model, null, 2),
);
await writeAtomic(this.path, payload);
// The atomic writer's temp file inherits the default mode; tighten the
// renamed file to 0600. The 0700 directory already keeps other users
// out during the brief window before this runs.
await chmod(this.path, FILE_MODE);
}
getCollection(id: number): Collection | undefined {
return this.collections.get(id);
}
listCollections(): Collection[] {
return [...this.collections.values()];
}
putCollection(collection: Collection): void {
this.collections.set(collection.id, collection);
}
// Removing a collection also drops its file memberships, and any record
// that it or one of its files could not be decrypted: a file record is
// only meaningful as part of a collection the cache still knows about.
deleteCollection(id: number): void {
this.collections.delete(id);
for (const [key, file] of this.files) {
if (file.collectionID === id) {
this.files.delete(key);
}
}
this.undecryptable = this.undecryptable.filter(
(record) => record.collectionID !== id,
);
}
getFile(collectionID: number, fileID: number): EnteFile | undefined {
return this.files.get(fileKey(collectionID, fileID));
}
// The membership of a file its record is read from (`representative`), or
// undefined. Any membership could fetch the bytes, but the content cache
// resolves a fileID to a file this way so that it dates the save path
// from the same membership as `photo.savePath`.
getFileByID(fileID: number): EnteFile | undefined {
const memberships: EnteFile[] = [];
for (const file of this.files.values()) {
if (file.id === fileID) memberships.push(file);
}
return memberships.length > 0 ? representative(memberships) : undefined;
}
listFiles(collectionID: number): EnteFile[] {
return [...this.files.values()].filter(
(f) => f.collectionID === collectionID,
);
}
putFile(file: EnteFile): void {
this.files.set(fileKey(file.collectionID, file.id), file);
}
deleteFile(collectionID: number, fileID: number): void {
this.files.delete(fileKey(collectionID, fileID));
}
// Sorted by collection ID, then file ID, so the order does not depend on
// when each record was fetched.
listUndecryptable(): Undecryptable[] {
return [...this.undecryptable].sort(
(a, b) =>
a.collectionID - b.collectionID ||
(a.fileID ?? 0) - (b.fileID ?? 0),
);
}
// Replaces any earlier record of the same collection or file.
putUndecryptable(record: Undecryptable): void {
this.deleteUndecryptable(record.collectionID, record.fileID);
this.undecryptable.push(record);
}
// Forgets the record of the collection, or with `fileID` of that file in
// it. Returns whether there was one.
deleteUndecryptable(collectionID: number, fileID?: number): boolean {
const before = this.undecryptable.length;
this.undecryptable = this.undecryptable.filter(
(record) =>
record.collectionID !== collectionID ||
record.fileID !== fileID,
);
return this.undecryptable.length < before;
}
}
src/metadata-backup.ts
+162 -164
View File
@@ -1,176 +1,115 @@
import { gunzipSync } from "node:zlib";
import {
mkdirSync,
mkdtempSync,
readFileSync,
rmSync,
writeFileSync,
} from "node:fs";
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import * as jpeg from "jpeg-js";
import exifReader from "exif-reader";
import { clipKeywords } from "./clip-keywords.js";
import type { Client } from "./client.js";
import { decryptBlob, fromBase64 } from "./crypto/index.js";
import { describeError } from "./errors.js";
import { readExifTags } from "./exif.js";
import type { Library, Photo } from "./library/index.js";
import {
MAX_NAME_BYTES,
sanitizeFileName,
shortenToBytes,
} from "./filename.js";
import {
fetchMLDataBatch,
MLDATA_BATCH_SIZE,
type MLData,
} from "./mldata-fetch.js";
import type { EnteFile } from "./model/types.js";
export type ProgressCallback = (message: string) => void;
// Reports one line of the dump's progress. `level` is "error" when the line
// reports ML data that could not be read, and is left out for every other
// line.
export type ProgressCallback = (message: string, level?: "error") => void;
export interface MetadataBackupOptions {
exif?: boolean;
onProgress?: ProgressCallback;
}
const sanitizePath = (name: string): string =>
name.replace(/[/\\:*?"<>|]/g, "_").replace(/^\.+/, "_");
interface RawRemoteFileData {
fileID: number;
encryptedData: string;
decryptionHeader: string;
updatedAt?: number;
}
const fetchMLDataForFiles = async (
client: Client,
fileIDs: number[],
fileKeys: Map<number, Uint8Array>,
): Promise<Map<number, Record<string, unknown>>> => {
const api = client.getApiClient();
const result = new Map<number, Record<string, unknown>>();
const batchSize = 200;
for (let i = 0; i < fileIDs.length; i += batchSize) {
const batch = fileIDs.slice(i, i + batchSize);
const { data } = await api.postJSON<{ data: RawRemoteFileData[] }>(
"/files/data/fetch",
{ type: "mldata", fileIDs: batch },
);
for (const entry of data ?? []) {
const key = fileKeys.get(entry.fileID);
if (!key) continue;
try {
const decrypted = decryptBlob(
fromBase64(entry.encryptedData),
fromBase64(entry.decryptionHeader),
key,
);
const jsonStr = gunzipSync(Buffer.from(decrypted)).toString(
"utf-8",
);
result.set(entry.fileID, JSON.parse(jsonStr));
} catch {
// Corrupted ML data for this file; skip it
}
}
}
return result;
};
// Extract the raw EXIF APP1 segment from JPEG bytes. Returns the EXIF
// data buffer (starting after the APP1 length field, at the "Exif\0\0"
// header) or undefined if no APP1 marker is found.
const extractExifFromJpeg = (buf: Uint8Array): Buffer | undefined => {
if (buf[0] !== 0xff || buf[1] !== 0xd8) return undefined;
let offset = 2;
while (offset < buf.length - 1) {
if (buf[offset] !== 0xff) return undefined;
const marker = buf[offset + 1]!;
if (marker === 0xda) break; // start of scan, no more markers
if (offset + 3 >= buf.length) break;
const len = (buf[offset + 2]! << 8) | buf[offset + 3]!;
if (marker === 0xe1) {
// APP1 — check for "Exif\0\0" header
if (
buf[offset + 4] === 0x45 &&
buf[offset + 5] === 0x78 &&
buf[offset + 6] === 0x69 &&
buf[offset + 7] === 0x66
) {
return Buffer.from(
buf.buffer,
buf.byteOffset + offset + 4,
len - 2,
);
}
}
offset += 2 + len;
}
return undefined;
};
const extractImageMetadata = (
// Extract dimensions, EXIF and XMP from a file's bytes. `exif` is the EXIF tags
// exifreader returns, from any image format it reads. When it finds an EXIF
// block but reads no tag from it, the record keeps the block's bytes, base64,
// in `exifRaw`, with the reason in `exifError`.
export const extractImageMetadata = (
fileBytes: Uint8Array,
): Record<string, unknown> | undefined => {
const result: Record<string, unknown> = {};
// Try to get dimensions from JPEG decode
try {
const result: Record<string, unknown> = {};
// Try to get dimensions from JPEG decode
try {
const decoded = jpeg.decode(fileBytes, {
useTArray: true,
formatAsRGBA: false,
});
result.format = "jpeg";
result.width = decoded.width;
result.height = decoded.height;
} catch {
// Not a JPEG or corrupt; still try EXIF extraction
}
const exifBuf = extractExifFromJpeg(fileBytes);
if (exifBuf) {
try {
result.exif = exifReader(exifBuf);
} catch {
result.exifRaw = exifBuf.toString("base64");
}
}
// Extract XMP (look for "http://ns.adobe.com/xap" in the bytes)
const xmpStart = Buffer.from(fileBytes).indexOf("<?xpacket begin");
if (xmpStart !== -1) {
const xmpEnd = Buffer.from(fileBytes).indexOf(
"<?xpacket end",
xmpStart,
);
if (xmpEnd !== -1) {
const end = Buffer.from(fileBytes).indexOf("?>", xmpEnd);
result.xmp = Buffer.from(fileBytes)
.subarray(xmpStart, end !== -1 ? end + 2 : xmpEnd + 50)
.toString("utf-8");
}
}
return Object.keys(result).length > 0 ? result : undefined;
const decoded = jpeg.decode(fileBytes, {
useTArray: true,
formatAsRGBA: false,
});
result.format = "jpeg";
result.width = decoded.width;
result.height = decoded.height;
} catch {
return undefined;
// Not every original is a JPEG (PNG, HEIC, video), so a failed decode
// is expected and only means no dimensions; unreadable EXIF is still
// reported below through `exifError`.
}
const tags = readExifTags(fileBytes);
if (tags?.exif && Object.keys(tags.exif).length > 0) {
result.exif = tags.exif;
} else if (tags?.exif) {
const block = tags.metadataRange?.blocks.find((b) => b.type === "exif");
if (block) {
result.exifRaw = Buffer.from(
fileBytes.subarray(block.start, block.end),
).toString("base64");
}
result.exifError = "no tag could be read from the EXIF block";
}
// Extract XMP (look for "http://ns.adobe.com/xap" in the bytes)
const xmpStart = Buffer.from(fileBytes).indexOf("<?xpacket begin");
if (xmpStart !== -1) {
const xmpEnd = Buffer.from(fileBytes).indexOf(
"<?xpacket end",
xmpStart,
);
if (xmpEnd !== -1) {
const end = Buffer.from(fileBytes).indexOf("?>", xmpEnd);
result.xmp = Buffer.from(fileBytes)
.subarray(xmpStart, end !== -1 ? end + 2 : xmpEnd + 50)
.toString("utf-8");
}
}
return Object.keys(result).length > 0 ? result : undefined;
};
// Read a file's original bytes through the library's content cache and extract
// its embedded image metadata. The bytes come from `photo.original()` — the
// same on-disk cache the rest of the library fills — rather than a fresh
// per-call download to a throwaway temp file. For a live photo, its `path` is
// the image.
const extractExif = async (
client: Client,
file: EnteFile,
photo: Photo,
): Promise<Record<string, unknown> | undefined> => {
const tmpDir = mkdtempSync(join(tmpdir(), "quak-exif-"));
try {
const origPath = join(tmpDir, "original");
await client.downloadFile(file, origPath);
const fileBytes = new Uint8Array(readFileSync(origPath));
return extractImageMetadata(fileBytes);
} catch {
return undefined;
} finally {
rmSync(tmpDir, { recursive: true, force: true });
}
const { path } = await photo.original();
const fileBytes = new Uint8Array(readFileSync(path));
return extractImageMetadata(fileBytes);
};
// Dump every decrypted metadata layer the account holds into a directory tree
// of plain JSON: account, per-collection, and per-file records including the
// private and public magic metadata and the ML data, with the keywords of its
// CLIP embedding in place of the embedding (see clip-keywords.ts). Collections
// and files are enumerated from the library's cache, which the caller refreshes
// first. Returns how many files have no ML data because their ML data request
// failed or their ML data did not decrypt, gunzip or parse; they are still
// written, with `mlDataError` in place of `mlData`.
export const runMetadataBackup = async (
lib: Library,
client: Client,
outDir: string,
opts?: MetadataBackupOptions,
): Promise<void> => {
): Promise<{ filesWithMLDataError: number }> => {
const log = opts?.onProgress ?? (() => {});
const wantExif = opts?.exif ?? false;
@@ -184,14 +123,28 @@ export const runMetadataBackup = async (
);
log("Fetching collections...");
const collections = await client.listCollections();
const allFiles: { file: EnteFile; colDirName: string }[] = [];
// Enumerate through the library's read surface. Each album carries its
// photos, but the full decrypted `Collection`/`EnteFile` records (with the
// magic-metadata layers this dump exists to preserve) come from the
// library's by-id accessors.
const allFiles: { file: EnteFile; photo: Photo; colDirName: string }[] = [];
const fileKeys = new Map<number, Uint8Array>();
const seenFileIDs = new Set<number>();
for (const col of collections) {
const dirName = `${col.id}-${sanitizePath(col.name || "unnamed")}`;
for (const album of lib.albums.list()) {
const col = lib.getCollection(album.collectionID);
if (!col) continue;
// The album's ID keeps the folders apart, so the name after it can be
// cut short to fit in a file name.
const prefix = `${col.id}-`;
const dirName =
prefix +
shortenToBytes(
sanitizeFileName(col.name, "unnamed"),
MAX_NAME_BYTES - prefix.length,
);
const colDir = join(outDir, "collections", dirName);
mkdirSync(colDir, { recursive: true });
@@ -215,11 +168,13 @@ export const runMetadataBackup = async (
);
log(`[${col.name}] Fetching files...`);
const files = await client.listFiles(col.id, col.key);
log(`[${col.name}] ${files.length} file(s)`);
const photos = album.photos.list();
log(`[${col.name}] ${photos.length} file(s)`);
for (const file of files) {
allFiles.push({ file, colDirName: dirName });
for (const photo of photos) {
const file = lib.getFile(col.id, photo.fileID);
if (!file) continue;
allFiles.push({ file, photo, colDirName: dirName });
if (!seenFileIDs.has(file.id)) {
fileKeys.set(file.id, file.key);
seenFileIDs.add(file.id);
@@ -227,16 +182,43 @@ export const runMetadataBackup = async (
}
}
// One failed request (retries exhausted) must not end the dump: its files
// get the reason in `mlDataError` and the other batches go on. So does a
// file whose ML data does not decrypt, gunzip or parse.
log("Fetching ML data (face detections, CLIP embeddings)...");
const mlDataMap = await fetchMLDataForFiles(
client,
[...fileKeys.keys()],
fileKeys,
);
const mlDataMap = new Map<number, MLData>();
const mlDataErrors = new Map<number, string>();
const fileIDs = [...fileKeys.keys()];
for (let i = 0; i < fileIDs.length; i += MLDATA_BATCH_SIZE) {
const batch = fileIDs.slice(i, i + MLDATA_BATCH_SIZE);
try {
const { payloads, errors } = await fetchMLDataBatch(
client.getApiClient(),
batch,
fileKeys,
);
for (const [id, payload] of payloads) mlDataMap.set(id, payload);
for (const [id, err] of errors) {
const reason = describeError(err);
log(
`ML data of file ${id} could not be read: ${reason}`,
"error",
);
mlDataErrors.set(id, reason);
}
} catch (err) {
const reason = err instanceof Error ? err.message : String(err);
log(
`ML data request for ${batch.length} file(s) failed: ${reason}`,
"error",
);
for (const id of batch) mlDataErrors.set(id, reason);
}
}
log(`Got ML data for ${mlDataMap.size} file(s)`);
const writtenFileIDs = new Set<number>();
for (const { file, colDirName } of allFiles) {
for (const { file, photo, colDirName } of allFiles) {
const colDir = join(outDir, "collections", colDirName);
const fileMeta: Record<string, unknown> = {
@@ -250,13 +232,28 @@ export const runMetadataBackup = async (
if (file.pubMagicMetadata)
fileMeta.pubMagicMetadata = file.pubMagicMetadata;
// The ML data without the CLIP embedding, and the keywords that
// embedding gives.
const ml = mlDataMap.get(file.id);
if (ml) fileMeta.mlData = ml;
if (ml) {
const { clip: _clip, ...mlDataWithoutClip } = ml;
fileMeta.mlData = mlDataWithoutClip;
const isVideo = file.metadata.fileType === "video";
const keywords = clipKeywords(ml, isVideo);
if (keywords) fileMeta.keywords = keywords;
}
const mlError = mlDataErrors.get(file.id);
if (mlError) fileMeta.mlDataError = mlError;
if (wantExif && !writtenFileIDs.has(file.id)) {
log(`[${file.metadata.title}] Extracting EXIF...`);
const exifData = await extractExif(client, file);
if (exifData) fileMeta.imageMetadata = exifData;
try {
const exifData = await extractExif(photo);
if (exifData) fileMeta.imageMetadata = exifData;
} catch (err) {
fileMeta.imageMetadataError =
err instanceof Error ? err.message : String(err);
}
}
writtenFileIDs.add(file.id);
@@ -267,4 +264,5 @@ export const runMetadataBackup = async (
}
log("Metadata backup complete.");
return { filesWithMLDataError: mlDataErrors.size };
};
src/mldata-fetch.ts
+79
View File
@@ -0,0 +1,79 @@
// Fetch and decrypt Ente's per-file machine-learning data ("magic" search
// data: face detections + CLIP embeddings).
//
// The data lives behind `/files/data/fetch` with `type: "mldata"`. Each entry
// comes back encrypted under the file's own key and gzipped; decrypting and
// gunzipping yields the JSON payload
// `{ face: { faces: [...] }, clip: { embedding } }`. Ente caps a request at 200
// ids, so callers that want many at once split them into batches of
// `MLDATA_BATCH_SIZE` and call `fetchMLDataBatch` once per batch.
import { gunzipSync } from "node:zlib";
import type { ApiClient } from "./api/client.js";
import { decryptBlob, fromBase64 } from "./crypto/index.js";
// The most ids one `/files/data/fetch` request may carry.
export const MLDATA_BATCH_SIZE = 200;
// The decrypted, gunzipped per-file payload. Its concrete shape is Ente's; the
// store keeps the whole object verbatim and each consumer reads the fields it
// needs, so it stays an open record rather than a fixed interface.
export type MLData = Record<string, unknown>;
interface RawRemoteFileData {
fileID: number;
encryptedData: string;
decryptionHeader: string;
updatedAt?: number;
}
// Decrypt one entry with its file key and gunzip the JSON payload. Throws why
// when the entry does not decrypt, gunzip or parse, or parses to something
// other than a JSON object, such as `null`, a number or an array.
const decodeEntry = (entry: RawRemoteFileData, key: Uint8Array): MLData => {
const decrypted = decryptBlob(
fromBase64(entry.encryptedData),
fromBase64(entry.decryptionHeader),
key,
);
const json = gunzipSync(Buffer.from(decrypted)).toString("utf-8");
const payload: unknown = JSON.parse(json);
if (
typeof payload !== "object" ||
payload === null ||
Array.isArray(payload)
) {
throw new Error("not a JSON object");
}
return payload as MLData;
};
// Fetch ML data for up to `MLDATA_BATCH_SIZE` ids in a single request. This is
// the unit the request pools schedule; callers with more ids split them into
// batches and submit each batch to the pool. Returns the payload of each file
// whose entry decodes and, in `errors`, why not for each file whose entry does
// not, so one corrupt file never fails a whole batch. An entry for a file
// whose key is not known, which quak did not ask for, is skipped.
export const fetchMLDataBatch = async (
api: ApiClient,
fileIDs: number[],
fileKeys: Map<number, Uint8Array>,
): Promise<{ payloads: Map<number, MLData>; errors: Map<number, unknown> }> => {
const { data } = await api.postJSON<{ data: RawRemoteFileData[] }>(
"/files/data/fetch",
{ type: "mldata", fileIDs },
);
const payloads = new Map<number, MLData>();
const errors = new Map<number, unknown>();
for (const entry of data ?? []) {
const key = fileKeys.get(entry.fileID);
if (!key) continue;
try {
payloads.set(entry.fileID, decodeEntry(entry, key));
} catch (err) {
errors.set(entry.fileID, err);
}
}
return { payloads, errors };
};
src/model/decrypt.ts
+42 -4
View File
@@ -34,6 +34,28 @@ const FILE_TYPE_MAP: Record<number, FileType> = {
const parseFileType = (n: number): FileType => FILE_TYPE_MAP[n] ?? "unknown";
// The hash the uploading client recorded for the original's bytes, read the
// way the upstream client's `metadataHash` reads it: `hash` if present,
// otherwise, for a live photo from an older client that wrote the two parts
// separately, `<imageHash>:<videoHash>`. A field that is not a non-empty
// string counts as absent, and a file with no hash at all is normal.
const expectedHash = (json: Record<string, unknown>): string | undefined => {
const text = (v: unknown): string | undefined =>
typeof v === "string" && v !== "" ? v : undefined;
const hash = text(json.hash);
if (hash !== undefined) return hash;
const imageHash = text(json.imageHash);
const videoHash = text(json.videoHash);
if (
json.fileType === 2 &&
imageHash !== undefined &&
videoHash !== undefined
) {
return `${imageHash}:${videoHash}`;
}
return undefined;
};
export const decryptCollection = (
raw: RawCollection,
keys: KeyMaterial,
@@ -98,15 +120,24 @@ export const decryptFile = (
key,
);
const metadataJSON = JSON.parse(new TextDecoder().decode(metadataBytes));
if (
typeof metadataJSON !== "object" ||
metadataJSON === null ||
Array.isArray(metadataJSON)
) {
throw new Error(`file ${raw.id}: metadata is not a JSON object`);
}
const metadata: FileMetadata = {
title: metadataJSON.title ?? "",
// The server controls this JSON: a title that is missing or not a
// string becomes "", never an arbitrary value.
title: typeof metadataJSON.title === "string" ? metadataJSON.title : "",
fileType: parseFileType(metadataJSON.fileType ?? -1),
creationTime: metadataJSON.creationTime ?? 0,
modificationTime: metadataJSON.modificationTime ?? 0,
latitude: metadataJSON.latitude,
longitude: metadataJSON.longitude,
hash: metadataJSON.hash,
hash: expectedHash(metadataJSON),
};
const magicMetadata = decryptMagicMetadata(raw.magicMetadata, key);
@@ -120,9 +151,16 @@ export const decryptFile = (
metadata,
magicMetadata,
pubMagicMetadata,
file: { decryptionHeader: raw.file.decryptionHeader },
thumbnail: { decryptionHeader: raw.thumbnail.decryptionHeader },
file: {
decryptionHeader: raw.file.decryptionHeader,
size: raw.info?.fileSize,
},
thumbnail: {
decryptionHeader: raw.thumbnail.decryptionHeader,
size: raw.info?.thumbSize,
},
updationTime: raw.updationTime,
isDeleted: raw.isDeleted,
};
};
src/model/types.ts
+6
View File
@@ -29,6 +29,9 @@ export interface FileMetadata {
modificationTime: Microseconds;
latitude?: number;
longitude?: number;
// The content hash the uploader recorded (see `expectedHash` in
// decrypt.ts); `downloadFile` refuses an original that does not match it.
// Absent for files from very old clients.
hash?: string;
}
@@ -48,6 +51,9 @@ export interface EnteFile {
file: FileBlob;
thumbnail: FileBlob;
updationTime: Microseconds;
// Set from the diff row's flag. Live files decode with it absent/false;
// deleted rows are filtered out before decryptFile, so it is not set here.
isDeleted?: boolean;
}
// The key material a logged-in client holds, everything needed to decrypt
src/retry.ts
+71 -25
View File
@@ -10,7 +10,7 @@
// declining to retry a transient one costs a single file that the next run
// picks up anyway. When the evidence is ambiguous, fail fast.
import { ApiError, TruncatedStreamError } from "./errors.js";
import { ApiError, causeChain, TruncatedStreamError } from "./errors.js";
export interface RetryOptions {
// Total calls, not retries: `attempts: 1` disables retrying.
@@ -24,6 +24,9 @@ export interface RetryOptions {
sleep?: (ms: number) => Promise<void>;
// Injected so the jitter is reproducible under test.
random?: () => number;
// Called before each wait with the error that failed attempt number
// `attempt` and how long the wait is, for a caller that logs retries.
onRetry?: (err: unknown, attempt: number, delayMs: number) => void;
}
export type ResolvedRetryOptions = Required<RetryOptions>;
@@ -35,6 +38,17 @@ export const DEFAULT_RETRY_OPTIONS: ResolvedRetryOptions = {
sleep: (ms: number): Promise<void> =>
new Promise((resolve) => setTimeout(resolve, ms)),
random: Math.random,
onRetry: () => {},
};
// For a run nobody is watching, such as `quak backup` from cron. A request that
// keeps failing waits at most 243 s in all before it gives up, usually about
// half that, since each wait is drawn at random below its ceiling.
export const UNATTENDED_RETRY_OPTIONS: ResolvedRetryOptions = {
...DEFAULT_RETRY_OPTIONS,
attempts: 10,
baseDelayMs: 1_000,
maxDelayMs: 60_000,
};
export const resolveRetryOptions = (
@@ -45,9 +59,18 @@ export const resolveRetryOptions = (
maxDelayMs: opts?.maxDelayMs ?? DEFAULT_RETRY_OPTIONS.maxDelayMs,
sleep: opts?.sleep ?? DEFAULT_RETRY_OPTIONS.sleep,
random: opts?.random ?? DEFAULT_RETRY_OPTIONS.random,
onRetry: opts?.onRetry ?? DEFAULT_RETRY_OPTIONS.onRetry,
});
// Transport failures: the request did not complete, for reasons below HTTP.
//
// The errnos are Node's. The binary `make install` builds runs under bun,
// whose `fetch` rejects with a plain `Error` that has no `cause` and a `code`
// of bun's own. Bun gives `ConnectionRefused` for a refused connection and for
// a TLS handshake that failed, and `ConnectionRefused` or `FailedToOpenSocket`
// for a name that does not resolve. For a connection reset or closed before
// the response ended, it gives Node's `ECONNRESET`. These were found by
// running that binary, built with bun 1.3.13, against listeners on 127.0.0.1.
const TRANSPORT_CODES = new Set([
"ECONNRESET",
"ECONNABORTED",
@@ -60,6 +83,9 @@ const TRANSPORT_CODES = new Set([
"ENETUNREACH",
"ENETRESET",
"ENETDOWN",
// Bun's.
"ConnectionRefused",
"FailedToOpenSocket",
]);
// The subset of the above that can only be reported before a TCP connection
@@ -77,31 +103,36 @@ const TRANSPORT_CODES = new Set([
// ambiguity this set exists to exclude. They stay in `TRANSPORT_CODES`, so
// they remain retryable for idempotent calls; only replay eligibility is
// narrowed. See `isSafeToReplay`.
//
// Bun's `ConnectionRefused` and `FailedToOpenSocket` are absent too, so a
// `POST` or `PUT` is not replayed on them. When a connection bun kept open from
// an earlier request closes before any answer, bun sends the request again on
// a new connection, and if that one fails it gives one of these codes although
// the server may already have the request.
const CONNECT_CODES = new Set(["ENOTFOUND", "EAI_AGAIN", "ECONNREFUSED"]);
// `cause` is an arbitrary user-settable property and nothing prevents it from
// forming a cycle, so the walk is bounded. Hanging the process would be a
// worse outcome than any misclassification.
const MAX_CAUSE_DEPTH = 8;
// Collect every `code` in an error's cause chain. undici does not put the
// errno on the error it throws — it hangs the underlying socket error off
// `cause`, sometimes more than one level down — so a classifier that only read
// the top-level error would see a bare `Error` and call every dropped
// connection permanent.
const causeCodes = (err: unknown): string[] => {
// connection permanent. `complete` is false when the walk stopped at the
// depth limit with more of the chain still below it.
const causeCodes = (err: unknown): { codes: string[]; complete: boolean } => {
const { chain, complete } = causeChain(err);
const codes: string[] = [];
let current: unknown = err;
for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
if (current === null || typeof current !== "object") break;
const { code, cause } = current as { code?: unknown; cause?: unknown };
for (const link of chain) {
const { code } = link as { code?: unknown };
if (typeof code === "string") codes.push(code);
if (cause === current) break;
current = cause;
}
return codes;
return { codes, complete };
};
// Does the error's cause chain hold the code of a transport failure, one of
// `TRANSPORT_CODES`? `isRetryable` retries such an error, and `classify` in
// `src/backup.ts` calls it transient.
export const hasTransportCode = (err: unknown): boolean =>
causeCodes(err).codes.some((code) => TRANSPORT_CODES.has(code));
const isAbort = (err: unknown): boolean => {
if (err === null || typeof err !== "object") return false;
const { name } = err as { name?: unknown };
@@ -143,17 +174,18 @@ export const isRetryable = (err: unknown): boolean => {
// deliberately literal. Demanding a recognised `cause` instead would
// classify real network failures as permanent and fail backups that should
// have succeeded; the cost of the imprecision is bounded by the attempt
// count.
// count. Bun's fetch rejects with a plain `Error` instead, which is
// recognised by its code (see `TRANSPORT_CODES`).
if (err instanceof TypeError) return true;
return causeCodes(err).some((code) => TRANSPORT_CODES.has(code));
return hasTransportCode(err);
};
// Could the first attempt already have taken effect on the server?
//
// `isRetryable` is the wrong question for a request that changes state.
// quak's non-idempotent calls are `/users/srp/create-session`,
// `/users/two-factor/verify` — which consumes one of a small number of 2FA
// attempts — and `/files/thumbnail`. They are replayed only on the failures in
// `postJSON` and `putJSON` use this for every `POST` and `PUT` listed in the
// README under "Endpoints used"; verifying a second factor, for one, consumes
// one of a small number of attempts. They are replayed only on the failures in
// `CONNECT_CODES`, which establish that no TCP connection to the server ever
// existed: there was no address to connect to, or the peer refused the
// connection outright. A request byte cannot have been transmitted, so the
@@ -161,10 +193,22 @@ export const isRetryable = (err: unknown): boolean => {
//
// Everything else is ambiguous. A 5xx proves the server did process the
// request. A reset or a broken pipe can arrive after it was fully sent and
// acted on. A routing errno can be delivered on an established socket. A
// deadline says nothing at all about the server's state.
export const isSafeToReplay = (err: unknown): boolean =>
isRetryable(err) && causeCodes(err).some((code) => CONNECT_CODES.has(code));
// acted on. A routing errno can be delivered on an established socket. Bun's
// `ConnectionRefused` and `FailedToOpenSocket` can come from a new connection
// bun opened to resend a request that already went out on a connection it kept
// open. A deadline says nothing at all about the server's state. So every code in the
// cause chain must be one of `CONNECT_CODES`: one other code anywhere in the
// chain is doubt, and doubt is not replayed. A chain longer than the walk is
// doubt too: the links below the limit were never read.
export const isSafeToReplay = (err: unknown): boolean => {
const { codes, complete } = causeCodes(err);
return (
isRetryable(err) &&
complete &&
codes.length > 0 &&
codes.every((code) => CONNECT_CODES.has(code))
);
};
export interface WithRetryOptions extends RetryOptions {
isRetryable?: (err: unknown) => boolean;
@@ -195,7 +239,9 @@ export const withRetry = async <T>(
// and its status intact.
throw err;
}
await policy.sleep(backoffMs(attempt, policy));
const delayMs = backoffMs(attempt, policy);
policy.onRetry(err, attempt, delayMs);
await policy.sleep(delayMs);
}
}
};
src/thumbnails.ts
+202 -82
View File
@@ -2,16 +2,26 @@ import { createHash } from "node:crypto";
import { readFileSync } from "node:fs";
import * as jpeg from "jpeg-js";
import type { Client } from "./client.js";
import type { Library } from "./library/index.js";
import { ApiError } from "./api/client.js";
import { encryptBlob, toBase64 } from "./crypto/index.js";
import { downloadFile } from "./download/index.js";
import type { EnteFile } from "./model/types.js";
import { mkdtempSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
const THUMB_MAX_DIMENSION = 720;
const THUMB_JPEG_QUALITY = 50;
// The server refuses a thumbnail larger than the one it already records for the
// file (`thumbnail.size`, the encrypted size), so these encodings are tried
// from largest to smallest and the first that fits is uploaded.
const THUMB_ENCODINGS = [
{ maxDimension: 720, quality: 50 },
{ maxDimension: 720, quality: 30 },
{ maxDimension: 480, quality: 30 },
{ maxDimension: 320, quality: 20 },
{ maxDimension: 160, quality: 20 },
];
// The server accepts a new thumbnail only from the file's owner, so files other
// people own in albums shared with this account are never checked or repaired.
const NOT_OWNED_REASON =
"owned by another account (only the owner can replace its thumbnail)";
export interface MissingThumbnailInfo {
fileID: number;
@@ -20,34 +30,64 @@ export interface MissingThumbnailInfo {
reason: string;
}
// Three outcomes, not two. "fixed": a thumbnail was generated and uploaded.
// "failed": something went wrong (download, encode, upload) and the file still
// has no thumbnail. "skipped": the server would refuse any thumbnail for the
// file or this helper cannot regenerate it — a file another account owns, a
// recorded thumbnail size nothing fits within, a video, or an image that is
// not a baseline JPEG. Skipped is a deliberate, expected outcome, not an error
// (issue #17): the repair path is JPEG-only because `jpeg-js` is, and a PNG or
// HEIC is left for a format-aware tool rather than reported as a failure.
export type ThumbnailFixStatus = "fixed" | "skipped" | "failed";
export interface ThumbnailFixResult {
fileID: number;
title: string;
collection: string;
success: boolean;
error?: string;
status: ThumbnailFixStatus;
// Why the file was skipped or failed; unset when it was fixed.
reason?: string;
}
export type ProgressCallback = (message: string) => void;
// Reports one line of progress. `level` is "error" when the line reports a
// file that could not be checked or fixed, and is left out for every other
// line.
export type ProgressCallback = (message: string, level?: "error") => void;
// Enumerate every file the library knows about, newest album first, each file
// once, and report those whose server-side thumbnail is missing. "Missing" is
// only two answers: an empty body, or a 404. Any other error reaching this
// point has already exhausted its retries — a failing server, a dropped
// connection, a deadline — and says nothing about whether the thumbnail
// exists, so it is logged and the file is left unreported. That distinction is
// what stops `fix-missing-thumbnails` from regenerating and uploading over
// thumbnails that were fine all along while the CDN was briefly returning 500s.
// Files another account owns are logged as skipped and not checked.
export const listMissingThumbnails = async (
lib: Library,
client: Client,
onProgress?: ProgressCallback,
): Promise<MissingThumbnailInfo[]> => {
const log = onProgress ?? (() => {});
const api = client.getApiClient();
const { userID } = client.whoami();
const missing: MissingThumbnailInfo[] = [];
const seen = new Set<number>();
const collections = await client.listCollections();
for (const col of collections) {
log(`[${col.name}] Checking thumbnails...`);
const files = await client.listFiles(col.id, col.key);
for (const file of files) {
if (seen.has(file.id)) continue;
seen.add(file.id);
for (const album of lib.albums.list()) {
log(`[${album.name}] Checking thumbnails...`);
for (const photo of album.photos.list()) {
if (seen.has(photo.fileID)) continue;
seen.add(photo.fileID);
const file = lib.getFile(album.collectionID, photo.fileID);
if (file && file.ownerID !== userID) {
log(
`[${album.name}] Skipping ${photo.title}: ${NOT_OWNED_REASON}`,
);
continue;
}
try {
const api = client.getApiClient();
const stream = await api.getThumbnailStream(file.id);
const stream = await api.getThumbnailStream(photo.fileID);
const reader = stream.getReader();
let totalBytes = 0;
for (;;) {
@@ -57,35 +97,24 @@ export const listMissingThumbnails = async (
}
if (totalBytes === 0) {
missing.push({
fileID: file.id,
title: file.metadata.title,
collection: col.name,
fileID: photo.fileID,
title: photo.title,
collection: album.name,
reason: "empty thumbnail (0 bytes)",
});
}
} catch (err) {
// A 404 is the server stating the thumbnail is not there:
// that, and an empty body, are the only two answers that mean
// "missing". Anything else reaching this point is a failure
// that already exhausted its retries — a failing server, a
// dropped connection, a deadline — and says nothing about
// whether the thumbnail exists.
//
// The distinction is what stops `helper
// fix-missing-thumbnails` from downloading originals,
// regenerating thumbnails and uploading them over thumbnails
// that were fine all along, because the CDN was briefly
// returning 500s while this ran.
if (err instanceof ApiError && err.status === 404) {
missing.push({
fileID: file.id,
title: file.metadata.title,
collection: col.name,
fileID: photo.fileID,
title: photo.title,
collection: album.name,
reason: "thumbnail not found (HTTP 404)",
});
} else {
log(
`[${col.name}] Could not check ${file.metadata.title}: ${err instanceof Error ? err.message : String(err)} (not reported as missing)`,
`[${album.name}] Could not check ${photo.title}: ${err instanceof Error ? err.message : String(err)} (not reported as missing)`,
"error",
);
}
}
@@ -94,7 +123,7 @@ export const listMissingThumbnails = async (
return missing;
};
// Bilinear resize of RGBA pixel buffer
// Bilinear resize of an RGBA pixel buffer.
const resizeRGBA = (
src: Uint8Array,
srcW: number,
@@ -133,17 +162,13 @@ const resizeRGBA = (
return dst;
};
const generateThumbnail = (fileBytes: Uint8Array): Uint8Array => {
const decoded = jpeg.decode(fileBytes, {
useTArray: true,
formatAsRGBA: true,
});
const generateThumbnail = (
decoded: { data: Uint8Array; width: number; height: number },
maxDimension: number,
quality: number,
): Uint8Array => {
const { width: srcW, height: srcH } = decoded;
const scale = Math.min(
THUMB_MAX_DIMENSION / srcW,
THUMB_MAX_DIMENSION / srcH,
1,
);
const scale = Math.min(maxDimension / srcW, maxDimension / srcH, 1);
const dstW = Math.round(srcW * scale);
const dstH = Math.round(srcH * scale);
@@ -156,12 +181,59 @@ const generateThumbnail = (fileBytes: Uint8Array): Uint8Array => {
const encoded = jpeg.encode(
{ data: pixels, width: dstW, height: dstH },
THUMB_JPEG_QUALITY,
quality,
);
return new Uint8Array(encoded.data);
};
// A baseline/JFIF JPEG starts with the SOI marker 0xFFD8. `jpeg-js` decodes
// only JPEG, so this signature check is what separates a file the helper can
// regenerate from one it must skip: a PNG, HEIC, or the odd non-image byte
// stream all fail this and are reported as skipped rather than crashing the
// decoder into an opaque failure (issue #17).
const isJpeg = (bytes: Uint8Array): boolean =>
bytes.length >= 2 && bytes[0] === 0xff && bytes[1] === 0xd8;
// The reason a file cannot have a JPEG thumbnail regenerated for it, known from
// its record alone before any bytes are fetched, or undefined when it might. A
// still image still has to be checked against its actual bytes once
// downloaded.
const reasonToSkip = (file: EnteFile, userID: number): string | undefined => {
if (file.ownerID !== userID) {
return NOT_OWNED_REASON;
}
if (file.metadata.fileType !== "image") {
return `unsupported file type: ${file.metadata.fileType} (only JPEG images can be regenerated)`;
}
if (!file.thumbnail.size) {
return `recorded thumbnail size is ${file.thumbnail.size ?? "unknown"} (the server refuses a thumbnail larger than the one it records)`;
}
return undefined;
};
// Encrypt the largest encoding of the decoded image whose ciphertext is no
// larger than `maxSize`, or return undefined when even the smallest is larger.
const encryptThumbnailWithin = (
decoded: { data: Uint8Array; width: number; height: number },
key: Uint8Array,
maxSize: number,
): { header: Uint8Array; ciphertext: Uint8Array } | undefined => {
for (const { maxDimension, quality } of THUMB_ENCODINGS) {
const thumbJpeg = generateThumbnail(decoded, maxDimension, quality);
const encrypted = encryptBlob(thumbJpeg, key);
if (encrypted.ciphertext.length <= maxSize) return encrypted;
}
return undefined;
};
// Regenerate and upload a thumbnail for each requested file. Originals are read
// through the library's content cache (`photo.original()`); the generated
// thumbnail is JPEG-encoded, encrypted under the file's own key, and registered
// with the server — the encrypt-and-upload path is unchanged. Each file is
// resolved to one outcome (fixed / skipped / failed) and a failure on one file
// never stops the others.
export const fixMissingThumbnails = async (
lib: Library,
client: Client,
fileIDs: number[],
onProgress?: ProgressCallback,
@@ -169,20 +241,26 @@ export const fixMissingThumbnails = async (
const log = onProgress ?? (() => {});
const results: ThumbnailFixResult[] = [];
const api = client.getApiClient();
const { userID } = client.whoami();
const collections = await client.listCollections();
// Resolve each requested fileID to its file record and owning album by
// enumerating the library, each file taken from the first album that holds
// it. The raw `EnteFile` carries the per-file key the thumbnail is
// encrypted under, which the projected records deliberately do not.
const wanted = new Set(fileIDs);
const fileMap = new Map<
number,
{ file: EnteFile; collectionName: string }
>();
for (const col of collections) {
const files = await client.listFiles(col.id, col.key);
for (const file of files) {
if (fileIDs.includes(file.id) && !fileMap.has(file.id)) {
fileMap.set(file.id, {
for (const album of lib.albums.list()) {
for (const photo of album.photos.list()) {
if (!wanted.has(photo.fileID) || fileMap.has(photo.fileID))
continue;
const file = lib.getFile(album.collectionID, photo.fileID);
if (file) {
fileMap.set(photo.fileID, {
file,
collectionName: col.name,
collectionName: album.name,
});
}
}
@@ -195,33 +273,78 @@ export const fixMissingThumbnails = async (
fileID,
title: "unknown",
collection: "unknown",
success: false,
error: "file not found in any collection",
status: "failed",
reason: "file not found in any collection",
});
continue;
}
const { file, collectionName } = entry;
const tmpDir = mkdtempSync(join(tmpdir(), "quak-thumb-"));
const title = file.metadata.title;
const skipReason = reasonToSkip(file, userID);
if (skipReason) {
log(`[${collectionName}] Skipping ${title}: ${skipReason}`);
results.push({
fileID,
title,
collection: collectionName,
status: "skipped",
reason: skipReason,
});
continue;
}
const maxSize = file.thumbnail.size!;
try {
log(
`[${collectionName}] Downloading ${file.metadata.title} for thumbnail generation...`,
);
const origPath = join(tmpDir, "original");
await downloadFile(api, file, origPath);
const photo = lib.photos.byID({ fileID });
if (!photo) {
throw new Error("file not present in the library cache");
}
log(
`[${collectionName}] Generating thumbnail for ${file.metadata.title}...`,
`[${collectionName}] Downloading ${title} for thumbnail generation...`,
);
const fileBytes = readFileSync(origPath);
const thumbJpeg = generateThumbnail(new Uint8Array(fileBytes));
const { path } = await photo.original();
const fileBytes = new Uint8Array(readFileSync(path));
if (!isJpeg(fileBytes)) {
const reason =
"unsupported image format (only baseline JPEG can be regenerated)";
log(`[${collectionName}] Skipping ${title}: ${reason}`);
results.push({
fileID,
title,
collection: collectionName,
status: "skipped",
reason,
});
continue;
}
log(`[${collectionName}] Generating thumbnail for ${title}...`);
const decoded = jpeg.decode(fileBytes, {
useTArray: true,
formatAsRGBA: true,
});
const fitting = encryptThumbnailWithin(decoded, file.key, maxSize);
if (!fitting) {
const reason = `no thumbnail encoding fits the recorded thumbnail size of ${maxSize} bytes`;
log(`[${collectionName}] Skipping ${title}: ${reason}`);
results.push({
fileID,
title,
collection: collectionName,
status: "skipped",
reason,
});
continue;
}
const { header, ciphertext } = fitting;
log(
`[${collectionName}] Encrypting and uploading thumbnail (${thumbJpeg.length} bytes)...`,
`[${collectionName}] Uploading thumbnail (${ciphertext.length} bytes)...`,
);
const { header, ciphertext } = encryptBlob(thumbJpeg, file.key);
const md5 = createHash("md5").update(ciphertext).digest("base64");
const { objectKey, url } = await api.getUploadURL(
ciphertext.length,
@@ -230,28 +353,25 @@ export const fixMissingThumbnails = async (
await api.putFile(url, ciphertext);
await api.updateThumbnail(file.id, objectKey, toBase64(header));
log(
`[${collectionName}] Thumbnail uploaded for ${file.metadata.title}`,
);
log(`[${collectionName}] Thumbnail uploaded for ${title}`);
results.push({
fileID,
title: file.metadata.title,
title,
collection: collectionName,
success: true,
status: "fixed",
});
} catch (err) {
log(
`[${collectionName}] FAILED ${file.metadata.title}: ${err instanceof Error ? err.message : err}`,
`[${collectionName}] FAILED ${title}: ${err instanceof Error ? err.message : err}`,
"error",
);
results.push({
fileID,
title: file.metadata.title,
title,
collection: collectionName,
success: false,
error: err instanceof Error ? err.message : String(err),
status: "failed",
reason: err instanceof Error ? err.message : String(err),
});
} finally {
rmSync(tmpDir, { recursive: true, force: true });
}
}
test/api/client.test.ts
+389 -28
View File
@@ -38,14 +38,18 @@
* the network. The fake records every call for assertion.
*/
import { describe, expect, it } from "vitest";
import { describe, expect, it, vi } from "vitest";
import {
ApiClient,
ApiError,
DEFAULT_DOWNLOAD_TIMEOUT_MS,
DEFAULT_REQUEST_TIMEOUT_MS,
} from "../../src/api/client.js";
import type { RetryOptions } from "../../src/retry.js";
import {
isRetryable,
isSafeToReplay,
type RetryOptions,
} from "../../src/retry.js";
// ---------------------------------------------------------------------------
// Test helpers
@@ -385,6 +389,103 @@ describe("ApiClient custom origins", () => {
});
});
describe("ApiClient request URLs", () => {
it("accepts a path with or without a leading slash", async () => {
const { fetch, calls } = recordingFetch(
jsonResponse({}),
jsonResponse({}),
jsonResponse({}),
);
const client = new ApiClient({ fetch });
await client.getJSON("health");
await client.postJSON("users/ott", {});
await client.putJSON("/files/thumbnail", {});
expect(calls.map((c) => c.url)).toEqual([
"https://api.ente.io/health",
"https://api.ente.io/users/ott",
"https://api.ente.io/files/thumbnail",
]);
});
it("accepts an apiOrigin with a trailing slash", async () => {
const { fetch, calls } = recordingFetch(jsonResponse({}));
const client = new ApiClient({
fetch,
apiOrigin: "https://my-ente.example.com/",
});
await client.getJSON("/health");
expect(calls[0]!.url).toBe("https://my-ente.example.com/health");
});
it("keeps a base path in a self-hosted apiOrigin for every request", async () => {
const body = new Uint8Array([1]);
const { fetch, calls } = recordingFetch(
jsonResponse({}),
jsonResponse({}),
jsonResponse({}),
streamResponse(body),
streamResponse(body),
);
const client = new ApiClient({
fetch,
apiOrigin: "https://example.com/ente/",
});
await client.getJSON("/collections/v2", { sinceTime: 0 });
await client.postJSON("/users/ott", {});
await client.putJSON("/files/thumbnail", {});
await client.getFileStream(99);
await client.getThumbnailStream(77);
expect(calls.map((c) => c.url)).toEqual([
"https://example.com/ente/collections/v2?sinceTime=0",
"https://example.com/ente/users/ott",
"https://example.com/ente/files/thumbnail",
"https://example.com/ente/files/download/99",
"https://example.com/ente/files/preview/77",
]);
});
it("percent-encodes query parameters and skips undefined ones", async () => {
const { fetch, calls } = recordingFetch(jsonResponse({}));
const client = new ApiClient({ fetch });
await client.getJSON("/search", {
q: "a&b=c/d é",
limit: 5,
cursor: undefined,
});
const url = new URL(calls[0]!.url);
expect(url.pathname).toBe("/search");
expect(url.search).toBe("?q=a%26b%3Dc%2Fd+%C3%A9&limit=5");
expect(url.searchParams.get("q")).toBe("a&b=c/d é");
});
it("rejects a path that carries its own query string", async () => {
const { fetch, calls } = recordingFetch();
const client = new ApiClient({ fetch });
await expect(client.getJSON("/diff?sinceTime=0")).rejects.toThrow(
/must not contain "\?"/,
);
await expect(client.postJSON("/users/ott?x=1", {})).rejects.toThrow(
/must not contain "\?"/,
);
expect(calls).toHaveLength(0);
});
it("rejects a path that carries a fragment", async () => {
const { fetch, calls } = recordingFetch();
const client = new ApiClient({ fetch });
await expect(client.getJSON("/diff#top")).rejects.toThrow(
/must not contain "\?" or "#"/,
);
expect(calls).toHaveLength(0);
});
});
describe("ApiError", () => {
it("throws ApiError on 4xx with status, code, requestID", async () => {
const { fetch } = recordingFetch(
@@ -639,17 +740,33 @@ describe("ApiClient retries", () => {
expect(policy.baseDelayMs).toBe(7);
expect(policy.maxDelayMs).toBe(11);
});
it("does not let a caller change its settings through that policy", async () => {
const { fetch, calls } = scriptedFetch(
textResponse("boom", 500),
textResponse("boom", 500),
textResponse("boom", 500),
);
const client = new ApiClient({
fetch,
retry: { ...noWait, attempts: 2 },
});
client.getRetryOptions().attempts = 3;
expect(client.getRetryOptions().attempts).toBe(2);
await expect(client.getJSON("/x")).rejects.toBeInstanceOf(ApiError);
expect(calls).toHaveLength(2);
});
});
describe("ApiClient timeouts", () => {
it("ships bounded default deadlines", () => {
// Asserted here so the README and the code cannot drift. Two numbers
// rather than one, because a deadline that is sane for a JSON call is
// nowhere near enough for a multi-gigabyte body, and a deadline long
// enough for that body would let a hung API call stall a backup for
// ten minutes.
// Asserted here so the README and the code cannot drift. The request
// deadline bounds a whole JSON call; the download deadline is an idle
// one, measured from the last byte that arrived.
expect(DEFAULT_REQUEST_TIMEOUT_MS).toBe(30_000);
expect(DEFAULT_DOWNLOAD_TIMEOUT_MS).toBe(600_000);
expect(DEFAULT_DOWNLOAD_TIMEOUT_MS).toBe(60_000);
});
it("attaches an abort signal to every request", async () => {
@@ -693,6 +810,51 @@ describe("ApiClient timeouts", () => {
expect(new Set(signals).size).toBe(3);
}, 5000);
it("gives every retrying entry point a fresh deadline per attempt", async () => {
// A refused connection is retried by every entry point, the
// non-idempotent ones included. If the deadline were created once,
// outside the retry, both attempts would carry the same signal.
const entryPoints: [
string,
() => Response,
(c: ApiClient) => unknown,
][] = [
["getJSON", () => jsonResponse({}), (c) => c.getJSON("/a")],
["postJSON", () => jsonResponse({}), (c) => c.postJSON("/b", {})],
["putJSON", () => jsonResponse({}), (c) => c.putJSON("/c", {})],
[
"putFile",
() => new Response(null, { status: 200 }),
(c) => c.putFile("https://s3.example/x", new Uint8Array([1])),
],
[
"getFileStream",
() => streamResponse(new Uint8Array([1])),
(c) => c.getFileStream(1),
],
[
"getThumbnailStream",
() => streamResponse(new Uint8Array([1])),
(c) => c.getThumbnailStream(1),
],
];
for (const [name, success, call] of entryPoints) {
const { fetch, calls } = scriptedFetch(
errnoError("ECONNREFUSED", "connect ECONNREFUSED"),
success(),
);
const client = new ApiClient({ fetch, retry: noWait });
await call(client);
expect(calls, name).toHaveLength(2);
const [first, second] = calls.map((c) => c.init?.signal);
expect(first, name).toBeInstanceOf(AbortSignal);
expect(second, name).toBeInstanceOf(AbortSignal);
expect(second, name).not.toBe(first);
}
});
it("recovers when a later attempt answers in time", async () => {
const { fetch, calls } = scriptedFetch(HANG, jsonResponse({ ok: 1 }));
const client = new ApiClient({
@@ -718,25 +880,91 @@ describe("ApiClient timeouts", () => {
// that never produces a chunk and never observes the signal, so the
// only thing that can unblock the read is quak's own enforcement of
// the deadline over the stream it hands out.
const stalling = new Response(
new ReadableStream<Uint8Array>({
pull: () => new Promise<void>(() => {}),
}),
{ status: 200 },
);
const { fetch } = scriptedFetch(stalling);
const client = new ApiClient({
fetch,
downloadTimeoutMs: 20,
retry: { ...noWait, attempts: 1 },
});
//
// The clock is faked, so the test runs under the real default
// deadline and waits for nothing.
vi.useFakeTimers();
try {
const stalling = new Response(
new ReadableStream<Uint8Array>({
pull: () => new Promise<void>(() => {}),
}),
{ status: 200 },
);
const { fetch } = scriptedFetch(stalling);
const client = new ApiClient({
fetch,
retry: { ...noWait, attempts: 1 },
});
const stream = await client.getFileStream(42);
const err: unknown = await readAll(stream).catch((e: unknown) => e);
const stream = await client.getFileStream(42);
let settled = false;
const result = readAll(stream).then(
(n) => n,
(e: unknown) => e,
);
void result.finally(() => {
settled = true;
});
expect(err).toBeInstanceOf(Error);
expect((err as Error).name).toBe("TimeoutError");
}, 5000);
await vi.advanceTimersByTimeAsync(DEFAULT_DOWNLOAD_TIMEOUT_MS - 1);
expect(settled).toBe(false);
await vi.advanceTimersByTimeAsync(1);
const err = await result;
expect(err).toBeInstanceOf(Error);
expect((err as Error).name).toBe("TimeoutError");
// Classified as every deadline is: retried by the idempotent
// downloads, never replayed for a POST or PUT.
expect(isRetryable(err)).toBe(true);
expect(isSafeToReplay(err)).toBe(false);
} finally {
vi.useRealTimers();
}
});
it("does not abort a slow body that keeps making progress", async () => {
// A deadline over the whole transfer would cut off a large video on a
// slow link however steadily it was arriving. The deadline restarts
// with every chunk, so a body that sends one byte every 600 ms for
// well over the 1000 ms deadline completes.
vi.useFakeTimers();
try {
let sent = 0;
const trickling = new Response(
new ReadableStream<Uint8Array>({
async pull(controller) {
await new Promise((resolve) =>
setTimeout(resolve, 600),
);
if (sent === 10) {
controller.close();
return;
}
controller.enqueue(new Uint8Array([sent++]));
},
}),
{ status: 200 },
);
const { fetch } = scriptedFetch(trickling);
const client = new ApiClient({
fetch,
downloadTimeoutMs: 1000,
retry: { ...noWait, attempts: 1 },
});
const stream = await client.getFileStream(42);
const result = readAll(stream).then(
(n) => n,
(e: unknown) => e,
);
await vi.advanceTimersByTimeAsync(11 * 600);
expect(await result).toBe(10);
} finally {
vi.useRealTimers();
}
});
it("lets a body that arrives in time through untouched", async () => {
// The counterpart to the previous test: enforcing the deadline over
@@ -762,6 +990,52 @@ describe("ApiClient timeouts", () => {
}
expect(joined).toEqual(payload);
});
it("leaves no timer pending after a download completes or fails", async () => {
vi.useFakeTimers();
try {
const { fetch } = scriptedFetch(
streamResponse(new Uint8Array([1, 2, 3])),
textResponse("gone", 404),
);
const client = new ApiClient({
fetch,
retry: { ...noWait, attempts: 1 },
});
expect(await readAll(await client.getFileStream(1))).toBe(3);
expect(vi.getTimerCount()).toBe(0);
await expect(client.getFileStream(2)).rejects.toBeInstanceOf(
ApiError,
);
expect(vi.getTimerCount()).toBe(0);
} finally {
vi.useRealTimers();
}
});
it("never lets the download timer keep the process alive", async () => {
const spy = vi.spyOn(globalThis, "setTimeout");
try {
const { fetch } = scriptedFetch(
streamResponse(new Uint8Array([1])),
);
const client = new ApiClient({
fetch,
downloadTimeoutMs: 12_345,
retry: noWait,
});
const stream = await client.getFileStream(1);
const i = spy.mock.calls.findIndex((call) => call[1] === 12_345);
const timer = spy.mock.results[i]!.value as NodeJS.Timeout;
expect(timer.hasRef()).toBe(false);
await stream.cancel();
} finally {
spy.mockRestore();
}
});
});
describe("ApiClient error typing", () => {
@@ -820,9 +1094,8 @@ describe("ApiClient error typing", () => {
describe("ApiClient non-idempotent requests", () => {
/**
* `postJSON` and `putJSON` carry quak's only requests that change server
* state: `/users/srp/create-session`, `/users/two-factor/verify` — which
* consumes one of a small number of 2FA attempts — and `/files/thumbnail`.
* `postJSON` and `putJSON` carry quak's requests that can change server
* state; the README lists them under "Endpoints used".
*
* They are retried only on a failure that establishes no TCP connection to
* the server ever existed — DNS produced no address, or the peer refused
@@ -922,4 +1195,92 @@ describe("ApiClient non-idempotent requests", () => {
await refusedClient.updateThumbnail(1, "key", "header");
expect(refused.calls).toHaveLength(2);
});
it("does not follow or replay a redirect on POST or PUT", async () => {
// The origin has already received a request it answers with a
// redirect, so following it would let a refused connection to the
// redirect target pass for a request that never went out.
for (const send of [
(c: ApiClient) => c.postJSON("/users/ott", {}),
(c: ApiClient) => c.putJSON("/files/thumbnail", {}),
]) {
const { fetch, calls } = scriptedFetch(
new Response(null, {
status: 307,
headers: { location: "https://elsewhere.example/" },
}),
jsonResponse({}),
);
const client = new ApiClient({ fetch, retry: noWait });
const err: unknown = await send(client).catch((e: unknown) => e);
expect(calls[0]?.init?.redirect).toBe("manual");
expect(err).toBeInstanceOf(ApiError);
expect((err as ApiError).status).toBe(307);
expect(calls).toHaveLength(1);
}
});
});
// `onRequest` hears of each request the client sends, as its response arrives
// or as it fails. The CLI logs these with `-vv`. A retried request is reported
// once for each attempt.
describe("ApiClient onRequest", () => {
it("reports each request with its status, or with the error it failed with", async () => {
const refused = new TypeError("fetch failed");
const { fetch } = scriptedFetch(
refused,
textResponse("unavailable", 503),
jsonResponse({ collections: [] }),
jsonResponse({}),
streamResponse(new Uint8Array([1, 2, 3])),
);
const requests: unknown[] = [];
const client = new ApiClient({
fetch,
retry: noWait,
onRequest: (request) => requests.push(request),
});
await client.getJSON("/collections/v2", { sinceTime: 0 });
await client.postJSON("/users/logout", {});
await readAll(await client.getFileStream(7));
const collections = "https://api.ente.io/collections/v2?sinceTime=0";
expect(requests).toEqual([
{ method: "GET", url: collections, error: refused },
{ method: "GET", url: collections, status: 503 },
{ method: "GET", url: collections, status: 200 },
{
method: "POST",
url: "https://api.ente.io/users/logout",
status: 200,
},
{
method: "GET",
url: "https://files.ente.io/?fileID=7",
status: 200,
},
]);
});
it("reports a presigned upload without its query, which holds the upload's signature", async () => {
const { fetch } = scriptedFetch(new Response(null, { status: 200 }));
const requests: unknown[] = [];
const client = new ApiClient({
fetch,
retry: noWait,
onRequest: (request) => requests.push(request),
});
await client.putFile(
"https://s3.example/obj?X-Amz-Signature=abc",
new Uint8Array([1, 2]),
);
expect(requests).toEqual([
{ method: "PUT", url: "https://s3.example/obj", status: 200 },
]);
});
});
test/cli/backup-concurrency.test.ts
+608
View File
@@ -0,0 +1,608 @@
/**
* How many originals `lib.backup()` works on at once.
*
* The backup puts up to the content pool's limit of originals at their save
* paths at the same time: 5 for a library opened with the default
* `RequestPools`, as `quak backup` opens it, or the `contentConcurrency` of the
* pools a library was opened with. Everything the backup does for one original
* counts against that limit, hashing it for `verify` included, so however large
* the account is, the backup has no more than that many originals open at once.
*
* As in `backup-end-to-end.test.ts`, the real `Client` and `Library` run
* against `buildMetaFetch` from `test/ente-server.ts`, a fake of the Ente HTTP
* API serving data this file encrypts with libsodium. Here a test can also hold
* each original's download until it lets it go, to see how many the backup has
* in flight at once.
*
* The account, user 42, has one album of its own, Holiday, of eight photos,
* all taken at noon on 1 March 2024, each with an original of its own.
*/
import {
existsSync,
lstatSync,
mkdtempSync,
readdirSync,
readFileSync,
readlinkSync,
rmSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import sodium from "libsodium-wrappers-sumo";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import type { BackupOptions } from "../../src/backup.js";
import { Client, type ClientSnapshot } from "../../src/client.js";
import { init, toBase64 } from "../../src/crypto/index.js";
import { Library } from "../../src/library/index.js";
import { RequestPools } from "../../src/library/pools.js";
import {
buildAccount,
buildMetaFetch,
encryptSecretbox,
encryptStreamBlob,
} from "../ente-server.js";
import { blake2b } from "../live-photo.js";
const EMAIL = "backup@example.com";
const PASSWORD = "backup-password";
// The fake server logs every account in as user 42.
const USER_ID = 42;
// Noon on 1 March 2024 in the time zone of the machine running the test, in
// microseconds, as Ente records times.
const TAKEN = new Date(2024, 2, 1, 12).getTime() * 1000;
// The eight photos' IDs. Photo `id` is titled `photo-<id>.jpg`, and its
// original is a few bytes naming it.
const PHOTO_IDS = [101, 102, 103, 104, 105, 106, 107, 108];
const originalOf = (id: number): Uint8Array =>
new TextEncoder().encode(`the photo ${id}`);
// Where a backup to `dir` saves photo `id`'s original.
const savedAt = (dir: string, id: number): string =>
join(dir, "2024/2024-03/2024-03-01", `2024-03-01.${id}.jpg`);
// The fake server, the session it logs the account in with, and the
// directory every backup below is made in.
let server: typeof globalThis.fetch;
let session: ClientSnapshot;
let root: string;
// The fake server, holding the account encrypted as Ente's clients upload it.
const buildServer = async () => {
const account = await buildAccount(PASSWORD);
// The album as `/collections/v2` lists it: its key encrypted with the
// account's master key, its name with its own key.
const albumKey = sodium.crypto_secretbox_keygen();
const encryptedAlbumKey = encryptSecretbox(albumKey, account.masterKey);
const encryptedName = encryptSecretbox(
new TextEncoder().encode("Holiday"),
albumKey,
);
const album = {
id: 10,
owner: { id: USER_ID },
encryptedKey: toBase64(encryptedAlbumKey.ciphertext),
keyDecryptionNonce: toBase64(encryptedAlbumKey.nonce),
encryptedName: toBase64(encryptedName.ciphertext),
nameDecryptionNonce: toBase64(encryptedName.nonce),
type: "album",
updationTime: TAKEN,
};
// Each photo as the album's `/collections/v2/diff` lists it: its key
// encrypted with the album's key, and its metadata, holding the hash of
// its original, encrypted with its own key. Its original, encrypted with
// its own key too, is what the files CDN serves for it.
const ciphertexts: Record<number, Uint8Array> = {};
const photos = PHOTO_IDS.map((id) => {
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const original = encryptStreamBlob(originalOf(id), key);
ciphertexts[id] = original.ciphertext;
const encryptedKey = encryptSecretbox(key, albumKey);
const metadata = encryptStreamBlob(
new TextEncoder().encode(
JSON.stringify({
title: `photo-${id}.jpg`,
fileType: 0,
hash: blake2b(originalOf(id)),
creationTime: TAKEN,
modificationTime: TAKEN,
}),
),
key,
);
return {
id,
collectionID: 10,
ownerID: USER_ID,
encryptedKey: toBase64(encryptedKey.ciphertext),
keyDecryptionNonce: toBase64(encryptedKey.nonce),
metadata: {
encryptedData: toBase64(metadata.ciphertext),
decryptionHeader: toBase64(metadata.header),
},
file: { decryptionHeader: toBase64(original.header) },
thumbnail: {
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
},
updationTime: TAKEN,
};
});
return buildMetaFetch({
verifier: account.verifier,
srpAttributes: account.srpAttributes,
keyAttributes: account.keyAttributes,
encryptedToken: account.encryptedToken,
collections: [album],
filesByCollection: { 10: photos },
encryptedMLData: {},
fileCiphertexts: ciphertexts,
});
};
beforeAll(async () => {
await init();
await sodium.ready;
root = mkdtempSync(join(tmpdir(), "quak-backup-concurrency-"));
server = await buildServer();
const client = await Client.login({
email: EMAIL,
password: PASSWORD,
apiOptions: { fetch: server },
});
session = client.toJSON();
});
afterAll(() => {
rmSync(root, { recursive: true, force: true });
});
// The ID of the photo whose original a request downloads, or undefined for any
// other request. quak downloads an original from
// `https://files.ente.io/?fileID=<id>`.
const downloadOf = (input: RequestInfo | URL): number | undefined => {
const url = new URL(String(input));
return url.hostname === "files.ente.io"
? Number(url.searchParams.get("fileID"))
: undefined;
};
// The fake server, except that it answers 404 to every download of photo
// `failing`'s original.
const failingFor =
(failing: number): typeof globalThis.fetch =>
async (input, init) =>
downloadOf(input) === failing
? new Response("not found", { status: 404 })
: server(input, init);
// The fake server, holding each download of an original until the test lets
// it go. `held` lists the downloads waiting, the first to arrive first;
// `most()` is the largest number that ever waited at once. A held download of
// photo `failing`'s original is answered 404 once it is let go.
const holdingServer = (failing?: number) => {
const held: { id: number; release: () => void }[] = [];
let most = 0;
const fetch: typeof globalThis.fetch = async (input, init) => {
const id = downloadOf(input);
if (id !== undefined) {
await new Promise<void>((release) => {
held.push({ id, release });
most = Math.max(most, held.length);
});
if (id === failing) {
return new Response("not found", { status: 404 });
}
}
return server(input, init);
};
return { fetch, held, most: () => most };
};
// Let the held downloads go one at a time, the first to arrive first, each
// once `limit` of them wait, or all the downloads still to come when there
// are fewer. Each of the eight photos is downloaded once. With a backup that
// downloads fewer than `limit` at once, this fails waiting for them. Each wait
// allows five seconds, for a loaded machine's disk to sync the files written.
const releaseOneAtATime = async (
held: { release: () => void }[],
limit: number,
): Promise<void> => {
for (let left = PHOTO_IDS.length; left > 0; left--) {
await vi.waitFor(
() => expect(held).toHaveLength(Math.min(limit, left)),
{ interval: 5, timeout: 5000 },
);
held.shift()!.release();
}
};
// A library of the account opened as `quak backup` opens one, with no
// precache and no background refresh, sending its requests through `fetch`.
// It backs up to `dir` and keeps its cache beside it. Its request pools are
// `pools`, or by default the library's own.
const openLibrary = (
dir: string,
fetch: typeof globalThis.fetch,
pools?: RequestPools,
): Promise<Library> =>
Library.open({
client: Client.fromJSON(session, { fetch }),
downloadDirectory: dir,
cacheDirectory: `${dir}-cache`,
refreshIntervalSeconds: Infinity,
precacheThumbnails: false,
precacheOriginals: false,
pools,
});
// Every entry under `dir`, relative to it, sorted, a symlink as
// `name -> target`.
const listTree = (dir: string): string[] =>
readdirSync(dir, { recursive: true })
.sort()
.map((name) => {
const path = join(dir, name);
return lstatSync(path).isSymbolicLink()
? `${name} -> ${readlinkSync(path)}`
: name;
});
const expectStored = (dir: string, ids: number[]): void => {
for (const id of ids) {
expect(readFileSync(savedAt(dir, id))).toEqual(
Buffer.from(originalOf(id)),
);
}
};
describe("lib.backup()", () => {
// A library opened with the default pools, as `quak backup` opens one,
// downloads up to five originals at once. As soon as one finishes, the
// backup starts on the next photo, so five stay in flight until fewer
// than five are left.
it("downloads five originals at once, and never more, with the default pools", async () => {
const dir = mkdtempSync(join(root, "backup-"));
const holding = holdingServer();
const lib = await openLibrary(dir, holding.fetch);
try {
const backup = lib.backup();
await releaseOneAtATime(holding.held, 5);
const result = await backup;
expect(holding.most()).toBe(5);
expect(result).toMatchObject({
totalFiles: 8,
downloaded: 8,
failed: 0,
});
expectStored(dir, PHOTO_IDS);
} finally {
await lib.close();
}
});
// The limit is the content pool's, so a library opened with pools of
// another `contentConcurrency` backs up that many at once.
it("downloads as many originals at once as the content pool of the pools it was opened with allows", async () => {
const dir = mkdtempSync(join(root, "backup-"));
const holding = holdingServer();
const lib = await openLibrary(
dir,
holding.fetch,
new RequestPools({ contentConcurrency: 2 }),
);
try {
const backup = lib.backup();
await releaseOneAtATime(holding.held, 2);
const result = await backup;
expect(holding.most()).toBe(2);
expect(result).toMatchObject({ downloaded: 8, failed: 0 });
expectStored(dir, PHOTO_IDS);
} finally {
await lib.close();
}
});
// With `verify`, hashing an original already stored is part of the work
// on it, and counts against the limit too: a large backup never has more
// than five originals open for hashing. Here every stored original has
// changed on disk, so each is hashed, found not to match, and downloaded
// again. While five downloads wait, the backup has hashed only those five
// photos' originals; it opens the next one once a download finishes.
it("with verify, hashes no more originals ahead of their downloads than the limit", async () => {
const dir = mkdtempSync(join(root, "backup-"));
const first = await openLibrary(dir, server);
try {
await first.backup();
} finally {
await first.close();
}
for (const id of PHOTO_IDS) {
writeFileSync(savedAt(dir, id), "not the photo any more");
}
const holding = holdingServer();
const lib = await openLibrary(dir, holding.fetch);
const lines: string[] = [];
const mismatches = () =>
lines.filter((line) => line.startsWith("MISMATCH"));
try {
const backup = lib.backup({
verify: true,
onProgress: (line) => lines.push(line),
});
await vi.waitFor(() => expect(holding.held).toHaveLength(5), {
interval: 5,
timeout: 5000,
});
expect(mismatches()).toHaveLength(5);
await releaseOneAtATime(holding.held, 5);
const result = await backup;
expect(holding.most()).toBe(5);
expect(mismatches()).toHaveLength(8);
expect(result).toMatchObject({
downloaded: 8,
verified: 0,
mismatched: 8,
failed: 0,
});
expectStored(dir, PHOTO_IDS);
} finally {
await lib.close();
}
});
// The server answers 404 for photo 103's original. A 404 is not retried,
// so that download fails at once, while other downloads are still in
// flight. The backup records the failure and goes on: the downloads
// beside it finish, the photos after it are downloaded, and only photo
// 103 is missing from the backup and in failures.json.
it("goes on with the other originals when one fails", async () => {
const dir = mkdtempSync(join(root, "backup-"));
const holding = holdingServer(103);
const lib = await openLibrary(dir, holding.fetch);
try {
const backup = lib.backup();
await releaseOneAtATime(holding.held, 5);
const result = await backup;
expect(result).toMatchObject({
totalFiles: 8,
downloaded: 7,
failed: 1,
});
expect(result.errors.map((e) => e.fileID)).toEqual([103]);
expectStored(
dir,
PHOTO_IDS.filter((id) => id !== 103),
);
expect(existsSync(savedAt(dir, 103))).toBe(false);
const failures = JSON.parse(
readFileSync(join(dir, "failures.json"), "utf-8"),
);
expect(Object.keys(failures.files)).toEqual(["103"]);
} finally {
await lib.close();
}
});
// A throw from `onProgress` fails the backup: `lib.backup()` rejects with
// it. From then on the backup starts no further original. It finishes the
// ones it is working on, and rejects only once they are in place, so
// nothing is written into the backup after it has ended and removed its
// lock. Here the first five downloads wait. Photo 103's is let go first
// and answered 404, and the line reporting that failure throws while the
// other four wait. Only those five downloads are held: a backup that went
// on would store the other three photos.
it("when onProgress throws, starts no further original, and rejects once the originals it was working on are in place", async () => {
const dir = mkdtempSync(join(root, "backup-"));
const downloads: number[] = [];
const held: { id: number; release: () => void }[] = [];
const fetch: typeof globalThis.fetch = async (input, init) => {
const id = downloadOf(input);
if (id !== undefined) {
downloads.push(id);
if (downloads.length <= 5) {
await new Promise<void>((release) =>
held.push({ id, release }),
);
}
if (id === 103) {
return new Response("not found", { status: 404 });
}
}
return server(input, init);
};
const lib = await openLibrary(dir, fetch);
const lines: string[] = [];
try {
const backup = lib.backup({
onProgress: (line) => {
lines.push(line);
if (line.startsWith("FAILED")) {
throw new Error("the progress output is closed");
}
},
});
// The photos whose originals are in place, and whether the lock
// is there, at the moment `lib.backup()` rejects.
let storedWhenRejected: number[] = [];
let lockedWhenRejected = true;
const rejection = backup.catch((err: unknown) => {
storedWhenRejected = PHOTO_IDS.filter((id) =>
existsSync(savedAt(dir, id)),
);
lockedWhenRejected = existsSync(join(dir, "backup.lock"));
return err;
});
await vi.waitFor(() => expect(held).toHaveLength(5), {
interval: 5,
timeout: 5000,
});
const at103 = held.findIndex((h) => h.id === 103);
held.splice(at103, 1)[0]!.release();
await vi.waitFor(
() =>
expect(
lines.some((line) => line.startsWith("FAILED")),
).toBe(true),
{ interval: 5, timeout: 5000 },
);
for (const download of held) download.release();
expect(await rejection).toEqual(
new Error("the progress output is closed"),
);
expect(downloads.sort()).toEqual([101, 102, 103, 104, 105]);
expect(storedWhenRejected).toEqual([101, 102, 104, 105]);
expect(lockedWhenRejected).toBe(false);
} finally {
await lib.close();
}
});
// Once the backup's lock is lost, another backup may be writing to the
// directory, so the backup starts no further original. It finishes the
// ones it is working on, and `lib.backup()` rejects once they are all in
// place. Here `backup.lock` is removed while the first five downloads
// wait, and proper-lockfile's timer finds it gone. As in
// `backup-end-to-end.test.ts`, only `setTimeout` and `clearTimeout` are
// faked, so the test can fire that timer without waiting 5 seconds. Only
// the first five downloads are held: a backup that went on would store
// the other three photos.
it("when its lock is lost, starts no further original, and rejects once the originals it was working on are in place", async () => {
const dir = mkdtempSync(join(root, "backup-"));
const lockPath = join(dir, "backup.lock");
const downloads: number[] = [];
const held: (() => void)[] = [];
const fetch: typeof globalThis.fetch = async (input, init) => {
const id = downloadOf(input);
if (id !== undefined) {
downloads.push(id);
if (downloads.length <= 5) {
await new Promise<void>((release) => held.push(release));
}
}
return server(input, init);
};
const lib = await openLibrary(dir, fetch);
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
try {
const backup = lib.backup();
// The photos whose originals are in place at the moment
// `lib.backup()` rejects.
let storedWhenRejected: number[] = [];
const rejection = backup.catch((err: unknown) => {
storedWhenRejected = PHOTO_IDS.filter((id) =>
existsSync(savedAt(dir, id)),
);
return err;
});
await vi.waitFor(() => expect(held).toHaveLength(5), {
interval: 5,
timeout: 5000,
});
rmSync(lockPath, { recursive: true });
// Fire proper-lockfile's timer, and wait for the look at the lock
// on disk that it starts to finish: Node lists a file system call
// in progress as an `FSReqCallback`.
vi.advanceTimersByTime(5_000);
while (process.getActiveResourcesInfo().includes("FSReqCallback")) {
await new Promise((resolve) => setImmediate(resolve));
}
for (const release of held) release();
expect(await rejection).toEqual(
new Error(
`lost the lock on ${dir}: ENOENT: no such file or directory, stat '${lockPath}'`,
),
);
expect(downloads.sort()).toEqual([101, 102, 103, 104, 105]);
expect(storedWhenRejected).toEqual([101, 102, 103, 104, 105]);
} finally {
vi.useRealTimers();
await lib.close();
}
});
// Working on several originals at once changes only how long a backup
// takes. The same two runs, with a content pool of one, which backs up
// one original at a time, and with the default pools, end the same way:
// the same result, the same progress lines (in another order), the same
// failures.json, and the same tree. The first run stores every photo but
// 103, whose download is answered 404. Then two stored originals change
// on disk, and a run with `verify` puts them back and tries 103 again.
it("ends as a backup of one original at a time does", async () => {
const twoRuns = async (pools: RequestPools) => {
const dir = mkdtempSync(join(root, "backup-"));
const lines: string[] = [];
const backup = async (opts: BackupOptions) => {
const lib = await openLibrary(dir, failingFor(103), pools);
try {
return await lib.backup({
...opts,
onProgress: (line) => lines.push(line),
});
} finally {
await lib.close();
}
};
const first = await backup({});
writeFileSync(savedAt(dir, 101), "not the photo any more");
writeFileSync(savedAt(dir, 105), "not the photo any more");
const second = await backup({ verify: true });
// Each entry of failures.json without the time it was last
// tried, which differs from run to run.
const { files } = JSON.parse(
readFileSync(join(dir, "failures.json"), "utf-8"),
) as { files: Record<string, { lastTriedAt?: number }> };
for (const entry of Object.values(files)) delete entry.lastTriedAt;
return {
first,
second,
lines: lines.sort(),
failures: files,
tree: listTree(dir),
};
};
const oneAtATime = await twoRuns(
new RequestPools({ contentConcurrency: 1 }),
);
const fiveAtATime = await twoRuns(new RequestPools());
expect(oneAtATime.first).toMatchObject({
totalFiles: 8,
downloaded: 7,
skipped: 0,
failed: 1,
});
expect(oneAtATime.second).toMatchObject({
totalFiles: 8,
downloaded: 2,
skipped: 5,
verified: 5,
mismatched: 2,
unchecked: 0,
failed: 1,
});
expect(oneAtATime.failures).toEqual({
"103": expect.objectContaining({ fileID: 103, attempts: 2 }),
});
expect(fiveAtATime).toEqual(oneAtATime);
});
});
test/cli/backup-end-to-end.test.ts
+1236
View File
File diff suppressed because it is too large Load Diff
test/cli/backup-undecryptable.test.ts
+573
View File
@@ -0,0 +1,573 @@
/**
* `quak backup` over an account holding an album or a file quak cannot decrypt
* (issue 181).
*
* The server sends every album and file encrypted, and a refresh decrypts each
* one it fetches. A record whose key no key of the account opens, or whose
* metadata is not JSON, must not fail the refresh: a refresh that fails commits
* nothing, so `quak backup` would store nothing, on every run, until the
* server's copy changed. Instead the refresh leaves that record out, and the
* library keeps a list of what it left out in its cache. It has to: the next
* refresh asks the server only for what changed since the last one, so it will
* not see the record again.
*
* The backup stores every other file. It records each file it could not
* decrypt in `failures.json`, prints one line for each album it could not
* decrypt, and exits 1 while any remain. Once the server sends a copy that
* decrypts, or deletes the record, the next run drops it and can exit 0. The
* read commands answer from what did decrypt.
*
* These tests run the commands with a real saved session over a stand-in Ente
* server, which serves albums, files and originals encrypted with libsodium as
* the real server does. A record is broken the way a corrupt one arrives: its
* key replaced by bytes no key opens, or its metadata text that is not JSON.
*/
import {
existsSync,
mkdtempSync,
readdirSync,
readFileSync,
rmSync,
} from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import sodium from "libsodium-wrappers-sumo";
import { afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest";
import {
backupCommand,
collectionsCommand,
filesCommand,
saveSession,
type CliContext,
} from "../../src/cli-commands.js";
import { stderrLog } from "../../src/cli-log.js";
import { loadSession } from "../../src/cli-session.js";
import { Client, type ClientSnapshot } from "../../src/client.js";
import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
import { Library } from "../../src/library/index.js";
const USER_ID = 42;
// Every file is taken at noon local time on 2026-03-01, in microseconds as Ente
// stores times, so its original is saved in the folder `DAY` whatever the time
// zone of the machine running the tests.
const TAKEN = new Date(2026, 2, 1, 12).getTime() * 1000;
const DAY = join("2026", "2026-03", "2026-03-01");
type Row = { id: number; updationTime: number } & Record<string, unknown>;
// `plaintext` sealed under `key` with a fresh nonce, both base64, as Ente
// sends a key or an album name.
const secretbox = (
plaintext: Uint8Array,
key: Uint8Array,
): { ciphertext: string; nonce: string } => {
const nonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
return {
ciphertext: toBase64(
sodium.crypto_secretbox_easy(plaintext, nonce, key),
),
nonce: toBase64(nonce),
};
};
// What a corrupt row carries in place of its sealed key: random bytes of the
// same length, which no key opens.
const corruptKey = (): string =>
toBase64(
sodium.randombytes_buf(
sodium.crypto_secretbox_KEYBYTES + sodium.crypto_secretbox_MACBYTES,
),
);
const json = (body: unknown): Response =>
new Response(JSON.stringify(body), {
status: 200,
headers: { "content-type": "application/json" },
});
// A stand-in Ente server for one account. It holds each album, and each file in
// an album, as the row `/collections/v2` and `/collections/v2/diff` serve, and
// answers a request with the rows whose `updationTime` is past the request's
// `sinceTime`, as the real server does. A change to a file also moves its
// album's `updationTime`, which is what tells a client to list the album's
// files again. Originals are served by file ID, as the files CDN serves them.
class FakeEnte {
private readonly masterKey = sodium.crypto_secretbox_keygen();
private readonly keyPair = sodium.crypto_box_keypair();
private readonly albums = new Map<number, Row>();
private readonly albumKeys = new Map<number, Uint8Array>();
// Each album's file rows, by album ID and then file ID.
private readonly files = new Map<number, Map<number, Row>>();
// Each file's key and its original as served, by file ID. A file in two
// albums is one original under one key, sealed under each album's key.
private readonly originals = new Map<
number,
{ key: Uint8Array; header: string; body: Uint8Array }
>();
// The session `quak login` would have saved for the account.
snapshot(): ClientSnapshot {
return {
email: "user@example.com",
userID: USER_ID,
token: "token",
masterKey: toBase64(this.masterKey),
secretKey: toBase64(this.keyPair.privateKey),
publicKey: toBase64(this.keyPair.publicKey),
};
}
// Album `id`, named `name`, as of time `t`. With `corrupt`, its key cannot
// be decrypted. An album keeps its key from one call to the next, so its
// files' rows still decrypt once it is put back without `corrupt`.
putAlbum(id: number, name: string, t: number, corrupt = false): void {
const key = this.albumKeys.get(id) ?? sodium.crypto_secretbox_keygen();
this.albumKeys.set(id, key);
const sealedKey = secretbox(key, this.masterKey);
const sealedName = secretbox(new TextEncoder().encode(name), key);
this.albums.set(id, {
id,
owner: { id: USER_ID },
encryptedKey: corrupt ? corruptKey() : sealedKey.ciphertext,
keyDecryptionNonce: sealedKey.nonce,
encryptedName: sealedName.ciphertext,
nameDecryptionNonce: sealedName.nonce,
type: "album",
updationTime: t,
});
if (!this.files.has(id)) this.files.set(id, new Map());
}
// File `id`, titled `photo-<id>.jpg`, in album `albumID` as of time `t`.
// With `corruptKey`, its key cannot be decrypted. With `metadata`, its
// metadata decrypts to that text instead of the JSON a client uploads.
putFile(
albumID: number,
id: number,
t: number,
opts: { corruptKey?: boolean; metadata?: string } = {},
): void {
const original = this.originalOf(id);
const sealedKey = secretbox(original.key, this.albumKeys.get(albumID)!);
const metadata =
opts.metadata ??
JSON.stringify({
title: `photo-${id}.jpg`,
fileType: 0,
creationTime: TAKEN,
modificationTime: TAKEN,
});
const sealedMetadata = encryptBlob(
new TextEncoder().encode(metadata),
original.key,
);
this.files.get(albumID)!.set(id, {
id,
collectionID: albumID,
ownerID: USER_ID,
encryptedKey: opts.corruptKey ? corruptKey() : sealedKey.ciphertext,
keyDecryptionNonce: sealedKey.nonce,
metadata: {
encryptedData: toBase64(sealedMetadata.ciphertext),
decryptionHeader: toBase64(sealedMetadata.header),
},
file: { decryptionHeader: original.header },
thumbnail: { decryptionHeader: original.header },
updationTime: t,
});
this.moveAlbum(albumID, t);
}
// A deleted record stays on the server as a tombstone holding only its
// ID, its time and `isDeleted`.
deleteFile(albumID: number, id: number, t: number): void {
this.files
.get(albumID)!
.set(id, { id, updationTime: t, isDeleted: true });
this.moveAlbum(albumID, t);
}
deleteAlbum(id: number, t: number): void {
this.albums.set(id, { id, updationTime: t, isDeleted: true });
}
// The `fetch` the saved session's client is given.
readonly fetch = (async (input: string | URL | Request) => {
const url = new URL(input instanceof Request ? input.url : input);
const since = Number(url.searchParams.get("sinceTime"));
const changed = (rows: Iterable<Row>): Row[] =>
[...rows].filter((row) => row.updationTime > since);
switch (url.pathname) {
case "/collections/v2":
return json({ collections: changed(this.albums.values()) });
case "/collections/v2/diff": {
const albumID = Number(url.searchParams.get("collectionID"));
const rows = this.files.get(albumID)?.values() ?? [];
return json({ diff: changed(rows), hasMore: false });
}
case "/files/data/fetch":
// No file has ML data.
return json({ data: [] });
default: {
const fileID = Number(url.searchParams.get("fileID"));
const original = this.originals.get(fileID);
return original
? new Response(original.body)
: new Response("not found", { status: 404 });
}
}
}) as typeof globalThis.fetch;
private originalOf(id: number): {
key: Uint8Array;
header: string;
body: Uint8Array;
} {
const known = this.originals.get(id);
if (known) return known;
const key = sodium.crypto_secretbox_keygen();
const { header, ciphertext } = encryptBlob(
new TextEncoder().encode(`original of ${id}`),
key,
);
const original = { key, header: toBase64(header), body: ciphertext };
this.originals.set(id, original);
return original;
}
private moveAlbum(albumID: number, t: number): void {
this.albums.set(albumID, {
...this.albums.get(albumID)!,
updationTime: t,
});
}
}
// Album 1, "Trip", holds file 10, which decrypts, file 11, whose key does not,
// and file 12, whose metadata is not JSON. Album 2, "Family", holds file 20,
// but the album's own key does not decrypt. All of it as of time 1.
const brokenAccount = (): FakeEnte => {
const ente = new FakeEnte();
ente.putAlbum(1, "Trip", 1);
ente.putFile(1, 10, 1);
ente.putFile(1, 11, 1, { corruptKey: true });
ente.putFile(1, 12, 1, { metadata: "not json" });
ente.putAlbum(2, "Family", 1, true);
ente.putFile(2, 20, 1);
return ente;
};
// Album 1, "Trip", holds file 10, and album 2, "Family", holds file 20. Both
// files decrypt, but album 2's own key does not, so the album is the only
// record left out. All of it as of time 1.
const brokenAlbumAccount = (): FakeEnte => {
const ente = new FakeEnte();
ente.putAlbum(1, "Trip", 1);
ente.putFile(1, 10, 1);
ente.putAlbum(2, "Family", 1, true);
ente.putFile(2, 20, 1);
return ente;
};
// Collects everything written to it.
class Output {
text = "";
write(text: string): void {
this.text += text;
}
}
let root: string;
let stdout: Output;
let stderr: Output;
beforeAll(async () => {
await init();
await sodium.ready;
});
beforeEach(() => {
root = mkdtempSync(join(tmpdir(), "quak-undecryptable-"));
});
afterEach(() => {
rmSync(root, { recursive: true, force: true });
});
// A context for one command, with fresh output, whose saved session talks to
// `ente`. Every run shares one cache directory, as runs of `quak` do.
const context = (ente: FakeEnte): CliContext => {
stdout = new Output();
stderr = new Output();
saveSession(join(root, "session"), ente.snapshot());
return {
stdout,
log: stderrLog(stderr, "info"),
sessionDir: join(root, "session"),
cacheDir: join(root, "cache"),
loadSession: (path) =>
loadSession(path, { fetch: ente.fetch, retry: { attempts: 1 } }),
login: async () => {
throw new Error("login not expected");
},
prompt: async () => {
throw new Error("prompt not expected");
},
promptSecret: async () => {
throw new Error("prompt not expected");
},
};
};
const backupDir = (): string => join(root, "backup");
// Runs `quak backup` into the backup directory and returns its exit code.
const backup = (ente: FakeEnte, opts: { json?: boolean } = {}) =>
backupCommand(context(ente), backupDir(), opts);
// The backup's `failures.json` entries by file ID, or {} when it has none.
const failures = (): Record<string, { error: string; attempts: number }> => {
const path = join(backupDir(), "failures.json");
return existsSync(path)
? JSON.parse(readFileSync(path, "utf-8")).files
: {};
};
// Whether file `id`'s original is at its save path in the backup.
const stored = (id: number): boolean =>
existsSync(join(backupDir(), DAY, `2026-03-01.${id}.jpg`));
// The IDs of the files that album `name`'s JSON lists.
const albumFiles = (name: string): number[] =>
JSON.parse(
readFileSync(join(backupDir(), "collections", `${name}.json`), "utf-8"),
).files.map((f: { id: number }) => f.id);
describe("quak backup with records it cannot decrypt", () => {
it("backs up every other file, records each file it cannot decrypt in failures.json, names each such album, and exits 1", async () => {
expect(await backup(brokenAccount())).toBe(1);
expect(stored(10)).toBe(true);
// Without its album's key, file 20 cannot even be listed.
expect(stored(20)).toBe(false);
expect(failures()).toEqual({
"11": expect.objectContaining({
fileID: 11,
error: expect.stringMatching(
/^could not be decrypted: .*wrong secret key/,
),
}),
"12": expect.objectContaining({
fileID: 12,
error: expect.stringMatching(/^could not be decrypted: .*JSON/),
}),
});
expect(stderr.text).toMatch(
/^FAILED album 2: could not be decrypted: .*wrong secret key/m,
);
expect(stderr.text).toContain(" Failed: 2\n");
});
// The album is an error line. The last line is fatal and counts the files
// and the album, so `-q`, which prints fatal lines only, still says why
// the backup exits 1.
it("logs the album at error, and ends with one fatal line counting the files and the album", async () => {
const ctx = {
...context(brokenAccount()),
log: stderrLog(stderr, "error"),
};
expect(await backupCommand(ctx, backupDir(), {})).toBe(1);
expect(stderr.text).toMatch(
/^FAILED album 2: could not be decrypted: /,
);
expect(stderr.text.split("\n").at(-2)).toBe(
`quak: 2 file(s) and 1 album(s) failed; see ${join(backupDir(), "failures.json")} and the FAILED lines`,
);
});
it("names each album it cannot decrypt in the --json result", async () => {
expect(await backup(brokenAccount(), { json: true })).toBe(1);
expect(JSON.parse(stdout.text)).toMatchObject({
downloaded: 1,
failed: 2,
undecryptableAlbums: [2],
});
});
it("still reports them on the next run, though the server does not send them again", async () => {
const ente = brokenAccount();
await backup(ente);
// Nothing changed on the server, so this run's refresh gets no rows at
// all. What it reports, it remembered from the first run.
expect(await backup(ente)).toBe(1);
expect(Object.keys(failures())).toEqual(["11", "12"]);
expect(failures()["11"]!.attempts).toBe(2);
expect(stderr.text).toMatch(/^FAILED album 2: could not be decrypted/m);
});
it("drops each once the server sends a copy that decrypts, and then exits 0", async () => {
const ente = brokenAccount();
await backup(ente);
// At time 2 files 11 and 12 and album 2 change on the server, and now
// each decrypts. Album 2 brings file 20 with it.
ente.putFile(1, 11, 2);
ente.putFile(1, 12, 2);
ente.putAlbum(2, "Family", 2);
expect(await backup(ente)).toBe(0);
expect([11, 12, 20].map(stored)).toEqual([true, true, true]);
expect(existsSync(join(backupDir(), "failures.json"))).toBe(false);
expect(stderr.text).not.toContain("FAILED");
});
it("drops each once the server deletes it, and then exits 0", async () => {
const ente = brokenAccount();
await backup(ente);
ente.deleteFile(1, 11, 2);
ente.deleteFile(1, 12, 2);
ente.deleteAlbum(2, 2);
expect(await backup(ente)).toBe(0);
expect(existsSync(join(backupDir(), "failures.json"))).toBe(false);
expect(stderr.text).not.toContain("FAILED");
});
it("exits 1 when the only record it cannot decrypt is an album, and 0 once the album decrypts", async () => {
const ente = brokenAlbumAccount();
// No file is in `failures.json`: the album alone keeps the exit code
// at 1.
expect(await backup(ente)).toBe(1);
expect(failures()).toEqual({});
ente.putAlbum(2, "Family", 2);
expect(await backup(ente)).toBe(0);
expect(stored(20)).toBe(true);
});
it("exits 1 when the only record it cannot decrypt is an album, and 0 once the album is deleted", async () => {
const ente = brokenAlbumAccount();
expect(await backup(ente)).toBe(1);
expect(failures()).toEqual({});
ente.deleteAlbum(2, 2);
expect(await backup(ente)).toBe(0);
});
it("removes what an earlier run linked for an album or a file that no longer decrypts, and keeps the originals", async () => {
// At time 1 everything decrypts: the first run links files 10 and 11
// in Trip and file 20 in Family.
const ente = new FakeEnte();
ente.putAlbum(1, "Trip", 1);
ente.putFile(1, 10, 1);
ente.putFile(1, 11, 1);
ente.putAlbum(2, "Family", 1);
ente.putFile(2, 20, 1);
expect(await backup(ente)).toBe(0);
const collections = join(backupDir(), "collections");
expect(readdirSync(join(collections, "Trip")).sort()).toEqual([
"photo-10.jpg",
"photo-11.jpg",
]);
expect(readdirSync(join(collections, "Family"))).toEqual([
"photo-20.jpg",
]);
// At time 2 the server's copy of file 11 in Trip, and album 2 itself,
// no longer decrypt. A run builds the album folders, their JSON and
// their links from what the library holds, so it removes them as if
// the album and the file had been deleted: Family's folder and JSON
// go, and Trip neither links nor lists file 11. The originals stay at
// their save paths, and the run exits 1.
ente.putFile(1, 11, 2, { corruptKey: true });
ente.putAlbum(2, "Family", 2, true);
expect(await backup(ente)).toBe(1);
expect(readdirSync(collections).sort()).toEqual(["Trip", "Trip.json"]);
expect(readdirSync(join(collections, "Trip"))).toEqual([
"photo-10.jpg",
]);
expect(albumFiles("Trip")).toEqual([10]);
expect([10, 11, 20].map(stored)).toEqual([true, true, true]);
expect(Object.keys(failures())).toEqual(["11"]);
});
it("backs up a file that does not decrypt in one album but does in another, and does not report it", async () => {
// File 30 is in albums 1 and 3, each holding its key sealed under the
// album's own key. In album 1 that is corrupt; in album 3 it is not,
// and one copy that decrypts is all the backup needs.
const ente = new FakeEnte();
ente.putAlbum(1, "Trip", 1);
ente.putFile(1, 30, 1, { corruptKey: true });
ente.putAlbum(3, "Favorites", 1);
ente.putFile(3, 30, 1);
expect(await backup(ente)).toBe(0);
expect(stored(30)).toBe(true);
expect(failures()).toEqual({});
// Album 1 has no copy of file 30 that decrypts, so it neither links
// the original nor lists the file; album 3 lists it.
expect(readdirSync(join(backupDir(), "collections", "Trip"))).toEqual(
[],
);
expect(albumFiles("Trip")).toEqual([]);
expect(albumFiles("Favorites")).toEqual([30]);
});
it("backs up that file in a run limited to the album where it does not decrypt", async () => {
// The same account, backed up by `lib.backup()` limited to album 1,
// "Trip", as `quak backup` cannot be. File 30 is in Trip, so the run
// must store it, from its copy in album 3 that the run otherwise
// skips.
const ente = new FakeEnte();
ente.putAlbum(1, "Trip", 1);
ente.putFile(1, 30, 1, { corruptKey: true });
ente.putAlbum(3, "Favorites", 1);
ente.putFile(3, 30, 1);
const lib = await Library.open({
client: Client.fromJSON(ente.snapshot(), {
fetch: ente.fetch,
retry: { attempts: 1 },
}),
downloadDirectory: backupDir(),
cacheDirectory: join(root, "cache"),
precacheThumbnails: false,
precacheOriginals: false,
});
try {
const result = await lib.backup({
downloadDirectory: backupDir(),
onlyAlbumNames: ["Trip"],
});
expect(result).toMatchObject({ downloaded: 1, failed: 0 });
} finally {
await lib.close();
}
expect(stored(30)).toBe(true);
expect(failures()).toEqual({});
});
});
describe("the read commands with records quak cannot decrypt", () => {
it("quak collections and quak files answer from what decrypts", async () => {
const ente = brokenAccount();
expect(await collectionsCommand(context(ente), {})).toBe(0);
expect(stdout.text).toBe("1\talbum\tTrip\n");
expect(await filesCommand(context(ente), "1", {})).toBe(0);
expect(stdout.text).toBe("10\timage\tphoto-10.jpg\n");
});
});
test/cli/backup.test.ts
+2380 -389
View File
File diff suppressed because it is too large Load Diff
test/cli/bin.test.ts
+97
View File
@@ -0,0 +1,97 @@
/**
* Tests for the retry options `bin/quak.ts` loads the saved session with
* (`quak backup` gets the unattended ones, every other command the default),
* each with the callbacks that log its requests and retries, and for
* `quak files` taking its collection ID as an optional argument.
*
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
* replaced by one that records the options it is given and reports no session,
* so the command stops before it makes any request.
*/
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { afterAll, afterEach, describe, expect, it, vi } from "vitest";
import type { ApiClientOptions } from "../../src/api/client.js";
import { UNATTENDED_RETRY_OPTIONS } from "../../src/retry.js";
const loaded = vi.hoisted(() => [] as (ApiClientOptions | undefined)[]);
vi.mock("../../src/cli-session.js", () => ({
loadSession: (_path: string, apiOptions?: ApiClientOptions) => {
loaded.push(apiOptions);
return null;
},
}));
const argv = process.argv;
const dir = mkdtempSync(join(tmpdir(), "quak-bin-test-"));
// Run `quak <args>` to completion and return what it wrote on stderr, which is
// kept off the terminal. The exit is swallowed.
const quak = async (...args: string[]): Promise<string> => {
let written = "";
vi.spyOn(process.stderr, "write").mockImplementation((text) => {
written += String(text);
return true;
});
vi.spyOn(process, "exit").mockImplementation(() => undefined as never);
process.argv = ["node", "quak", ...args];
vi.resetModules();
await import("../../bin/quak.js");
return written;
};
afterEach(() => {
process.argv = argv;
loaded.length = 0;
vi.restoreAllMocks();
});
afterAll(() => {
rmSync(dir, { recursive: true, force: true });
});
describe("bin/quak.ts session loading", () => {
it("loads the session for backup with the unattended retry options", async () => {
await quak("backup", dir);
// `vi.resetModules()` gave `bin/quak.ts` its own copy of
// `src/retry.ts`, whose `sleep` is a different function, so the
// options are compared by their numbers.
const { attempts, baseDelayMs, maxDelayMs } = UNATTENDED_RETRY_OPTIONS;
expect(loaded).toEqual([
{
retry: expect.objectContaining({
attempts,
baseDelayMs,
maxDelayMs,
onRetry: expect.any(Function),
}),
onRequest: expect.any(Function),
},
]);
});
it("loads the session for another command with the default retry options", async () => {
await quak("collections");
expect(loaded).toEqual([
{
retry: { onRetry: expect.any(Function) },
onRequest: expect.any(Function),
},
]);
});
});
// The command line takes `files` with or without a collection ID, and the
// command runs as far as its session check either way (issue #211).
describe("bin/quak.ts files", () => {
it("runs without a collection ID", async () => {
expect(await quak("files")).toMatch(/^Not logged in/);
});
it("runs with a collection ID", async () => {
expect(await quak("files", "7")).toMatch(/^Not logged in/);
});
});
test/cli/commands.test.ts
+1836
View File
File diff suppressed because it is too large Load Diff
test/cli/log.test.ts
+357
View File
@@ -0,0 +1,357 @@
/**
* Tests for the CLI's log levels, set with `-v`/`--verbose` and `-q`/`--quiet`
* (`src/cli-log.ts`, issue #212).
*
* Every line quak prints on stderr is a log line at one of six levels, from
* the quietest: fatal, error, warning, info, debug, trace. A run prints the
* lines at its own level and at every quieter one. The level is info unless
* asked otherwise: `-v` raises it to debug, `-vv` to trace, and more repeats
* stay at trace. `-q` lowers it to fatal, so a run prints only the line a
* command prints as it fails with a non-zero exit code. stdout holds the
* command's output at every level.
*
* Most tests run `bin/quak.ts`, as the smoke test does, against an account of
* one album holding one photo: the saved session is in a temporary directory,
* and `fetch` is a fake of the Ente server that serves the album and the photo
* encrypted as Ente's clients encrypt them. Each run has a cache directory of
* its own, so its refresh fetches the album's files, as on a first run.
*/
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import sodium from "libsodium-wrappers-sumo";
import {
afterAll,
afterEach,
beforeAll,
describe,
expect,
it,
vi,
} from "vitest";
import { ApiClient } from "../../src/api/client.js";
import { saveSession } from "../../src/cli-commands.js";
import { logRefresh, logRequests, type Log } from "../../src/cli-log.js";
import { init, toBase64 } from "../../src/crypto/index.js";
import { encryptSecretbox, encryptStreamBlob } from "../ente-server.js";
// `bin/quak.ts` reads the saved session from the data directory env-paths
// gives it, which here is a temporary directory made before the first run.
const paths = vi.hoisted(() => ({ data: "" }));
vi.mock("env-paths", () => ({ default: () => paths }));
const USER_ID = 42;
const root = mkdtempSync(join(tmpdir(), "quak-log-test-"));
const argv = process.argv;
class ExitCalled extends Error {}
// Run `quak <args>` to its exit, with a cache of its own, and return its exit
// code and what it wrote on stdout and on stderr, which are kept off the
// terminal. `process.exit` is stubbed to throw, so nothing runs after it, as
// nothing does after a real exit.
const quak = async (
...args: string[]
): Promise<{ code: number; stdout: string; stderr: string }> => {
let stdout = "";
let stderr = "";
let code = NaN;
vi.spyOn(process.stdout, "write").mockImplementation((text) => {
stdout += String(text);
return true;
});
vi.spyOn(process.stderr, "write").mockImplementation((text) => {
stderr += String(text);
return true;
});
vi.spyOn(process, "exit").mockImplementation((exitCode) => {
code = Number(exitCode ?? 0);
throw new ExitCalled();
});
const cache = mkdtempSync(join(root, "cache-"));
process.argv = ["node", "quak", "--cache-dir", cache, ...args];
vi.resetModules();
await expect(import("../../bin/quak.js")).rejects.toBeInstanceOf(
ExitCalled,
);
return { code, stdout, stderr };
};
// Album 1, "Trips", of user 42, holding photo 100, "beach.jpg". The album's
// key is encrypted with the account's master key, the photo's key with the
// album's, and the photo's metadata with its own key. Only the session and
// these two records are needed: the tests never log in.
beforeAll(async () => {
await init();
await sodium.ready;
const masterKey = sodium.crypto_secretbox_keygen();
const keyPair = sodium.crypto_box_keypair();
paths.data = join(root, "session");
saveSession(paths.data, {
email: "log@example.com",
userID: USER_ID,
token: "token",
masterKey: toBase64(masterKey),
secretKey: toBase64(keyPair.privateKey),
publicKey: toBase64(keyPair.publicKey),
});
const albumKey = sodium.crypto_secretbox_keygen();
const albumKeyBox = encryptSecretbox(albumKey, masterKey);
const name = encryptSecretbox(new TextEncoder().encode("Trips"), albumKey);
const album = {
id: 1,
owner: { id: USER_ID },
encryptedKey: toBase64(albumKeyBox.ciphertext),
keyDecryptionNonce: toBase64(albumKeyBox.nonce),
encryptedName: toBase64(name.ciphertext),
nameDecryptionNonce: toBase64(name.nonce),
type: "album",
updationTime: 1,
};
const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const photoKeyBox = encryptSecretbox(photoKey, albumKey);
const metadata = encryptStreamBlob(
new TextEncoder().encode(
JSON.stringify({
title: "beach.jpg",
fileType: 0,
creationTime: 1000,
modificationTime: 1000,
}),
),
photoKey,
);
const photo = {
id: 100,
collectionID: 1,
ownerID: USER_ID,
encryptedKey: toBase64(photoKeyBox.ciphertext),
keyDecryptionNonce: toBase64(photoKeyBox.nonce),
metadata: {
encryptedData: toBase64(metadata.ciphertext),
decryptionHeader: toBase64(metadata.header),
},
// These tests fetch no original and no thumbnail.
file: { decryptionHeader: "" },
thumbnail: { decryptionHeader: "" },
updationTime: 1,
};
const json = (body: unknown): Response =>
new Response(JSON.stringify(body), {
headers: { "content-type": "application/json" },
});
vi.stubGlobal("fetch", async (input: string) => {
const { pathname } = new URL(input);
if (pathname === "/collections/v2") {
return json({ collections: [album] });
}
if (pathname === "/collections/v2/diff") {
return json({ diff: [photo], hasMore: false });
}
return new Response("not found", { status: 404 });
});
});
afterEach(() => {
process.argv = argv;
vi.restoreAllMocks();
});
afterAll(() => {
vi.unstubAllGlobals();
rmSync(root, { recursive: true, force: true });
});
// What `quak files` writes on stdout at every level.
const FILES = "100\timage\tbeach.jpg\n";
// What it writes on stderr at each level. Info is the refresh's progress, one
// line for the one album whose files it fetched. Debug adds the start and end
// of the refresh, and trace each request with the status of its answer.
const INFO = "Fetched the files of 1 of 1 collections\n";
const DEBUG = "Refresh started\n" + INFO + "Refresh done\n";
const TRACE =
"Refresh started\n" +
"GET https://api.ente.io/collections/v2?sinceTime=0: HTTP 200\n" +
"GET https://api.ente.io/collections/v2/diff?collectionID=1&sinceTime=0: HTTP 200\n" +
INFO +
"Refresh done\n";
describe("the log level", () => {
it("is info by default", async () => {
expect(await quak("files")).toEqual({
code: 0,
stdout: FILES,
stderr: INFO,
});
});
it("is debug with -v", async () => {
expect(await quak("-v", "files")).toEqual({
code: 0,
stdout: FILES,
stderr: DEBUG,
});
});
it("is trace with -vv", async () => {
expect(await quak("-vv", "files")).toEqual({
code: 0,
stdout: FILES,
stderr: TRACE,
});
});
it("stays at trace with -vvv", async () => {
expect((await quak("-vvv", "files")).stderr).toBe(TRACE);
});
it("counts --verbose --verbose as -vv", async () => {
expect((await quak("--verbose", "--verbose", "files")).stderr).toBe(
TRACE,
);
});
// Both are options of the program, taken before or after the command's
// name, as `--cache-dir` is.
it("is set by an option after the command's name too", async () => {
expect((await quak("files", "-v")).stderr).toBe(DEBUG);
expect((await quak("files", "--json", "-vv")).stderr).toBe(TRACE);
expect((await quak("files", "-q")).stderr).toBe("");
});
});
describe("-q", () => {
it("prints nothing on stderr when the command works", async () => {
expect(await quak("-q", "files")).toEqual({
code: 0,
stdout: FILES,
stderr: "",
});
});
// The line saying why the command failed is the fatal one, and it is
// printed.
it("prints only the line a failing command exits with", async () => {
expect(await quak("-q", "files", "9")).toEqual({
code: 1,
stdout: "",
stderr: "Collection 9 not found\n",
});
});
it("is refused with -v, in one line, before the command runs", async () => {
for (const args of [
["-q", "-v", "files"],
["files", "--verbose", "--quiet"],
]) {
const { code, stdout, stderr } = await quak(...args);
expect(code).toBe(1);
expect(stdout).toBe("");
expect(stderr).toBe(
"error: option '-q, --quiet' cannot be used with option '-v, --verbose'\n",
);
}
});
});
// `--help` lists both options, and each command's help lists them too, since
// each command takes them.
describe("--help", () => {
it("lists -v and -q for the program and for each command", async () => {
for (const args of [["--help"], ["files", "--help"]]) {
const { stdout } = await quak(...args);
expect(stdout).toContain("-v, --verbose");
expect(stdout).toContain("-q, --quiet");
}
});
});
// The library writes no lines itself. Its layers report through callbacks,
// and the CLI logs what they report.
describe("what the library reports", () => {
const collect = (): { log: Log; lines: string[] } => {
const lines: string[] = [];
return { log: (level, line) => lines.push(`${level}: ${line}`), lines };
};
// A request answered 503 and retried: each attempt is a request, logged
// at trace with its status, and the retry is logged at debug with how
// long it waits and why.
it("logs each request at trace and each retry at debug", async () => {
const { log, lines } = collect();
const answers = [
new Response("unavailable", { status: 503 }),
new Response("{}", {
headers: { "content-type": "application/json" },
}),
];
const client = new ApiClient({
...logRequests(log, {
sleep: async () => {},
random: () => 0.5,
}),
fetch: async () => answers.shift()!,
});
await client.getJSON("/collections/v2", { sinceTime: 0 });
const url = "https://api.ente.io/collections/v2?sinceTime=0";
expect(lines).toEqual([
`trace: GET ${url}: HTTP 503`,
"debug: Attempt 1 failed, retrying in 250 ms: HTTP 503",
`trace: GET ${url}: HTTP 200`,
]);
});
it("logs a request that failed with its error", async () => {
const { log, lines } = collect();
const client = new ApiClient({
...logRequests(log, { attempts: 1 }),
fetch: async () => {
throw new TypeError("fetch failed", {
cause: new Error("connect ECONNREFUSED 1.2.3.4:443"),
});
},
});
await expect(client.getJSON("/collections/v2")).rejects.toThrow();
expect(lines).toEqual([
"trace: GET https://api.ente.io/collections/v2: fetch failed: connect ECONNREFUSED 1.2.3.4:443",
]);
});
// Each operation is named in plain words, as the README names it.
it("logs a refresh's progress at info, and its start and end at debug", () => {
const { log, lines } = collect();
const onProgress = logRefresh(log);
onProgress({ operation: "refresh", status: "started" });
onProgress({
operation: "refresh",
status: "progress",
fetched: 1,
total: 2,
});
onProgress({
operation: "refresh",
status: "failed",
error: "HTTP 503",
});
onProgress({ operation: "fetchMLData", status: "started" });
onProgress({ operation: "fetchMLData", status: "done", fetched: 3 });
expect(lines).toEqual([
"debug: Refresh started",
"info: Fetched the files of 1 of 2 collections",
"debug: Refresh failed: HTTP 503",
"debug: ML data fetch started",
"debug: ML data fetch done",
]);
});
});
test/cli/metadata-backup.test.ts
+349 -242
View File
@@ -2,13 +2,17 @@
* Tests for `quak backup-metadata <dir>`.
*
* This command dumps all decrypted account metadata into a directory
* tree of plain JSON files, without downloading any file content. It
* is fast (no multi-megabyte downloads) and produces a complete
* plaintext record of every collection name, file title, creation
* date, GPS coordinate, camera model, caption, face label, and any
* other metadata the Ente clients have attached.
* tree of plain JSON files, without downloading any file content (unless
* `--exif` is given). It is fast and produces a complete plaintext record of
* every collection name, file title, creation date, GPS coordinate, camera
* model, caption, face label, and any other metadata the Ente clients have
* attached.
*
* Layout:
* As of issue #52 it runs on the library API: `runMetadataBackup(lib, client,
* dir)` enumerates collections and files from the library's cache rather than
* scanning the client directly, and `--exif` reads each original through the
* library's content cache (`photo.original()`). The ML fetch is unchanged. The
* output tree is identical:
*
* <dir>/
* account.json { email, userID }
@@ -33,96 +37,57 @@ import {
import { join } from "node:path";
import { tmpdir } from "node:os";
import sodium from "libsodium-wrappers-sumo";
import { SRP, SrpServer } from "fast-srp-hap";
import { beforeAll, afterAll, describe, expect, it } from "vitest";
import {
init,
toBase64,
deriveKEK,
deriveLoginSubkey,
encryptBlob,
} from "../../src/crypto/index.js";
import { beforeAll, afterAll, describe, expect, it, vi } from "vitest";
import { init, toBase64, encryptBlob } from "../../src/crypto/index.js";
import * as jpegJs from "jpeg-js";
import { Client } from "../../src/client.js";
import { runMetadataBackup } from "../../src/metadata-backup.js";
import type { KeyAttributes } from "../../src/auth/types.js";
import { DISCOVER_CATEGORIES } from "../../src/clip-keyword-vectors.js";
import { Library } from "../../src/library/index.js";
import {
runMetadataBackup,
type MetadataBackupOptions,
} from "../../src/metadata-backup.js";
import { backupMetadataCommand } from "../../src/cli-commands.js";
import { stderrLog, type LogLevel } from "../../src/cli-log.js";
import {
buildAccount,
buildMetaFetch,
encryptSecretbox,
encryptStreamBlob,
type MetaMockState,
} from "../ente-server.js";
// One file per ML data request, so each file of the mock account is fetched in
// a request of its own and one of them can fail on its own.
vi.mock("../../src/mldata-fetch.js", async (importOriginal) => ({
...(await importOriginal<typeof import("../../src/mldata-fetch.js")>()),
MLDATA_BATCH_SIZE: 1,
}));
const TEST_EMAIL = "metabackup@example.com";
const TEST_PASSWORD = "metapass";
const TEST_OPS = 2;
const TEST_MEM = 64 * 1024 * 1024;
interface MetaMockState {
verifier: Buffer;
srpAttributes: Record<string, unknown>;
keyAttributes: KeyAttributes;
encryptedToken: string;
collections: Record<string, unknown>[];
filesByCollection: Record<number, Record<string, unknown>[]>;
// For ML data and EXIF tests
encryptedMLData: Record<
number,
{ encryptedData: string; decryptionHeader: string }
>;
fileCiphertexts: Record<number, Uint8Array>;
}
// The CLIP embedding of a photo of a sunset (see clip-keywords.test.ts), which
// the ML data of beach.jpg holds.
const SUNSET_EMBEDDING: number[] = JSON.parse(
readFileSync(
new URL("../clip-keywords-sunset.json", import.meta.url),
"utf-8",
),
).embedding;
// Identity, a Discover category of Ente's photos app that leaves out videos.
const IDENTITY = DISCOVER_CATEGORIES.find((c) => c.title === "Identity")!;
// ML data holding faces and no CLIP embedding, which file 302 has.
const FACES_ONLY = { face: { faces: [{ faceID: "face-def", score: 0.9 }] } };
let mock: MetaMockState;
let testDir: string;
const encryptSecretbox = (plaintext: Uint8Array, key: Uint8Array) => {
const nonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
const ciphertext = sodium.crypto_secretbox_easy(plaintext, nonce, key);
return { ciphertext, nonce };
};
const encryptStreamBlob = (plaintext: Uint8Array, key: Uint8Array) => {
const push = sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const ciphertext = sodium.crypto_secretstream_xchacha20poly1305_push(
push.state,
plaintext,
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
);
return { ciphertext, header: push.header };
};
const buildMetaMock = async (): Promise<MetaMockState> => {
const kekSalt = sodium.randombytes_buf(sodium.crypto_pwhash_SALTBYTES);
const kek = await deriveKEK(TEST_PASSWORD, kekSalt, TEST_OPS, TEST_MEM);
const loginSubKeyBytes = deriveLoginSubkey(kek);
const srpUserID = "meta-srp";
const srpSalt = sodium.randombytes_buf(16);
const verifier = SRP.computeVerifier(
SRP.params["4096"],
Buffer.from(srpSalt),
Buffer.from(srpUserID),
Buffer.from(loginSubKeyBytes),
);
const masterKey = sodium.randombytes_buf(32);
const { ciphertext: encMK, nonce: mkNonce } = encryptSecretbox(
masterKey,
kek,
);
const kp = sodium.crypto_box_keypair();
const { ciphertext: encSK, nonce: skNonce } = encryptSecretbox(
kp.privateKey,
masterKey,
);
const tokenBytes = sodium.randombytes_buf(32);
const encToken = sodium.crypto_box_seal(tokenBytes, kp.publicKey);
const keyAttributes: KeyAttributes = {
kekSalt: toBase64(kekSalt),
encryptedKey: toBase64(encMK),
keyDecryptionNonce: toBase64(mkNonce),
publicKey: toBase64(kp.publicKey),
encryptedSecretKey: toBase64(encSK),
secretKeyDecryptionNonce: toBase64(skNonce),
memLimit: TEST_MEM,
opsLimit: TEST_OPS,
};
const account = await buildAccount(TEST_PASSWORD);
const masterKey = account.masterKey;
// Collection 1: "Vacation" with collection-level pubMagicMetadata
const ck1 = sodium.crypto_secretbox_keygen();
@@ -157,14 +122,15 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
},
};
// Collection 2: "Work" with no magic metadata
// Collection 2: "../Work" with no magic metadata. The server chose a name
// that tries to climb out of the backup directory.
const ck2 = sodium.crypto_secretbox_keygen();
const { ciphertext: encCK2, nonce: ck2N } = encryptSecretbox(
ck2,
masterKey,
);
const { ciphertext: encCN2, nonce: cn2N } = encryptSecretbox(
new TextEncoder().encode("Work"),
new TextEncoder().encode("../Work"),
ck2,
);
const rawColl2 = {
@@ -254,6 +220,65 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
updationTime: 1710000000000000,
};
// A file in coll 10 with the ML data given. `fileType` is Ente's number: 0
// an image, 1 a video.
const vacationFile = (
id: number,
title: string,
fileType: number,
mlData: object,
) => {
const fileKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { ciphertext: encKey, nonce } = encryptSecretbox(fileKey, ck1);
const meta = encryptStreamBlob(
new TextEncoder().encode(
JSON.stringify({
title,
fileType,
creationTime: 1700000000000000,
modificationTime: 1700000000000000,
}),
),
fileKey,
);
const ml = encryptBlob(
new Uint8Array(gzipSync(Buffer.from(JSON.stringify(mlData)))),
fileKey,
);
return {
raw: {
id,
collectionID: 10,
ownerID: 42,
encryptedKey: toBase64(encKey),
keyDecryptionNonce: toBase64(nonce),
metadata: {
encryptedData: toBase64(meta.ciphertext),
decryptionHeader: toBase64(meta.header),
},
file: {
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
},
thumbnail: {
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
},
updationTime: 1700000000000000,
},
mlData: {
encryptedData: toBase64(ml.ciphertext),
decryptionHeader: toBase64(ml.header),
},
};
};
// Files 300 and 301 in coll 10: a photo and a video whose ML data holds
// only a CLIP embedding, Identity's own text embedding.
const identityML = { clip: { embedding: IDENTITY.textEmbedding } };
const identityPhoto = vacationFile(300, "passport.jpg", 0, identityML);
const identityVideo = vacationFile(301, "passport.mp4", 1, identityML);
// File 302 in coll 10: a photo whose ML data holds faces and no CLIP
// embedding.
const facesPhoto = vacationFile(302, "party.jpg", 0, FACES_ONLY);
// Encrypt ML data for file 100 (gzipped JSON, encrypted with file key)
const mlPayload = JSON.stringify({
face: {
@@ -280,7 +305,7 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
clip: {
version: 1,
client: "test",
embedding: [0.5, 0.6, 0.7],
embedding: SUNSET_EMBEDDING,
},
});
const gzipped = gzipSync(Buffer.from(mlPayload));
@@ -315,110 +340,33 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
rawFile1.file.decryptionHeader = toBase64(filePush1.header);
return {
verifier,
srpAttributes: {
srpUserID,
srpSalt: toBase64(srpSalt),
memLimit: TEST_MEM,
opsLimit: TEST_OPS,
kekSalt: toBase64(kekSalt),
isEmailMFAEnabled: false,
},
keyAttributes,
encryptedToken: toBase64(encToken),
verifier: account.verifier,
srpAttributes: account.srpAttributes,
keyAttributes: account.keyAttributes,
encryptedToken: account.encryptedToken,
collections: [rawColl1, rawColl2],
filesByCollection: { 10: [rawFile1], 20: [rawFile2] },
filesByCollection: {
10: [
rawFile1,
identityPhoto.raw,
identityVideo.raw,
facesPhoto.raw,
],
20: [rawFile2],
},
encryptedMLData: {
100: {
encryptedData: toBase64(mlCiphertext),
decryptionHeader: toBase64(mlHeader),
},
300: identityPhoto.mlData,
301: identityVideo.mlData,
302: facesPhoto.mlData,
},
fileCiphertexts: { 100: encFileBody1 },
};
};
const buildMetaFetch = (m: MetaMockState) => {
let srpServer: SrpServer;
return (async (
input: RequestInfo | URL,
init?: RequestInit,
): Promise<Response> => {
const url =
typeof input === "string"
? input
: input instanceof URL
? input.href
: input.url;
const path = new URL(url).pathname;
const json = (body: unknown) =>
new Response(JSON.stringify(body), {
status: 200,
headers: { "content-type": "application/json" },
});
if (path === "/users/srp/attributes")
return json({ attributes: m.srpAttributes });
if (path === "/users/srp/create-session") {
const body = JSON.parse(init?.body as string);
const serverKey = await SRP.genKey();
srpServer = new SrpServer(
SRP.params["4096"],
m.verifier,
serverKey,
);
const B = srpServer.computeB();
srpServer.setA(Buffer.from(body.srpA, "base64"));
return json({ sessionID: "s1", srpB: B.toString("base64") });
}
if (path === "/users/srp/verify-session") {
const body = JSON.parse(init?.body as string);
srpServer.checkM1(Buffer.from(body.srpM1, "base64"));
return json({
srpM2: srpServer.computeM2().toString("base64"),
id: 42,
keyAttributes: m.keyAttributes,
encryptedToken: m.encryptedToken,
});
}
if (path === "/collections/v2")
return json({ collections: m.collections });
if (path === "/collections/v2/diff") {
const collID = Number(
new URL(url).searchParams.get("collectionID"),
);
return json({
diff: m.filesByCollection[collID] ?? [],
hasMore: false,
});
}
if (path === "/files/data/fetch") {
const body = JSON.parse(init?.body as string);
const data = (body.fileIDs as number[])
.filter((id: number) => m.encryptedMLData[id])
.map((id: number) => ({
fileID: id,
...m.encryptedMLData[id],
updatedAt: 1700000000000000,
}));
return json({ data });
}
if (
url.includes("files.ente.io") ||
path.startsWith("/files/download/")
) {
const parsed = new URL(url);
const fileID = Number(
parsed.searchParams.get("fileID") ?? path.split("/").pop(),
);
const ct = m.fileCiphertexts[fileID];
if (ct) return new Response(ct, { status: 200 });
return new Response("not found", { status: 404 });
}
return new Response("not found", { status: 404 });
}) as typeof globalThis.fetch;
};
beforeAll(async () => {
await init();
await sodium.ready;
@@ -431,16 +379,50 @@ afterAll(() => {
rmSync(testDir, { recursive: true, force: true });
});
// Log in against the mock and open a library over its cache. The point commands
// open the library with the background precache off and a long refresh interval;
// the same here keeps the test deterministic (no thumbnail/original prefetch it
// did not ask for, no second refresh mid-test).
const openLib = async (client: Client): Promise<Library> =>
Library.open({
// The library client omits `fetchMLData`, matching how the CLI opens
// point commands: `runMetadataBackup` fetches ML data itself through
// the client, so the library's background backfill would only be a
// redundant second pass over the same endpoint.
client: {
whoami: () => client.whoami(),
collectionsSince: (args) => client.collectionsSince(args),
filesSince: (args) => client.filesSince(args),
contentSource: () => client.contentSource(),
},
cacheDirectory: mkdtempSync(join(testDir, "cache-")),
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
});
// Run one metadata backup end to end: fresh client, fresh library, then close.
const runBackup = async (
outDir: string,
opts?: MetadataBackupOptions,
): Promise<void> => {
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
const lib = await openLib(client);
try {
await runMetadataBackup(lib, client, outDir, opts);
} finally {
lib.close();
}
};
describe("quak backup-metadata", () => {
it("writes account.json with email and userID", async () => {
const outDir = join(testDir, "full");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runBackup(outDir);
const account = JSON.parse(
readFileSync(join(outDir, "account.json"), "utf-8"),
@@ -451,16 +433,11 @@ describe("quak backup-metadata", () => {
it("creates per-collection directories with _collection.json", async () => {
const outDir = join(testDir, "collections");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runBackup(outDir);
const collDirs = readdirSync(join(outDir, "collections"));
expect(collDirs.length).toBe(2);
// "../Work" is sanitized into one directory name.
expect(collDirs.sort()).toEqual(["10-Vacation", "20-__Work"]);
// Find the Vacation collection dir (prefixed with ID)
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
@@ -478,13 +455,7 @@ describe("quak backup-metadata", () => {
it("decrypts collection-level pubMagicMetadata", async () => {
const outDir = join(testDir, "coll-magic");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runBackup(outDir);
const collDirs = readdirSync(join(outDir, "collections"));
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
@@ -501,13 +472,7 @@ describe("quak backup-metadata", () => {
it("writes per-file JSON with all three metadata layers", async () => {
const outDir = join(testDir, "file-meta");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runBackup(outDir);
const collDirs = readdirSync(join(outDir, "collections"));
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
@@ -526,13 +491,7 @@ describe("quak backup-metadata", () => {
it("handles files with no magic metadata gracefully", async () => {
const outDir = join(testDir, "no-magic");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runBackup(outDir);
const collDirs = readdirSync(join(outDir, "collections"));
const workDir = collDirs.find((d) => d.includes("Work"))!;
@@ -550,14 +509,8 @@ describe("quak backup-metadata", () => {
it("is incremental: second run does not fail", async () => {
const outDir = join(testDir, "incremental");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runMetadataBackup(client, outDir);
await runBackup(outDir);
await runBackup(outDir);
const account = JSON.parse(
readFileSync(join(outDir, "account.json"), "utf-8"),
@@ -565,15 +518,11 @@ describe("quak backup-metadata", () => {
expect(account.email).toBe(TEST_EMAIL);
});
it("fetches and decrypts ML data by default", async () => {
// The CLIP embedding is left out of the ML data, and the keywords it gives
// are written in its place.
it("fetches and decrypts ML data by default, with keywords for the CLIP embedding", async () => {
const outDir = join(testDir, "ml-data");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runBackup(outDir);
const collDirs = readdirSync(join(outDir, "collections"));
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
@@ -591,19 +540,46 @@ describe("quak backup-metadata", () => {
expect(fileMeta.mlData.face.faces[0].faceID).toBe("face-abc");
expect(fileMeta.mlData.face.faces[0].score).toBeCloseTo(0.98);
expect(fileMeta.mlData.face.faces[0].detection.box.x).toBeCloseTo(0.1);
expect(fileMeta.mlData.clip).toBeDefined();
expect(fileMeta.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]);
expect(fileMeta.mlData).not.toHaveProperty("clip");
expect(fileMeta.keywords).toEqual(["Sunset"]);
});
// Ente's photos app keeps videos out of some Discover categories, Identity
// among them. Files 300 and 301, a photo and a video, share one CLIP
// embedding, Identity's own text embedding, so the photo gets the keyword
// Identity and the video does not.
it("keeps a video out of a category that leaves out videos", async () => {
expect(IDENTITY.showVideo).toBe(false);
const outDir = join(testDir, "video-keywords");
await runBackup(outDir);
const vacation = join(outDir, "collections", "10-Vacation");
const fileJSON = (fileID: number) =>
JSON.parse(readFileSync(join(vacation, `${fileID}.json`), "utf-8"));
expect(fileJSON(300).keywords).toContain("Identity");
expect(fileJSON(301)).toHaveProperty("keywords");
expect(fileJSON(301).keywords).not.toContain("Identity");
});
// A file's ML data can hold faces and no CLIP embedding. File 302's JSON
// then has the ML data and no keywords.
it("writes no keywords for ML data without a CLIP embedding", async () => {
const outDir = join(testDir, "no-clip");
await runBackup(outDir);
const fileMeta = JSON.parse(
readFileSync(
join(outDir, "collections", "10-Vacation", "302.json"),
"utf-8",
),
);
expect(fileMeta.mlData).toEqual(FACES_ONLY);
expect(fileMeta).not.toHaveProperty("keywords");
});
it("extracts EXIF from downloaded files when --exif is set", async () => {
const outDir = join(testDir, "exif-data");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir, { exif: true });
await runBackup(outDir, { exif: true });
const collDirs = readdirSync(join(outDir, "collections"));
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
@@ -619,5 +595,136 @@ describe("quak backup-metadata", () => {
expect(fileMeta.imageMetadata.format).toBe("jpeg");
expect(fileMeta.imageMetadata.width).toBe(100);
expect(fileMeta.imageMetadata.height).toBe(80);
expect(fileMeta.imageMetadataError).toBeUndefined();
// File 200 has no original on the mock server, so extraction fails
// and the reason is recorded instead of the field being left out.
const workDir = collDirs.find((d) => d.includes("Work"))!;
const failedMeta = JSON.parse(
readFileSync(
join(outDir, "collections", workDir, "200.json"),
"utf-8",
),
);
expect(failedMeta.imageMetadata).toBeUndefined();
expect(failedMeta.imageMetadataError).toEqual(expect.any(String));
});
});
describe("quak backup-metadata when a file's ML data cannot be read", () => {
// Run the CLI command against `fetch`, the mock unless given, logging at
// `level`, and return its exit code, stderr and output directory.
const runCommand = async (
fetch = buildMetaFetch(mock),
level: LogLevel = "info",
) => {
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: {
fetch,
retry: { sleep: async () => {} },
},
});
const outDir = mkdtempSync(join(testDir, "ml-fail-"));
let stderr = "";
const code = await backupMetadataCommand(
{
stdout: { write: () => true },
log: stderrLog(
{ write: (text: string) => (stderr += text) },
level,
),
sessionDir: testDir,
cacheDir: mkdtempSync(join(testDir, "cache-")),
loadSession: () => client,
},
outDir,
{},
);
return { code, stderr, outDir };
};
// The failed request is logged at error, so it is the first line printed
// at that level.
it("writes every file, marks the failed batch's files, and exits 1", async () => {
const { code, stderr, outDir } = await runCommand(
buildMetaFetch(mock, 200),
"error",
);
expect(code).toBe(1);
expect(stderr).toMatch(/^ML data request for 1 file\(s\) failed: /);
const ok = JSON.parse(
readFileSync(
join(outDir, "collections", "10-Vacation", "100.json"),
"utf-8",
),
);
expect(ok.keywords).toEqual(["Sunset"]);
expect(ok.mlDataError).toBeUndefined();
const failed = JSON.parse(
readFileSync(
join(outDir, "collections", "20-__Work", "200.json"),
"utf-8",
),
);
expect(failed.metadata.title).toBe("diagram.png");
expect(failed.mlData).toBeUndefined();
expect(failed.mlDataError).toContain("500");
});
// The server sends ML data for diagram.png encrypted under a key that is
// not the file's, so quak cannot decrypt it. That is logged at error, and
// the last line is fatal, so `-q`, which prints fatal lines only, still
// says why the command exits 1.
it("writes the reason for ML data that does not decrypt, and exits 1", async () => {
const otherKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptBlob(
new Uint8Array(gzipSync(Buffer.from("{}"))),
otherKey,
);
const { code, stderr, outDir } = await runCommand(
buildMetaFetch({
...mock,
encryptedMLData: {
...mock.encryptedMLData,
200: {
encryptedData: toBase64(ciphertext),
decryptionHeader: toBase64(header),
},
},
}),
"error",
);
expect(code).toBe(1);
const reason = "secretstream chunk authentication failed";
expect(stderr).toBe(
`ML data of file 200 could not be read: ${reason}\n` +
`quak: the ML data of 1 file(s) could not be read; see mlDataError in their JSON in ${outDir}\n`,
);
const failed = JSON.parse(
readFileSync(
join(outDir, "collections", "20-__Work", "200.json"),
"utf-8",
),
);
expect(failed.mlData).toBeUndefined();
expect(failed.mlDataError).toBe(reason);
const ok = JSON.parse(
readFileSync(
join(outDir, "collections", "10-Vacation", "100.json"),
"utf-8",
),
);
expect(ok.keywords).toEqual(["Sunset"]);
});
it("exits 0 when every ML data request succeeds", async () => {
const { code } = await runCommand();
expect(code).toBe(0);
});
});
test/cli/metadata-exif.test.ts
+368
View File
@@ -0,0 +1,368 @@
/**
* Tests for reading EXIF (`src/exif.ts`) and the image metadata
* `quak backup-metadata --exif` records.
*
* The originals come from users' libraries, so a truncated or corrupt file
* must neither hang the read nor throw out of it: `readAllExifTags` gives `{}`,
* and `backup-metadata` tells an EXIF block it cannot read apart from a file
* that simply has no EXIF, carrying the reason in `exifError`. Each JPEG below
* is a short hand-built byte array; the HEIC is a real file.
*/
import { describe, expect, it } from "vitest";
import {
readAllExifTags,
readExifTags,
readPhotoExif,
} from "../../src/exif.js";
import { extractImageMetadata } from "../../src/metadata-backup.js";
import { HEIC_WITH_EXIF } from "../exif-heic.js";
const SOI = [0xff, 0xd8]; // start of image
const SOS = [0xff, 0xda, 0x00, 0x02]; // start of scan
const EXIF_HEADER = [0x45, 0x78, 0x69, 0x66, 0x00, 0x00]; // "Exif\0\0"
const APP0 = [0xff, 0xe0, 0x00, 0x04, 0x00, 0x00];
const ZERO_LENGTH_APP0 = [0xff, 0xe0, 0x00, 0x00];
// A big-endian TIFF block with one IFD entry: Orientation (0x0112), SHORT, 6.
const TIFF_ORIENTATION_6 = [
0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08, 0x00, 0x01, 0x01, 0x12,
0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00,
];
// A big-endian TIFF block holding Orientation 6 and DateTimeOriginal
// "0000:00:00 00:00:00", which a camera with an unset clock writes.
const TIFF_UNSET_DATE = [
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
// The first IFD, at 8: two entries, then no next IFD.
...[0x00, 0x02],
// Orientation (0x0112), SHORT, 6.
...[0x01, 0x12, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x06, 0x00, 0x00],
// The Exif IFD's offset (0x8769), LONG, 38.
...[0x87, 0x69, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x26],
...[0x00, 0x00, 0x00, 0x00],
// The Exif IFD, at 38: one entry, then no next IFD.
...[0x00, 0x01],
// DateTimeOriginal (0x9003), 20 ASCII bytes at 56.
...[0x90, 0x03, 0x00, 0x02, 0x00, 0x00, 0x00, 0x14, 0x00, 0x00, 0x00, 0x38],
...[0x00, 0x00, 0x00, 0x00],
...new TextEncoder().encode("0000:00:00 00:00:00\0"), // at 56
];
// A big-endian TIFF block holding a GPSAltitude of 12.5 m and no
// GPSAltitudeRef.
const TIFF_ALTITUDE_WITHOUT_REF = [
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
// The first IFD, at 8: one entry, then no next IFD.
...[0x00, 0x01],
// The GPS IFD's offset (0x8825), LONG, 26.
...[0x88, 0x25, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x1a],
...[0x00, 0x00, 0x00, 0x00],
// The GPS IFD, at 26: one entry, then no next IFD.
...[0x00, 0x01],
// GPSAltitude (0x0006), one RATIONAL at 44.
...[0x00, 0x06, 0x00, 0x05, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x2c],
...[0x00, 0x00, 0x00, 0x00],
...[0x00, 0x00, 0x00, 0x19, 0x00, 0x00, 0x00, 0x02], // 25/2, at 44
];
// A big-endian TIFF block holding a GPSLatitude of 33° 30' 0" and a
// GPSLatitudeRef of "S".
const TIFF_SOUTHERN_LATITUDE = [
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
// The first IFD, at 8: one entry, then no next IFD.
...[0x00, 0x01],
// The GPS IFD's offset (0x8825), LONG, 26.
...[0x88, 0x25, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x1a],
...[0x00, 0x00, 0x00, 0x00],
// The GPS IFD, at 26: two entries, then no next IFD.
...[0x00, 0x02],
// GPSLatitudeRef (0x0001), 2 ASCII bytes: "S".
...[0x00, 0x01, 0x00, 0x02, 0x00, 0x00, 0x00, 0x02, 0x53, 0x00, 0x00, 0x00],
// GPSLatitude (0x0002), three RATIONALs at 56.
...[0x00, 0x02, 0x00, 0x05, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x38],
...[0x00, 0x00, 0x00, 0x00],
...[0x00, 0x00, 0x00, 0x21, 0x00, 0x00, 0x00, 0x01], // 33/1, at 56
...[0x00, 0x00, 0x00, 0x1e, 0x00, 0x00, 0x00, 0x01], // 30/1
...[0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01], // 0/1
];
// A big-endian TIFF block holding a GPSLatitude of 40° 26' 46" and a
// GPSLongitude of 79° 58' 56", and neither GPSLatitudeRef nor GPSLongitudeRef.
const TIFF_POSITION_WITHOUT_REFS = [
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
// The first IFD, at 8: one entry, then no next IFD.
...[0x00, 0x01],
// The GPS IFD's offset (0x8825), LONG, 26.
...[0x88, 0x25, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x1a],
...[0x00, 0x00, 0x00, 0x00],
// The GPS IFD, at 26: two entries, then no next IFD.
...[0x00, 0x02],
// GPSLatitude (0x0002), three RATIONALs at 56.
...[0x00, 0x02, 0x00, 0x05, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x38],
// GPSLongitude (0x0004), three RATIONALs at 80.
...[0x00, 0x04, 0x00, 0x05, 0x00, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x50],
...[0x00, 0x00, 0x00, 0x00],
...[0x00, 0x00, 0x00, 0x28, 0x00, 0x00, 0x00, 0x01], // 40/1, at 56
...[0x00, 0x00, 0x00, 0x1a, 0x00, 0x00, 0x00, 0x01], // 26/1
...[0x00, 0x00, 0x00, 0x2e, 0x00, 0x00, 0x00, 0x01], // 46/1
...[0x00, 0x00, 0x00, 0x4f, 0x00, 0x00, 0x00, 0x01], // 79/1, at 80
...[0x00, 0x00, 0x00, 0x3a, 0x00, 0x00, 0x00, 0x01], // 58/1
...[0x00, 0x00, 0x00, 0x38, 0x00, 0x00, 0x00, 0x01], // 56/1
];
// A big-endian TIFF block holding Orientation 6 and a Make whose value lies
// past the end of the file.
const TIFF_MAKE_PAST_END = [
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
// The first IFD, at 8: two entries, then no next IFD.
...[0x00, 0x02],
// Make (0x010f), 6 ASCII bytes at 4096.
...[0x01, 0x0f, 0x00, 0x02, 0x00, 0x00, 0x00, 0x06, 0x00, 0x00, 0x10, 0x00],
// Orientation (0x0112), SHORT, 6.
...[0x01, 0x12, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x06, 0x00, 0x00],
...[0x00, 0x00, 0x00, 0x00],
];
// A big-endian TIFF block with one IFD entry that exifreader has no name for:
// tag 0xc000 (49152), SHORT, 7.
const TIFF_UNNAMED_TAG = [
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
// The first IFD, at 8: one entry, then no next IFD.
...[0x00, 0x01],
// Tag 0xc000, SHORT, 7.
...[0xc0, 0x00, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x07, 0x00, 0x00],
...[0x00, 0x00, 0x00, 0x00],
];
// A big-endian TIFF block holding Orientation 6, and a thumbnail IFD holding
// its own Orientation 1 and a 4-byte JPEG thumbnail.
const TIFF_WITH_THUMBNAIL = [
...[0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08], // the first IFD at 8
// The first IFD, at 8: one entry, then the thumbnail IFD at 26.
...[0x00, 0x01],
// Orientation (0x0112), SHORT, 6.
...[0x01, 0x12, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x06, 0x00, 0x00],
...[0x00, 0x00, 0x00, 0x1a],
// The thumbnail IFD, at 26: three entries, then no next IFD.
...[0x00, 0x03],
// Orientation (0x0112), SHORT, 1.
...[0x01, 0x12, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x01, 0x00, 0x00],
// JPEGInterchangeFormat (0x0201), LONG: the thumbnail is at 68.
...[0x02, 0x01, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x44],
// JPEGInterchangeFormatLength (0x0202), LONG: 4 bytes.
...[0x02, 0x02, 0x00, 0x04, 0x00, 0x00, 0x00, 0x01, 0x00, 0x00, 0x00, 0x04],
...[0x00, 0x00, 0x00, 0x00],
...[0xff, 0xd8, 0xff, 0xd9], // the thumbnail, at 68: an empty JPEG
];
// An APP1 segment whose length field matches its data.
const app1 = (data: number[]): number[] => {
const len = data.length + 2;
return [0xff, 0xe1, len >> 8, len & 0xff, ...data];
};
const bytes = (...parts: number[][]): Uint8Array =>
new Uint8Array(parts.flat());
describe("readAllExifTags", () => {
it("keys a tag exifreader has no name for by its number", () => {
const data = [...EXIF_HEADER, ...TIFF_UNNAMED_TAG];
expect(readAllExifTags(bytes(SOI, app1(data), SOS))).toStrictEqual({
"undefined-49152": {
id: 49152,
value: 7,
description: 7,
computed: 7,
},
});
});
it("puts the thumbnail's tags under Thumbnail, without its image", () => {
const input = bytes(
SOI,
app1([...EXIF_HEADER, ...TIFF_WITH_THUMBNAIL]),
SOS,
);
// exifreader finds the thumbnail's image.
expect(readExifTags(input)?.Thumbnail?.type).toBe("image/jpeg");
const tags = readAllExifTags(input);
expect(tags.Orientation?.value).toBe(6);
expect(Object.keys(tags.Thumbnail ?? {}).sort()).toEqual([
"JPEGInterchangeFormat",
"JPEGInterchangeFormatLength",
"Orientation",
]);
expect(tags.Thumbnail?.Orientation?.value).toBe(1);
expect(readPhotoExif(tags)).toStrictEqual({ orientation: 6 });
});
});
describe("readPhotoExif", () => {
it("reads the common fields of a valid JPEG", () => {
const data = [...EXIF_HEADER, ...TIFF_ORIENTATION_6];
expect(
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
).toStrictEqual({
orientation: 6,
});
});
it("gives no dateTimeOriginal for a DateTimeOriginal of 0000:00:00 00:00:00", () => {
const data = [...EXIF_HEADER, ...TIFF_UNSET_DATE];
expect(
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
).toStrictEqual({
orientation: 6,
});
});
it("reads a GPSAltitude without GPSAltitudeRef as above sea level", () => {
const data = [...EXIF_HEADER, ...TIFF_ALTITUDE_WITHOUT_REF];
expect(
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
).toStrictEqual({
gpsAltitude: 12.5,
});
});
it("reads a GPSLatitude with GPSLatitudeRef S as south of the equator", () => {
const data = [...EXIF_HEADER, ...TIFF_SOUTHERN_LATITUDE];
expect(
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
).toStrictEqual({
gpsLatitude: -33.5,
});
});
it("gives no gpsLatitude or gpsLongitude without their reference tags", () => {
const data = [...EXIF_HEADER, ...TIFF_POSITION_WITHOUT_REFS];
const tags = readAllExifTags(bytes(SOI, app1(data), SOS));
// The position is read; only its hemisphere is unknown.
expect(tags.GPSLatitude?.computed).toStrictEqual([40, 26, 46]);
expect(tags.GPSLongitude?.computed).toStrictEqual([79, 58, 56]);
expect(readPhotoExif(tags)).toStrictEqual({});
});
it("gives no make for a Make whose value lies past the end of the file", () => {
const data = [...EXIF_HEADER, ...TIFF_MAKE_PAST_END];
expect(
readPhotoExif(readAllExifTags(bytes(SOI, app1(data), SOS))),
).toStrictEqual({
orientation: 6,
});
});
it.each([
[
"a file that is not an image",
new TextEncoder().encode("just some text, not an image"),
],
[
"a PNG without EXIF",
bytes([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]),
],
["a JPEG without EXIF", bytes(SOI, APP0, SOS)],
// A length under 8 cannot hold the six-byte "Exif\0\0" header. This one
// has length 7 and holds only "Exif\0", so a read past its end would
// take the next segment's bytes as EXIF.
[
"an APP1 segment too short to hold the Exif header",
bytes(SOI, app1(EXIF_HEADER.slice(0, 5)), SOS),
],
[
"an APP1 segment holding just the Exif header",
bytes(SOI, app1(EXIF_HEADER), SOS),
],
[
"a JPEG truncated inside a segment header",
bytes(SOI, [0xff, 0xe1, 0x00]),
],
["a JPEG that ends before the image data", bytes(SOI, APP0)],
// A length of 0 would step a scan by 2 bytes at a time through the
// rest of the file, reading garbage as markers.
[
"a zero-length segment",
bytes(
SOI,
ZERO_LENGTH_APP0,
ZERO_LENGTH_APP0,
ZERO_LENGTH_APP0,
ZERO_LENGTH_APP0,
SOS,
),
],
["a segment length of 1", bytes(SOI, [0xff, 0xe0, 0x00, 0x01], SOS)],
// APP1 claims 0x4000 bytes but only the "Exif\0\0" header follows.
[
"a segment length that runs past the end of the file",
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
],
// "XX" where the TIFF byte order belongs.
[
"an EXIF block that cannot be parsed",
bytes(SOI, app1([...EXIF_HEADER, 0x58, 0x58]), SOS),
],
])("returns no tags and no fields for %s", (_, input) => {
expect(readAllExifTags(input)).toStrictEqual({});
expect(readPhotoExif(readAllExifTags(input))).toStrictEqual({});
});
});
describe("extractImageMetadata", () => {
it("parses EXIF from a valid JPEG", () => {
const meta = extractImageMetadata(
bytes(SOI, app1([...EXIF_HEADER, ...TIFF_ORIENTATION_6]), SOS),
);
expect(meta?.exifError).toBeUndefined();
expect(meta?.exif).toMatchObject({ Orientation: { value: 6 } });
});
it("parses EXIF from a HEIC", () => {
const meta = extractImageMetadata(HEIC_WITH_EXIF);
expect(meta?.exifError).toBeUndefined();
expect(meta?.exif).toMatchObject({
Make: { value: ["Canon"] },
Model: { value: ["EOS R5"] },
DateTimeOriginal: { value: ["2021:07:15 14:30:00"] },
Orientation: { value: 6 },
GPSLatitudeRef: { value: ["N"] },
});
});
it("keys a tag exifreader has no name for by its number", () => {
const meta = extractImageMetadata(
bytes(SOI, app1([...EXIF_HEADER, ...TIFF_UNNAMED_TAG]), SOS),
);
expect(meta?.exifError).toBeUndefined();
expect(meta?.exif).toMatchObject({ "undefined-49152": { value: 7 } });
});
it("returns nothing for a file that is not an image", () => {
const text = new TextEncoder().encode("just some text, not an image");
expect(extractImageMetadata(text)).toBeUndefined();
});
it("records the reason when the JPEG is malformed", () => {
const meta = extractImageMetadata(
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
);
expect(meta?.exif).toBeUndefined();
expect(meta?.exifError).toBe(
"no tag could be read from the EXIF block",
);
});
it("keeps the raw bytes and the reason when EXIF cannot be parsed", () => {
// The raw bytes are the whole EXIF block as exifreader finds it: for a
// JPEG, the APP1 segment, marker and length included.
const segment = app1([...EXIF_HEADER, 0x58, 0x58]);
const meta = extractImageMetadata(bytes(SOI, segment, SOS));
expect(meta?.exif).toBeUndefined();
expect(meta?.exifRaw).toBe(Buffer.from(segment).toString("base64"));
expect(meta?.exifError).toBe(
"no tag could be read from the EXIF block",
);
});
});
test/cli/output.test.ts
+94
View File
@@ -0,0 +1,94 @@
// The CLI presents a file by its own decrypted metadata, not the PhotoRecord
// projection (issue #52). For a renamed file the two disagree: the projection
// prefers `editedName` and reports `editedTime` in milliseconds, while the CLI
// must print the raw `metadata.title` and `metadata.creationTime` (microseconds)
// and name downloads after the raw title, byte-identical to the pre-library CLI.
//
// This locks in that contrast: the shared output helpers emit the raw values,
// and the projection of the same file emits the edited ones — so a regression
// that re-sourced the CLI from the projection would fail here.
import { describe, it, expect } from "vitest";
import {
fileListRow,
fileListLine,
originalName,
thumbnailName,
} from "../../src/cli-output.js";
import { deriveRecords } from "../../src/library/records.js";
import type { EnteFile } from "../../src/model/types.js";
// Microseconds, as Ente stores times.
const RAW_CREATION = 1700000000000000;
const EDITED_TIME = 1710000000000000;
const RAW_TITLE = "IMG_0001.HEIC";
const EDITED_NAME = "Sunset.heic";
// A file the user has renamed and re-dated: basic metadata holds the original
// title and capture time; public magic metadata holds the edits.
const renamedFile: EnteFile = {
id: 100,
collectionID: 10,
ownerID: 42,
key: new Uint8Array(),
metadata: {
title: RAW_TITLE,
fileType: "image",
creationTime: RAW_CREATION,
modificationTime: RAW_CREATION,
},
pubMagicMetadata: { editedName: EDITED_NAME, editedTime: EDITED_TIME },
file: { decryptionHeader: "" },
thumbnail: { decryptionHeader: "" },
updationTime: RAW_CREATION,
};
describe("CLI file output (issue #52)", () => {
it("emits the raw title and microsecond creationTime for --json", () => {
expect(fileListRow(renamedFile)).toEqual({
id: 100,
title: RAW_TITLE,
fileType: "image",
creationTime: RAW_CREATION,
collectionID: 10,
});
});
it("emits the raw title in the human column", () => {
expect(fileListLine(renamedFile)).toBe(`100\timage\t${RAW_TITLE}`);
});
it("names downloads after the raw title", () => {
expect(originalName(renamedFile)).toBe(RAW_TITLE);
expect(thumbnailName(renamedFile)).toBe(`thumb_${RAW_TITLE}`);
});
it("sanitizes the title when naming `quak get` downloads", () => {
// Without `--out`, the server-supplied title names the file, so it must
// not be able to point outside the working directory.
const hostile = {
...renamedFile,
metadata: { ...renamedFile.metadata, title: "../../.bashrc" },
};
expect(originalName(hostile)).toBe("__.._.bashrc");
expect(thumbnailName(hostile)).toBe("thumb___.._.bashrc");
const untitled = {
...renamedFile,
metadata: { ...renamedFile.metadata, title: "" },
};
expect(originalName(untitled)).toBe("file-100");
expect(thumbnailName(untitled)).toBe("thumb_file-100");
});
it("does not use the editedName/editedTime projection", () => {
const record = deriveRecords([], [renamedFile]).photos.get(100);
// The projection prefers the edits and reports milliseconds; the CLI
// helpers above deliberately do not.
expect(record?.title).toBe(EDITED_NAME);
expect(record?.takenAt).toBe(Math.floor(EDITED_TIME / 1000));
expect(fileListRow(renamedFile).title).not.toBe(record?.title);
expect(fileListRow(renamedFile).creationTime).not.toBe(record?.takenAt);
});
});
test/cli/read.test.ts
+274
View File
@@ -0,0 +1,274 @@
/**
* Tests for the CLI read helpers (`src/cli-read.ts`, owner amendment to
* issue #36, issue #52).
*
* The `collections`, `files`, `get`, and `get-thumb` commands must answer for
* current server state, not the local cache. `files` without a collection ID,
* `get` and `get-thumb` force a `Library.fresh()` round-trip before they read.
* The stand-in library below serves nothing until `fresh()` has been awaited.
* A helper that read without refreshing would come back empty and fail here.
* `collections` asks the library for the collection list alone
* (`freshCollections()`, issue #210). It must not wait for a refresh, which
* would fetch the files of every collection. `files COLLECTION` reads that list
* too, then brings that one collection up to date (`refreshCollection()`,
* issue #217). The stand-in serves a collection's files once `fresh()` has run,
* or once `refreshCollection()` has resolved to `true` for that collection.
* When `refreshCollection()` resolves to `false`, `files COLLECTION` forces a
* refresh instead.
*
* `collections` and `files` also list in the library's enumeration order
* (`listCollections`/`listFiles`) — the order the pre-library CLI printed — not
* the albums/photos projection's newest-first order. The fixtures are seeded in
* an enumeration order that a newest-first sort would rearrange, so a
* regression to the projection order would fail here too. Field values still
* come from the raw metadata via `cli-output.ts`.
*/
import { describe, it, expect } from "vitest";
import {
freshCollections,
freshFiles,
freshFile,
type FreshReadLibrary,
} from "../../src/cli-read.js";
import { fileListRow } from "../../src/cli-output.js";
import type { Photo } from "../../src/library/index.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
const collection = (id: number, updationTime: number): Collection => ({
id,
ownerID: 42,
key: new Uint8Array(),
name: `album-${id}`,
type: "album",
updationTime,
isShared: false,
});
// Microseconds, as Ente stores times.
const file = (
id: number,
collectionID: number,
creationTime: number,
): EnteFile => ({
id,
collectionID,
ownerID: 42,
key: new Uint8Array(),
metadata: {
title: `file-${id}.jpg`,
fileType: "image",
creationTime,
modificationTime: creationTime,
},
file: { decryptionHeader: "" },
thumbnail: { decryptionHeader: "" },
updationTime: creationTime,
});
// A library that reveals its records only after `fresh()` has been awaited, and
// serves them in the enumeration order it was given. `freshCollections()`
// answers the collection list in that order without a refresh. When
// `refreshCollection()` resolves to true, it reveals the files of that one
// collection.
// `photos.byID` returns a stand-in `Photo` carrying just the fileID the helper
// passes through.
class FakeLibrary implements FreshReadLibrary {
freshCalls = 0;
freshCollectionsCalls = 0;
refreshCollectionIDs: number[] = [];
// What `refreshCollection()` resolves to. With false it reveals nothing,
// as a library that has had a refresh does for a collection it lacks.
refreshCollectionHolds = true;
private refreshed = false;
constructor(
private readonly collections: Collection[],
private readonly files: EnteFile[],
) {}
async fresh(): Promise<unknown> {
this.freshCalls++;
this.refreshed = true;
return {};
}
async freshCollections(): Promise<Collection[]> {
this.freshCollectionsCalls++;
return this.collections;
}
async refreshCollection(collection: Collection): Promise<boolean> {
this.refreshCollectionIDs.push(collection.id);
return this.refreshCollectionHolds;
}
getCollection(id: number): Collection | undefined {
if (!this.refreshed) return undefined;
return this.collections.find((c) => c.id === id);
}
listCollections(): Collection[] {
return this.refreshed ? this.collections : [];
}
listFiles(collectionID: number): EnteFile[] {
const current =
this.refreshed ||
(this.refreshCollectionHolds &&
this.refreshCollectionIDs.includes(collectionID));
return current
? this.files.filter((f) => f.collectionID === collectionID)
: [];
}
getFileByID(fileID: number): EnteFile | undefined {
if (!this.refreshed) return undefined;
return this.files.find((f) => f.id === fileID);
}
photos = {
byID: ({ fileID }: { fileID: number }): Photo | undefined => {
if (!this.refreshed) return undefined;
if (!this.files.some((f) => f.id === fileID)) return undefined;
return { fileID } as unknown as Photo;
},
};
}
describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
it("freshCollections reads the collection list alone, in enumeration order", async () => {
// Enumeration order 2, 1, 3; a newest-first sort would be 3, 2, 1.
const lib = new FakeLibrary(
[collection(2, 200), collection(1, 300), collection(3, 100)],
[],
);
const rows = await freshCollections(lib);
expect(lib.freshCollectionsCalls).toBe(1);
expect(lib.freshCalls).toBe(0);
expect(rows.map((c) => c.id)).toEqual([2, 1, 3]);
// The projection's newest-first order is a different sequence, so this
// is not accidentally that order.
const newestFirst = [...rows]
.sort((a, b) => b.updationTime - a.updationTime)
.map((c) => c.id);
expect(newestFirst).toEqual([1, 2, 3]);
expect(rows.map((c) => c.id)).not.toEqual(newestFirst);
});
it("freshFiles brings the collection alone up to date, lists in enumeration order, keeps raw fields", async () => {
// Enumeration order by id 10, 11, 12; creationTimes ascending, so a
// newest-first sort would reverse them. Collection 2's file is not
// listed.
const files = [
file(10, 1, 1_700_000_000_000_000),
file(11, 1, 1_700_000_000_000_001),
file(12, 1, 1_700_000_000_000_002),
file(20, 2, 1_700_000_000_000_003),
];
const lib = new FakeLibrary(
[collection(1, 100), collection(2, 100)],
files,
);
const rows = await freshFiles(lib, 1);
expect(lib.freshCollectionsCalls).toBe(1);
expect(lib.refreshCollectionIDs).toEqual([1]);
expect(lib.freshCalls).toBe(0);
expect(rows?.map((f) => f.id)).toEqual([10, 11, 12]);
// Field values come from raw metadata: microsecond creationTime and the
// raw title, unchanged.
expect(rows?.map(fileListRow)).toEqual([
{
id: 10,
title: "file-10.jpg",
fileType: "image",
creationTime: 1_700_000_000_000_000,
collectionID: 1,
},
{
id: 11,
title: "file-11.jpg",
fileType: "image",
creationTime: 1_700_000_000_000_001,
collectionID: 1,
},
{
id: 12,
title: "file-12.jpg",
fileType: "image",
creationTime: 1_700_000_000_000_002,
collectionID: 1,
},
]);
});
// Without a collection ID, `files` lists every file in the account once
// (issue #211). File 11 is in collections 1 and 2; collection 2 is
// enumerated first, so 11 is listed once, with collection 2's record.
it("freshFiles without a collection ID lists every file once, from the first collection holding it", async () => {
const files = [
file(10, 1, 1_700_000_000_000_000),
file(11, 1, 1_700_000_000_000_001),
file(20, 2, 1_700_000_000_000_002),
file(11, 2, 1_700_000_000_000_001),
];
const lib = new FakeLibrary(
[collection(2, 100), collection(1, 200)],
files,
);
const rows = await freshFiles(lib);
expect(lib.freshCalls).toBe(1);
expect(rows?.map((f) => [f.id, f.collectionID])).toEqual([
[20, 2],
[11, 2],
[10, 1],
]);
});
it("freshFiles forces a refresh when refreshCollection resolves to false", async () => {
const lib = new FakeLibrary(
[collection(1, 100)],
[file(10, 1, 1_700_000_000_000_000)],
);
lib.refreshCollectionHolds = false;
const rows = await freshFiles(lib, 1);
expect(lib.refreshCollectionIDs).toEqual([1]);
expect(lib.freshCalls).toBe(1);
expect(rows?.map((f) => f.id)).toEqual([10]);
});
it("freshFiles returns undefined for an unknown collection", async () => {
const lib = new FakeLibrary([collection(1, 100)], []);
const rows = await freshFiles(lib, 999);
expect(lib.freshCollectionsCalls).toBe(1);
expect(lib.refreshCollectionIDs).toEqual([]);
expect(lib.freshCalls).toBe(0);
expect(rows).toBeUndefined();
});
it("freshFile refreshes first, then resolves the photo and its raw record", async () => {
const f = file(10, 1, 1_700_000_000_000_000);
const lib = new FakeLibrary([collection(1, 100)], [f]);
const resolved = await freshFile(lib, 10);
expect(lib.freshCalls).toBe(1);
expect(resolved?.photo.fileID).toBe(10);
expect(resolved?.file.metadata.title).toBe("file-10.jpg");
expect(resolved?.file.metadata.creationTime).toBe(
1_700_000_000_000_000,
);
});
it("freshFile returns undefined for an unknown file", async () => {
const lib = new FakeLibrary([collection(1, 100)], []);
const resolved = await freshFile(lib, 404);
expect(lib.freshCalls).toBe(1);
expect(resolved).toBeUndefined();
});
});
test/cli/run.test.ts
+168
View File
@@ -0,0 +1,168 @@
/**
* Tests for `run` in `src/cli-run.ts`, which every CLI command goes through.
*/
import { PassThrough } from "node:stream";
import { describe, it, expect } from "vitest";
import { ApiError } from "../../src/api/client.js";
import { stderrLog } from "../../src/cli-log.js";
import { run } from "../../src/cli-run.js";
// A stream whose written text is kept in `text`; writes finish at once, so
// nothing is left waiting to drain.
const collector = (): { stream: PassThrough; text: () => string } => {
const stream = new PassThrough();
const chunks: string[] = [];
stream.on("data", (chunk: Buffer) => chunks.push(chunk.toString()));
return { stream, text: () => chunks.join("") };
};
const runToExit = async (
command: Promise<number>,
): Promise<{ code: number; stdout: string; stderr: string }> => {
const stdout = collector();
const stderr = collector();
const code = await new Promise<number>((resolve) => {
void run(
command,
stdout.stream,
stderr.stream,
stderrLog(stderr.stream, "info"),
resolve,
);
});
return { code, stdout: stdout.text(), stderr: stderr.text() };
};
describe("run", () => {
it("exits with the code the command returns", async () => {
const result = await runToExit(Promise.resolve(3));
expect(result).toEqual({ code: 3, stdout: "", stderr: "" });
});
it("prints a thrown error as one line without a stack trace and exits 1", async () => {
const failing = async (): Promise<number> => {
throw new Error(
"ENOTDIR: not a directory, mkdir '/dev/null/x/originals'",
);
};
const result = await runToExit(failing());
expect(result.code).toBe(1);
expect(result.stdout).toBe("");
expect(result.stderr).toBe(
"quak: ENOTDIR: not a directory, mkdir '/dev/null/x/originals'\n",
);
});
// Node's `fetch` rejects every failure below HTTP with a TypeError that
// says only "fetch failed", and hangs the reason, with its errno, off
// `cause`. The printed line carries the reason, so a cron log tells a
// refused connection from a name that did not resolve.
it("prints the reason a fetch failed, from the error's cause", async () => {
const refused = new TypeError("fetch failed", {
cause: Object.assign(
new Error("connect ECONNREFUSED 1.2.3.4:443"),
{ code: "ECONNREFUSED" },
),
});
const result = await runToExit(Promise.reject(refused));
expect(result).toEqual({
code: 1,
stdout: "",
stderr: "quak: fetch failed: connect ECONNREFUSED 1.2.3.4:443\n",
});
});
// When the host has several addresses and every one refuses, Node's
// `fetch` gives as the cause an AggregateError with an empty message and
// one error per address. The printed line gives each address's reason.
it("prints the reason for each address a fetch could not reach", async () => {
const refusedAt = (address: string): Error =>
Object.assign(new Error(`connect ECONNREFUSED ${address}`), {
code: "ECONNREFUSED",
});
const refused = new TypeError("fetch failed", {
cause: Object.assign(
new AggregateError([
refusedAt("2001:db8::1:443"),
refusedAt("1.2.3.4:443"),
]),
{ code: "ECONNREFUSED" },
),
});
const result = await runToExit(Promise.reject(refused));
expect(result).toEqual({
code: 1,
stdout: "",
stderr: "quak: fetch failed: connect ECONNREFUSED 2001:db8::1:443, connect ECONNREFUSED 1.2.3.4:443\n",
});
});
// An error that already quotes its cause's message is printed as it is.
it("does not print a cause's message twice", async () => {
const reset = new Error("read ECONNRESET");
const wrapped = new Error(`download: file 7: ${reset.message}`, {
cause: reset,
});
const result = await runToExit(Promise.reject(wrapped));
expect(result.stderr).toBe("quak: download: file 7: read ECONNRESET\n");
});
// OpenSSL ends its message for a failed TLS connection, such as one to a
// server that answers in plain HTTP, with a newline. The printed line is
// still one line.
it("prints a cause's message without the white space around it", async () => {
const wrongVersion = new TypeError("fetch failed", {
cause: Object.assign(
new Error(
"C0B73A55A47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:\n",
),
{ code: "ERR_SSL_WRONG_VERSION_NUMBER" },
),
});
const result = await runToExit(Promise.reject(wrongVersion));
expect(result.stderr).toBe(
"quak: fetch failed: C0B73A55A47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:\n",
);
});
it("prints a thrown value that is not an Error", async () => {
const result = await runToExit(Promise.reject("offline"));
expect(result).toEqual({
code: 1,
stdout: "",
stderr: "quak: offline\n",
});
});
// A cause that is not an Error, such as a thrown string, is printed as a
// thrown value that is not an Error is.
it("prints a cause that is not an Error", async () => {
const wrapped = new Error("ML data", { cause: "offline" });
const result = await runToExit(Promise.reject(wrapped));
expect(result.stderr).toBe("quak: ML data: offline\n");
});
it("on a 401 from the server says to run quak login, on one line, and exits 3", async () => {
const result = await runToExit(
Promise.reject(new ApiError("unauthorized", 401)),
);
expect(result).toEqual({
code: 3,
stdout: "",
stderr: `quak: the saved session is no longer valid; run "quak login"\n`,
});
});
it("prints another HTTP error as it is and exits 1", async () => {
const result = await runToExit(
Promise.reject(new ApiError("forbidden", 403)),
);
expect(result).toEqual({
code: 1,
stdout: "",
stderr: "quak: forbidden\n",
});
});
});
test/client/enumeration.test.ts
+483
View File
@@ -0,0 +1,483 @@
/**
* Tests for the resumable, deletion-aware enumeration variants on `Client`:
* `collectionsSince` and `filesSince`.
*
* The whole-account methods `listCollections` / `listFiles` always start at
* `sinceTime: 0` and hide deletions. The cache refresh needs the opposite:
* start from a saved cursor, learn what was deleted, and get back a cursor to
* resume from next time. These two methods provide that.
*
* The return shape keeps live records and tombstones apart — `collections` /
* `files` are decrypted live records, `deleted` is a plain list of the ids the
* server tombstoned. A tombstone carries no decryptable key or metadata, so it
* is a bare id rather than a hollowed-out `Collection` / `EnteFile`. A live row
* that does not decrypt, or whose metadata is not JSON, is listed under
* `undecryptable` with the reason, so one bad row does not fail the call.
*
* All tests inject a fake `fetch` and drive a real `Client` (built with
* `Client.fromJSON`) so the decryption path runs for real. Live rows are built
* with libsodium exactly as the server would encrypt them; tombstone rows carry
* only the fields the code reads (`id`, `updationTime`, `isDeleted`), because
* they are never decrypted.
*/
import sodium from "libsodium-wrappers-sumo";
import { beforeAll, describe, expect, it } from "vitest";
import { init, toBase64 } from "../../src/crypto/index.js";
import { Client, type ClientSnapshot } from "../../src/client.js";
// ---------------------------------------------------------------------------
// Fixtures
// ---------------------------------------------------------------------------
const USER_ID = 42;
interface Keys {
masterKey: Uint8Array;
publicKey: Uint8Array;
secretKey: Uint8Array;
}
const buildKeys = (): Keys => {
const kp = sodium.crypto_box_keypair();
return {
masterKey: sodium.crypto_secretbox_keygen(),
publicKey: kp.publicKey,
secretKey: kp.privateKey,
};
};
const snapshotFor = (keys: Keys): ClientSnapshot => ({
email: "user@example.com",
userID: USER_ID,
token: "test-token",
masterKey: toBase64(keys.masterKey),
secretKey: toBase64(keys.secretKey),
publicKey: toBase64(keys.publicKey),
});
const secretboxEncrypt = (
plaintext: Uint8Array,
key: Uint8Array,
): { ciphertext: Uint8Array; nonce: Uint8Array } => {
const nonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
return {
ciphertext: sodium.crypto_secretbox_easy(plaintext, nonce, key),
nonce,
};
};
/** An owned collection row as the server sends it, keyed under the master key. */
const ownedCollectionRow = (
masterKey: Uint8Array,
opts: { id: number; name: string; updationTime: number },
): Record<string, unknown> => {
const collectionKey = sodium.crypto_secretbox_keygen();
const { ciphertext: encKey, nonce: keyNonce } = secretboxEncrypt(
collectionKey,
masterKey,
);
const { ciphertext: encName, nonce: nameNonce } = secretboxEncrypt(
new TextEncoder().encode(opts.name),
collectionKey,
);
return {
id: opts.id,
owner: { id: USER_ID },
encryptedKey: toBase64(encKey),
keyDecryptionNonce: toBase64(keyNonce),
encryptedName: toBase64(encName),
nameDecryptionNonce: toBase64(nameNonce),
type: "album",
updationTime: opts.updationTime,
};
};
/**
* A live file row inside a collection, keyed under that collection's key. Its
* metadata decrypts to `metadata` when given, otherwise to the JSON a client
* uploads for `title`.
*/
const fileRow = (
collectionKey: Uint8Array,
opts: {
id: number;
title: string;
updationTime: number;
metadata?: string;
},
): Record<string, unknown> => {
const fileKey = sodium.crypto_secretbox_keygen();
const { ciphertext: encFileKey, nonce: fileKeyNonce } = secretboxEncrypt(
fileKey,
collectionKey,
);
const metadata =
opts.metadata ??
JSON.stringify({
title: opts.title,
fileType: 0,
creationTime: opts.updationTime,
modificationTime: opts.updationTime,
});
const push =
sodium.crypto_secretstream_xchacha20poly1305_init_push(fileKey);
const encMeta = sodium.crypto_secretstream_xchacha20poly1305_push(
push.state,
new TextEncoder().encode(metadata),
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
);
return {
id: opts.id,
collectionID: 1,
ownerID: USER_ID,
encryptedKey: toBase64(encFileKey),
keyDecryptionNonce: toBase64(fileKeyNonce),
metadata: {
encryptedData: toBase64(encMeta),
decryptionHeader: toBase64(push.header),
},
file: { decryptionHeader: toBase64(sodium.randombytes_buf(24)) },
thumbnail: { decryptionHeader: toBase64(sodium.randombytes_buf(24)) },
updationTime: opts.updationTime,
};
};
/**
* `row` with its sealed key replaced by random bytes of the same length, as a
* corrupt row arrives: no key opens it, so the row cannot be decrypted.
*/
const withCorruptKey = (
row: Record<string, unknown>,
): Record<string, unknown> => ({
...row,
encryptedKey: toBase64(
sodium.randombytes_buf(
sodium.crypto_secretbox_KEYBYTES + sodium.crypto_secretbox_MACBYTES,
),
),
});
/** A tombstone row. Never decrypted, so only these fields are ever read. */
const tombstoneRow = (
id: number,
updationTime: number,
): Record<string, unknown> => ({
id,
updationTime,
isDeleted: true,
});
const jsonResponse = (body: unknown): Response =>
new Response(JSON.stringify(body), {
status: 200,
headers: { "content-type": "application/json" },
});
/**
* A fetch that serves canned responses in order and records the `sinceTime`
* query parameter each request carried, so tests can prove the cursor is
* threaded from one page (and one call) to the next.
*/
const recordingFetch = (
...responses: Response[]
): { fetch: typeof globalThis.fetch; sinceTimes: (string | null)[] } => {
const sinceTimes: (string | null)[] = [];
let i = 0;
const fake = async (input: RequestInfo | URL): Promise<Response> => {
const url =
typeof input === "string"
? input
: input instanceof URL
? input.href
: input.url;
sinceTimes.push(new URL(url).searchParams.get("sinceTime"));
if (i >= responses.length) {
throw new Error(`recordingFetch: no response for call #${i}`);
}
return responses[i++]!;
};
return { fetch: fake as typeof globalThis.fetch, sinceTimes };
};
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
describe("Client.filesSince", () => {
beforeAll(async () => {
await init();
await sodium.ready;
});
it("pages from the given cursor, decrypts live rows, and collects tombstones", async () => {
const keys = buildKeys();
const collectionKey = sodium.crypto_secretbox_keygen();
// Page 1 mixes a live file and a tombstone; the tombstone has the
// higher updationTime, so it — not the live row — sets the cursor the
// second page must be fetched from.
const { fetch, sinceTimes } = recordingFetch(
jsonResponse({
diff: [
fileRow(collectionKey, {
id: 1001,
title: "first.jpg",
updationTime: 100,
}),
tombstoneRow(1002, 150),
],
hasMore: true,
}),
jsonResponse({
diff: [
fileRow(collectionKey, {
id: 1003,
title: "second.jpg",
updationTime: 200,
}),
],
hasMore: false,
}),
);
const client = Client.fromJSON(snapshotFor(keys), { fetch });
const { files, deleted, cursor } = await client.filesSince({
collectionID: 1,
collectionKey,
sinceTime: 0,
});
expect(files.map((f) => f.id)).toEqual([1001, 1003]);
expect(files.map((f) => f.metadata.title)).toEqual([
"first.jpg",
"second.jpg",
]);
expect(deleted).toEqual([1002]);
expect(cursor).toBe(200);
// First request started at the caller's cursor; the second resumed
// from the max updationTime seen on the first page (the tombstone's).
expect(sinceTimes).toEqual(["0", "150"]);
});
it("fetches only newer rows when the returned cursor is passed back in", async () => {
const keys = buildKeys();
const collectionKey = sodium.crypto_secretbox_keygen();
const { fetch, sinceTimes } = recordingFetch(
jsonResponse({ diff: [], hasMore: false }),
);
const client = Client.fromJSON(snapshotFor(keys), { fetch });
const result = await client.filesSince({
collectionID: 1,
collectionKey,
sinceTime: 200,
});
expect(result.files).toEqual([]);
expect(result.deleted).toEqual([]);
// An empty diff advances nothing: the cursor falls back to the input.
expect(result.cursor).toBe(200);
expect(sinceTimes).toEqual(["200"]);
});
it("stops and throws when the server claims more but does not advance (#7)", async () => {
const keys = buildKeys();
const collectionKey = sodium.crypto_secretbox_keygen();
// hasMore is true, but the page's max updationTime (50) does not exceed
// the cursor the request was made with (50). Following hasMore here
// would refetch this same page forever.
const { fetch, sinceTimes } = recordingFetch(
jsonResponse({ diff: [tombstoneRow(1, 50)], hasMore: true }),
jsonResponse({ diff: [tombstoneRow(1, 50)], hasMore: true }),
);
const client = Client.fromJSON(snapshotFor(keys), { fetch });
await expect(
client.filesSince({
collectionID: 1,
collectionKey,
sinceTime: 50,
}),
).rejects.toThrow(/not advance|non-advancing/i);
// It gave up after the first page rather than looping.
expect(sinceTimes).toEqual(["50"]);
});
it("leaves out each file it cannot decrypt or parse, with the reason, and decrypts the rest", async () => {
const keys = buildKeys();
const collectionKey = sodium.crypto_secretbox_keygen();
// File 1002's key is sealed under no key quak holds, and file 1003's
// metadata decrypts to text that is not JSON. Neither may cost the
// caller file 1001: each is left out of `files` and reported under
// `undecryptable` with the reason instead of failing the call.
const { fetch } = recordingFetch(
jsonResponse({
diff: [
fileRow(collectionKey, {
id: 1001,
title: "fine.jpg",
updationTime: 100,
}),
withCorruptKey(
fileRow(collectionKey, {
id: 1002,
title: "bad-key.jpg",
updationTime: 200,
}),
),
fileRow(collectionKey, {
id: 1003,
title: "bad-metadata.jpg",
updationTime: 300,
metadata: "not json",
}),
],
hasMore: false,
}),
);
const client = Client.fromJSON(snapshotFor(keys), { fetch });
const { files, undecryptable, cursor } = await client.filesSince({
collectionID: 1,
collectionKey,
sinceTime: 0,
});
expect(files.map((f) => f.id)).toEqual([1001]);
expect(undecryptable).toEqual([
{ id: 1002, error: expect.stringContaining("wrong secret key") },
{ id: 1003, error: expect.stringContaining("JSON") },
]);
// The cursor moves past the two left out, as past any other row, so
// the server does not send them again until its copy changes. The
// caller has to keep what it was told here.
expect(cursor).toBe(300);
});
});
describe("Client.collectionsSince", () => {
beforeAll(async () => {
await init();
await sodium.ready;
});
it("decrypts live collections, collects tombstones, and returns a cursor", async () => {
const keys = buildKeys();
const { fetch, sinceTimes } = recordingFetch(
jsonResponse({
collections: [
ownedCollectionRow(keys.masterKey, {
id: 1,
name: "Vacation",
updationTime: 100,
}),
tombstoneRow(3, 150),
],
}),
);
const client = Client.fromJSON(snapshotFor(keys), { fetch });
const { collections, deleted, cursor } = await client.collectionsSince({
sinceTime: 0,
});
expect(collections.map((c) => c.id)).toEqual([1]);
expect(collections[0]!.name).toBe("Vacation");
expect(deleted).toEqual([3]);
// The tombstone's updationTime advances the cursor too, so the next
// sync starts after it rather than seeing it again.
expect(cursor).toBe(150);
expect(sinceTimes).toEqual(["0"]);
});
it("falls back to the input cursor on an empty response", async () => {
const keys = buildKeys();
const { fetch, sinceTimes } = recordingFetch(
jsonResponse({ collections: [] }),
);
const client = Client.fromJSON(snapshotFor(keys), { fetch });
const result = await client.collectionsSince({ sinceTime: 150 });
expect(result.collections).toEqual([]);
expect(result.deleted).toEqual([]);
expect(result.cursor).toBe(150);
expect(sinceTimes).toEqual(["150"]);
});
it("leaves out an album it cannot decrypt, with the reason, and decrypts the rest", async () => {
const keys = buildKeys();
// Album 2's key is sealed under no key quak holds. Without its key
// neither its name nor its files can be read, so it is left out and
// reported under `undecryptable`; album 1 is unaffected.
const { fetch } = recordingFetch(
jsonResponse({
collections: [
ownedCollectionRow(keys.masterKey, {
id: 1,
name: "Vacation",
updationTime: 100,
}),
withCorruptKey(
ownedCollectionRow(keys.masterKey, {
id: 2,
name: "Broken",
updationTime: 200,
}),
),
],
}),
);
const client = Client.fromJSON(snapshotFor(keys), { fetch });
const { collections, undecryptable, cursor } =
await client.collectionsSince({ sinceTime: 0 });
expect(collections.map((c) => c.name)).toEqual(["Vacation"]);
expect(undecryptable).toEqual([
{ id: 2, error: expect.stringContaining("wrong secret key") },
]);
expect(cursor).toBe(200);
});
});
describe("Client list wrappers still hide deletions", () => {
beforeAll(async () => {
await init();
await sodium.ready;
});
it("listFiles drops tombstones and returns only live files", async () => {
const keys = buildKeys();
const collectionKey = sodium.crypto_secretbox_keygen();
const { fetch, sinceTimes } = recordingFetch(
jsonResponse({
diff: [
fileRow(collectionKey, {
id: 7,
title: "keep.jpg",
updationTime: 100,
}),
tombstoneRow(8, 150),
],
hasMore: false,
}),
);
const client = Client.fromJSON(snapshotFor(keys), { fetch });
const files = await client.listFiles(1, collectionKey);
expect(files.map((f) => f.id)).toEqual([7]);
// The wrapper starts a full enumeration from zero.
expect(sinceTimes).toEqual(["0"]);
});
});
test/client/session.test.ts
+215
View File
@@ -0,0 +1,215 @@
/**
* Tests for the client session lifecycle: `toJSON`, `fromJSON`, `logout`, and
* the CLI's `loadSession`, which reads the saved session file back into a
* client.
*/
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import sodium from "libsodium-wrappers-sumo";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { init, toBase64 } from "../../src/crypto/index.js";
import { Client, type ClientSnapshot } from "../../src/client.js";
import { loadSession } from "../../src/cli-session.js";
import {
DEFAULT_RETRY_OPTIONS,
UNATTENDED_RETRY_OPTIONS,
} from "../../src/retry.js";
const validSnapshot = (): ClientSnapshot => {
const kp = sodium.crypto_box_keypair();
return {
email: "user@example.com",
userID: 42,
token: "test-token",
masterKey: toBase64(sodium.crypto_secretbox_keygen()),
secretKey: toBase64(kp.privateKey),
publicKey: toBase64(kp.publicKey),
};
};
// The client's key buffers are private; the tests read them to prove that
// logout wipes them.
const keyBuffers = (client: Client): Uint8Array[] => [
client["masterKey"],
client["secretKey"],
client["publicKey"],
];
beforeAll(async () => {
await init();
});
describe("Client.toJSON", () => {
it("round-trips through fromJSON unchanged", () => {
const snapshot = validSnapshot();
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
});
it("throws instead of emitting a snapshot without a token", () => {
const client = Client.fromJSON(validSnapshot());
client.getApiClient().clearAuthToken();
expect(() => client.toJSON()).toThrow(/no auth token/);
});
});
describe("Client.fromJSON", () => {
const shortKey = toBase64(new Uint8Array(16));
it.each([
["email", undefined],
["email", 7],
["email", ""],
["token", undefined],
["token", null],
["token", ""],
["userID", undefined],
["userID", "42"],
["userID", 4.2],
["masterKey", undefined],
["masterKey", 7],
["masterKey", "not base64!"],
["masterKey", shortKey],
["secretKey", undefined],
["secretKey", "not base64!"],
["secretKey", shortKey],
["publicKey", undefined],
["publicKey", "not base64!"],
["publicKey", shortKey],
])("rejects %s = %j, naming the field", (field, value) => {
const snapshot: Record<string, unknown> = { ...validSnapshot() };
snapshot[field] = value;
expect(() => Client.fromJSON(snapshot)).toThrow(
new RegExp(`^Invalid session data: ${field} `),
);
});
it.each([null, "a string", 42])("rejects a non-object %j", (value) => {
expect(() => Client.fromJSON(value)).toThrow(
/^Invalid session data: not a JSON object/,
);
});
});
describe("Client.logout", () => {
it("zeroes the key buffers and clears the token", () => {
const client = Client.fromJSON(validSnapshot());
const api = client.getApiClient();
const keys = keyBuffers(client);
client.logout();
for (const key of keys) {
expect(key.length).toBe(32);
expect(key.every((b) => b === 0)).toBe(true);
}
expect(api.getAuthToken()).toBeUndefined();
});
it("makes every later operation throw", async () => {
const client = Client.fromJSON(validSnapshot());
client.logout();
expect(() => client.whoami()).toThrow(/logged out/);
expect(() => client.toJSON()).toThrow(/logged out/);
expect(() => client.getApiClient()).toThrow(/logged out/);
expect(() => client.contentSource()).toThrow(/logged out/);
await expect(client.listCollections()).rejects.toThrow(/logged out/);
await expect(client.collectionsSince({ sinceTime: 0 })).rejects.toThrow(
/logged out/,
);
await expect(
client.filesSince({
collectionID: 1,
collectionKey: new Uint8Array(32),
sinceTime: 0,
}),
).rejects.toThrow(/logged out/);
await expect(
client.fetchMLData({ fileIDs: [1], fileKeys: new Map() }),
).rejects.toThrow(/logged out/);
});
it("stops a listing in flight from decrypting with the zeroed keys", async () => {
// The server answers only after the client has logged out. If the
// listing went on to decrypt this row with all-zero keys it would fail
// with a decryption error, not the logged-out one.
const row = {
id: 1,
owner: { id: 42 },
encryptedKey: toBase64(new Uint8Array(48)),
keyDecryptionNonce: toBase64(new Uint8Array(24)),
updationTime: 1,
};
const client: Client = Client.fromJSON(validSnapshot(), {
fetch: async () => {
client.logout();
return new Response(JSON.stringify({ collections: [row] }), {
status: 200,
headers: { "content-type": "application/json" },
});
},
});
await expect(client.listCollections()).rejects.toThrow(/logged out/);
});
});
describe("loadSession", () => {
let dir: string;
beforeAll(() => {
dir = mkdtempSync(join(tmpdir(), "quak-session-test-"));
});
afterAll(() => {
rmSync(dir, { recursive: true, force: true });
});
it("returns null when there is no session file", () => {
expect(loadSession(join(dir, "missing.json"))).toBeNull();
});
it("restores a client from a valid session file", () => {
const path = join(dir, "valid.json");
writeFileSync(path, JSON.stringify(validSnapshot()));
expect(loadSession(path)!.whoami()).toEqual({
email: "user@example.com",
userID: 42,
});
});
it("gives the restored client the retry options it is passed", () => {
const path = join(dir, "retry.json");
writeFileSync(path, JSON.stringify(validSnapshot()));
const retryOptions = (client: Client | null) =>
client!.getApiClient().getRetryOptions();
expect(
retryOptions(
loadSession(path, { retry: UNATTENDED_RETRY_OPTIONS }),
),
).toEqual(UNATTENDED_RETRY_OPTIONS);
expect(retryOptions(loadSession(path))).toEqual(DEFAULT_RETRY_OPTIONS);
});
it("says the file is corrupt when it is not JSON", () => {
const path = join(dir, "truncated.json");
writeFileSync(path, '{"email": "user@exa');
expect(() => loadSession(path)).toThrow(
`Session file ${path} is corrupt`,
);
});
it("says the file is corrupt and names the bad field", () => {
const path = join(dir, "bad-key.json");
writeFileSync(
path,
JSON.stringify({ ...validSnapshot(), secretKey: "AAAA" }),
);
expect(() => loadSession(path)).toThrow(
new RegExp(`^Session file ${path} is corrupt: .*secretKey`),
);
});
});
test/clip-keywords-sunset.json
+109
View File
@@ -0,0 +1,109 @@
{
"photo": "https://upload.wikimedia.org/wikipedia/commons/d/db/Sunset_off_the_Massachusetts_coast_-_NOAA.jpg",
"sha256": "be62f3289c9bf7b75af68cd6978021d54e8a531d5d4cc493f6e9fbf5e1ab74d3",
"embedding": [
-0.0021123015, -0.04437615, -0.062354308, 0.006249992, 0.022141185,
0.011159019, 0.01227392, -0.00030532264, 0.03186437, 0.09196922,
0.039111234, -0.037084445, -0.032962628, -0.019820206, -0.013141825,
0.0047183544, 0.017137334, 0.046581306, 0.0047622235, 0.04982521,
0.023128154, 0.032080866, 0.037709303, 0.019241406, 0.0032100384,
-0.014275843, 0.049337693, 0.025953151, 0.014263995, -0.0062328726,
-0.040504098, -0.015083303, -0.008123875, -0.006233639, -0.05352683,
-0.0048083956, 0.0763486, -0.013155067, 0.025114346, 0.022016674,
0.0001588023, -0.010341572, -0.0037518279, -0.019181008, 0.04549337,
-0.0010725886, 0.004636866, 0.017177915, -0.0117317885, 0.0026097358,
0.030738998, 0.013158572, -0.08856403, -0.0121391015, 0.011477229,
-0.0033104012, 0.035524756, -0.018607879, -0.0071739843, -0.031906743,
0.0056713386, -0.0044195866, 0.00292471, 0.00061686034, -0.043075223,
0.09040427, 0.001522537, -0.011445652, 0.009609886, -0.0053201183,
-0.0057452377, 0.00972853, -0.0024397348, -0.021255083, 0.015893811,
-0.006020548, -0.09428023, 0.014277583, -0.017975299, -0.011878315,
0.011825521, 0.00014203975, -0.0020437292, -0.05305343, -0.024840152,
-0.011336872, -0.002796541, -0.00036171873, 0.004530499, 0.044486605,
0.021602718, -0.01292857, 0.024530916, 0.019559948, -0.039928272,
-0.0681174, 0.049101, 0.02461532, 0.0365531, -0.04070358, -0.011011939,
-0.0027188526, -0.020370876, 0.012904856, 0.043157287, -0.051658224,
-0.3726012, 0.019549074, -0.0499252, -0.04800559, -0.011493859,
0.02893256, 0.060341954, 0.0013929688, -0.051283866, -0.024286803,
0.044124287, -0.005995806, -0.062817656, 0.007369614, -0.021470634,
-0.028147545, 0.00411659, 0.04234773, 0.03787647, 0.008579682,
-0.038568925, -0.027498025, -0.028416378, 0.0022012168, 0.010906191,
-0.013745787, 0.018267838, 0.013531097, -0.061559845, 0.004682517,
-0.005993303, 0.0028113858, -0.0028286583, 0.0026029332, -0.06693403,
-0.0072381413, -0.02419584, 0.0037009937, -0.0011296495, -0.030310456,
0.0005999609, 0.009945354, 0.024915686, 0.038672045, 0.023009432,
-0.05093051, -0.09101409, -0.06686167, -0.06211481, -0.022600474,
0.0024902367, -0.0067171026, -0.016769778, -0.0032803628, 0.008623083,
-0.0016110836, 0.014925073, 0.033811048, 0.012829449, -0.052575592,
0.03728281, -0.0051201633, 0.021238064, -0.031086605, -0.0043402985,
0.25054494, -0.028396713, -0.010725523, -0.023445267, 0.021751683,
0.003130971, 0.012270548, -0.014836475, 0.014466885, 0.0021478906,
-0.088153616, 0.015356315, 0.039639276, -0.040283255, 0.014923924,
-0.029303031, -0.004475201, 0.047299225, -0.06726505, -0.012491676,
-0.002080582, 0.03506341, 0.04146718, 0.0074114986, -0.009582633,
0.006623749, -0.05454898, -0.0034387247, -0.014780821, 0.03698328,
-0.034441486, 0.22032379, -0.021792699, 0.008001907, 0.0341019,
0.0025784625, -0.0071689896, -0.014151225, -0.050547987, -0.030372301,
-0.03288388, -0.042987447, -0.0033656165, 0.021502906, -0.0030374376,
-0.024413, -0.03506603, 0.030891743, 0.0069994032, 0.004887366,
0.030707076, 0.0065348223, 0.0008240555, 0.016809713, -0.024151647,
-0.0040458064, 0.044567548, 0.03908212, -0.061782435, 0.0026002831,
0.02003242, -0.0115451, -0.045135964, -0.0663542, 0.005240975,
0.0010777109, -0.009276002, -0.011919081, 0.027654119, -0.013634551,
0.0129154315, -0.013195512, 0.040571272, -0.099290505, -0.04656693,
-0.20979854, 0.04181004, -0.06769166, -0.04121946, 0.018260444,
0.005617506, -0.15694222, 0.032472793, 0.014189598, 0.008345103,
-0.041118417, 0.0074527003, -0.011494104, 0.004906474, -0.0027100458,
0.0009994594, -0.041061398, -0.045274835, 0.015187793, 0.034275036,
-0.02813289, 0.02098019, -0.015043198, 0.03716667, 0.013542642,
-0.008104835, 0.014965486, -0.027978558, 0.021163663, -0.006942737,
-0.034969814, -0.038961593, 0.016802894, -0.012318175, -0.047821593,
-0.01933216, 0.0064432374, 0.0322358, -0.020437784, 0.02659447,
-0.03022022, -0.019443465, -0.094178624, -0.017009107, 0.03860671,
-0.00023695534, -0.033089854, -0.014677462, 0.012356399, 0.018040499,
-0.19360113, 0.011710817, -0.019871691, 0.020124774, 0.024948105,
-0.024573864, -0.028349072, -0.041731454, 0.032646254, 0.015098165,
0.0018447012, -0.01129152, -0.02481687, 0.036048, 0.20797811,
0.04317757, -0.017239755, 0.033431605, -0.011384422, 0.0099174455,
-0.19739223, -0.008901062, -0.056426458, 0.014591864, 0.0044474266,
0.008209326, -0.012241819, 0.042553645, -0.021552453, 0.008611734,
0.048944127, -0.013872049, 0.019179989, 0.008183264, -0.053223096,
-0.027327595, -0.0033542586, -0.012353858, -0.010941209, 0.010212546,
0.029714134, -0.011385593, -0.0013529479, 0.014505385, 0.01148663,
0.012613636, -0.0074430504, -0.014008101, -0.028146038, -0.042778917,
0.02143165, 0.016812252, -0.06286215, 0.02085532, -0.035128277,
0.0241282, 0.01397908, -0.034992814, 0.013153766, 0.0068236147,
0.013594184, -0.06859393, 0.051849406, 0.029675119, -0.0037699796,
-0.00013216575, 0.02916647, 0.031859048, -0.015006051, 0.0001439966,
0.0026584554, -0.0046948595, 0.039754897, 0.034873154, 0.0012870439,
-0.011516082, -0.074089095, 0.019241294, 0.00076195505, 0.0027334464,
-0.036206458, -0.0011109262, -0.048838835, -0.00204685, 0.01179991,
0.0068136845, 0.009852495, 0.017308988, 0.03159545, 0.058803353,
0.038116634, 0.008836143, -0.010460637, -0.010994099, -0.0012926942,
-0.048370842, 0.037696566, -0.08093239, -0.0041242186, -0.005866373,
0.02048607, -0.12549639, 0.011705739, -0.0021425558, 0.018078592,
-0.038228087, 0.004245971, -0.11560507, 0.01381203, -0.02734893,
0.029595908, 0.101082794, -0.042842254, -0.08397301, -0.1546237,
-0.037624564, 0.0069606556, -0.008020479, 0.0043268185, 0.016504597,
0.044554908, -0.05240209, 0.020519335, -0.050994243, -0.019861093,
0.022066997, -0.0188911, 0.012919696, -0.022872692, -0.012961374,
-0.020071112, 0.0074553066, 0.06402014, 0.04065211, 0.014139608,
0.06221546, -0.02574805, 0.010114545, 0.020619567, 0.032264773,
-0.009148828, 0.022849955, -0.016593235, -0.032456633, -0.006737303,
-0.0037493357, -0.065912865, -0.0024181998, -0.0037048983, -0.019304488,
0.009085037, 0.057529982, -0.0378338, 0.01792042, 0.01885012,
-0.0043600523, 0.017382368, -0.027792772, 0.006749118, -0.0053297766,
-0.011628587, 0.015736576, 0.024801968, 0.011543913, -0.01159873,
-0.000759976, 0.042183932, 0.05764386, -0.06422416, 0.010577374,
-0.02261249, 0.02282793, -0.05141075, -0.11287078, -0.020670144,
-0.020953743, 0.030885734, -0.012597732, -0.121472664, 0.028447151,
-0.0415664, -0.010061708, 0.06441338, 0.0271207, -0.0016693748,
-0.0045694243, 0.020342957, 0.030528478, 0.007129517, -0.0029600926,
-0.007061397, 0.06036486, -0.004736018, -0.0090786135, -0.07601116,
-0.039556272, -0.051269546, -0.019473685, 0.029858151, 0.028464556,
0.012424476, 0.013742254, 0.008742927, 0.006065751, -0.009892663,
0.07699999, -0.003443197, 0.0027863425, 0.029197995, 0.0211013,
0.01098379, 0.016684106, -0.028917195, 0.008122037, 0.0265241,
-0.04461223
]
}
test/clip-keywords.test.ts
+129
View File
@@ -0,0 +1,129 @@
// `clipKeywords`: the keywords quak writes for a file in place of its CLIP
// embedding.
//
// Ente's ML data for a file holds a CLIP embedding, 512 numbers its image
// model gave the photo. Ente's photos app puts a file in a category of its
// Discover section when the cosine similarity of that embedding and the text
// embedding of the category's query is at least the category's minimum score,
// and keeps videos out of some categories. `clipKeywords` applies that rule
// and returns the titles of the categories a file falls in, highest score
// first; `quak backup` and `quak backup-metadata` write them as `keywords`.
import { readFileSync } from "node:fs";
import { describe, expect, it } from "vitest";
import { DISCOVER_CATEGORIES } from "../src/clip-keyword-vectors.js";
import { clipKeywords } from "../src/clip-keywords.js";
// 512 numbers: the ones given, then zeros.
const vector = (...start: number[]): number[] => [
...start,
...new Array<number>(512 - start.length).fill(0),
];
const withEmbedding = (embedding: unknown) => ({
face: { faces: [] },
clip: { embedding },
});
// Two made-up categories, each with a text embedding along one axis. The
// embedding [3, 4] has a cosine similarity of 3/5 with "Across" and 4/5 with
// "Up". Videos are kept out of "Across".
const ACROSS = {
title: "Across",
query: "across",
minScore: 0.6,
showVideo: false,
textEmbedding: vector(1),
};
const UP = {
title: "Up",
query: "up",
minScore: 0.5,
showVideo: true,
textEmbedding: vector(0, 1),
};
describe("clipKeywords", () => {
it("gives each category whose minimum score the similarity reaches, highest score first", () => {
expect(
clipKeywords(withEmbedding(vector(3, 4)), false, [ACROSS, UP]),
).toEqual(["Up", "Across"]);
});
it("leaves out a category when the similarity is just below its minimum score", () => {
const stricter = { ...ACROSS, minScore: 0.6 + 1e-9 };
expect(
clipKeywords(withEmbedding(vector(3, 4)), false, [stricter, UP]),
).toEqual(["Up"]);
});
it("keeps a video out of a category that leaves out videos", () => {
expect(
clipKeywords(withEmbedding(vector(3, 4)), true, [ACROSS, UP]),
).toEqual(["Up"]);
});
it("gives an empty list for an embedding that falls in no category", () => {
expect(
clipKeywords(withEmbedding(vector(0, 0, 1)), false, [ACROSS, UP]),
).toEqual([]);
});
it.each([
["no CLIP embedding", { face: { faces: [] } }],
["a CLIP part that is not an object", { clip: null }],
["a CLIP part without an embedding", { clip: {} }],
["an embedding of 511 numbers", withEmbedding(vector(3, 4).slice(1))],
["an embedding with NaN in it", withEmbedding(vector(3, NaN))],
["an embedding with Infinity in it", withEmbedding(vector(Infinity))],
["an embedding of strings", withEmbedding(vector(3, 4).map(String))],
["an embedding of zeros", withEmbedding(vector())],
])("gives no keywords for ML data with %s", (_, mlData) => {
expect(clipKeywords(mlData, false, [ACROSS, UP])).toBeUndefined();
});
});
// The CLIP embedding Ente's image model gives a public-domain NOAA photo of a
// sunset, made by bin/clip-keyword-vectors from the photo at `photo`.
const SUNSET: { photo: string; sha256: string; embedding: number[] } =
JSON.parse(
readFileSync(
new URL("clip-keywords-sunset.json", import.meta.url),
"utf-8",
),
);
describe("Ente's Discover categories", () => {
it("are the 16 of Ente's photos app, each with a text embedding of norm 1", () => {
expect(DISCOVER_CATEGORIES.map((c) => c.title)).toEqual([
"Identity",
"Screenshots",
"QR Codes",
"Receipts",
"Notes",
"Memes",
"Visiting Cards",
"Babies",
"Pets",
"Selfies",
"Wallpapers",
"Food",
"Celebrations",
"Sunset",
"Hills",
"Greenery",
]);
for (const category of DISCOVER_CATEGORIES) {
expect(category.textEmbedding).toHaveLength(512);
expect(Math.hypot(...category.textEmbedding)).toBeCloseTo(1, 5);
}
});
it("give a photo of a sunset the keyword Sunset and no other", () => {
expect(
clipKeywords({ clip: { embedding: SUNSET.embedding } }, false),
).toEqual(["Sunset"]);
});
});
test/crypto/hash.test.ts
+33
View File
@@ -0,0 +1,33 @@
import { beforeAll, describe, expect, it } from "vitest";
import {
chunkHashFinal,
chunkHashInit,
chunkHashUpdate,
init,
} from "../../src/crypto/index.js";
beforeAll(async () => {
await init();
});
describe("content hash", () => {
// RFC 7693 Appendix A: BLAKE2b-512 of "abc".
const abc = Buffer.from(
"ba80a53f981c4d0d6a2797b69f12f6e94c212f14685ac4b74b12bb6fdbffa2d1" +
"7d87c5392aab792dc252d5de4533cc9518d38aa8dbf1925ab92386edd4009923",
"hex",
).toString("base64");
it("is unkeyed BLAKE2b-512 in standard base64", () => {
const state = chunkHashInit();
chunkHashUpdate(state, new TextEncoder().encode("abc"));
expect(chunkHashFinal(state)).toBe(abc);
});
it("gives the same hash when the input arrives in chunks", () => {
const state = chunkHashInit();
chunkHashUpdate(state, new TextEncoder().encode("a"));
chunkHashUpdate(state, new TextEncoder().encode("bc"));
expect(chunkHashFinal(state)).toBe(abc);
});
});
test/crypto/kdf.test.ts
+4 -4
View File
@@ -29,10 +29,10 @@ describe("crypto.deriveKEK (Argon2id)", () => {
});
/**
* Cheap parameters used so the test suite stays under the 30-second
* budget. The real production parameters Ente uses are larger
* (memLimit up to 1 GiB, opsLimit 3-16). The algorithm is the same
* regardless of parameters.
* Cheap parameters used so the test suite stays under the 90-second
* `timeout` in the `test` phase of the `Dockerfile`. The real production
* parameters Ente uses are larger (memLimit up to 1 GiB, opsLimit 3-16).
* The algorithm is the same regardless of parameters.
*/
const TEST_OPS = 2;
const TEST_MEM = 64 * 1024 * 1024; // 64 MiB
test/download/download.test.ts
+1042 -19
View File
File diff suppressed because it is too large Load Diff
test/ente-server.ts
+197
View File
@@ -0,0 +1,197 @@
/**
* A fake of the Ente HTTP API, for tests that run quak's real `Client` against
* a server.
*
* `buildMetaFetch` is a `fetch` that answers the requests quak makes, from a
* `MetaMockState` the test builds: the SRP login, `/collections/v2`, each
* collection's `/collections/v2/diff`, the ML data of `/files/data/fetch`, and
* each file's content from the files CDN, a 404 for a file it holds none for.
* The test encrypts everything in that state with libsodium, as Ente's clients
* do, so quak decrypts it with its own code. Nothing hits the network.
*
* `buildAccount` builds what the server holds to log an account in, and the
* keys a test encrypts that account's collections with. Call `init()` from
* `src/crypto` and await `sodium.ready` before using either.
*/
import sodium from "libsodium-wrappers-sumo";
import { SRP, SrpServer } from "fast-srp-hap";
import { toBase64, deriveKEK, deriveLoginSubkey } from "../src/crypto/index.js";
import type { KeyAttributes } from "../src/auth/types.js";
const TEST_OPS = 2;
const TEST_MEM = 64 * 1024 * 1024;
export interface MetaMockState {
verifier: Buffer;
srpAttributes: Record<string, unknown>;
keyAttributes: KeyAttributes;
encryptedToken: string;
collections: Record<string, unknown>[];
filesByCollection: Record<number, Record<string, unknown>[]>;
// For ML data and EXIF tests
encryptedMLData: Record<
number,
{ encryptedData: string; decryptionHeader: string }
>;
fileCiphertexts: Record<number, Uint8Array>;
}
export const encryptSecretbox = (plaintext: Uint8Array, key: Uint8Array) => {
const nonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
const ciphertext = sodium.crypto_secretbox_easy(plaintext, nonce, key);
return { ciphertext, nonce };
};
export const encryptStreamBlob = (plaintext: Uint8Array, key: Uint8Array) => {
const push = sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const ciphertext = sodium.crypto_secretstream_xchacha20poly1305_push(
push.state,
plaintext,
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
);
return { ciphertext, header: push.header };
};
// An account whose password is `password`. The first four fields are those of
// `MetaMockState`; with the master key the test encrypts the keys of the
// account's own collections, and to the public key it seals the keys of the
// collections other accounts share with it.
export const buildAccount = async (password: string) => {
const kekSalt = sodium.randombytes_buf(sodium.crypto_pwhash_SALTBYTES);
const kek = await deriveKEK(password, kekSalt, TEST_OPS, TEST_MEM);
const loginSubKeyBytes = deriveLoginSubkey(kek);
const srpUserID = "meta-srp";
const srpSalt = sodium.randombytes_buf(16);
const verifier = SRP.computeVerifier(
SRP.params["4096"],
Buffer.from(srpSalt),
Buffer.from(srpUserID),
Buffer.from(loginSubKeyBytes),
);
const masterKey = sodium.randombytes_buf(32);
const { ciphertext: encMK, nonce: mkNonce } = encryptSecretbox(
masterKey,
kek,
);
const kp = sodium.crypto_box_keypair();
const { ciphertext: encSK, nonce: skNonce } = encryptSecretbox(
kp.privateKey,
masterKey,
);
const tokenBytes = sodium.randombytes_buf(32);
const encToken = sodium.crypto_box_seal(tokenBytes, kp.publicKey);
const keyAttributes: KeyAttributes = {
kekSalt: toBase64(kekSalt),
encryptedKey: toBase64(encMK),
keyDecryptionNonce: toBase64(mkNonce),
publicKey: toBase64(kp.publicKey),
encryptedSecretKey: toBase64(encSK),
secretKeyDecryptionNonce: toBase64(skNonce),
memLimit: TEST_MEM,
opsLimit: TEST_OPS,
};
return {
verifier,
srpAttributes: {
srpUserID,
srpSalt: toBase64(srpSalt),
memLimit: TEST_MEM,
opsLimit: TEST_OPS,
kekSalt: toBase64(kekSalt),
isEmailMFAEnabled: false,
},
keyAttributes,
encryptedToken: toBase64(encToken),
masterKey,
publicKey: kp.publicKey,
};
};
// `failMLDataFor`: answer 500 to every ML data request that asks for this file.
export const buildMetaFetch = (m: MetaMockState, failMLDataFor?: number) => {
let srpServer: SrpServer;
return (async (
input: RequestInfo | URL,
init?: RequestInit,
): Promise<Response> => {
const url =
typeof input === "string"
? input
: input instanceof URL
? input.href
: input.url;
const path = new URL(url).pathname;
const json = (body: unknown) =>
new Response(JSON.stringify(body), {
status: 200,
headers: { "content-type": "application/json" },
});
if (path === "/users/srp/attributes")
return json({ attributes: m.srpAttributes });
if (path === "/users/srp/create-session") {
const body = JSON.parse(init?.body as string);
const serverKey = await SRP.genKey();
srpServer = new SrpServer(
SRP.params["4096"],
m.verifier,
serverKey,
);
const B = srpServer.computeB();
srpServer.setA(Buffer.from(body.srpA, "base64"));
return json({ sessionID: "s1", srpB: B.toString("base64") });
}
if (path === "/users/srp/verify-session") {
const body = JSON.parse(init?.body as string);
srpServer.checkM1(Buffer.from(body.srpM1, "base64"));
return json({
srpM2: srpServer.computeM2().toString("base64"),
id: 42,
keyAttributes: m.keyAttributes,
encryptedToken: m.encryptedToken,
});
}
if (path === "/collections/v2")
return json({ collections: m.collections });
if (path === "/collections/v2/diff") {
const collID = Number(
new URL(url).searchParams.get("collectionID"),
);
return json({
diff: m.filesByCollection[collID] ?? [],
hasMore: false,
});
}
if (path === "/files/data/fetch") {
const body = JSON.parse(init?.body as string);
if ((body.fileIDs as number[]).includes(failMLDataFor!))
return new Response("server error", { status: 500 });
const data = (body.fileIDs as number[])
.filter((id: number) => m.encryptedMLData[id])
.map((id: number) => ({
fileID: id,
...m.encryptedMLData[id],
updatedAt: 1700000000000000,
}));
return json({ data });
}
if (
url.includes("files.ente.io") ||
path.startsWith("/files/download/")
) {
const parsed = new URL(url);
const fileID = Number(
parsed.searchParams.get("fileID") ?? path.split("/").pop(),
);
const ct = m.fileCiphertexts[fileID];
if (ct) return new Response(ct, { status: 200 });
return new Response("not found", { status: 404 });
}
return new Response("not found", { status: 404 });
}) as typeof globalThis.fetch;
};
test/examples/download-albums.test.ts
+306
View File
@@ -0,0 +1,306 @@
/**
* The example script `examples/download-albums.ts` (issue #144), run twice
* against a stand-in account, as a user would run it twice.
*
* The account has two albums sharing one photo, and one of its photos is a
* live photo whose image is a HEIC with EXIF. The first run puts every
* original at its save path, writes each photo's metadata beside it and each
* album's photos under `albums/`. Before the second run the account gains an
* album holding a new photo. The second run downloads that photo and writes
* its files and the new album's, and fetches nothing else and changes no
* other file.
*/
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import {
existsSync,
mkdtempSync,
readdirSync,
readFileSync,
rmSync,
statSync,
utimesSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { downloadAlbums } from "../../examples/download-albums.js";
import { Library, type ContentSource } from "../../src/index.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
import { readAllExifTags } from "../../src/exif.js";
import { HEIC_WITH_EXIF } from "../exif-heic.js";
import {
asLivePhoto,
cdnSource,
livePhotoHash,
livePhotoZip,
VIDEO,
} from "../live-photo.js";
const USER_ID = 7;
// Every photo is taken at noon local time on 2026-03-01, so the machine's time
// zone cannot move it to another day; it is saved in the folder `DAY`. Ente
// stores times in microseconds.
const TAKEN_MS = new Date(2026, 2, 1, 12).getTime();
const DAY = join("2026", "2026-03", "2026-03-01");
const collection = (id: number, name: string): Collection => ({
id,
ownerID: USER_ID,
key: new Uint8Array([id]),
name,
type: "album",
updationTime: 1,
isShared: false,
});
const file = (id: number, collectionID: number): EnteFile => ({
id,
collectionID,
ownerID: USER_ID,
key: new Uint8Array([id]),
metadata: {
title: `file-${id}.jpg`,
fileType: "image",
creationTime: TAKEN_MS * 1000,
modificationTime: TAKEN_MS * 1000,
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime: 1,
});
let root: string;
beforeEach(() => {
root = mkdtempSync(join(tmpdir(), "quak-download-albums-"));
});
afterEach(() => {
if (root && existsSync(root))
rmSync(root, { recursive: true, force: true });
});
// Every file and directory under `dir`, by path.
const entries = (dir: string): string[] =>
readdirSync(dir, { recursive: true, encoding: "utf-8" });
// Set the modification time of everything under `dir` to the epoch, so that
// anything written there afterwards has a later one, however soon it comes.
const backdate = (dir: string): void => {
for (const name of entries(dir)) utimesSync(join(dir, name), 0, 0);
};
// The modification time of everything under `dir`, by path.
const mtimes = (dir: string): Map<string, number> =>
new Map(
entries(dir).map((name) => [name, statSync(join(dir, name)).mtimeMs]),
);
const readJSON = (path: string): unknown =>
JSON.parse(readFileSync(path, "utf-8"));
describe("examples/download-albums.ts", () => {
it("downloads every album's photos with their metadata, and on a second run only what the account gained", async () => {
// Album 1, "Trip", holds photos 1 and 2. Album 2, "Family", holds
// photo 2 and photo 3, a live photo.
const live = await asLivePhoto(
file(3, 2),
livePhotoZip({ "image.heic": HEIC_WITH_EXIF, "video.mov": VIDEO }),
livePhotoHash(HEIC_WITH_EXIF, VIDEO),
);
const collections = [collection(1, "Trip"), collection(2, "Family")];
const filesByAlbum = new Map<number, EnteFile[]>([
[1, [file(1, 1), file(2, 1)]],
[2, [file(2, 2), live.file]],
]);
const client = {
whoami: () => ({ email: "u@example.com", userID: USER_ID }),
collectionsSince: async (): Promise<CollectionsPage> => ({
collections: [...collections],
deleted: [],
cursor: 1,
}),
filesSince: async (args: {
collectionID: number;
}): Promise<FilesPage> => ({
files: filesByAlbum.get(args.collectionID) ?? [],
deleted: [],
cursor: 1,
}),
};
// Photo 3 comes from a stand-in server, encrypted as Ente serves a
// live photo. Any other original is a few bytes naming its photo.
// `calls` counts every fetch, thumbnails included.
const server = cdnSource(new Map([[3, live.body]]));
let calls = 0;
const source: ContentSource = {
original: async (args) => {
calls++;
if (args.file.id === 3) return server.original(args);
const bytes = `original-${args.file.id}`;
writeFileSync(args.destination, bytes);
return { bytesWritten: bytes.length };
},
thumbnail: async (args) => {
calls++;
return server.thumbnail(args);
},
};
const dir = join(root, "photos");
const open = (): Promise<Library> =>
Library.open({
client,
cacheDirectory: join(root, "cache"),
downloadDirectory: dir,
contentSource: source,
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
// The test counts fetches, so the free space of the disk it
// runs on must not shrink the cache and evict photo 1.
freeBelowBytes: 0,
});
const first = await open();
// The cache already holds photo 1's original, so its record names a
// cache path, which the metadata leaves out. download() copies it
// from the cache rather than fetching it again.
await first.photos.byID({ fileID: 1 })!.original();
expect(await downloadAlbums(first, dir)).toEqual({
downloaded: 3,
alreadyLocal: 0,
});
await first.close();
expect(calls).toBe(3);
// Each original at its save path, a live photo as its image, its video
// and the file naming them, and each photo's metadata beside it.
const day = join(dir, DAY);
expect(readdirSync(day).sort()).toEqual([
"2026-03-01.1.jpg",
"2026-03-01.1.jpg.json",
"2026-03-01.2.jpg",
"2026-03-01.2.jpg.json",
"2026-03-01.3.heic",
"2026-03-01.3.heic.json",
"2026-03-01.3.livephoto.json",
"2026-03-01.3.mov",
]);
expect(readFileSync(join(day, "2026-03-01.1.jpg"), "utf-8")).toBe(
"original-1",
);
expect(readFileSync(join(day, "2026-03-01.2.jpg"), "utf-8")).toBe(
"original-2",
);
expect(readFileSync(join(day, "2026-03-01.3.heic"))).toEqual(
Buffer.from(HEIC_WITH_EXIF),
);
expect(readFileSync(join(day, "2026-03-01.3.mov"))).toEqual(
Buffer.from(VIDEO),
);
// The metadata is the photo's record and its EXIF tags. The originals
// of photos 1 and 2 are not image data, so they have no EXIF tags.
// Photo 3's are every tag of its image, as `photo.exif()` returns
// them. `record` holds the fields the three records share.
const record = {
takenAt: TAKEN_MS,
modifiedAt: TAKEN_MS,
fileType: "image",
isArchived: false,
isHidden: false,
};
expect(readJSON(join(day, "2026-03-01.1.jpg.json"))).toEqual({
...record,
fileID: 1,
albumIDs: [1],
title: "file-1.jpg",
exif: {},
});
expect(readJSON(join(day, "2026-03-01.2.jpg.json"))).toEqual({
...record,
fileID: 2,
albumIDs: [1, 2],
title: "file-2.jpg",
exif: {},
});
expect(readJSON(join(day, "2026-03-01.3.heic.json"))).toEqual({
...record,
fileID: 3,
albumIDs: [2],
title: "file-3.jpg",
fileType: "livePhoto",
hash: livePhotoHash(HEIC_WITH_EXIF, VIDEO),
exif: readAllExifTags(HEIC_WITH_EXIF),
});
// Each album's photos, newest first, by save path relative to `dir`.
// Photo 2 is in both.
expect(readdirSync(join(dir, "albums")).sort()).toEqual([
"1.json",
"2.json",
]);
expect(readJSON(join(dir, "albums", "1.json"))).toEqual({
collectionID: 1,
name: "Trip",
savePaths: [
join(DAY, "2026-03-01.2.jpg"),
join(DAY, "2026-03-01.1.jpg"),
],
});
expect(readJSON(join(dir, "albums", "2.json"))).toEqual({
collectionID: 2,
name: "Family",
savePaths: [
join(DAY, "2026-03-01.3.heic"),
join(DAY, "2026-03-01.2.jpg"),
],
});
// Before the second run the account gains album 3, "Garden", holding
// a new photo 4. The second run, with a newly opened library, opens
// the cache written by the first and still downloads photo 4. It finds
// the other photos already local and fetches nothing else.
collections.push(collection(3, "Garden"));
filesByAlbum.set(3, [file(4, 3)]);
backdate(dir);
const before = mtimes(dir);
const second = await open();
expect(await downloadAlbums(second, dir)).toEqual({
downloaded: 1,
alreadyLocal: 3,
});
await second.close();
expect(calls).toBe(4);
expect(readFileSync(join(day, "2026-03-01.4.jpg"), "utf-8")).toBe(
"original-4",
);
expect(readJSON(join(dir, "albums", "3.json"))).toEqual({
collectionID: 3,
name: "Garden",
savePaths: [join(DAY, "2026-03-01.4.jpg")],
});
// The second run adds only photo 4's files and album 3's, and
// rewrites, renames or removes no file from the first run. The two
// directories that gain a file are the only other changes.
const after = mtimes(dir);
expect(
[...after.keys()].filter((name) => !before.has(name)).sort(),
).toEqual([
join(DAY, "2026-03-01.4.jpg"),
join(DAY, "2026-03-01.4.jpg.json"),
join("albums", "3.json"),
]);
for (const [name, mtime] of before) {
if (name === DAY || name === "albums") continue;
expect(after.get(name), name).toBe(mtime);
}
});
});
test/exif-heic.ts
+27
View File
@@ -0,0 +1,27 @@
/**
* `exif.heic`, beside this file: a real 64x64 HEIC whose EXIF holds the same
* values as the hand-built JPEG in `exif-jpeg.ts`, for the tests of `exif()`,
* `backup-metadata --exif` and the image metadata `quak backup` records.
*
* It was made once, in a throwaway node:22-alpine container (Alpine 3.23.3),
* with libheif 1.23.0 and exiftool 13.55:
*
* apk add libheif-tools exiftool imagemagick
* magick -size 64x64 gradient:red-blue -depth 8 in.png
* heif-enc -q 30 -o exif.heic in.png
* exiftool -overwrite_original \
* -Make=Canon -Model="EOS R5" -LensModel="RF50mm F1.8 STM" \
* -DateTimeOriginal="2021:07:15 14:30:00" -OffsetTimeOriginal="+02:00" \
* -ExposureTime=1/250 -FNumber=2.8 -ISO=400 -FocalLength=50 \
* -Orientation#=6 \
* -GPSLatitude="40 26 46" -GPSLatitudeRef=N \
* -GPSLongitude="79 58 56" -GPSLongitudeRef=W \
* -GPSAltitude=12.5 -GPSAltitudeRef#=1 \
* exif.heic
*/
import { readFileSync } from "node:fs";
export const HEIC_WITH_EXIF = new Uint8Array(
readFileSync(new URL("exif.heic", import.meta.url)),
);
test/exif-jpeg.ts
+89
View File
@@ -0,0 +1,89 @@
/**
* Hand-built JPEGs for the tests of `exif()` and of the image metadata
* `quak backup` records: one whose EXIF holds the same values as `exif.heic`
* (see `exif-heic.ts`), and one whose EXIF cannot be parsed.
*/
// Big-endian bytes for the hand-built JPEG below.
const u16 = (n: number): number[] => [n >> 8, n & 0xff];
const u32 = (n: number): number[] => [...u16(n >>> 16), ...u16(n & 0xffff)];
const ascii = (s: string): number[] => [...new TextEncoder().encode(s), 0];
const rational = (num: number, den: number): number[] => [
...u32(num),
...u32(den),
];
// One IFD entry: tag, type (1 BYTE, 2 ASCII, 3 SHORT, 4 LONG, 5 RATIONAL),
// count, then the value when it fits in 4 bytes, else its offset.
const entry = (
tag: number,
type: number,
count: number,
value: number[],
): number[] => [...u16(tag), ...u16(type), ...u32(count), ...value];
// The TIFF block of a JPEG's EXIF segment, holding every field `Photo`'s typed
// methods return: the camera in the first IFD, the exposure in the Exif IFD,
// and a GPS position of 40°26'46" N, 79°58'56" W, 12.5 m below sea level.
// Offsets count from the start of this block.
const TIFF = [
...[0x4d, 0x4d, 0x00, 0x2a], // big-endian TIFF
...u32(8), // the first IFD's offset
// The first IFD, at 8: five entries, then no next IFD.
...u16(5),
...entry(0x010f, 2, 6, u32(74)), // Make
...entry(0x0110, 2, 7, u32(80)), // Model
...entry(0x0112, 3, 1, [...u16(6), 0, 0]), // Orientation
...entry(0x8769, 4, 1, u32(88)), // the Exif IFD's offset
...entry(0x8825, 4, 1, u32(246)), // the GPS IFD's offset
...u32(0),
...ascii("Canon"), // at 74
...ascii("EOS R5"), // at 80
0, // a pad byte
// The Exif IFD, at 88: seven entries, then no next IFD.
...u16(7),
...entry(0x829a, 5, 1, u32(178)), // ExposureTime
...entry(0x829d, 5, 1, u32(186)), // FNumber
...entry(0x8827, 3, 1, [...u16(400), 0, 0]), // ISOSpeedRatings
...entry(0x9003, 2, 20, u32(194)), // DateTimeOriginal
...entry(0x9011, 2, 7, u32(214)), // OffsetTimeOriginal
...entry(0x920a, 5, 1, u32(222)), // FocalLength
...entry(0xa434, 2, 16, u32(230)), // LensModel
...u32(0),
...rational(1, 250), // at 178
...rational(28, 10), // at 186
...ascii("2021:07:15 14:30:00"), // at 194
...ascii("+02:00"), // at 214
0, // a pad byte
...rational(50, 1), // at 222
...ascii("RF50mm F1.8 STM"), // at 230
// The GPS IFD, at 246: six entries, then no next IFD.
...u16(6),
...entry(0x0001, 2, 2, [...ascii("N"), 0, 0]), // GPSLatitudeRef
...entry(0x0002, 5, 3, u32(324)), // GPSLatitude
...entry(0x0003, 2, 2, [...ascii("W"), 0, 0]), // GPSLongitudeRef
...entry(0x0004, 5, 3, u32(348)), // GPSLongitude
...entry(0x0005, 1, 1, [1, 0, 0, 0]), // GPSAltitudeRef: below sea level
...entry(0x0006, 5, 1, u32(372)), // GPSAltitude
...u32(0),
...[...rational(40, 1), ...rational(26, 1), ...rational(46, 1)], // at 324
...[...rational(79, 1), ...rational(58, 1), ...rational(56, 1)], // at 348
...rational(25, 2), // at 372
];
export const JPEG_WITH_EXIF = new Uint8Array([
...[0xff, 0xd8], // start of image
...[0xff, 0xe1, ...u16(2 + 6 + TIFF.length)], // APP1 and its length
...[...ascii("Exif"), 0], // "Exif\0\0"
...TIFF,
...[0xff, 0xda, 0x00, 0x02], // start of scan
]);
// A JPEG whose EXIF segment is laid out correctly but holds "XX" where the TIFF
// byte order belongs, so exifreader cannot parse it.
export const JPEG_WITH_BAD_EXIF = new Uint8Array([
...[0xff, 0xd8], // start of image
...[0xff, 0xe1, ...u16(2 + 6 + 2)], // APP1 and its length
...[...ascii("Exif"), 0], // "Exif\0\0"
...[0x58, 0x58], // "XX"
...[0xff, 0xda, 0x00, 0x02], // start of scan
]);
test/exif.heic
BIN
View File
Binary file not shown.
test/filename/filename.test.ts
+228
View File
@@ -0,0 +1,228 @@
// File names built from server-supplied metadata.
//
// quak does not trust the server. A file's title and a collection's name are
// decrypted from data the server hands us, and a hostile server (or a
// compromised account) can set them to anything. quak uses them to name files
// on disk: `quak get` without `--out`, `downloadFile` without `outPath`, the
// backup's symlink and collection directories, and the extension of every file
// in the originals cache. Each of those goes through `sanitizeFileName` or
// `safeExtension`, so a title can only ever name one file inside the directory
// the caller chose.
//
// A path the user supplies (`--out`, `outPath`) is never sanitized: the caller
// is trusted, the server is not.
import { describe, expect, it } from "vitest";
import {
defaultFileName,
safeExtension,
sanitizeFileName,
shortenKeepingExtension,
shortenToBytes,
} from "../../src/filename.js";
const FALLBACK = "file-42";
describe("sanitizeFileName", () => {
it("passes a normal title through unchanged", () => {
expect(sanitizeFileName("IMG_0001.HEIC", FALLBACK)).toBe(
"IMG_0001.HEIC",
);
expect(sanitizeFileName("Holiday 2024 (1).jpg", FALLBACK)).toBe(
"Holiday 2024 (1).jpg",
);
expect(sanitizeFileName("café.jpg", FALLBACK)).toBe("café.jpg");
});
it("cannot climb out of the directory with ../", () => {
// Without sanitizing, this would overwrite the user's SSH keys.
expect(sanitizeFileName("../../.ssh/authorized_keys", FALLBACK)).toBe(
"__.._.ssh_authorized_keys",
);
expect(sanitizeFileName("..", FALLBACK)).toBe("_");
expect(sanitizeFileName("..\\..\\x", FALLBACK)).toBe("__.._x");
});
it("cannot name an absolute path", () => {
expect(sanitizeFileName("/etc/passwd", FALLBACK)).toBe("_etc_passwd");
expect(sanitizeFileName("C:\\Windows\\x.dll", FALLBACK)).toBe(
"C__Windows_x.dll",
);
});
it("replaces embedded separators, so the name stays one file", () => {
expect(sanitizeFileName("a/b\\c.jpg", FALLBACK)).toBe("a_b_c.jpg");
});
it("replaces NUL and other control characters", () => {
// A NUL truncates the path in C code and makes Node's fs throw.
expect(sanitizeFileName("evil\0.jpg", FALLBACK)).toBe("evil_.jpg");
expect(sanitizeFileName("line\nbreak\x7f.jpg", FALLBACK)).toBe(
"line_break_.jpg",
);
});
it("does not produce a hidden file", () => {
expect(sanitizeFileName(".bashrc", FALLBACK)).toBe("_bashrc");
});
it("does not produce a Windows device name", () => {
expect(sanitizeFileName("CON", FALLBACK)).toBe("_CON");
expect(sanitizeFileName("nul.txt", FALLBACK)).toBe("_nul.txt");
expect(sanitizeFileName("LPT1", FALLBACK)).toBe("_LPT1");
// Only the exact names are reserved.
expect(sanitizeFileName("console.jpg", FALLBACK)).toBe("console.jpg");
});
it("falls back to the given name for an empty title", () => {
expect(sanitizeFileName("", FALLBACK)).toBe(FALLBACK);
});
});
describe("safeExtension", () => {
it("keeps a normal extension", () => {
expect(safeExtension("IMG_0001.HEIC")).toBe(".HEIC");
expect(safeExtension("clip.mp4")).toBe(".mp4");
});
it("uses .bin when there is no extension", () => {
expect(safeExtension("")).toBe(".bin");
expect(safeExtension("README")).toBe(".bin");
});
it("uses .bin when the extension holds anything but letters and digits", () => {
expect(safeExtension("x.j\\..\\pg")).toBe(".bin");
expect(safeExtension("x.jp g")).toBe(".bin");
expect(safeExtension("x.jpg\0")).toBe(".bin");
});
it("keeps an extension of 16 letters and digits, and uses .bin for a longer one", () => {
expect(safeExtension(`x.${"e".repeat(16)}`)).toBe(`.${"e".repeat(16)}`);
expect(safeExtension(`x.${"e".repeat(17)}`)).toBe(".bin");
expect(safeExtension(`b.${"e".repeat(300)}`)).toBe(".bin");
});
});
describe("shortenToBytes", () => {
it("returns a name that fits unchanged", () => {
expect(shortenToBytes("beach", 5)).toBe("beach");
expect(shortenToBytes("café", 5)).toBe("café");
expect(shortenToBytes("", 0)).toBe("");
});
it("cuts a name to the number of bytes given", () => {
expect(shortenToBytes("L".repeat(300), 255)).toBe("L".repeat(255));
});
// In UTF-8 "é" takes two bytes and an emoji four, which a JavaScript
// string holds as two halves.
it("cuts between two characters, never inside one", () => {
expect(shortenToBytes("ééé", 5)).toBe("éé");
expect(shortenToBytes("a\u{1F600}b", 4)).toBe("a");
expect(shortenToBytes("a\u{1F600}b", 5)).toBe("a\u{1F600}");
expect(Buffer.byteLength(shortenToBytes("é".repeat(200), 255))).toBe(
254,
);
});
});
describe("shortenKeepingExtension", () => {
it("returns a name that fits unchanged", () => {
expect(shortenKeepingExtension("beach.jpg", 9)).toBe("beach.jpg");
expect(shortenKeepingExtension("beach", 5)).toBe("beach");
});
it("cuts the part before the extension short and keeps the extension", () => {
expect(shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255)).toBe(
`${"t".repeat(251)}.jpg`,
);
expect(shortenKeepingExtension("é".repeat(150), 255)).toBe(
"é".repeat(127),
);
});
// A name cut down to its extension would start with a dot, and so be
// hidden.
it("keeps the first character before the extension, cutting the extension short to leave room for it", () => {
expect(shortenKeepingExtension(`b.${"e".repeat(300)}`, 255)).toBe(
`b.${"e".repeat(253)}`,
);
expect(shortenKeepingExtension(`ab.${"e".repeat(300)}`, 255)).toBe(
`a.${"e".repeat(253)}`,
);
expect(shortenKeepingExtension(`é.${"e".repeat(300)}`, 255)).toBe(
`é.${"e".repeat(252)}`,
);
// "€" takes three bytes, so the extension is cut to 253 and the
// part before it keeps the two bytes left over, for the "a".
expect(shortenKeepingExtension(`a.${"€".repeat(90)}`, 255)).toBe(
`a.${"€".repeat(84)}`,
);
});
it("puts `insert` right before the extension and leaves room for it", () => {
expect(shortenKeepingExtension("beach.jpg", 255, " (7)")).toBe(
"beach (7).jpg",
);
expect(
shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255, " (7)"),
).toBe(`${"t".repeat(247)} (7).jpg`);
expect(
shortenKeepingExtension(`b.${"e".repeat(300)}`, 255, " (7)"),
).toBe(`b (7).${"e".repeat(249)}`);
});
});
describe("defaultFileName", () => {
it("is the title when it is safe and fits", () => {
expect(defaultFileName("IMG_0001.HEIC", 42)).toBe("IMG_0001.HEIC");
});
it("replaces unsafe characters, and is file-<id> for an empty title", () => {
expect(defaultFileName("../x.jpg", 42)).toBe("__x.jpg");
expect(defaultFileName("", 42)).toBe("file-42");
});
it("cuts a title over 255 bytes short, keeping its extension", () => {
expect(defaultFileName(`${"t".repeat(296)}.jpg`, 42)).toBe(
`${"t".repeat(251)}.jpg`,
);
expect(defaultFileName(`b.${"e".repeat(300)}`, 42)).toBe(
`b.${"e".repeat(253)}`,
);
});
it("puts `prefix` before the title and leaves room for it", () => {
expect(defaultFileName("beach.jpg", 42, { prefix: "thumb_" })).toBe(
"thumb_beach.jpg",
);
expect(
defaultFileName(`${"t".repeat(296)}.jpg`, 42, { prefix: "thumb_" }),
).toBe(`thumb_${"t".repeat(245)}.jpg`);
});
// A live photo's image and video are each named with their own
// extension, and the name is cut short once that extension is in it.
it("puts `ext` in place of the title's extension before cutting the name short", () => {
expect(defaultFileName("IMG_0001.HEIC", 42, { ext: ".mov" })).toBe(
"IMG_0001.mov",
);
expect(defaultFileName("", 42, { ext: ".mov" })).toBe("file-42.mov");
expect(
defaultFileName(`${"t".repeat(251)}.jpg`, 42, { ext: ".heic" }),
).toBe(`${"t".repeat(250)}.heic`);
expect(defaultFileName("t".repeat(300), 42, { ext: ".heic" })).toBe(
`${"t".repeat(250)}.heic`,
);
});
// Cutting `CONSOLE` or `NULL` short before a long extension leaves `CON`
// or `NUL`, so the name gets the `_` `sanitizeFileName` gives a device
// name, and the part before the extension gives up a byte for it.
it("never returns a Windows device name, even once cut short", () => {
const ext = `.${"e".repeat(251)}`;
expect(defaultFileName(`CONSOLE${ext}`, 42)).toBe(`_CO${ext}`);
expect(defaultFileName(`NULL${ext}`, 42)).toBe(`_NU${ext}`);
});
});
test/library/content-eviction.test.ts
+359
View File
@@ -0,0 +1,359 @@
/**
* Tests for the originals cache size limit and LRU eviction (issue #47),
* layered on the on-disk content cache (#46).
*
* Only `cacheDirectory/originals` is bounded and evicted. Before each original
* write the effective limit is
* min(cacheOriginalsMaxBytes, bytesUsedByOriginals + bytesFree - freeBelowBytes)
* with `bytesFree` read from `statfs` on the volume holding `cacheDirectory`.
* The limit therefore falls as the disk fills and rises as space returns. When
* a write would cross the limit, least-recently-used originals are removed until
* it fits; pinned files are skipped, and if only pinned files remain the write
* proceeds over-limit. Last-use is the file `mtime`, bumped whenever a read
* returns an original's path, so ordering survives a restart with no ledger.
*
* `statfs` is injected so the adaptive limit is exercised deterministically:
* `bsize` is 1, so `bavail` is the free byte count the formula sees. The source
* writes a controllable number of bytes per file, and tests set each stored
* file's `mtime` explicitly so LRU order does not depend on wall-clock timing.
*/
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { mkdtempSync, rmSync, existsSync, statSync, utimesSync } from "node:fs";
import { writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import {
ContentCache,
type ContentSource,
type StatFsFn,
} from "../../src/library/content.js";
import { RequestPools } from "../../src/library/pools.js";
import type { EnteFile } from "../../src/model/types.js";
const file = (id: number): EnteFile => ({
id,
collectionID: 1,
ownerID: 1,
key: new Uint8Array([id & 0xff]),
metadata: {
title: `file-${id}.jpg`,
fileType: "image",
creationTime: 0,
modificationTime: 0,
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime: 0,
});
// A source that writes a controllable number of bytes per file (default 10).
class SizedSource implements ContentSource {
sizeFor = new Map<number, number>();
async original(args: {
file: EnteFile;
destination: string;
}): Promise<{ bytesWritten: number }> {
const n = this.sizeFor.get(args.file.id) ?? 10;
await writeFile(args.destination, Buffer.alloc(n, 1));
return { bytesWritten: n };
}
async thumbnail(args: {
file: EnteFile;
destination: string;
}): Promise<{ bytesWritten: number }> {
await writeFile(args.destination, Buffer.alloc(10, 1));
return { bytesWritten: 10 };
}
}
// A source whose original() writes 10 bytes but blocks on a gate before
// returning, so two fetches can be held in flight together. `bothStarted`
// resolves once both originals have entered, and `release()` lets them finish.
class GatedSource implements ContentSource {
private openGate!: () => void;
private readonly gate = new Promise<void>((r) => (this.openGate = r));
private inFlight = 0;
private reachedTwo!: () => void;
readonly bothStarted = new Promise<void>((r) => (this.reachedTwo = r));
async original(args: {
file: EnteFile;
destination: string;
}): Promise<{ bytesWritten: number }> {
this.inFlight += 1;
if (this.inFlight === 2) this.reachedTwo();
await this.gate;
await writeFile(args.destination, Buffer.alloc(10, 1));
return { bytesWritten: 10 };
}
async thumbnail(args: {
file: EnteFile;
destination: string;
}): Promise<{ bytesWritten: number }> {
await writeFile(args.destination, Buffer.alloc(10, 1));
return { bytesWritten: 10 };
}
release(): void {
this.openGate();
}
}
let root: string;
let cacheDir: string;
beforeEach(() => {
root = mkdtempSync(join(tmpdir(), "quak-eviction-"));
cacheDir = join(root, "cache");
});
afterEach(() => {
if (root && existsSync(root))
rmSync(root, { recursive: true, force: true });
});
const originalPath = (id: number): string =>
join(cacheDir, "originals", `${id}.jpg`);
// Pin an original's mtime to a fixed second-resolution instant so LRU order is
// deterministic. Lower `seconds` = older = evicted first.
const setMtime = (id: number, seconds: number): void => {
utimesSync(originalPath(id), seconds, seconds);
};
const buildCache = (args: {
source?: ContentSource;
files?: EnteFile[];
statfs: StatFsFn;
cacheOriginalsMaxBytes?: number;
freeBelowBytes?: number;
isPinned?: (fileID: number) => boolean;
}): { cache: ContentCache; source: SizedSource } => {
const source = (args.source as SizedSource) ?? new SizedSource();
const byID = new Map<number, EnteFile>();
for (const f of args.files ?? [file(1), file(2), file(3), file(4), file(5)])
byID.set(f.id, f);
const cache = new ContentCache({
pools: new RequestPools(),
source,
cacheDirectory: cacheDir,
downloadDirectory: join(root, "photos"),
getFile: (id) => byID.get(id),
statfs: args.statfs,
cacheOriginalsMaxBytes: args.cacheOriginalsMaxBytes,
freeBelowBytes: args.freeBelowBytes,
isPinned: args.isPinned,
});
return { cache, source };
};
// Plenty of free space, so the configured max governs the limit unless a test
// dials it down. `bsize` of 1 makes `bavail` the free byte count.
const abundantFree: StatFsFn = async () => ({
bsize: 1,
bavail: 1_000_000_000,
});
describe("originals eviction", () => {
it("removes least-recently-used originals to fit the limit", async () => {
// Each original is 10 bytes; a 25-byte cap holds two.
const { cache } = buildCache({
statfs: abundantFree,
cacheOriginalsMaxBytes: 25,
freeBelowBytes: 0,
});
await cache.open();
await cache.original(1);
setMtime(1, 1000);
await cache.original(2);
setMtime(2, 2000);
// Writing the third (total 30 > 25) evicts the oldest, file 1.
await cache.original(3);
expect(existsSync(originalPath(1))).toBe(false);
expect(existsSync(originalPath(2))).toBe(true);
expect(existsSync(originalPath(3))).toBe(true);
expect(cache.pathsFor(1).originalPath).toBeUndefined();
expect(cache.originalsStatus().usedBytes).toBe(20);
});
it("skips pinned files, evicting the oldest unpinned instead", async () => {
const { cache } = buildCache({
statfs: abundantFree,
cacheOriginalsMaxBytes: 25,
freeBelowBytes: 0,
isPinned: (id) => id === 1,
});
await cache.open();
await cache.original(1);
setMtime(1, 1000); // oldest, but pinned
await cache.original(2);
setMtime(2, 2000);
await cache.original(3);
// File 1 is oldest but pinned, so file 2 is evicted instead.
expect(existsSync(originalPath(1))).toBe(true);
expect(existsSync(originalPath(2))).toBe(false);
expect(existsSync(originalPath(3))).toBe(true);
});
it("proceeds over-limit when only pinned files remain", async () => {
// A 5-byte cap cannot hold even one 10-byte original.
const { cache } = buildCache({
statfs: abundantFree,
cacheOriginalsMaxBytes: 5,
freeBelowBytes: 0,
isPinned: () => true,
});
await cache.open();
const result = await cache.original(1);
expect(existsSync(result.path)).toBe(true);
const status = cache.originalsStatus();
expect(status.usedBytes).toBe(10);
expect(status.usedBytes).toBeGreaterThan(status.limitBytes ?? 0);
});
it("keeps a just-written original larger than the limit, evicting it only on a later write", async () => {
// A 5-byte cap cannot hold even one 10-byte original, but a fetch must
// never evict the file it just wrote and is about to return.
const { cache } = buildCache({
statfs: abundantFree,
cacheOriginalsMaxBytes: 5,
freeBelowBytes: 0,
});
await cache.open();
const result = await cache.original(1);
// The just-written original survives its own over-budget write and the
// returned path exists on disk.
expect(existsSync(result.path)).toBe(true);
expect(existsSync(originalPath(1))).toBe(true);
expect(cache.originalsStatus().usedBytes).toBe(10);
setMtime(1, 1000);
// A later write finds file 1 eligible and evicts it to make room, while
// the newly written file 2 is itself kept over-limit.
await cache.original(2);
expect(existsSync(originalPath(1))).toBe(false);
expect(existsSync(originalPath(2))).toBe(true);
expect(cache.originalsStatus().usedBytes).toBe(10);
});
it("adapts the limit down as the disk fills and up as space returns", async () => {
// The configured max is generous; free space drives the limit. Eviction
// only kicks in once free space falls below the protected reserve.
let free = 1_000_000;
const statfs: StatFsFn = async () => ({ bsize: 1, bavail: free });
const { cache } = buildCache({
statfs,
cacheOriginalsMaxBytes: 1000,
freeBelowBytes: 100,
});
await cache.open();
// Ample free space: the limit is the configured max and nothing is
// evicted as three 10-byte originals accumulate.
await cache.original(1);
setMtime(1, 1000);
await cache.original(2);
setMtime(2, 2000);
await cache.original(3);
setMtime(3, 3000);
expect(cache.originalsStatus().limitBytes).toBe(1000);
expect(cache.originalsStatus().usedBytes).toBe(30);
// The disk fills: only 95 bytes free, below the 100-byte reserve. The
// next write (used 40) sees limit = min(1000, 40 + 95 - 100) = 35 and
// evicts the oldest original (file 1) to fit.
free = 95;
await cache.original(4);
expect(cache.originalsStatus().limitBytes).toBe(35);
expect(existsSync(originalPath(1))).toBe(false);
expect(cache.originalsStatus().usedBytes).toBe(30);
setMtime(4, 4000);
// Space returns: the limit rises back to the configured max and the
// next write is kept without eviction.
free = 1_000_000;
await cache.original(5);
expect(cache.originalsStatus().limitBytes).toBe(1000);
expect(existsSync(originalPath(4))).toBe(true);
expect(existsSync(originalPath(5))).toBe(true);
});
it("bumps an original's mtime when a read returns its path", async () => {
const { cache } = buildCache({
statfs: abundantFree,
cacheOriginalsMaxBytes: 1000,
freeBelowBytes: 0,
});
await cache.open();
await cache.original(1);
// Age the file well into the past.
setMtime(1, 1000);
expect(statSync(originalPath(1)).mtimeMs).toBeLessThan(2_000_000);
// A second read is a cache hit that must touch the file.
const before = Date.now();
await cache.original(1);
expect(statSync(originalPath(1)).mtimeMs).toBeGreaterThanOrEqual(
before - 2000,
);
});
it("keeps both originals when two over-budget fetches race", async () => {
// A 15-byte cap holds one 10-byte original but not two. Two fetches are
// held in flight together; when both stored files cross the limit,
// neither eviction pass may delete the sibling whose path has not yet
// been returned, so both survive over-limit.
const source = new GatedSource();
const { cache } = buildCache({
source,
statfs: abundantFree,
cacheOriginalsMaxBytes: 15,
freeBelowBytes: 0,
});
await cache.open();
const p1 = cache.original(1);
const p2 = cache.original(2);
await source.bothStarted; // both downloads are in flight before either stores
source.release();
const [r1, r2] = await Promise.all([p1, p2]);
// Both returned paths exist on disk even though together they exceed the
// cap; nothing was evicted out from under a fetch still in progress.
expect(existsSync(r1.path)).toBe(true);
expect(existsSync(r2.path)).toBe(true);
expect(existsSync(originalPath(1))).toBe(true);
expect(existsSync(originalPath(2))).toBe(true);
expect(cache.originalsStatus().usedBytes).toBe(20);
});
it("never counts or evicts thumbnails", async () => {
const { cache } = buildCache({
statfs: abundantFree,
cacheOriginalsMaxBytes: 5,
freeBelowBytes: 0,
});
await cache.open();
await cache.thumbnail(1);
await cache.thumbnail(2);
expect(cache.pathsFor(1).thumbnailPath).toBeDefined();
expect(cache.pathsFor(2).thumbnailPath).toBeDefined();
expect(cache.originalsStatus().usedBytes).toBe(0);
});
});
test/library/content-library.test.ts
+746
View File
@@ -0,0 +1,746 @@
/**
* Integration between `Library` and the content cache (issue #46).
*
* The cache itself is covered in `content.test.ts`; this file locks the wiring:
* `Library.open` builds the cache from a content source, `lib.photos` hands out
* `Photo` objects that fetch through it, `lib.thumbnails.ensure` drives it, and
* a cached path shows up on the projected record. A library opened without a
* content source leaves those methods throwing rather than silently doing
* nothing. It also covers a `Photo`'s `savePath`, `isLocal`, `download()`,
* `content()`, `exif()` and the methods that each return one EXIF field.
*/
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import {
mkdtempSync,
readdirSync,
readFileSync,
rmSync,
existsSync,
statSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { Library, type LibraryOptions } from "../../src/library/index.js";
import type { ContentSource } from "../../src/library/content.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
import { readPhotoExif, type PhotoExif } from "../../src/exif.js";
import { HEIC_WITH_EXIF } from "../exif-heic.js";
import { JPEG_WITH_BAD_EXIF, JPEG_WITH_EXIF } from "../exif-jpeg.js";
import {
asLivePhoto,
cdnSource,
IMAGE,
livePhotoHash,
livePhotoZip,
VIDEO,
} from "../live-photo.js";
const USER_ID = 7;
// Every file is taken at noon local time on 2026-03-01, in microseconds as Ente
// stores times, so the machine's time zone cannot move it to another day; it
// is saved in the folder `DAY`.
const TAKEN = new Date(2026, 2, 1, 12).getTime() * 1000;
const DAY = join("2026", "2026-03", "2026-03-01");
const collection = (id: number): Collection => ({
id,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
name: `album-${id}`,
type: "album",
updationTime: 1,
isShared: false,
});
const file = (id: number, collectionID: number): EnteFile => ({
id,
collectionID,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
metadata: {
title: `file-${id}.jpg`,
fileType: "image",
creationTime: TAKEN,
modificationTime: 1,
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime: 1,
});
// A metadata-only client serving one album with one file, once.
class MockClient {
served = false;
whoami(): { email: string; userID: number } {
return { email: "u@example.com", userID: USER_ID };
}
async collectionsSince(): Promise<CollectionsPage> {
if (this.served) return { collections: [], deleted: [], cursor: 1 };
this.served = true;
return { collections: [collection(1)], deleted: [], cursor: 1 };
}
async filesSince(): Promise<FilesPage> {
return { files: [file(1, 1)], deleted: [], cursor: 1 };
}
}
// A content source that writes `original` as every original and a marker file
// as every thumbnail, and counts the fetches of each.
const stubSource = (
original: string | Uint8Array = "orig-bytes",
): ContentSource & {
originalCalls: () => number;
thumbCalls: () => number;
} => {
let originalCalls = 0;
let thumbCalls = 0;
return {
originalCalls: () => originalCalls,
thumbCalls: () => thumbCalls,
original: async ({ destination }) => {
originalCalls++;
writeFileSync(destination, original);
return { bytesWritten: original.length };
},
thumbnail: async ({ destination }) => {
thumbCalls++;
writeFileSync(destination, "thumb");
return { bytesWritten: 5 };
},
};
};
let root: string;
beforeEach(() => {
root = mkdtempSync(join(tmpdir(), "quak-content-lib-"));
});
afterEach(() => {
if (root && existsSync(root))
rmSync(root, { recursive: true, force: true });
});
describe("Library content wiring", () => {
it("fetches a thumbnail through a Photo and records its cache path", async () => {
const source = stubSource();
const lib = await Library.open({
client: new MockClient(),
cacheDirectory: join(root, "cache"),
contentSource: source,
refreshIntervalSeconds: 3600,
// On-demand wiring only; the background precache (#48) is covered
// in precache.test.ts and would race the exact-count assertions.
precacheThumbnails: false,
precacheOriginals: false,
});
const photo = lib.photos.byID({ fileID: 1 });
expect(photo).toBeDefined();
const result = await photo!.thumbnail();
expect(source.thumbCalls()).toBe(1);
expect(result.path).toBe(join(root, "cache", "thumbnails", "1.jpg"));
expect(existsSync(result.path)).toBe(true);
// The cached path is now on the projected record.
expect(lib.photos.byID({ fileID: 1 })!.record().thumbnailPath).toBe(
result.path,
);
await lib.close();
});
it("drives thumbnails.ensure through the cache", async () => {
const source = stubSource();
const lib = await Library.open({
client: new MockClient(),
cacheDirectory: join(root, "cache"),
contentSource: source,
refreshIntervalSeconds: 3600,
// On-demand wiring only; the background precache (#48) is covered
// in precache.test.ts and would race the exact-count assertions.
precacheThumbnails: false,
precacheOriginals: false,
});
const results = await lib.thumbnails.ensure({
fileIDs: [1],
priority: "visible",
});
expect(results).toEqual([
{ fileID: 1, path: join(root, "cache", "thumbnails", "1.jpg") },
]);
await lib.close();
});
it("does not take a live photo's image or video with no JSON file as its original when it opens, and precaches both", async () => {
const { file: live, body } = await asLivePhoto(file(1, 1));
class LiveClient extends MockClient {
override async filesSince(): Promise<FilesPage> {
return { files: [live], deleted: [], cursor: 1 };
}
}
const originals = join(root, "cache", "originals");
const open = (opts: Partial<LibraryOptions>): Promise<Library> =>
Library.open({
client: new LiveClient(),
cacheDirectory: join(root, "cache"),
contentSource: cdnSource(new Map([[1, body]])),
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
...opts,
});
// A first run records the library, so the next one knows that file 1
// is a live photo when it opens the cache.
await (await open({})).close();
writeFileSync(join(originals, "1.heic"), "an image");
writeFileSync(join(originals, "1.mov"), "a video");
let precached!: () => void;
const done = new Promise<void>((r) => (precached = r));
const lib = await open({
precacheOriginals: true,
onProgress: (e) => {
if (e.operation === "precacheOriginals" && e.status === "done")
precached();
},
});
expect(
lib.photos.byID({ fileID: 1 })!.record().originalPath,
).toBeUndefined();
await done;
expect(readdirSync(originals).sort()).toEqual([
"1.heic",
"1.livephoto.json",
"1.mov",
]);
expect(readFileSync(join(originals, "1.heic"))).toEqual(
Buffer.from(IMAGE),
);
expect(readFileSync(join(originals, "1.mov"))).toEqual(
Buffer.from(VIDEO),
);
expect(
JSON.parse(
readFileSync(join(originals, "1.livephoto.json"), "utf-8"),
),
).toEqual({ image: "1.heic", video: "1.mov" });
expect(lib.photos.byID({ fileID: 1 })!.record().originalPath).toBe(
join(originals, "1.heic"),
);
await lib.close();
});
it("throws from content methods when opened without a content source", async () => {
const lib = await Library.open({
client: new MockClient(),
cacheDirectory: join(root, "cache"),
refreshIntervalSeconds: 3600,
});
await expect(
lib.photos.byID({ fileID: 1 })!.thumbnail(),
).rejects.toThrow(/content cache/i);
await expect(
lib.photos.byID({ fileID: 1 })!.download(),
).rejects.toThrow(/content cache/i);
await expect(
lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
).rejects.toThrow(/content cache/i);
await lib.close();
});
});
describe("Photo save path, local copy, content and EXIF", () => {
// The same account, with `files` in its album instead.
class FilesClient extends MockClient {
constructor(private readonly files: EnteFile[]) {
super();
}
override async filesSince(): Promise<FilesPage> {
return { files: this.files, deleted: [], cursor: 1 };
}
}
const open = (opts: Partial<LibraryOptions> = {}): Promise<Library> =>
Library.open({
client: new MockClient(),
cacheDirectory: join(root, "cache"),
downloadDirectory: join(root, "backup"),
contentSource: stubSource(),
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
...opts,
});
it("names where a backup writes the original, which is local once the backup has written it", async () => {
const lib = await open();
const photo = lib.photos.byID({ fileID: 1 })!;
const savePath = join(root, "backup", DAY, "2026-03-01.1.jpg");
expect(photo.savePath).toBe(savePath);
expect(photo.isLocal).toBe(false);
await lib.backup();
expect(photo.savePath).toBe(savePath);
expect(existsSync(savePath)).toBe(true);
expect(photo.isLocal).toBe(true);
await lib.close();
});
it("returns the original's bytes, and a copy only in the cache is not local", async () => {
const lib = await open();
const photo = lib.photos.byID({ fileID: 1 })!;
expect(await photo.content()).toEqual(Buffer.from("orig-bytes"));
expect(existsSync(join(root, "cache", "originals", "1.jpg"))).toBe(
true,
);
expect(existsSync(photo.savePath)).toBe(false);
expect(photo.isLocal).toBe(false);
await lib.close();
});
it("saves under photos/ in the working directory at open without a download directory", async () => {
const cwd = vi.spyOn(process, "cwd").mockReturnValue(root);
const lib = await open({ downloadDirectory: undefined });
cwd.mockRestore();
const photo = lib.photos.byID({ fileID: 1 })!;
expect(lib.downloadDirectory).toBe(join(root, "photos"));
expect(photo.savePath).toBe(
join(root, "photos", DAY, "2026-03-01.1.jpg"),
);
expect(photo.isLocal).toBe(false);
await lib.close();
});
it("refuses an empty download directory", async () => {
await expect(open({ downloadDirectory: "" })).rejects.toThrow(
/downloadDirectory is empty/,
);
});
it("keeps a photo's save path after a refresh removes its file", async () => {
// The same account, whose album is deleted on the second refresh.
class AlbumDeletedClient extends MockClient {
override async collectionsSince(): Promise<CollectionsPage> {
if (!this.served) return super.collectionsSince();
return { collections: [], deleted: [1], cursor: 2 };
}
}
const lib = await open({ client: new AlbumDeletedClient() });
const photo = lib.photos.byID({ fileID: 1 })!;
await photo.download();
await lib.fresh();
expect(lib.photos.byID({ fileID: 1 })).toBeUndefined();
expect(photo.savePath).toBe(
join(root, "backup", DAY, "2026-03-01.1.jpg"),
);
expect(photo.isLocal).toBe(true);
await lib.close();
});
it("dates the save path and download() by the membership its takenAt comes from", async () => {
// One file in two albums. The date edited in Ente has reached album 2's
// copy, synced later, but album 1 still has an earlier edit.
const older = file(1, 1);
older.pubMagicMetadata = {
editedTime: new Date(2026, 1, 1, 12).getTime() * 1000,
};
const newer = file(1, 2);
newer.updationTime = 2;
newer.pubMagicMetadata = {
editedTime: new Date(2026, 3, 15, 12).getTime() * 1000,
};
const client = {
whoami: () => ({ email: "u@example.com", userID: USER_ID }),
collectionsSince: async (): Promise<CollectionsPage> => ({
collections: [collection(1), collection(2)],
deleted: [],
cursor: 1,
}),
filesSince: async (args: {
collectionID: number;
}): Promise<FilesPage> => ({
files: [args.collectionID === 1 ? older : newer],
deleted: [],
cursor: 1,
}),
};
const lib = await open({ client });
const photo = lib.photos.byID({ fileID: 1 })!;
expect(photo.takenAt).toBe(new Date(2026, 3, 15, 12).getTime());
expect(photo.savePath).toBe(
join(
root,
"backup",
"2026",
"2026-04",
"2026-04-15",
"2026-04-15.1.jpg",
),
);
// download() writes to that same path, so the photo is then local.
const saved = await photo.download();
expect(saved.path).toBe(photo.savePath);
expect(photo.isLocal).toBe(true);
expect(existsSync(join(root, "backup", "2026", "2026-02"))).toBe(false);
await lib.close();
});
it("downloads a photo held across a refresh that edits its date to the save path it names", async () => {
// The same account, whose second refresh brings a date edited in Ente.
const edited = file(1, 1);
edited.updationTime = 2;
edited.pubMagicMetadata = {
editedTime: new Date(2026, 3, 15, 12).getTime() * 1000,
};
class DateEditedClient extends MockClient {
refreshes = 0;
override async collectionsSince(): Promise<CollectionsPage> {
this.refreshes++;
return {
collections: [
{ ...collection(1), updationTime: this.refreshes },
],
deleted: [],
cursor: this.refreshes,
};
}
override async filesSince(): Promise<FilesPage> {
return {
files: [this.refreshes === 1 ? file(1, 1) : edited],
deleted: [],
cursor: this.refreshes,
};
}
}
const lib = await open({ client: new DateEditedClient() });
const photo = lib.photos.byID({ fileID: 1 })!;
await lib.fresh();
expect(lib.photos.byID({ fileID: 1 })!.takenAt).toBe(
new Date(2026, 3, 15, 12).getTime(),
);
const saved = await photo.download();
expect(saved.path).toBe(photo.savePath);
expect(saved.path).toBe(join(root, "backup", DAY, "2026-03-01.1.jpg"));
expect(photo.isLocal).toBe(true);
await lib.close();
});
it("has a save path, and is not local, without a content source", async () => {
const lib = await open({ contentSource: undefined });
const photo = lib.photos.byID({ fileID: 1 })!;
expect(photo.savePath).toBe(
join(root, "backup", DAY, "2026-03-01.1.jpg"),
);
expect(photo.isLocal).toBe(false);
await lib.close();
});
it("gives a live photo's image as its save path once a backup has stored it", async () => {
const { file: live, body } = await asLivePhoto(file(1, 1));
const lib = await open({
client: new FilesClient([live]),
contentSource: cdnSource(new Map([[1, body]])),
});
const photo = lib.photos.byID({ fileID: 1 })!;
const day = join(root, "backup", DAY);
// Until then the name comes from the title, file-1.jpg; the backup
// stores the image with the extension found inside the live photo.
expect(photo.savePath).toBe(join(day, "2026-03-01.1.jpg"));
await lib.backup();
expect(photo.savePath).toBe(join(day, "2026-03-01.1.heic"));
expect(photo.isLocal).toBe(true);
expect(await photo.content()).toEqual(Buffer.from(IMAGE));
await lib.close();
});
it("downloads an original the cache holds by copying it, without fetching it again", async () => {
const source = stubSource();
const lib = await open({ contentSource: source });
const photo = lib.photos.byID({ fileID: 1 })!;
await photo.original();
expect(source.originalCalls()).toBe(1);
expect(photo.isLocal).toBe(false);
const saved = await photo.download();
expect(source.originalCalls()).toBe(1);
expect(saved).toEqual({
path: photo.savePath,
bytes: "orig-bytes".length,
});
expect(readFileSync(photo.savePath, "utf-8")).toBe("orig-bytes");
expect(photo.isLocal).toBe(true);
// The cache keeps its own copy.
expect(existsSync(join(root, "cache", "originals", "1.jpg"))).toBe(
true,
);
await lib.close();
});
it("downloads an original the cache does not hold straight to its save path", async () => {
const source = stubSource();
const lib = await open({ contentSource: source });
const photo = lib.photos.byID({ fileID: 1 })!;
const saved = await photo.download();
expect(source.originalCalls()).toBe(1);
expect(saved.path).toBe(join(root, "backup", DAY, "2026-03-01.1.jpg"));
expect(readFileSync(saved.path, "utf-8")).toBe("orig-bytes");
expect(photo.isLocal).toBe(true);
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
await lib.close();
});
it("does nothing when download() finds the original already at its save path", async () => {
const source = stubSource();
const lib = await open({ contentSource: source });
const photo = lib.photos.byID({ fileID: 1 })!;
const first = await photo.download();
const written = statSync(first.path).ino;
const second = await photo.download();
expect(second).toEqual(first);
expect(source.originalCalls()).toBe(1);
expect(statSync(second.path).ino).toBe(written);
await lib.close();
});
it("downloads a live photo the cache holds as its image, its video and the JSON file naming them", async () => {
const { file: live, body } = await asLivePhoto(file(1, 1));
const lib = await open({
client: new FilesClient([live]),
contentSource: cdnSource(new Map([[1, body]])),
});
const photo = lib.photos.byID({ fileID: 1 })!;
await photo.original();
const day = join(root, "backup", DAY);
const saved = await photo.download();
expect(saved).toEqual({
path: join(day, "2026-03-01.1.heic"),
videoPath: join(day, "2026-03-01.1.mov"),
bytes: IMAGE.length,
});
expect(readdirSync(day).sort()).toEqual([
"2026-03-01.1.heic",
"2026-03-01.1.livephoto.json",
"2026-03-01.1.mov",
]);
expect(
JSON.parse(
readFileSync(join(day, "2026-03-01.1.livephoto.json"), "utf-8"),
),
).toEqual({ image: "2026-03-01.1.heic", video: "2026-03-01.1.mov" });
expect(photo.savePath).toBe(saved.path);
expect(photo.isLocal).toBe(true);
await lib.close();
});
// Every tag in JPEG_WITH_EXIF, by name, in the order of its IFDs.
const jpegTags = [
"Make",
"Model",
"Orientation",
"Exif IFD Pointer",
"GPS Info IFD Pointer",
"ExposureTime",
"FNumber",
"ISOSpeedRatings",
"DateTimeOriginal",
"OffsetTimeOriginal",
"FocalLength",
"LensModel",
"GPSLatitudeRef",
"GPSLatitude",
"GPSLongitudeRef",
"GPSLongitude",
"GPSAltitudeRef",
"GPSAltitude",
];
// HEIC_WITH_EXIF holds those and the tags exiftool adds to every file.
const heicTags = [
...jpegTags,
"YCbCrPositioning",
"ExifVersion",
"ComponentsConfiguration",
"ColorSpace",
"GPSVersionID",
];
it.each([
[
"JPEG",
JPEG_WITH_EXIF,
jpegTags,
{
"Exif IFD Pointer": { value: 88 },
GPSLatitudeRef: { value: ["N"], description: "North latitude" },
},
],
[
"HEIC",
HEIC_WITH_EXIF,
heicTags,
{
ColorSpace: { value: 0xffff, description: "Uncalibrated" },
ExifVersion: { description: "0232" },
},
],
])(
"returns every EXIF tag of a %s original, by name",
async (_, bytes, names, others) => {
const lib = await open({ contentSource: stubSource(bytes) });
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
expect(Object.keys(exif).sort()).toEqual([...names].sort());
// Tags outside the thirteen fields, as exifreader decodes them.
expect(exif).toMatchObject(others);
await lib.close();
},
);
it("picks the common EXIF fields from a JPEG original's tags", async () => {
const lib = await open({ contentSource: stubSource(JPEG_WITH_EXIF) });
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
expect(readPhotoExif(exif)).toStrictEqual({
make: "Canon",
model: "EOS R5",
lensModel: "RF50mm F1.8 STM",
// The camera's clock reading, held in the Date's UTC fields.
dateTimeOriginal: new Date(Date.UTC(2021, 6, 15, 14, 30)),
offsetTimeOriginal: "+02:00",
exposureTime: 1 / 250,
fNumber: 2.8,
iso: 400,
focalLength: 50,
orientation: 6,
gpsLatitude: 40 + 26 / 60 + 46 / 3600,
gpsLongitude: -(79 + 58 / 60 + 56 / 3600),
gpsAltitude: -12.5,
});
await lib.close();
});
// The fields picked from HEIC_WITH_EXIF's tags, and from JPEG_WITH_EXIF's,
// which hold the same values.
const heicFields: PhotoExif = {
make: "Canon",
model: "EOS R5",
lensModel: "RF50mm F1.8 STM",
dateTimeOriginal: new Date(Date.UTC(2021, 6, 15, 14, 30)),
offsetTimeOriginal: "+02:00",
exposureTime: 1 / 250,
fNumber: 2.8,
iso: 400,
focalLength: 50,
orientation: 6,
gpsLatitude: 40 + 26 / 60 + 46 / 3600,
gpsLongitude: -(79 + 58 / 60 + 56 / 3600),
gpsAltitude: -12.5,
};
it("picks the same common EXIF fields from a HEIC original's tags", async () => {
const lib = await open({ contentSource: stubSource(HEIC_WITH_EXIF) });
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
expect(readPhotoExif(exif)).toStrictEqual(heicFields);
await lib.close();
});
it("reads the EXIF of a live photo whose image is a HEIC", async () => {
const { file: live, body } = await asLivePhoto(
file(1, 1),
livePhotoZip({ "image.heic": HEIC_WITH_EXIF, "video.mov": VIDEO }),
livePhotoHash(HEIC_WITH_EXIF, VIDEO),
);
const lib = await open({
client: new FilesClient([live]),
contentSource: cdnSource(new Map([[1, body]])),
});
const exif = await lib.photos.byID({ fileID: 1 })!.exif();
expect(readPhotoExif(exif)).toStrictEqual(heicFields);
await lib.close();
});
// The build's type check, not this test, makes sure `Photo` has a method
// for every `PhotoExif` field, whatever the fixtures hold: `Photo`
// implements a type with one method per field. This test checks that each
// method gives the field picked from the tags exif() returns.
it.each([
["JPEG", JPEG_WITH_EXIF],
["HEIC", HEIC_WITH_EXIF],
])(
"has a method for each field, agreeing with the tags exif() returns, for a %s",
async (_, bytes) => {
const lib = await open({ contentSource: stubSource(bytes) });
const photo = lib.photos.byID({ fileID: 1 })!;
const fields = readPhotoExif(await photo.exif());
// The file holds every field, so every method is checked.
expect(fields).toStrictEqual(heicFields);
for (const [field, value] of Object.entries(fields)) {
expect(await photo[field as keyof PhotoExif]()).toStrictEqual(
value,
);
}
await lib.close();
},
);
it("returns no EXIF tags for an original that is not an image", async () => {
const lib = await open();
const photo = lib.photos.byID({ fileID: 1 })!;
expect(await photo.exif()).toStrictEqual({});
expect(await photo.dateTimeOriginal()).toBeUndefined();
await lib.close();
});
it("returns no EXIF tags for a JPEG whose EXIF cannot be parsed", async () => {
const lib = await open({
contentSource: stubSource(JPEG_WITH_BAD_EXIF),
});
expect(await lib.photos.byID({ fileID: 1 })!.exif()).toStrictEqual({});
await lib.close();
});
it("throws from exif() on a video without a content source, as the other content methods do", async () => {
const video = file(1, 1);
video.metadata.fileType = "video";
const lib = await open({
client: new FilesClient([video]),
contentSource: undefined,
});
await expect(lib.photos.byID({ fileID: 1 })!.exif()).rejects.toThrow(
/content cache/i,
);
await lib.close();
});
it("returns no EXIF tags for a video, without fetching it", async () => {
const video = file(1, 1);
video.metadata.fileType = "video";
const source = stubSource(JPEG_WITH_EXIF);
const lib = await open({
client: new FilesClient([video]),
contentSource: source,
});
expect(await lib.photos.byID({ fileID: 1 })!.exif()).toStrictEqual({});
expect(source.originalCalls()).toBe(0);
await lib.close();
});
});
test/library/content.test.ts
+724
View File
@@ -0,0 +1,724 @@
/**
* Tests for the on-disk content and thumbnail cache (issue #46).
*
* The cache keys stored bytes by `fileID` under `cacheDirectory`:
* `originals/<fileID>.<ext>` and `thumbnails/<fileID>.<ext>`. Its contract:
*
* 1. **Fetch once, then serve from disk.** The first `original`/`thumbnail`
* fetches through the request pool and stores the bytes; the next finds the
* file present and returns its path with a single `skipped` event and no
* network. A file already stored at its save path under the
* `downloadDirectory` counts as present too.
* 2. **Present-means-complete.** Content appears only by the streaming atomic
* writer's rename, so a file that exists is whole. The directory listing
* taken at `open()` is the record of what is cached, and the orphan temp
* files a crashed write may have left are reaped there.
* 3. **`thumbnails.ensure` drives the thumbnail pool with priority, dedup, and
* abort.** A `fileID` asked for twice downloads once; a visible request is
* served ahead of a background one; and an `AbortSignal` drops work still
* queued while letting an in-flight fetch finish.
*
* The `ContentSource` is a stand-in: it writes deterministic bytes to the
* destination and returns the count, so the cache logic is exercised with no
* crypto and no network. Ordering tests gate the stand-in on explicit deferreds
* and assert the persisted result, never a bare call or a timer.
*/
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import {
mkdtempSync,
rmSync,
existsSync,
writeFileSync,
mkdirSync,
readdirSync,
readFileSync,
statSync,
} from "node:fs";
import { spawnSync } from "node:child_process";
import { tmpdir } from "node:os";
import { join } from "node:path";
import {
ContentCache,
savePath,
type ContentSource,
type EnsureEvent,
} from "../../src/library/content.js";
import { RequestPools } from "../../src/library/pools.js";
import type { EnteFile } from "../../src/model/types.js";
import {
asLivePhoto,
cdnSource,
IMAGE,
livePhotoZip,
VIDEO,
} from "../live-photo.js";
const file = (id: number, title = `file-${id}.jpg`): EnteFile => ({
id,
collectionID: 1,
ownerID: 1,
key: new Uint8Array([id & 0xff]),
metadata: {
title,
fileType: "image",
creationTime: 0,
modificationTime: 0,
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime: 0,
});
// A deferred with externally callable resolve, used to gate the stand-in source
// so ordering is controlled by the test rather than by timing.
const deferred = (): { promise: Promise<void>; resolve: () => void } => {
let resolve!: () => void;
const promise = new Promise<void>((r) => {
resolve = r;
});
return { promise, resolve };
};
// A ContentSource that writes `${kind}:${fileID}` bytes to the destination and
// records every call. `gate` optionally blocks a call until released, and
// `completed` records the order in which fetches finished — the observable used
// by the priority and abort tests instead of a timer.
class StubSource implements ContentSource {
originalCalls: number[] = [];
thumbnailCalls: number[] = [];
completed: number[] = [];
emptyFor = new Set<number>();
gates = new Map<number, Promise<void>>();
private async run(
kind: "original" | "thumbnail",
file: EnteFile,
destination: string,
): Promise<{ bytesWritten: number }> {
const gate = this.gates.get(file.id);
if (gate) await gate;
const bytes = this.emptyFor.has(file.id)
? new Uint8Array(0)
: new TextEncoder().encode(`${kind}:${file.id}`);
writeFileSync(destination, bytes);
this.completed.push(file.id);
return { bytesWritten: bytes.length };
}
async original(args: {
file: EnteFile;
destination: string;
}): Promise<{ bytesWritten: number }> {
this.originalCalls.push(args.file.id);
return this.run("original", args.file, args.destination);
}
async thumbnail(args: {
file: EnteFile;
destination: string;
}): Promise<{ bytesWritten: number }> {
this.thumbnailCalls.push(args.file.id);
return this.run("thumbnail", args.file, args.destination);
}
}
let root: string;
let cacheDir: string;
beforeEach(() => {
root = mkdtempSync(join(tmpdir(), "quak-content-"));
cacheDir = join(root, "cache");
});
afterEach(() => {
if (root && existsSync(root))
rmSync(root, { recursive: true, force: true });
});
const buildCache = (
args: {
source?: ContentSource;
files?: EnteFile[];
pools?: RequestPools;
downloadDirectory?: string;
} = {},
): { cache: ContentCache; source: StubSource } => {
const source = (args.source as StubSource) ?? new StubSource();
const byID = new Map<number, EnteFile>();
for (const f of args.files ?? [file(1), file(2), file(3)])
byID.set(f.id, f);
const cache = new ContentCache({
pools: args.pools ?? new RequestPools(),
source,
cacheDirectory: cacheDir,
downloadDirectory: args.downloadDirectory ?? join(root, "photos"),
getFile: (id) => byID.get(id),
});
return { cache, source };
};
// Microseconds, as Ente stores times, for noon local time on a day, so the
// machine's time zone cannot move the photo to another day.
const noon = (year: number, month: number, day: number): number =>
new Date(year, month - 1, day, 12).getTime() * 1000;
describe("savePath", () => {
it("files an original by year, month and day under the root", () => {
const f = file(12345, "IMG_0001.HEIC");
f.metadata.creationTime = noon(2026, 3, 1);
expect(savePath("/photos", f)).toBe(
"/photos/2026/2026-03/2026-03-01/2026-03-01.12345.HEIC",
);
});
it("dates an original by the date the user set, when there is one", () => {
const f = file(7, "a.jpg");
f.metadata.creationTime = noon(2026, 3, 1);
f.pubMagicMetadata = { editedTime: noon(1999, 12, 31) };
expect(savePath("/photos", f)).toBe(
"/photos/1999/1999-12/1999-12-31/1999-12-31.7.jpg",
);
});
it("takes the extension from the title it was uploaded with, not a new name", () => {
const f = file(8, "upload");
f.metadata.creationTime = noon(2026, 3, 1);
f.pubMagicMetadata = { editedName: "renamed.png" };
expect(savePath("/photos", f)).toBe(
"/photos/2026/2026-03/2026-03-01/2026-03-01.8.bin",
);
});
});
describe("ContentCache.open", () => {
it("creates the cache directories with 0700 permissions", async () => {
const { cache } = buildCache();
await cache.open();
const originals = join(cacheDir, "originals");
const thumbnails = join(cacheDir, "thumbnails");
expect(existsSync(originals)).toBe(true);
expect(existsSync(thumbnails)).toBe(true);
expect(statSync(originals).mode & 0o777).toBe(0o700);
expect(statSync(thumbnails).mode & 0o777).toBe(0o700);
});
it("removes temp files of an exited process, keeping those of a running one and complete content", async () => {
const originals = join(cacheDir, "originals");
const thumbnails = join(cacheDir, "thumbnails");
mkdirSync(originals, { recursive: true });
mkdirSync(thumbnails, { recursive: true });
// A child that has already exited: its process ID is not running.
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
const orphan = join(originals, `.quak-${exitedPID}-abc123.tmp`);
const orphanThumb = join(thumbnails, `.quak-${exitedPID}-abc456.tmp`);
// This test's own process stands in for another process still
// downloading into the same cache.
const inProgress = join(originals, `.quak-${process.pid}-def123.tmp`);
const inProgressThumb = join(
thumbnails,
`.quak-${process.pid}-def456.tmp`,
);
const complete = join(originals, "1.jpg");
const thumb = join(thumbnails, "2.jpg");
writeFileSync(orphan, "half-written");
writeFileSync(orphanThumb, "half-written");
writeFileSync(inProgress, "half-written");
writeFileSync(inProgressThumb, "half-written");
writeFileSync(complete, "whole");
writeFileSync(thumb, "whole-thumb");
const { cache } = buildCache();
await cache.open();
expect(existsSync(orphan)).toBe(false);
expect(existsSync(orphanThumb)).toBe(false);
expect(existsSync(inProgress)).toBe(true);
expect(existsSync(inProgressThumb)).toBe(true);
expect(existsSync(complete)).toBe(true);
expect(existsSync(thumb)).toBe(true);
expect(cache.pathsFor(1)).toEqual({ originalPath: complete });
});
it("records already-cached files so their paths appear in pathsFor", async () => {
const originals = join(cacheDir, "originals");
const thumbnails = join(cacheDir, "thumbnails");
mkdirSync(originals, { recursive: true });
mkdirSync(thumbnails, { recursive: true });
writeFileSync(join(originals, "1.jpg"), "orig");
writeFileSync(join(thumbnails, "1.jpg"), "thumb");
const { cache } = buildCache();
await cache.open();
expect(cache.pathsFor(1)).toEqual({
originalPath: join(originals, "1.jpg"),
thumbnailPath: join(thumbnails, "1.jpg"),
});
expect(cache.pathsFor(2)).toEqual({});
});
});
describe("ContentCache.original / thumbnail", () => {
it("fetches once, then serves the cached file with a single skipped event", async () => {
const { cache, source } = buildCache();
await cache.open();
const events: string[] = [];
const first = await cache.original(1, {
onProgress: (e) => events.push(e.status),
});
expect(source.originalCalls).toEqual([1]);
expect(first.path).toBe(join(cacheDir, "originals", "1.jpg"));
expect(first.bytes).toBe("original:1".length);
expect(existsSync(first.path)).toBe(true);
expect(statSync(first.path).mode & 0o777).toBe(0o600);
expect(cache.pathsFor(1).originalPath).toBe(first.path);
const skips: string[] = [];
const second = await cache.original(1, {
onProgress: (e) => skips.push(e.status),
});
// No second download, and exactly one skipped event.
expect(source.originalCalls).toEqual([1]);
expect(second.path).toBe(first.path);
expect(skips).toEqual(["skipped"]);
});
it("takes only a letters-and-digits extension from the title", async () => {
// The title comes from the server; an extension such as `.\..\x`
// must not reach the cache file name, so it becomes `.bin`.
const { cache } = buildCache({
files: [file(1, "a.jpg"), file(2, "b.\\..\\x"), file(3, "")],
});
await cache.open();
expect((await cache.original(1)).path).toBe(
join(cacheDir, "originals", "1.jpg"),
);
expect((await cache.original(2)).path).toBe(
join(cacheDir, "originals", "2.bin"),
);
expect((await cache.original(3)).path).toBe(
join(cacheDir, "originals", "3.bin"),
);
});
it("serves a file already present in the download directory without fetching", async () => {
const downloadDirectory = join(root, "backup");
const day = join(downloadDirectory, "2026", "2026-03", "2026-03-01");
mkdirSync(day, { recursive: true });
const backupPath = join(day, "2026-03-01.1.jpg");
writeFileSync(backupPath, "from-backup");
const f = file(1);
f.metadata.creationTime = noon(2026, 3, 1);
const { cache, source } = buildCache({
downloadDirectory,
files: [f],
});
await cache.open();
const events: EnsureEvent["status"][] = [];
const result = await cache.original(1, {
onProgress: (e) => events.push(e.status),
});
expect(source.originalCalls).toEqual([]);
expect(result.path).toBe(backupPath);
expect(result.bytes).toBe("from-backup".length);
expect(events).toEqual(["skipped"]);
});
it("fetches and caches a thumbnail", async () => {
const { cache, source } = buildCache();
await cache.open();
const result = await cache.thumbnail(2);
expect(source.thumbnailCalls).toEqual([2]);
expect(result.path).toBe(join(cacheDir, "thumbnails", "2.jpg"));
expect(existsSync(result.path)).toBe(true);
expect(cache.pathsFor(2).thumbnailPath).toBe(result.path);
});
it("shares one download between concurrent callers for the same file", async () => {
const { cache, source } = buildCache();
await cache.open();
const gate = deferred();
source.gates.set(1, gate.promise);
const a = cache.original(1);
const b = cache.original(1);
gate.resolve();
const [ra, rb] = await Promise.all([a, b]);
expect(source.originalCalls).toEqual([1]);
expect(ra.path).toBe(rb.path);
});
it("does not record a path when the fetched file is empty", async () => {
const { cache, source } = buildCache();
source.emptyFor.add(1);
await cache.open();
await expect(cache.original(1)).rejects.toThrow(/empty/i);
expect(cache.pathsFor(1).originalPath).toBeUndefined();
});
it("rejects an unknown file", async () => {
const { cache } = buildCache({ files: [] });
await cache.open();
await expect(cache.original(999)).rejects.toThrow(/unknown file/i);
});
});
describe("ContentCache.ensureThumbnails", () => {
it("downloads once for a file listed twice and reports every id", async () => {
const { cache, source } = buildCache();
await cache.open();
const results = await cache.ensureThumbnails({
fileIDs: [1, 1, 2],
priority: "visible",
});
expect(source.thumbnailCalls.sort()).toEqual([1, 2]);
expect(results).toEqual([
{ fileID: 1, path: join(cacheDir, "thumbnails", "1.jpg") },
{ fileID: 2, path: join(cacheDir, "thumbnails", "2.jpg") },
]);
});
it("skips present files and reports a skipped event", async () => {
const thumbnails = join(cacheDir, "thumbnails");
mkdirSync(thumbnails, { recursive: true });
writeFileSync(join(thumbnails, "1.jpg"), "present");
const { cache, source } = buildCache();
await cache.open();
const events: EnsureEvent[] = [];
const results = await cache.ensureThumbnails({
fileIDs: [1, 2],
priority: "ahead",
onProgress: (e) => events.push(e),
});
expect(source.thumbnailCalls).toEqual([2]);
expect(results).toEqual([
{ fileID: 1, path: join(thumbnails, "1.jpg") },
{ fileID: 2, path: join(thumbnails, "2.jpg") },
]);
expect(events).toContainEqual({
fileID: 1,
status: "skipped",
path: join(thumbnails, "1.jpg"),
});
});
it("serves a visible request ahead of an already-queued background one", async () => {
// One thumbnail slot, so exactly one fetch runs at a time and the rest
// wait in the pool. A background fetch takes the slot; a background and
// a visible fetch queue behind it. When the slot frees, the pool must
// pick the visible (on-demand) request ahead of the background one that
// was submitted first. The completion order is the observable.
const pools = new RequestPools({ thumbnailConcurrency: 1 });
const { cache, source } = buildCache({ pools });
await cache.open();
const gateA = deferred();
const gateB = deferred();
const gateC = deferred();
source.gates.set(1, gateA.promise);
source.gates.set(2, gateB.promise);
source.gates.set(3, gateC.promise);
const bgFirst = cache.ensureThumbnails({
fileIDs: [1],
priority: "background",
});
// Let fetch 1 take the only slot before the others queue.
await Promise.resolve();
const bgSecond = cache.ensureThumbnails({
fileIDs: [2],
priority: "background",
});
const visible = cache.ensureThumbnails({
fileIDs: [3],
priority: "visible",
});
gateA.resolve();
gateC.resolve();
gateB.resolve();
await Promise.all([bgFirst, bgSecond, visible]);
// 1 ran first (it held the slot). Of the two that were queued, the
// visible id 3 was served before the background id 2.
expect(source.completed).toEqual([1, 3, 2]);
});
it("drops queued work on abort but keeps an in-flight fetch", async () => {
const pools = new RequestPools({ thumbnailConcurrency: 1 });
const { cache, source } = buildCache({ pools });
await cache.open();
const gate = deferred();
source.gates.set(1, gate.promise);
const controller = new AbortController();
const pending = cache.ensureThumbnails({
fileIDs: [1, 2],
priority: "ahead",
signal: controller.signal,
});
// Fetch 1 is in flight (holds the slot); 2 is queued.
await Promise.resolve();
controller.abort();
gate.resolve();
const results = await pending;
// The in-flight fetch finished and is kept; the queued one was dropped
// before it ran.
expect(source.thumbnailCalls).toEqual([1]);
expect(results).toEqual([
{ fileID: 1, path: join(cacheDir, "thumbnails", "1.jpg") },
{ fileID: 2, error: "aborted" },
]);
});
it("captures a per-file failure without failing the batch", async () => {
const { cache } = buildCache({ files: [file(1)] });
await cache.open();
const results = await cache.ensureThumbnails({
fileIDs: [1, 2],
priority: "background",
});
expect(results[0]).toEqual({
fileID: 1,
path: join(cacheDir, "thumbnails", "1.jpg"),
});
expect(results[1]?.fileID).toBe(2);
expect(results[1]?.error).toMatch(/unknown file/i);
});
});
// A live photo's original is two files, its image and its video, which a
// photo viewer can open, and a JSON file naming them: the two are named with
// the extensions from inside the ZIP, so the names alone do not say which is
// which. These tests download a live photo ZIP through the real download
// layer (test/live-photo.ts).
describe("ContentCache live photos", () => {
const originals = (): string => join(cacheDir, "originals");
// A cache over the stand-in server, which holds `bodies` by file ID.
const cacheOf = (
files: EnteFile[],
bodies: Map<number, Uint8Array>,
): ContentCache => buildCache({ files, source: cdnSource(bodies) }).cache;
it("stores a live photo as its image and its video and a JSON file naming them", async () => {
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
const cache = cacheOf([live], new Map([[5, body]]));
await cache.open();
const result = await cache.original(5);
expect(result).toEqual({
path: join(originals(), "5.heic"),
videoPath: join(originals(), "5.mov"),
bytes: IMAGE.length,
});
expect(readFileSync(result.path)).toEqual(Buffer.from(IMAGE));
expect(readFileSync(result.videoPath!)).toEqual(Buffer.from(VIDEO));
expect(statSync(result.videoPath!).mode & 0o777).toBe(0o600);
expect(
JSON.parse(
readFileSync(join(originals(), "5.livephoto.json"), "utf-8"),
),
).toEqual({ image: "5.heic", video: "5.mov" });
expect(cache.pathsFor(5)).toEqual({ originalPath: result.path });
});
it("serves a stored live photo from disk after the cache is opened again", async () => {
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
const first = cacheOf([live], new Map([[5, body]]));
await first.open();
const stored = await first.original(5);
// This server has nothing, so a fetch would fail.
const second = cacheOf([live], new Map());
await second.open();
const events: string[] = [];
const served = await second.original(5, {
onProgress: (e) => events.push(e.status),
});
expect(served).toEqual(stored);
expect(events).toEqual(["skipped"]);
});
it("leaves the image and video another process has just stored when it opens before their JSON file is written", async () => {
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
const server = cdnSource(new Map([[5, body]]));
let second: ContentCache | undefined;
// Once the image and video are in place, and before the first cache
// writes the JSON file naming them, a second cache opens.
const source: ContentSource = {
original: async (args) => {
const stored = await server.original(args);
second = cacheOf([live], new Map());
await second.open((fileID) => fileID === 5);
return stored;
},
thumbnail: (args) => server.thumbnail(args),
};
const first = buildCache({ files: [live], source }).cache;
await first.open((fileID) => fileID === 5);
const result = await first.original(5);
expect(readdirSync(originals()).sort()).toEqual([
"5.heic",
"5.livephoto.json",
"5.mov",
]);
expect(readFileSync(result.path)).toEqual(Buffer.from(IMAGE));
expect(readFileSync(result.videoPath!)).toEqual(Buffer.from(VIDEO));
expect(second!.pathsFor(5)).toEqual({});
});
it("fetches a live photo's image and video again when the cache opened before knowing it is a live photo and no JSON file names them", async () => {
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
mkdirSync(originals(), { recursive: true });
writeFileSync(join(originals(), "5.heic"), "an image");
writeFileSync(join(originals(), "5.mov"), "a video");
const cache = cacheOf([live], new Map([[5, body]]));
// Opened without being told that file 5 is a live photo, the cache
// records one of the two files as its original, with no video.
await cache.open();
const events: string[] = [];
const result = await cache.original(5, {
onProgress: (e) => events.push(e.status),
});
expect(events.at(-1)).toBe("done");
expect(result).toEqual({
path: join(originals(), "5.heic"),
videoPath: join(originals(), "5.mov"),
bytes: IMAGE.length,
});
expect(readFileSync(result.path)).toEqual(Buffer.from(IMAGE));
expect(readFileSync(result.videoPath!)).toEqual(Buffer.from(VIDEO));
expect(
JSON.parse(
readFileSync(join(originals(), "5.livephoto.json"), "utf-8"),
),
).toEqual({ image: "5.heic", video: "5.mov" });
});
it.each(["missing", "empty"])(
"fetches a live photo again when the video its JSON file names is %s",
async (state) => {
const { file: live, body } = await asLivePhoto(
file(5, "IMG_5.HEIC"),
);
const cache = cacheOf([live], new Map([[5, body]]));
await cache.open();
const stored = await cache.original(5);
if (state === "missing") rmSync(stored.videoPath!);
else writeFileSync(stored.videoPath!, "");
const events: string[] = [];
const fetched = await cache.original(5, {
onProgress: (e) => events.push(e.status),
});
expect(events.at(-1)).toBe("done");
expect(fetched).toEqual(stored);
expect(readFileSync(fetched.videoPath!)).toEqual(
Buffer.from(VIDEO),
);
},
);
it("does not take a file outside originals/ that a live photo's JSON file names", async () => {
const { file: live, body } = await asLivePhoto(file(5, "IMG_5.HEIC"));
mkdirSync(originals(), { recursive: true });
writeFileSync(join(cacheDir, "5.heic"), "not this photo");
writeFileSync(join(originals(), "5.mov"), VIDEO);
writeFileSync(
join(originals(), "5.livephoto.json"),
JSON.stringify({ image: "../5.heic", video: "5.mov" }),
);
const cache = cacheOf([live], new Map([[5, body]]));
await cache.open((fileID) => fileID === 5);
const result = await cache.original(5);
expect(result.path).toBe(join(originals(), "5.heic"));
expect(readFileSync(result.path)).toEqual(Buffer.from(IMAGE));
expect(readFileSync(join(cacheDir, "5.heic"), "utf-8")).toBe(
"not this photo",
);
});
it("evicts a live photo's image, video and JSON file together", async () => {
const a = await asLivePhoto(file(5, "a.HEIC"));
const b = await asLivePhoto(file(6, "b.HEIC"));
const size = IMAGE.length + VIDEO.length;
const cache = new ContentCache({
pools: new RequestPools(),
source: cdnSource(
new Map([
[5, a.body],
[6, b.body],
]),
),
cacheDirectory: cacheDir,
downloadDirectory: join(root, "photos"),
getFile: (id) => [a.file, b.file].find((f) => f.id === id),
// Room for one live photo, on a disk with plenty free.
cacheOriginalsMaxBytes: size,
freeBelowBytes: 0,
statfs: async () => ({ bsize: 1, bavail: 1e12 }),
});
await cache.open();
await cache.original(5);
await cache.original(6);
expect(readdirSync(originals()).sort()).toEqual([
"6.heic",
"6.livephoto.json",
"6.mov",
]);
expect(cache.originalsStatus().usedBytes).toBe(size);
});
it("stores nothing when a live photo does not match its recorded hash", async () => {
const { file: live, body } = await asLivePhoto(
file(5, "IMG_5.HEIC"),
livePhotoZip(),
"not:the recorded hash",
);
const cache = cacheOf([live], new Map([[5, body]]));
await cache.open();
await expect(cache.original(5)).rejects.toThrow(
/file 5: content hash .* does not match/,
);
expect(readdirSync(originals())).toEqual([]);
expect(cache.pathsFor(5)).toEqual({});
});
});
test/library/fresh.test.ts
+611
View File
@@ -0,0 +1,611 @@
/**
* Tests for fresh reads (issue #75, an owner amendment to design #36).
*
* The default read namespaces answer from RAM and never touch the network; a
* background loop keeps the local copy current. `fresh()` adds an awaited path:
* it forces a refresh, waits for it to complete and persist, and only then
* hands back the `albums`/`photos`/`timeline` namespaces, guaranteeing the
* local copy reflects a completed server round-trip. The contracts here:
*
* 1. A fresh read observes a server change that a same-instant default read
* would miss (the default path has not refreshed yet).
* 2. Concurrent fresh reads coalesce onto one in-flight refresh — three
* concurrent `fresh()` calls make exactly one collections round-trip, not
* three.
* 3. A refresh that fails rejects the fresh read (currency was unavailable),
* while the default reads stay silent and keep serving the last good copy.
* 4. `freshCollections()` reads the collection list alone, with one request
* and no collection's files, and stores nothing, so the next refresh still
* fetches the files of every collection (issue #210).
* 5. `refreshCollection()` brings one collection up to date in the store
* (issue #217). It leaves the stored cursor where it was, so the next
* refresh still fetches the files of every other collection that changed.
* It never runs at the same time as a refresh. After a refresh, it stores
* no collection the store does not hold: not a new one, and not one a
* refresh deleted. `close()` waits for it. A library opened on a cache that
* only it wrote awaits its first refresh. That refresh replaces what it
* stored, in the server's order.
*
* The client is the same metadata-only mock the background-refresh tests use:
* no crypto, no network, scripted pages, and a record of each call. A long
* refresh interval keeps the background timer out of the way so each test's
* refreshes are exactly the ones it triggers.
*/
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { Library } from "../../src/library/index.js";
import { MetadataStore } from "../../src/library/store.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
const USER_ID = 42;
// Long enough that no background tick fires during a test; each test's
// refreshes are only the ones its own `fresh()` calls force.
const SLOW_INTERVAL = 3600;
const collection = (id: number, updationTime: number): Collection => ({
id,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
name: `album-${id}`,
type: "album",
updationTime,
isShared: false,
});
const file = (
id: number,
collectionID: number,
updationTime: number,
): EnteFile => ({
id,
collectionID,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
metadata: {
title: `file-${id}.jpg`,
fileType: "image",
creationTime: updationTime,
modificationTime: updationTime,
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime,
});
class MockClient {
userID = USER_ID;
failCollections = false;
collectionsQueue: CollectionsPage[] = [];
filesByCollection = new Map<number, FilesPage[]>();
collectionsSinceTimes: number[] = [];
filesCalls: { collectionID: number; sinceTime: number }[] = [];
whoami(): { email: string; userID: number } {
return { email: "user@example.com", userID: this.userID };
}
async collectionsSince(args: {
sinceTime: number;
}): Promise<CollectionsPage> {
this.collectionsSinceTimes.push(args.sinceTime);
if (this.failCollections) throw new Error("network down");
return (
this.collectionsQueue.shift() ?? {
collections: [],
deleted: [],
cursor: args.sinceTime,
}
);
}
async filesSince(args: {
collectionID: number;
collectionKey: Uint8Array;
sinceTime: number;
}): Promise<FilesPage> {
this.filesCalls.push({
collectionID: args.collectionID,
sinceTime: args.sinceTime,
});
const queue = this.filesByCollection.get(args.collectionID);
return (
queue?.shift() ?? {
files: [],
deleted: [],
cursor: args.sinceTime,
}
);
}
filesFor(collectionID: number, ...pages: FilesPage[]): void {
this.filesByCollection.set(collectionID, pages);
}
}
// A client seeded with one collection and one file, opened with an empty cache
// so the initial refresh is awaited and post-`open()` state is deterministic.
const openSeeded = async (
cacheDirectory: string,
): Promise<{ client: MockClient; lib: Library }> => {
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: SLOW_INTERVAL,
});
return { client, lib };
};
// Hold the client's next files request until `release()` is called. `started`
// resolves once that request has been sent.
const holdNextFilesRequest = (
client: MockClient,
): { started: Promise<void>; release: () => void } => {
let release!: () => void;
const held = new Promise<void>((r) => (release = r));
let sent!: () => void;
const started = new Promise<void>((r) => (sent = r));
const realFilesSince = client.filesSince.bind(client);
client.filesSince = async (args) => {
client.filesSince = realFilesSince;
const page = await realFilesSince(args);
sent();
await held;
return page;
};
return { started, release };
};
describe("Library.fresh", () => {
let dir: string;
let cacheDirectory: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), "quak-fresh-"));
cacheDirectory = join(dir, "cache");
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
it("observes a server change a same-instant default read would miss", async () => {
const { client, lib } = await openSeeded(cacheDirectory);
try {
// A new file appears on the server after open, advancing its
// collection so the next refresh re-enumerates it.
client.collectionsQueue.push({
collections: [collection(1, 200)],
deleted: [],
cursor: 200,
});
client.filesFor(1, {
files: [file(1002, 1, 190)],
deleted: [],
cursor: 190,
});
// A default read at this instant has not refreshed: it misses 1002.
expect(lib.photos.byID({ fileID: 1002 })).toBeUndefined();
// A fresh read forces the round-trip and sees it.
const reads = await lib.fresh();
expect(reads.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
// And the change is now live for the default namespaces too.
expect(lib.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
} finally {
await lib.close();
}
});
it("coalesces concurrent fresh reads onto one in-flight refresh", async () => {
const { client, lib } = await openSeeded(cacheDirectory);
try {
client.collectionsQueue.push({
collections: [collection(1, 200)],
deleted: [],
cursor: 200,
});
client.filesFor(1, {
files: [file(1002, 1, 190)],
deleted: [],
cursor: 190,
});
const collectionsBefore = client.collectionsSinceTimes.length;
const filesBefore = client.filesCalls.length;
// Gate the next collections fetch so all three fresh reads are in
// flight together before any of them completes.
let release: () => void = () => {};
const gate = new Promise<void>((r) => {
release = r;
});
const inner = client.collectionsSince.bind(client);
client.collectionsSince = async (args: { sinceTime: number }) => {
await gate;
return inner(args);
};
const all = Promise.all([lib.fresh(), lib.fresh(), lib.fresh()]);
release();
const [a, b, c] = await all;
// Exactly one collections round-trip and one file round-trip served
// all three fresh reads.
expect(client.collectionsSinceTimes.length).toBe(
collectionsBefore + 1,
);
expect(client.filesCalls.length).toBe(filesBefore + 1);
// All three observed the change.
for (const reads of [a, b, c]) {
expect(reads.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
}
} finally {
await lib.close();
}
});
it("rejects the fresh read when the refresh fails, leaving defaults intact", async () => {
const { client, lib } = await openSeeded(cacheDirectory);
try {
client.failCollections = true;
// Two concurrent fresh reads both reject, and share one failed
// round-trip rather than each making its own.
const collectionsBefore = client.collectionsSinceTimes.length;
const first = lib.fresh();
const second = lib.fresh();
await expect(first).rejects.toThrow(/network down/);
await expect(second).rejects.toThrow(/network down/);
expect(client.collectionsSinceTimes.length).toBe(
collectionsBefore + 1,
);
// The default reads never rejected: they still serve the last good
// copy, and the failure surfaced through status().
expect(lib.photos.byID({ fileID: 1001 })?.fileID).toBe(1001);
expect(lib.status().lastError).toMatch(/network down/);
// Recovery: once the server answers, a fresh read resolves current.
client.failCollections = false;
client.collectionsQueue.push({
collections: [collection(2, 300)],
deleted: [],
cursor: 300,
});
const reads = await lib.fresh();
expect(reads.albums.byID({ collectionID: 2 })?.collectionID).toBe(
2,
);
expect(lib.status().lastError).toBeUndefined();
} finally {
await lib.close();
}
});
});
describe("Library.freshCollections", () => {
let dir: string;
let cacheDirectory: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), "quak-fresh-collections-"));
cacheDirectory = join(dir, "cache");
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
// A refresh from a stored cursor past 0 fetches a collection's files only
// when the collection is new or its updationTime has advanced past the
// stored copy's. So does `refreshCollection()`. Neither would fetch the
// files of a collection stored here without them.
it("asks for the collection list alone and stores nothing", async () => {
const client = new MockClient();
const page = {
collections: [collection(2, 100), collection(1, 110)],
deleted: [],
cursor: 110,
};
client.collectionsQueue.push(page, page);
client.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
client.filesFor(2, {
files: [file(2001, 2, 95)],
deleted: [],
cursor: 95,
});
const lib = await Library.open({
client,
cacheDirectory,
refreshOnOpen: false,
refreshIntervalSeconds: SLOW_INTERVAL,
});
try {
// One request, the whole list in the server's order, and no
// collection's files.
const collections = await lib.freshCollections();
expect(collections.map((c) => c.id)).toEqual([2, 1]);
expect(collections.map((c) => c.name)).toEqual([
"album-2",
"album-1",
]);
expect(client.collectionsSinceTimes).toEqual([0]);
expect(client.filesCalls).toEqual([]);
expect(lib.listCollections()).toEqual([]);
// The next refresh still asks from the beginning and fetches the
// files of both collections.
const reads = await lib.fresh();
expect(client.collectionsSinceTimes).toEqual([0, 0]);
expect(reads.photos.byID({ fileID: 1001 })?.fileID).toBe(1001);
expect(reads.photos.byID({ fileID: 2001 })?.fileID).toBe(2001);
} finally {
await lib.close();
}
});
});
describe("Library.refreshCollection", () => {
let dir: string;
let cacheDirectory: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), "quak-refresh-collection-"));
cacheDirectory = join(dir, "cache");
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
// Collections 1 and 2 are stored at updationTime 100, with a stored cursor
// of 100, and then both change on the server.
it("fetches the files of one changed collection and leaves the others to the next refresh", async () => {
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 100), collection(2, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
client.filesFor(2, {
files: [file(2001, 2, 90)],
deleted: [],
cursor: 90,
});
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: SLOW_INTERVAL,
});
try {
client.filesCalls.length = 0;
client.filesFor(1, {
files: [file(1002, 1, 110)],
deleted: [],
cursor: 110,
});
client.filesFor(2, {
files: [file(2002, 2, 120)],
deleted: [],
cursor: 120,
});
// Collection 1's files alone, since the stored copy.
expect(await lib.refreshCollection(collection(1, 110))).toBe(true);
expect(client.filesCalls).toEqual([
{ collectionID: 1, sinceTime: 100 },
]);
expect(lib.listFiles(1).map((f) => f.id)).toEqual([1001, 1002]);
expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001]);
// The stored copy is now current, so no request.
await lib.refreshCollection(collection(1, 110));
expect(client.filesCalls).toHaveLength(1);
// The next refresh asks from the stored cursor, 100, so it still
// sees collection 2 changed and fetches its files.
client.collectionsQueue.push({
collections: [collection(1, 110), collection(2, 120)],
deleted: [],
cursor: 120,
});
await lib.fresh();
expect(client.collectionsSinceTimes).toEqual([0, 100]);
expect(client.filesCalls).toEqual([
{ collectionID: 1, sinceTime: 100 },
{ collectionID: 2, sinceTime: 100 },
]);
expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001, 2002]);
} finally {
await lib.close();
}
});
// Collection 1 is stored at 100. `refreshCollection()` asks for it at 110,
// which adds file 1002. While that files request waits, a fresh read
// starts a refresh. The server tells that refresh that collection 1 was
// deleted at 130.
it("holds off a refresh that starts during the files request, which then deletes the collection", async () => {
const { client, lib } = await openSeeded(cacheDirectory);
const files = holdNextFilesRequest(client);
try {
client.filesFor(1, {
files: [file(1002, 1, 110)],
deleted: [],
cursor: 110,
});
client.collectionsQueue.push({
collections: [],
deleted: [1],
cursor: 130,
});
const refreshing = lib.refreshCollection(collection(1, 110));
await files.started;
const reading = lib.fresh();
// The refresh has not sent its collections request yet.
await new Promise((r) => setImmediate(r));
expect(client.collectionsSinceTimes).toEqual([0]);
files.release();
await refreshing;
await reading;
expect(client.collectionsSinceTimes).toEqual([0, 100]);
expect(lib.getCollection(1)).toBeUndefined();
expect(lib.listFiles(1)).toEqual([]);
} finally {
files.release();
await lib.close();
}
});
// `freshCollections()` lists collection 1 at 110. Before
// `refreshCollection()` is called, a refresh learns that it was deleted at
// 130.
it("stores nothing for a collection that a refresh deleted before the call", async () => {
const { client, lib } = await openSeeded(cacheDirectory);
try {
client.collectionsQueue.push(
{ collections: [collection(1, 110)], deleted: [], cursor: 110 },
{ collections: [], deleted: [1], cursor: 130 },
);
const [listed] = await lib.freshCollections();
await lib.fresh();
client.filesCalls.length = 0;
expect(await lib.refreshCollection(listed)).toBe(false);
expect(client.filesCalls).toEqual([]);
expect(lib.getCollection(1)).toBeUndefined();
} finally {
await lib.close();
}
});
// A refresh has stored collection 1. Collection 2 is new on the server
// since then. Only the next refresh may add it, after collection 1.
it("stores nothing for a collection that is new since the last refresh", async () => {
const { client, lib } = await openSeeded(cacheDirectory);
try {
client.filesCalls.length = 0;
expect(await lib.refreshCollection(collection(2, 110))).toBe(false);
expect(client.filesCalls).toEqual([]);
expect(lib.getCollection(2)).toBeUndefined();
} finally {
await lib.close();
}
});
// The caller may remove the cache directory once `close()` resolves, so
// the cache write must have landed by then.
it("close() waits for a refreshCollection() in flight", async () => {
const { client, lib } = await openSeeded(cacheDirectory);
const files = holdNextFilesRequest(client);
client.filesFor(1, {
files: [file(1002, 1, 110)],
deleted: [],
cursor: 110,
});
const refreshing = lib.refreshCollection(collection(1, 110));
await files.started;
let closed = false;
const closing = lib.close().then(() => {
closed = true;
});
await new Promise((r) => setImmediate(r));
expect(closed).toBe(false);
files.release();
await closing;
const saved = await MetadataStore.load(
join(cacheDirectory, "metadata.json"),
);
expect(saved.listFiles(1).map((f) => f.id)).toEqual([1001, 1002]);
await refreshing;
});
// `quak files 2` on an empty cache stores collection 2 alone. A library
// opened later on that cache must not serve it as the whole account. Its
// first refresh replaces what the cache holds, in the server's order.
it("open() awaits the first refresh on a cache that only refreshCollection() wrote", async () => {
const client = new MockClient();
client.filesFor(2, {
files: [file(2001, 2, 90)],
deleted: [],
cursor: 90,
});
const first = await Library.open({
client,
cacheDirectory,
refreshOnOpen: false,
refreshIntervalSeconds: SLOW_INTERVAL,
});
expect(await first.refreshCollection(collection(2, 100))).toBe(true);
await first.close();
client.collectionsQueue.push({
collections: [collection(1, 100), collection(2, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
client.filesFor(2, {
files: [file(2001, 2, 90)],
deleted: [],
cursor: 90,
});
client.filesCalls.length = 0;
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: SLOW_INTERVAL,
});
try {
expect(lib.status().lastRefreshAt).toBeDefined();
expect(lib.listCollections().map((c) => c.id)).toEqual([1, 2]);
// It fetches collection 2's files again, from the beginning.
expect(client.filesCalls).toEqual([
{ collectionID: 1, sinceTime: 0 },
{ collectionID: 2, sinceTime: 0 },
]);
expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001]);
} finally {
await lib.close();
}
});
});
test/library/library.test.ts
+845
View File
@@ -0,0 +1,845 @@
/**
* Tests for `Library.open()` and its transparent background refresh loop.
*
* The library keeps the account's server state in a `MetadataStore` (issue
* #41) and pulls changes with the resumable, tombstone-aware enumerators on
* `Client` (issue #38: `collectionsSince` / `filesSince`). `open()` loads the
* cache, does one refresh, then refreshes again every `refreshIntervalSeconds`
* on a background timer. The design (#36) forbids an exposed `sync()`, a
* `serverReachable` flag, a `lib.refresh()` method, and a "before each read"
* mode. The contracts exercised here:
*
* 1. Reads are answered from RAM. A read never calls the client.
* 2. `open()` does an initial refresh, then the interval keeps refreshing;
* each refresh resumes from the stored cursor and applies diffs + tombstones.
* 3. The cache is rewritten only when a refresh actually changes something.
* 4. A failed refresh is invisible to reads: the last good data stays, the
* failure surfaces via `onProgress` ("failed") and `status()`, and a later
* success clears the error. `open()` itself resolves even when the first
* refresh fails (offline start from cache).
* 5. `close()` stops the timer and is idempotent, and its promise resolves
* only once an in-flight refresh has written the cache file.
* 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id.
* 7. `open()` branches on the stored cursor. At 0, as on an empty cache, it
* awaits the first refresh. Past 0 it serves the cache's copy at once and
* refreshes in the background, so a slow or dead server never stalls
* opening.
* 8. A save failure that leaves RAM ahead of disk keeps `status().lastError`
* set and keeps retrying the write; a later empty refresh does not clear it.
* 9. With `refreshOnOpen: false`, `open()` starts no refresh at all, and a
* refresh reports through `onProgress` each collection whose files it
* fetched.
*
* The client is a mock: no crypto, no network. It serves scripted pages and
* records the `sinceTime` each call carried so cursor threading is provable.
*
* On an empty cache `open()` awaits the initial refresh (including its cache
* write), so state right after `open()` is deterministic; the tests that
* inspect post-`open()` state seed no cache and rely on that. Tests for an
* existing-cache open seed a store first and prove `open()` returns without
* waiting for the network. The interval tests then use real timers with a
* short interval and `vi.waitFor`: a fake clock cannot settle the real
* fsync-and-rename cache write, and empty diffs never write, so the eventual
* state is stable to poll for.
*
* A refresh changes RAM before it writes the cache file, so a polled state can
* be visible while that write is still running. Every test therefore awaits
* `close()`, which waits for the in-flight refresh, before `afterEach` removes
* the directory.
*/
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import envPaths from "env-paths";
import { Library, type RefreshEvent } from "../../src/library/index.js";
import { MetadataStore } from "../../src/library/store.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
const USER_ID = 42;
// Short enough that a couple of ticks pass within a test, long enough not to
// spin; interval tests poll for the eventual state rather than counting ticks.
const FAST_INTERVAL = 0.02;
const collection = (
id: number,
updationTime: number,
name = `album-${id}`,
): Collection => ({
id,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
name,
type: "album",
updationTime,
isShared: false,
});
const file = (
id: number,
collectionID: number,
updationTime: number,
): EnteFile => ({
id,
collectionID,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
metadata: {
title: `file-${id}.jpg`,
fileType: "image",
creationTime: updationTime,
modificationTime: updationTime,
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime,
});
/**
* A mock `Client`. `collectionsSince` shifts one page off `collectionsQueue`
* per call (an empty diff that advances nothing when the queue runs dry);
* `filesSince` shifts from a per-collection queue. `failCollections` makes the
* next and all further collection fetches throw, to simulate an offline server.
*/
class MockClient {
userID = USER_ID;
failCollections = false;
collectionsQueue: CollectionsPage[] = [];
filesByCollection = new Map<number, FilesPage[]>();
collectionsSinceTimes: number[] = [];
filesCalls: { collectionID: number; sinceTime: number }[] = [];
whoami(): { email: string; userID: number } {
return { email: "user@example.com", userID: this.userID };
}
async collectionsSince(args: {
sinceTime: number;
}): Promise<CollectionsPage> {
this.collectionsSinceTimes.push(args.sinceTime);
if (this.failCollections) throw new Error("network down");
return (
this.collectionsQueue.shift() ?? {
collections: [],
deleted: [],
cursor: args.sinceTime,
}
);
}
async filesSince(args: {
collectionID: number;
collectionKey: Uint8Array;
sinceTime: number;
}): Promise<FilesPage> {
this.filesCalls.push({
collectionID: args.collectionID,
sinceTime: args.sinceTime,
});
const queue = this.filesByCollection.get(args.collectionID);
return (
queue?.shift() ?? {
files: [],
deleted: [],
cursor: args.sinceTime,
}
);
}
filesFor(collectionID: number, ...pages: FilesPage[]): void {
this.filesByCollection.set(collectionID, pages);
}
}
describe("Library.open and background refresh", () => {
let dir: string;
let cacheDirectory: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), "quak-library-"));
cacheDirectory = join(dir, "cache");
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
it("does an initial refresh and answers reads from the cache", async () => {
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90), file(1002, 1, 95)],
deleted: [],
cursor: 95,
});
const lib = await Library.open({ client, cacheDirectory });
try {
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
expect(lib.listFiles(1).map((f) => f.id)).toEqual([1001, 1002]);
expect(lib.getFile(1, 1001)?.metadata.title).toBe("file-1001.jpg");
const status = lib.status();
expect(status.userID).toBe(USER_ID);
expect(status.collections).toBe(1);
expect(status.files).toBe(2);
expect(status.lastRefreshAt).toBeGreaterThan(0);
expect(status.lastError).toBeUndefined();
// The initial refresh persisted the cache to disk.
const reloaded = await MetadataStore.load(
join(cacheDirectory, "metadata.json"),
);
expect(reloaded.getFile(1, 1001)?.id).toBe(1001);
expect(reloaded.collectionsSinceTime).toBe(100);
} finally {
await lib.close();
}
});
it("reads never call the client", async () => {
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
const lib = await Library.open({ client, cacheDirectory });
try {
const collectionCalls = client.collectionsSinceTimes.length;
const fileCalls = client.filesCalls.length;
lib.listCollections();
lib.getCollection(1);
lib.listFiles(1);
lib.getFile(1, 1001);
lib.status();
expect(client.collectionsSinceTimes.length).toBe(collectionCalls);
expect(client.filesCalls.length).toBe(fileCalls);
} finally {
await lib.close();
}
});
it("resumes each refresh from the stored cursor", async () => {
// Seed a cache with a cursor and a collection, as a prior run left it.
const path = join(cacheDirectory, "metadata.json");
const seed = await MetadataStore.load(path);
seed.userID = USER_ID;
seed.collectionsSinceTime = 500;
seed.putCollection(collection(1, 400));
seed.putFile(file(1001, 1, 400));
await seed.save();
const client = new MockClient();
// The collection's updationTime advances (400 -> 600), so its files are
// re-enumerated from the collection's stored updationTime (400).
client.collectionsQueue.push({
collections: [collection(1, 600)],
deleted: [],
cursor: 600,
});
client.filesFor(1, {
files: [file(1002, 1, 550)],
deleted: [],
cursor: 550,
});
// Opening from an existing cache serves the seeded copy at once and
// refreshes in the background, so the refresh's effects are polled for.
const lib = await Library.open({ client, cacheDirectory });
try {
await vi.waitFor(
() => {
// Collections resumed from the stored cursor, and files were
// re-enumerated from the stored collection updationTime.
expect(client.collectionsSinceTimes[0]).toBe(500);
expect(client.filesCalls).toEqual([
{ collectionID: 1, sinceTime: 400 },
]);
expect(lib.listFiles(1).map((f) => f.id)).toEqual([
1001, 1002,
]);
},
{ timeout: 2000, interval: 5 },
);
} finally {
await lib.close();
}
});
it("does not re-enumerate a collection whose updationTime did not advance", async () => {
const path = join(cacheDirectory, "metadata.json");
const seed = await MetadataStore.load(path);
seed.userID = USER_ID;
seed.collectionsSinceTime = 100;
seed.putCollection(collection(1, 400));
await seed.save();
const client = new MockClient();
// The collection comes back in the diff (its metadata changed) but at
// the same updationTime, so its files must not be re-fetched.
client.collectionsQueue.push({
collections: [collection(1, 400, "renamed")],
deleted: [],
cursor: 400,
});
// Existing cache: the rename lands via the background refresh.
const lib = await Library.open({ client, cacheDirectory });
try {
await vi.waitFor(
() => expect(lib.getCollection(1)?.name).toBe("renamed"),
{ timeout: 2000, interval: 5 },
);
// The collection's updationTime did not advance, so its files were
// never re-fetched.
expect(client.filesCalls).toEqual([]);
} finally {
await lib.close();
}
});
it("applies diffs and tombstones on the interval", async () => {
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 100), collection(2, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
client.filesFor(2, {
files: [file(2001, 2, 90)],
deleted: [],
cursor: 90,
});
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: FAST_INTERVAL,
});
try {
expect(lib.listCollections().map((c) => c.id)).toEqual([1, 2]);
expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001]);
// Next refresh: collection 2 is tombstoned; collection 1 gains a
// file and loses its old one.
client.filesFor(1, {
files: [file(1002, 1, 190)],
deleted: [1001],
cursor: 190,
});
client.collectionsQueue.push({
collections: [collection(1, 200)],
deleted: [2],
cursor: 200,
});
await vi.waitFor(
() => {
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
expect(lib.listFiles(1).map((f) => f.id)).toEqual([1002]);
// Collection 2's files went with it.
expect(lib.listFiles(2)).toEqual([]);
},
{ timeout: 2000, interval: 5 },
);
} finally {
await lib.close();
}
});
it("rewrites the cache only when a refresh changes something", async () => {
const saveSpy = vi.spyOn(MetadataStore.prototype, "save");
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: FAST_INTERVAL,
});
try {
// The initial refresh changed everything, so it saved once.
expect(saveSpy).toHaveBeenCalledTimes(1);
// Several empty-diff ticks pass; none of them may rewrite the file.
await new Promise((r) => setTimeout(r, FAST_INTERVAL * 1000 * 4));
expect(saveSpy).toHaveBeenCalledTimes(1);
// A real change triggers exactly one more rewrite; later empty ticks
// still do not, so the count settles at two.
client.collectionsQueue.push({
collections: [collection(2, 200)],
deleted: [],
cursor: 200,
});
await vi.waitFor(() => expect(saveSpy).toHaveBeenCalledTimes(2), {
timeout: 2000,
interval: 5,
});
await new Promise((r) => setTimeout(r, FAST_INTERVAL * 1000 * 4));
expect(saveSpy).toHaveBeenCalledTimes(2);
} finally {
await lib.close();
saveSpy.mockRestore();
}
});
it("keeps a failed refresh invisible to reads and recovers later", async () => {
const events: RefreshEvent[] = [];
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: FAST_INTERVAL,
onProgress: (e) => events.push(e),
});
try {
expect(lib.listFiles(1).map((f) => f.id)).toEqual([1001]);
// The server goes away; refreshes now fail.
client.failCollections = true;
await vi.waitFor(
() => expect(lib.status().lastError).toMatch(/network down/),
{ timeout: 2000, interval: 5 },
);
// Reads still see the last good data; the failure was reported.
expect(lib.listFiles(1).map((f) => f.id)).toEqual([1001]);
expect(
events.some(
(e) => e.operation === "refresh" && e.status === "failed",
),
).toBe(true);
// Recovery: a later refresh succeeds and clears the error.
client.failCollections = false;
client.collectionsQueue.push({
collections: [collection(2, 300)],
deleted: [],
cursor: 300,
});
await vi.waitFor(
() => {
expect(lib.status().lastError).toBeUndefined();
expect(lib.listCollections().map((c) => c.id)).toEqual([
1, 2,
]);
},
{ timeout: 2000, interval: 5 },
);
} finally {
await lib.close();
}
});
it("resolves open() even when the first refresh fails", async () => {
const client = new MockClient();
client.failCollections = true;
const events: RefreshEvent[] = [];
const lib = await Library.open({
client,
cacheDirectory,
onProgress: (e) => events.push(e),
});
try {
// Nothing was cached and the server is unreachable: reads are empty,
// but the library opened and the failure is on record.
expect(lib.listCollections()).toEqual([]);
expect(lib.status().lastError).toMatch(/network down/);
expect(lib.status().lastRefreshAt).toBeUndefined();
expect(
events.some(
(e) => e.operation === "refresh" && e.status === "failed",
),
).toBe(true);
} finally {
await lib.close();
}
});
it("opens from an existing cache without waiting for the first refresh", async () => {
// Seed a cache as a prior run left it.
const path = join(cacheDirectory, "metadata.json");
const seed = await MetadataStore.load(path);
seed.userID = USER_ID;
seed.collectionsSinceTime = 500;
seed.putCollection(collection(1, 400));
seed.putFile(file(1001, 1, 400));
await seed.save();
// The server does not answer this run's first refresh until the test
// is done with it.
let answerFirstFetch: (page: CollectionsPage) => void = () => {};
const client = new MockClient();
client.collectionsSince = () =>
new Promise<CollectionsPage>((resolve) => {
answerFirstFetch = resolve;
});
// open() must resolve from the cache without blocking on the network,
// and reads must serve the seeded copy.
const lib = await Library.open({ client, cacheDirectory });
try {
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
expect(lib.listFiles(1).map((f) => f.id)).toEqual([1001]);
// The first refresh is still outstanding: nothing has completed or
// failed yet.
expect(lib.status().lastRefreshAt).toBeUndefined();
expect(lib.status().lastError).toBeUndefined();
} finally {
// close() waits for the outstanding refresh, so let it finish.
answerFirstFetch({ collections: [], deleted: [], cursor: 500 });
await lib.close();
}
});
it("awaits the first refresh on a first run with an empty cache", async () => {
// No cache on disk: open() must not resolve until the first fetch does,
// so it never hands back an empty library it could have filled.
let releaseFirstFetch: (page: CollectionsPage) => void = () => {};
const gate = new Promise<CollectionsPage>((resolve) => {
releaseFirstFetch = resolve;
});
const client = new MockClient();
client.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
client.collectionsSince = async (args: { sinceTime: number }) => {
client.collectionsSinceTimes.push(args.sinceTime);
return gate;
};
let opened = false;
const openPromise = Library.open({ client, cacheDirectory }).then(
(l) => {
opened = true;
return l;
},
);
// While the first fetch is outstanding, open() has not resolved.
await new Promise((r) => setTimeout(r, 20));
expect(opened).toBe(false);
// Completing the fetch lets open() resolve with the data in place.
releaseFirstFetch({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
const lib = await openPromise;
try {
expect(opened).toBe(true);
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
expect(lib.listFiles(1).map((f) => f.id)).toEqual([1001]);
expect(lib.status().lastRefreshAt).toBeGreaterThan(0);
} finally {
await lib.close();
}
});
// A caller that asks for each refresh it needs, as the CLI's read commands
// do, opens without one: on an empty cache the refresh `open()` would
// await fetches the files of every collection, which `quak collections`
// never needs (issue #210).
it("starts no refresh as it opens when refreshOnOpen is false", async () => {
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
const lib = await Library.open({
client,
cacheDirectory,
refreshOnOpen: false,
refreshIntervalSeconds: Infinity,
});
try {
expect(client.collectionsSinceTimes).toEqual([]);
expect(lib.listCollections()).toEqual([]);
expect(lib.status().lastRefreshAt).toBeUndefined();
// The refresh comes when the caller asks for it.
await lib.fresh();
expect(client.collectionsSinceTimes).toEqual([0]);
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
} finally {
await lib.close();
}
});
// A refresh fetches the files of each collection that changed, one
// collection after another, and stores nothing until it has them all. On
// a large account that takes a while, so it reports each collection whose
// files it has fetched, and how many it will fetch in all (issue #210).
it("reports each collection whose files a refresh fetched", async () => {
const events: RefreshEvent[] = [];
const client = new MockClient();
client.collectionsQueue.push({
collections: [
collection(1, 100),
collection(2, 100),
collection(3, 100),
],
deleted: [],
cursor: 100,
});
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: Infinity,
onProgress: (e) => events.push(e),
});
try {
expect(events).toEqual([
{ operation: "refresh", status: "started" },
{
operation: "refresh",
status: "progress",
fetched: 1,
total: 3,
},
{
operation: "refresh",
status: "progress",
fetched: 2,
total: 3,
},
{
operation: "refresh",
status: "progress",
fetched: 3,
total: 3,
},
{ operation: "refresh", status: "done" },
]);
// Only collection 2 changed since, so the next refresh fetches
// its files alone and counts one.
events.length = 0;
client.collectionsQueue.push({
collections: [collection(2, 200)],
deleted: [],
cursor: 200,
});
await lib.fresh();
expect(events.filter((e) => e.status === "progress")).toEqual([
{
operation: "refresh",
status: "progress",
fetched: 1,
total: 1,
},
]);
} finally {
await lib.close();
}
});
it("keeps a save failure visible until a save actually succeeds", async () => {
const saveSpy = vi
.spyOn(MetadataStore.prototype, "save")
.mockRejectedValue(new Error("disk full"));
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: FAST_INTERVAL,
});
try {
// The initial refresh mutated RAM but its save failed, so the error
// is on record and no refresh has counted as successful.
expect(lib.status().lastError).toMatch(/disk full/);
expect(lib.status().lastRefreshAt).toBeUndefined();
// Empty-diff ticks pass. Each still retries the unsaved write and
// still fails, so the error never silently clears and the refresh
// clock never advances — RAM must not run ahead of disk unnoticed.
const savesBefore = saveSpy.mock.calls.length;
await new Promise((r) => setTimeout(r, FAST_INTERVAL * 1000 * 4));
expect(saveSpy.mock.calls.length).toBeGreaterThan(savesBefore);
expect(lib.status().lastError).toMatch(/disk full/);
expect(lib.status().lastRefreshAt).toBeUndefined();
// Once the disk recovers, the next tick persists the pending change
// and only then clears the error and advances the clock.
saveSpy.mockRestore();
await vi.waitFor(
() => {
expect(lib.status().lastError).toBeUndefined();
expect(lib.status().lastRefreshAt).toBeGreaterThan(0);
},
{ timeout: 2000, interval: 5 },
);
const reloaded = await MetadataStore.load(
join(cacheDirectory, "metadata.json"),
);
expect(reloaded.getFile(1, 1001)?.id).toBe(1001);
} finally {
await lib.close();
saveSpy.mockRestore();
}
});
it("close() stops the timer and is idempotent", async () => {
const client = new MockClient();
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: FAST_INTERVAL,
});
const callsAfterOpen = client.collectionsSinceTimes.length;
await lib.close();
await lib.close(); // second close must not throw
expect(lib.status().closed).toBe(true);
// No further refreshes fire once closed.
await new Promise((r) => setTimeout(r, FAST_INTERVAL * 1000 * 5));
expect(client.collectionsSinceTimes.length).toBe(callsAfterOpen);
});
it("close() resolves only after an in-flight refresh has written the cache", async () => {
const path = join(cacheDirectory, "metadata.json");
const seed = await MetadataStore.load(path);
seed.userID = USER_ID;
seed.collectionsSinceTime = 500;
seed.putCollection(collection(1, 400));
seed.putFile(file(1001, 1, 400));
await seed.save();
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 600)],
deleted: [],
cursor: 600,
});
client.filesFor(1, {
files: [file(1002, 1, 600)],
deleted: [],
cursor: 600,
});
// Hold the refresh's cache write until the test releases it.
const realSave = MetadataStore.prototype.save;
let releaseSave: () => void = () => {};
const saveHeld = new Promise<void>((resolve) => {
releaseSave = resolve;
});
const saveSpy = vi
.spyOn(MetadataStore.prototype, "save")
.mockImplementation(async function (this: MetadataStore) {
await saveHeld;
return realSave.call(this);
});
const lib = await Library.open({ client, cacheDirectory });
try {
await vi.waitFor(() => expect(saveSpy).toHaveBeenCalled(), {
timeout: 2000,
interval: 5,
});
let closed = false;
const closing = lib.close().then(() => {
closed = true;
});
await new Promise((r) => setTimeout(r, 50));
expect(closed).toBe(false);
releaseSave();
await closing;
const reloaded = await MetadataStore.load(path);
expect(reloaded.getFile(1, 1002)?.id).toBe(1002);
} finally {
releaseSave();
await lib.close();
saveSpy.mockRestore();
}
});
it("defaults cacheDirectory to the env-paths cache dir plus user id", async () => {
const xdg = join(dir, "xdg-cache");
const prev = process.env.XDG_CACHE_HOME;
process.env.XDG_CACHE_HOME = xdg;
try {
const client = new MockClient();
const lib = await Library.open({ client });
try {
const expected = join(
envPaths("quak", { suffix: "" }).cache,
String(USER_ID),
);
expect(lib.cacheDirectory).toBe(expected);
expect(lib.cacheDirectory.startsWith(xdg)).toBe(true);
expect(lib.cacheDirectory.endsWith(String(USER_ID))).toBe(true);
} finally {
await lib.close();
}
} finally {
if (prev === undefined) delete process.env.XDG_CACHE_HOME;
else process.env.XDG_CACHE_HOME = prev;
}
});
});
test/library/mldata.test.ts
+703
View File
@@ -0,0 +1,703 @@
/**
* Tests for the ML-data cache and its derived CLIP index (issue #49).
*
* Two layers are exercised:
*
* 1. `MLDataStore` on its own: storing one payload file per fileID (present
* means complete), building a `clip.f32` + `clip.json` index that reloads
* in a single read, rebuilding that index from the payloads when it is
* missing or disagrees with the files present, appending as new payloads
* arrive, overwriting a refetched file in place, and deciding what to
* (re)fetch as `updationTime` advances.
*
* 2. `Library` wiring: after each refresh the library fetches ML data through
* the metadata pool for every known file not yet cached, is incremental on
* later refreshes, and refetches a file whose `updationTime` advanced. A
* file whose ML data did not decode is reported with the reason, which
* stays listed until the next pass ends, and asked for again. Opening a
* cache directory written by another account starts empty, its ML data
* included (issue #104).
*
* Embedding values are chosen to be exactly representable as float32 so the
* round-trip through `clip.f32` compares equal.
*/
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import { existsSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { MLDataStore } from "../../src/library/mldata.js";
import { Library, type RefreshEvent } from "../../src/library/index.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js";
import type { MLData } from "../../src/mldata-fetch.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
// A payload shaped like Ente's: a CLIP embedding plus face data that only the
// on-disk payload carries (never the RAM index).
const payload = (embedding: number[]): MLData => ({
face: {
faces: [{ faceID: "f", detection: { box: { x: 0.5 } } }],
},
clip: { embedding },
});
describe("MLDataStore", () => {
let dir: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), "quak-mldata-"));
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
it("stores one payload file per fileID and builds a one-read index", async () => {
const store = await MLDataStore.open(dir);
const res = await store.storeFetched(
new Map([
[100, payload([0.5, 0.25, 0.75])],
[200, payload([1, -2, 0.5])],
]),
new Map([
[100, 10],
[200, 20],
]),
);
expect(res).toEqual({ stored: 2, indexed: 2 });
// One payload file per fileID, and the derived index files.
expect(existsSync(join(dir, "100.json"))).toBe(true);
expect(existsSync(join(dir, "200.json"))).toBe(true);
expect(existsSync(join(dir, "clip.f32"))).toBe(true);
expect(existsSync(join(dir, "clip.json"))).toBe(true);
// Reopening loads the index from disk in one read.
const reopened = await MLDataStore.open(dir);
const index = reopened.getIndex();
expect(index.fileIDs).toEqual([100, 200]);
expect(index.embeddingLength).toBe(3);
expect([...index.embeddings]).toEqual([0.5, 0.25, 0.75, 1, -2, 0.5]);
// The full payload (face boxes) is read back from disk on demand.
const full = await reopened.readPayload(100);
expect(full?.face).toBeDefined();
expect(await reopened.readPayload(999)).toBeUndefined();
});
it("rebuilds the index from payloads when it is missing", async () => {
const store = await MLDataStore.open(dir);
await store.storeFetched(
new Map([[100, payload([0.5, 0.25, 0.75])]]),
new Map([[100, 10]]),
);
// The derived index is lost but the payloads survive.
rmSync(join(dir, "clip.f32"));
rmSync(join(dir, "clip.json"));
const reopened = await MLDataStore.open(dir);
const index = reopened.getIndex();
expect(index.fileIDs).toEqual([100]);
expect([...index.embeddings]).toEqual([0.5, 0.25, 0.75]);
expect(existsSync(join(dir, "clip.f32"))).toBe(true);
});
it("rebuilds the index when it disagrees with the files present", async () => {
const store = await MLDataStore.open(dir);
await store.storeFetched(
new Map([
[100, payload([0.5, 0.25, 0.75])],
[200, payload([1, -2, 0.5])],
]),
new Map([
[100, 10],
[200, 20],
]),
);
// A payload disappears out from under the index (leaving it referencing
// a file no longer present); the index must be rebuilt from what is
// actually on disk.
rmSync(join(dir, "200.json"));
const reopened = await MLDataStore.open(dir);
expect(reopened.getIndex().fileIDs).toEqual([100]);
});
it("rebuilds the index when a payload on disk is missing from it", async () => {
const store = await MLDataStore.open(dir);
await store.storeFetched(
new Map([[100, payload([0.5, 0.25, 0.75])]]),
new Map([[100, 10]]),
);
// A crash between storeFetched renaming a payload into place and
// rewriting the index leaves the payload complete on disk but absent
// from clip.json. Write a second payload directly to reproduce that
// torn state without touching the index.
writeFileSync(
join(dir, "200.json"),
JSON.stringify(payload([1, -2, 0.5])),
);
// Reopening self-heals with no manual delete: the index is rebuilt from
// the payloads to include the orphaned embedding.
const reopened = await MLDataStore.open(dir);
const index = reopened.getIndex();
expect(index.fileIDs).toEqual([100, 200]);
expect([...index.embeddings]).toEqual([0.5, 0.25, 0.75, 1, -2, 0.5]);
});
it("appends new payloads and overwrites a refetched file in place", async () => {
const store = await MLDataStore.open(dir);
await store.storeFetched(
new Map([[100, payload([0.5, 0.25, 0.75])]]),
new Map([[100, 10]]),
);
// A later batch adds a new file: appended after the first.
await store.storeFetched(
new Map([[200, payload([1, -2, 0.5])]]),
new Map([[200, 20]]),
);
// Refetching 100 (its embedding changed) updates it in place, not a
// duplicate row.
await store.storeFetched(
new Map([[100, payload([9, 9, 9])]]),
new Map([[100, 30]]),
);
const index = store.getIndex();
expect(index.fileIDs).toEqual([100, 200]);
expect([...index.embeddings]).toEqual([9, 9, 9, 1, -2, 0.5]);
});
it("keeps a payload without a CLIP embedding out of the index", async () => {
const store = await MLDataStore.open(dir);
const res = await store.storeFetched(
new Map<number, MLData>([[100, { face: { faces: [] } }]]),
new Map([[100, 10]]),
);
expect(res.stored).toBe(1);
expect(res.indexed).toBe(0);
// The payload is still cached (present means complete).
expect(existsSync(join(dir, "100.json"))).toBe(true);
expect(store.getIndex().fileIDs).toEqual([]);
});
it("fetches only what is missing or has a newer updationTime", async () => {
const store = await MLDataStore.open(dir);
await store.storeFetched(
new Map([[100, payload([0.5, 0.25, 0.75])]]),
new Map([[100, 10]]),
);
// 100 is cached and current; 200 has never been fetched.
expect(
store.neededFor([
{ id: 100, updationTime: 10 },
{ id: 200, updationTime: 5 },
]),
).toEqual([200]);
// 100's updationTime advanced past what it was fetched at: refetch.
expect(store.neededFor([{ id: 100, updationTime: 15 }])).toEqual([100]);
// Nothing advanced: nothing to fetch.
expect(store.neededFor([{ id: 100, updationTime: 10 }])).toEqual([]);
});
it("survives a corrupt index without losing the payloads", async () => {
const store = await MLDataStore.open(dir);
await store.storeFetched(
new Map([[100, payload([0.5, 0.25, 0.75])]]),
new Map([[100, 10]]),
);
writeFileSync(join(dir, "clip.json"), "not json");
const reopened = await MLDataStore.open(dir);
expect(reopened.getIndex().fileIDs).toEqual([100]);
});
});
// --- Library wiring ---------------------------------------------------------
const USER_ID = 42;
const FAST_INTERVAL = 0.02;
const collection = (id: number, updationTime: number): Collection => ({
id,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
name: `album-${id}`,
type: "album",
updationTime,
isShared: false,
});
const file = (
id: number,
collectionID: number,
updationTime: number,
): EnteFile => ({
id,
collectionID,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
metadata: {
title: `file-${id}.jpg`,
fileType: "image",
creationTime: updationTime,
modificationTime: updationTime,
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime,
});
// A mock client that serves scripted collection/file pages and per-file ML
// payloads, recording every ML fetch request so incremental behaviour is
// provable. A file in `mlErrorByFile` has ML data that did not decode, for
// the reason given there.
class MLMockClient {
userID = USER_ID;
collectionsQueue: CollectionsPage[] = [];
filesByCollection = new Map<number, FilesPage[]>();
mlByFile = new Map<number, MLData>();
mlErrorByFile = new Map<number, Error>();
mlFetchCalls: number[][] = [];
whoami(): { email: string; userID: number } {
return { email: "user@example.com", userID: this.userID };
}
async collectionsSince(args: {
sinceTime: number;
}): Promise<CollectionsPage> {
return (
this.collectionsQueue.shift() ?? {
collections: [],
deleted: [],
cursor: args.sinceTime,
}
);
}
async filesSince(args: {
collectionID: number;
collectionKey: Uint8Array;
sinceTime: number;
}): Promise<FilesPage> {
const queue = this.filesByCollection.get(args.collectionID);
return (
queue?.shift() ?? {
files: [],
deleted: [],
cursor: args.sinceTime,
}
);
}
async fetchMLData(args: {
fileIDs: number[];
fileKeys: Map<number, Uint8Array>;
}): Promise<{
payloads: Map<number, MLData>;
errors: Map<number, unknown>;
}> {
this.mlFetchCalls.push([...args.fileIDs]);
const payloads = new Map<number, MLData>();
const errors = new Map<number, unknown>();
for (const id of args.fileIDs) {
const p = this.mlByFile.get(id);
if (p) payloads.set(id, p);
const error = this.mlErrorByFile.get(id);
if (error) errors.set(id, error);
}
return { payloads, errors };
}
filesFor(collectionID: number, ...pages: FilesPage[]): void {
this.filesByCollection.set(collectionID, pages);
}
}
describe("Library ML-data fetch on refresh", () => {
let dir: string;
let cacheDirectory: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), "quak-lib-mldata-"));
cacheDirectory = join(dir, "cache");
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
it("fetches, stores and indexes ML data for known files, then is incremental", async () => {
const client = new MLMockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90), file(1002, 1, 95)],
deleted: [],
cursor: 95,
});
client.mlByFile.set(1001, payload([0.5, 0.25, 0.75]));
client.mlByFile.set(1002, payload([1, -2, 0.5]));
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: FAST_INTERVAL,
});
try {
// Wait on `lastMLFetchAt`, set only once the pass has persisted the
// index and payloads — not on the in-RAM counts, which advance
// before `storeFetched` writes to disk, so the reopen below reads
// the committed index rather than racing the write.
await vi.waitFor(
() => {
expect(lib.status().lastMLFetchAt).toBeGreaterThan(0);
expect(lib.status().mlIndexed).toBe(2);
expect(lib.status().mlStored).toBe(2);
},
{ timeout: 2000, interval: 5 },
);
// Both files were fetched, in one batch.
expect(client.mlFetchCalls.flat().sort((a, b) => a - b)).toEqual([
1001, 1002,
]);
const callsAfterFirst = client.mlFetchCalls.length;
// The index is on disk and reloads to the same shape.
const reopened = await MLDataStore.open(
join(cacheDirectory, "mldata"),
);
expect(reopened.getIndex().fileIDs).toEqual([1001, 1002]);
// Later refreshes with nothing new must not refetch.
await new Promise((r) => setTimeout(r, FAST_INTERVAL * 1000 * 5));
expect(client.mlFetchCalls.length).toBe(callsAfterFirst);
} finally {
await lib.close();
}
});
it("refetches a file whose updationTime advanced", async () => {
const client = new MLMockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
client.mlByFile.set(1001, payload([0.5, 0.25, 0.75]));
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: FAST_INTERVAL,
});
try {
// Wait on `lastMLFetchAt`, set only after the first pass has
// persisted, not on `mlIndexed`, which is bumped in RAM before the
// write lands.
await vi.waitFor(
() => expect(lib.status().lastMLFetchAt).toBeGreaterThan(0),
{ timeout: 2000, interval: 5 },
);
const callsBefore = client.mlFetchCalls.length;
// The file changes on the server (updationTime advances) with a new
// embedding; the next refresh must refetch it.
client.mlByFile.set(1001, payload([9, 9, 9]));
client.collectionsQueue.push({
collections: [collection(1, 200)],
deleted: [],
cursor: 200,
});
client.filesFor(1, {
files: [file(1001, 1, 190)],
deleted: [],
cursor: 190,
});
// Poll the persisted index itself, not the fetch-call log: a call
// is recorded the instant the mock is entered, but `storeFetched`
// rewrites `clip.f32` only after it resolves, so an earlier reopen
// would read the pre-refetch vector. Reopening reads only committed
// (atomically renamed) files, so this sees the new embedding once —
// and only once — the store has written it.
await vi.waitFor(
async () => {
const reopened = await MLDataStore.open(
join(cacheDirectory, "mldata"),
);
expect([...reopened.getIndex().embeddings]).toEqual([
9, 9, 9,
]);
},
{ timeout: 2000, interval: 20 },
);
// The refetch really went back to the server for 1001.
expect(client.mlFetchCalls.length).toBeGreaterThan(callsBefore);
expect(client.mlFetchCalls.flat()).toContain(1001);
} finally {
await lib.close();
}
});
it("reports a file whose ML data did not decode through status() and onProgress, and asks for it again", async () => {
const client = new MLMockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90), file(1002, 1, 95)],
deleted: [],
cursor: 95,
});
client.mlByFile.set(1001, payload([0.5, 0.25, 0.75]));
client.mlErrorByFile.set(
1002,
new Error("secretstream chunk authentication failed"),
);
const events: RefreshEvent[] = [];
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: Infinity,
onProgress: (event) => events.push(event),
});
try {
await vi.waitFor(
() => expect(lib.status().lastMLFetchAt).toBeGreaterThan(0),
{ timeout: 2000, interval: 5 },
);
const reasons = [
{
fileID: 1002,
error: "secretstream chunk authentication failed",
},
];
expect(lib.status().mlDataErrors).toEqual(reasons);
expect(lib.status().mlStored).toBe(1);
expect(events.filter((e) => e.operation === "fetchMLData")).toEqual(
[
{ operation: "fetchMLData", status: "started" },
{
operation: "fetchMLData",
status: "done",
fetched: 1,
mlDataErrors: reasons,
},
],
);
// Nothing was cached for 1002, so the pass a refresh starts asks
// for it again, and once its ML data decodes the reason is gone.
client.mlErrorByFile.clear();
client.mlByFile.set(1002, payload([1, -2, 0.5]));
await lib.fresh();
await vi.waitFor(
() => {
expect(lib.status().mlStored).toBe(2);
expect(lib.status().mlDataErrors).toEqual([]);
},
{ timeout: 2000, interval: 5 },
);
expect(client.mlFetchCalls.at(-1)).toEqual([1002]);
} finally {
await lib.close();
}
});
it("still lists the last pass's reasons while the next pass runs", async () => {
const client = new MLMockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1002, 1, 95)],
deleted: [],
cursor: 95,
});
client.mlErrorByFile.set(
1002,
new Error("secretstream chunk authentication failed"),
);
const reasons = [
{ fileID: 1002, error: "secretstream chunk authentication failed" },
];
let release!: () => void;
const held = new Promise<void>((r) => (release = r));
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: Infinity,
});
try {
await vi.waitFor(
() => expect(lib.status().lastMLFetchAt).toBeGreaterThan(0),
{ timeout: 2000, interval: 5 },
);
expect(lib.status().mlDataErrors).toEqual(reasons);
// Hold the next ML data fetch open until the test releases it.
let fetchStarted!: () => void;
const started = new Promise<void>((r) => (fetchStarted = r));
const realFetch = client.fetchMLData.bind(client);
client.fetchMLData = async (args) => {
fetchStarted();
await held;
return realFetch(args);
};
client.mlErrorByFile.clear();
client.mlByFile.set(1002, payload([1, -2, 0.5]));
// The pass a refresh starts asks for 1002 again. While it is
// held, the reason from the pass before is still listed; once it
// ends, its own empty list takes the place of that one.
await lib.fresh();
await started;
expect(lib.status().mlDataErrors).toEqual(reasons);
release();
await vi.waitFor(
() => expect(lib.status().mlDataErrors).toEqual([]),
{ timeout: 2000, interval: 5 },
);
} finally {
release();
await lib.close();
}
});
it("close() resolves only after a running ML data fetch has stored its payloads", async () => {
const client = new MLMockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
client.mlByFile.set(1001, payload([0.5, 0.25, 0.75]));
// Hold the ML data fetch open until the test releases it.
let release!: () => void;
const held = new Promise<void>((r) => (release = r));
let fetchStarted!: () => void;
const started = new Promise<void>((r) => (fetchStarted = r));
const realFetch = client.fetchMLData.bind(client);
client.fetchMLData = async (args) => {
fetchStarted();
await held;
return realFetch(args);
};
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: 3600,
});
try {
await started;
let closed = false;
const closing = lib.close().then(() => {
closed = true;
});
await new Promise((r) => setTimeout(r, 20));
expect(closed).toBe(false);
release();
await closing;
expect(
existsSync(join(cacheDirectory, "mldata", "1001.json")),
).toBe(true);
} finally {
release();
await lib.close();
}
});
it("starts empty when the cache directory holds another account's cache", async () => {
// Account A fills the cache directory: metadata and ML data.
const clientA = new MLMockClient();
clientA.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
clientA.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
clientA.mlByFile.set(1001, payload([0.5, 0.25, 0.75]));
const libA = await Library.open({
client: clientA,
cacheDirectory,
refreshIntervalSeconds: 3600,
});
try {
await vi.waitFor(
() => expect(libA.status().lastMLFetchAt).toBeGreaterThan(0),
{ timeout: 2000, interval: 5 },
);
} finally {
await libA.close();
}
// Account B opens the same directory.
const clientB = new MLMockClient();
clientB.userID = USER_ID + 1;
const sinceTimes: number[] = [];
const realCollectionsSince = clientB.collectionsSince.bind(clientB);
clientB.collectionsSince = async (args) => {
sinceTimes.push(args.sinceTime);
return realCollectionsSince(args);
};
const libB = await Library.open({
client: clientB,
cacheDirectory,
refreshIntervalSeconds: 3600,
});
try {
expect(sinceTimes[0]).toBe(0);
expect(libB.status().userID).toBe(USER_ID + 1);
expect(libB.listCollections()).toEqual([]);
expect(libB.getFile(1, 1001)).toBeUndefined();
expect(await libB.mldata.forFile({ fileID: 1001 })).toBeUndefined();
expect(
libB.mldata.searchByEmbedding({ embedding: [0.5, 0.25, 0.75] }),
).toEqual([]);
expect(
existsSync(join(cacheDirectory, "mldata", "1001.json")),
).toBe(false);
} finally {
await libB.close();
}
});
});
test/library/mlsearch.test.ts
+108
View File
@@ -0,0 +1,108 @@
/**
* Tests for the content-similarity search surface over the CLIP index
* (issue #50).
*
* The surface is `lib.mldata`: `forFile` reads the full stored payload from
* disk, while `similar` and `searchByEmbedding` rank fileIDs by cosine
* similarity over the in-RAM `Float32Array` index alone (no disk, no network).
* The fixture uses axis-aligned vectors so the correct cosine ranking is
* obvious by inspection; cosine ignores magnitude, so `[2, 0, 0]` ranks above
* `[0.8, 0.6, 0]` for a `[1, 0, 0]` query.
*/
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { MLDataStore } from "../../src/library/mldata.js";
import { makeMLDataAPI, type MLDataAPI } from "../../src/library/mlsearch.js";
import type { MLData } from "../../src/mldata-fetch.js";
// A payload shaped like Ente's: a CLIP embedding plus face data that only the
// on-disk payload carries (never the RAM index).
const payload = (embedding: number[]): MLData => ({
face: { faces: [{ faceID: "f", detection: { box: { x: 0.5 } } }] },
clip: { embedding },
});
// A small fixture index. Directions are chosen so every cosine ranking below
// is unambiguous.
const fixture = (): Map<number, MLData> =>
new Map([
[10, payload([1, 0, 0])],
[20, payload([0.8, 0.6, 0])],
[30, payload([0, 1, 0])],
[40, payload([-1, 0, 0])],
[50, payload([2, 0, 0])],
]);
describe("lib.mldata content-similarity search", () => {
let dir: string;
let store: MLDataStore;
let api: MLDataAPI;
beforeEach(async () => {
dir = mkdtempSync(join(tmpdir(), "quak-mlsearch-"));
store = await MLDataStore.open(dir);
const updation = new Map([...fixture().keys()].map((id) => [id, 1]));
await store.storeFetched(fixture(), updation);
api = makeMLDataAPI(() => store);
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
it("forFile returns the whole stored payload, or undefined when uncached", async () => {
const full = await api.forFile({ fileID: 20 });
expect(full).toBeDefined();
// Face data lives only in the payload, never in the RAM index.
expect(full?.face).toBeDefined();
expect(full?.clip).toEqual({ embedding: [0.8, 0.6, 0] });
expect(await api.forFile({ fileID: 999 })).toBeUndefined();
});
it("similar ranks other files by cosine and excludes the query itself", () => {
// Query is file 10 = [1, 0, 0]. By cosine: 50 (1.0) > 20 (0.8) >
// 30 (0) > 40 (-1); 10 itself is left out.
const ranked = api.similar({ fileID: 10 });
expect(ranked.map((r) => r.fileID)).toEqual([50, 20, 30, 40]);
// Cosine ignores magnitude: [2,0,0] is a perfect match for [1,0,0].
expect(ranked[0]).toMatchObject({ fileID: 50 });
expect(ranked[0].score).toBeCloseTo(1, 5);
});
it("similar honours limit and returns [] for an unindexed file", () => {
expect(
api.similar({ fileID: 10, limit: 2 }).map((r) => r.fileID),
).toEqual([50, 20]);
expect(api.similar({ fileID: 999 })).toEqual([]);
});
it("searchByEmbedding ranks the index by cosine to the query vector", () => {
// Query [0, 1, 0]: 30 (1.0) > 20 (0.6) > {10, 40, 50} all 0, broken by
// ascending fileID.
const ranked = api.searchByEmbedding({ embedding: [0, 1, 0] });
expect(ranked.map((r) => r.fileID)).toEqual([30, 20, 10, 40, 50]);
expect(ranked[0].score).toBeCloseTo(1, 5);
expect(
api
.searchByEmbedding({ embedding: [0, 1, 0], limit: 2 })
.map((r) => r.fileID),
).toEqual([30, 20]);
});
it("searchByEmbedding returns [] for a wrong-length or zero query", () => {
expect(api.searchByEmbedding({ embedding: [1, 0] })).toEqual([]);
expect(api.searchByEmbedding({ embedding: [0, 0, 0] })).toEqual([]);
});
it("degrades to empty results when no ML store is present", async () => {
const none = makeMLDataAPI(() => undefined);
expect(await none.forFile({ fileID: 10 })).toBeUndefined();
expect(none.similar({ fileID: 10 })).toEqual([]);
expect(none.searchByEmbedding({ embedding: [1, 0, 0] })).toEqual([]);
});
});
test/library/pools.test.ts
+345
View File
@@ -0,0 +1,345 @@
/**
* Tests for `src/library/pools.ts` — the three bounded request pools (issue
* #45).
*
* A `BoundedPool` runs submitted tasks with a fixed concurrency cap. Within a
* pool, on-demand work runs before background work, and a task submitted with a
* key that a still-pending task already carries is not run twice — both callers
* share the one result. `RequestPools` bundles the three the design calls for
* (metadata 10, content 5, thumbnails 25); the pools are independent, so an
* idle pool never lends its slots to a busy one.
*
* ## How the tasks are controlled
*
* Every task here is a gate: it reports when it *starts* and then blocks until
* the test *releases* it, so the test decides exactly how many run at once and
* in what order they finish. A shared tracker counts how many tasks are running
* at any instant and records the peak, which is what the concurrency assertions
* read. No assertion is about wall-clock time.
*
* `drain()` returns a promise that settles on a macrotask, which flushes the
* microtask queue the pool schedules its starts on; the tests await it to let
* the pool react to a submission or a release before they inspect it.
*/
import { describe, it, expect } from "vitest";
import {
BoundedPool,
RequestPools,
DEFAULT_METADATA_CONCURRENCY,
DEFAULT_CONTENT_CONCURRENCY,
DEFAULT_THUMBNAIL_CONCURRENCY,
} from "../../src/library/pools.js";
// Settle on a macrotask so every microtask the pool queued has run.
const drain = (): Promise<void> =>
new Promise((resolve) => setTimeout(resolve, 0));
// A controllable task. `task` blocks until `release()` (resolve) or `fail()`
// (reject) is called; `startOrder` records the sequence in which tasks began.
interface Gate<T> {
task: () => Promise<T>;
release: (value: T) => void;
fail: (err: unknown) => void;
started: () => boolean;
runs: () => number;
}
// Tracks how many gated tasks are running concurrently across a whole test.
class Tracker {
active = 0;
peak = 0;
readonly starts: string[] = [];
gate<T>(label = ""): Gate<T> {
let settleResolve!: (value: T) => void;
let settleReject!: (err: unknown) => void;
const settled = new Promise<T>((resolve, reject) => {
settleResolve = resolve;
settleReject = reject;
});
let started = false;
let runs = 0;
const task = async (): Promise<T> => {
started = true;
runs++;
this.active++;
this.peak = Math.max(this.peak, this.active);
this.starts.push(label);
try {
return await settled;
} finally {
this.active--;
}
};
return {
task,
release: (value: T) => settleResolve(value),
fail: (err: unknown) => settleReject(err),
started: () => started,
runs: () => runs,
};
}
}
describe("BoundedPool concurrency cap", () => {
it("never runs more than `concurrency` tasks at once", async () => {
const pool = new BoundedPool(3);
const t = new Tracker();
const gates = Array.from({ length: 5 }, () => t.gate<void>());
const done = gates.map((g) => pool.run(g.task));
await drain();
// Three started, two queued behind the cap.
expect(t.active).toBe(3);
expect(gates.slice(0, 3).every((g) => g.started())).toBe(true);
expect(gates.slice(3).some((g) => g.started())).toBe(false);
// Finishing one admits exactly one more; the cap holds.
gates[0]!.release();
await drain();
expect(t.active).toBe(3);
expect(gates[3]!.started()).toBe(true);
expect(gates[4]!.started()).toBe(false);
for (const g of gates.slice(1)) g.release();
await Promise.all(done);
expect(t.peak).toBe(3);
});
it("rejects a non-positive or non-integer concurrency", () => {
expect(() => new BoundedPool(0)).toThrow(RangeError);
expect(() => new BoundedPool(-1)).toThrow(RangeError);
expect(() => new BoundedPool(2.5)).toThrow(RangeError);
});
});
describe("BoundedPool priority ordering", () => {
it("runs on-demand work before background, FIFO within a priority", async () => {
const pool = new BoundedPool(1);
const t = new Tracker();
const a = t.gate<void>("a");
const b = t.gate<void>("b");
const c = t.gate<void>("c");
const d = t.gate<void>("d");
// `a` takes the only slot; the rest queue.
void pool.run(a.task, { priority: "background" });
await drain();
void pool.run(b.task, { priority: "background" });
void pool.run(c.task, { priority: "on-demand" });
void pool.run(d.task, { priority: "background" });
await drain();
expect(t.starts).toEqual(["a"]);
// The on-demand `c` jumps ahead of the earlier-queued background `b`.
a.release();
await drain();
expect(t.starts).toEqual(["a", "c"]);
// Then background work drains in submission order: `b` before `d`.
c.release();
await drain();
expect(t.starts).toEqual(["a", "c", "b"]);
b.release();
await drain();
expect(t.starts).toEqual(["a", "c", "b", "d"]);
d.release();
});
it("defaults to background priority", async () => {
const pool = new BoundedPool(1);
const t = new Tracker();
const a = t.gate<void>("a");
const plain = t.gate<void>("plain");
const urgent = t.gate<void>("urgent");
void pool.run(a.task);
await drain();
void pool.run(plain.task); // no options -> background
void pool.run(urgent.task, { priority: "on-demand" });
await drain();
a.release();
await drain();
expect(t.starts).toEqual(["a", "urgent"]);
urgent.release();
plain.release();
});
});
describe("BoundedPool in-flight dedup", () => {
it("fetches a key once and hands both callers the same result", async () => {
const pool = new BoundedPool(5);
const t = new Tracker();
const g = t.gate<number>();
const first = pool.run(g.task, { key: 7 });
const second = pool.run(g.task, { key: 7 });
await drain();
expect(g.runs()).toBe(1);
expect(first).toBe(second);
g.release(99);
expect(await first).toBe(99);
expect(await second).toBe(99);
});
it("dedups only while in flight; a settled key runs again", async () => {
const pool = new BoundedPool(5);
const t = new Tracker();
const g1 = t.gate<number>();
const first = pool.run(g1.task, { key: 7 });
await drain();
g1.release(1);
expect(await first).toBe(1);
// The key is free again once its task settled.
const g2 = t.gate<number>();
const third = pool.run(g2.task, { key: 7 });
await drain();
expect(g2.started()).toBe(true);
g2.release(2);
expect(await third).toBe(2);
});
it("propagates a rejection to every deduped caller", async () => {
const pool = new BoundedPool(5);
const t = new Tracker();
const g = t.gate<number>();
const first = pool.run(g.task, { key: 7 });
const second = pool.run(g.task, { key: 7 });
await drain();
const boom = new Error("boom");
g.fail(boom);
await expect(first).rejects.toBe(boom);
await expect(second).rejects.toBe(boom);
// A failed key is also freed, so it may be retried by a fresh submit.
const g2 = t.gate<number>();
const retry = pool.run(g2.task, { key: 7 });
await drain();
expect(g2.started()).toBe(true);
g2.release(5);
expect(await retry).toBe(5);
});
});
describe("BoundedPool slot lifetime", () => {
it("holds one slot for a task's whole lifetime, retries included", async () => {
const pool = new BoundedPool(1);
const t = new Tracker();
// A task that internally makes two attempts before succeeding — the
// shape of a retrying request. It must occupy exactly one slot for the
// whole of that, so no other task may start until it finally settles.
const attempt1 = t.gate<void>("attempt1");
const attempt2 = t.gate<void>("attempt2");
const retrying = async (): Promise<void> => {
try {
await attempt1.task();
} catch {
await attempt2.task();
}
};
const other = t.gate<void>("other");
const running = pool.run(retrying);
await drain();
void pool.run(other.task);
await drain();
// First attempt is in flight and holds the only slot.
expect(t.starts).toEqual(["attempt1"]);
expect(other.started()).toBe(false);
// The retry is still the same task in the same slot; `other` waits.
attempt1.fail(new Error("transient"));
await drain();
expect(t.starts).toEqual(["attempt1", "attempt2"]);
expect(other.started()).toBe(false);
// Only when the whole task settles does the slot free.
attempt2.release();
await running;
await drain();
expect(other.started()).toBe(true);
other.release();
});
});
describe("RequestPools", () => {
it("exposes three pools at the design's default caps", () => {
expect(DEFAULT_METADATA_CONCURRENCY).toBe(10);
expect(DEFAULT_CONTENT_CONCURRENCY).toBe(5);
expect(DEFAULT_THUMBNAIL_CONCURRENCY).toBe(25);
const pools = new RequestPools();
expect(pools.metadata.concurrency).toBe(10);
expect(pools.content.concurrency).toBe(5);
expect(pools.thumbnails.concurrency).toBe(25);
});
it("takes overridden caps", () => {
const pools = new RequestPools({
metadataConcurrency: 1,
contentConcurrency: 2,
thumbnailConcurrency: 3,
});
expect(pools.metadata.concurrency).toBe(1);
expect(pools.content.concurrency).toBe(2);
expect(pools.thumbnails.concurrency).toBe(3);
});
it("keeps pools independent: an idle pool lends no slots", async () => {
const pools = new RequestPools({ contentConcurrency: 1 });
const t = new Tracker();
const c1 = t.gate<void>();
const c2 = t.gate<void>();
const c3 = t.gate<void>();
// The content pool is capped at 1. The thumbnail pool sits idle with 25
// free slots — none of which may be borrowed to run a second content
// task.
void pools.content.run(c1.task);
void pools.content.run(c2.task);
void pools.content.run(c3.task);
await drain();
expect(t.active).toBe(1);
c1.release();
await drain();
expect(t.active).toBe(1);
c2.release();
await drain();
expect(t.active).toBe(1);
c3.release();
await drain();
expect(t.peak).toBe(1);
});
it("runs different pools concurrently", async () => {
const pools = new RequestPools({
metadataConcurrency: 1,
contentConcurrency: 1,
});
const t = new Tracker();
const m = t.gate<void>();
const c = t.gate<void>();
void pools.metadata.run(m.task);
void pools.content.run(c.task);
await drain();
// One slot each, in two independent pools: both run at once.
expect(t.active).toBe(2);
m.release();
c.release();
});
});
test/library/precache.test.ts
+500
View File
@@ -0,0 +1,500 @@
/**
* The aggressive local precache (issue #48), driven from `Library.open`.
*
* Two background fills start with no caller input: every thumbnail in the
* account newest first, and the originals of the pinned set (the favorites
* album then the latest `precacheOriginalsDays` window ending at the newest
* file). Both run through the shared pools at background priority, so on-demand
* work always preempts them; both report through `status()`. The pinned set is
* the eviction predicate (#47), so a pinned original is never evicted and a
* file that leaves the set becomes an ordinary, evictable original.
*
* The unit tests drive `Precache` against a fake cache that records what it was
* asked to fetch (order and kind) with no pool or network; the integration
* tests drive the real wiring through `Library.open` with a stub content
* source, and the eviction test drives the real `ContentCache`.
*/
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { mkdtempSync, rmSync, existsSync, utimesSync } from "node:fs";
import { writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { Precache, type PrecacheCache } from "../../src/library/precache.js";
import {
deriveRecords,
type DerivedRecords,
} from "../../src/library/records.js";
import {
ContentCache,
type ContentSource,
type EnsureResult,
type StatFsFn,
} from "../../src/library/content.js";
import { RequestPools } from "../../src/library/pools.js";
import { Library } from "../../src/library/index.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js";
const DAY_MICROS = 24 * 60 * 60 * 1000 * 1000;
const collection = (
id: number,
type: Collection["type"] = "album",
): Collection => ({
id,
ownerID: 1,
key: new Uint8Array([id & 0xff]),
name: `album-${id}`,
type,
updationTime: 1,
isShared: false,
});
// A file whose creationTime (microseconds) places it `daysAgo` days before a
// fixed reference instant, so the latest-week window is deterministic.
const REFERENCE_MICROS = 1_000 * DAY_MICROS;
const file = (id: number, collectionID: number, daysAgo: number): EnteFile => ({
id,
collectionID,
ownerID: 1,
key: new Uint8Array([id & 0xff]),
metadata: {
title: `file-${id}.jpg`,
fileType: "image",
creationTime: REFERENCE_MICROS - daysAgo * DAY_MICROS,
modificationTime: 0,
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime: 1,
});
const records = (
collections: Collection[],
files: EnteFile[],
): DerivedRecords => deriveRecords(collections, files);
// A fake cache: records every fetch (kind + order) and reports the files it has
// stored via `pathsFor`. `presentThumbs`/`presentOriginals` seed already-cached
// files so the precache skips them with a single lookup.
class FakeCache implements PrecacheCache {
readonly thumbFetched: number[] = [];
readonly originalFetched: number[] = [];
readonly presentThumbs = new Set<number>();
readonly presentOriginals = new Set<number>();
pathsFor(fileID: number): {
originalPath?: string;
thumbnailPath?: string;
} {
const out: { originalPath?: string; thumbnailPath?: string } = {};
if (this.presentThumbs.has(fileID))
out.thumbnailPath = `/thumbs/${fileID}`;
if (this.presentOriginals.has(fileID))
out.originalPath = `/originals/${fileID}`;
return out;
}
async ensureThumbnails(args: {
fileIDs: number[];
priority: "background";
}): Promise<EnsureResult[]> {
return args.fileIDs.map((fileID) => {
this.thumbFetched.push(fileID);
this.presentThumbs.add(fileID);
return { fileID, path: `/thumbs/${fileID}` };
});
}
async ensureOriginals(args: {
fileIDs: number[];
}): Promise<EnsureResult[]> {
return args.fileIDs.map((fileID) => {
this.originalFetched.push(fileID);
this.presentOriginals.add(fileID);
return { fileID, path: `/originals/${fileID}` };
});
}
}
// Resolve once a predicate holds, polling the microtask queue; fails fast
// rather than hanging the suite.
const until = async (predicate: () => boolean): Promise<void> => {
for (let i = 0; i < 1000; i++) {
if (predicate()) return;
await new Promise((r) => setTimeout(r, 1));
}
throw new Error("condition not met in time");
};
describe("Precache unit", () => {
it("precaches every thumbnail newest first, skipping present ones", async () => {
const cols = [collection(1)];
const files = [
file(1, 1, 0),
file(2, 1, 1),
file(3, 1, 2),
file(4, 1, 3),
];
const cache = new FakeCache();
cache.presentThumbs.add(3); // already on disk: skipped
const pre = new Precache({ originals: false });
pre.bind(cache);
pre.update(records(cols, files));
pre.start();
await until(() => cache.thumbFetched.length === 3);
// Newest first (file 1 is newest), file 3 skipped by a lookup.
expect(cache.thumbFetched).toEqual([1, 2, 4]);
expect(cache.originalFetched).toEqual([]);
});
it("pins favorites then the latest-week window and precaches their originals in that order", async () => {
const cols = [collection(1), collection(2, "favorites")];
// File 10 is an old favorite (30 days old); files 1..3 are within the
// 7-day window; file 4 is outside it.
const files = [
file(1, 1, 0),
file(2, 1, 2),
file(3, 1, 6),
file(4, 1, 20),
file(10, 2, 30), // favorite, old
];
const cache = new FakeCache();
const pre = new Precache({ originalsDays: 7 });
pre.bind(cache);
pre.update(records(cols, files));
pre.start();
await until(() => cache.originalFetched.length === 4);
// Favorite (10) first, then the window newest-first (1, 2, 3). File 4
// is outside the window and never pinned.
expect(cache.originalFetched).toEqual([10, 1, 2, 3]);
expect(pre.isPinned(10)).toBe(true);
expect(pre.isPinned(1)).toBe(true);
expect(pre.isPinned(4)).toBe(false);
});
it("drops a file from the pinned set when the window moves past it", () => {
const cols = [collection(1)];
const cache = new FakeCache();
const pre = new Precache({ originalsDays: 7 });
pre.bind(cache);
pre.update(records(cols, [file(1, 1, 0), file(2, 1, 3)]));
expect(pre.isPinned(2)).toBe(true);
// A newer file arrives; the window's newest end moves forward so the
// 3-day-old file 2 (now 13 days behind the newest) falls out.
pre.update(
records(cols, [file(3, 1, -10), file(1, 1, 0), file(2, 1, 3)]),
);
expect(pre.isPinned(3)).toBe(true);
expect(pre.isPinned(2)).toBe(false);
});
it("reports progress through status()", async () => {
const cols = [collection(1), collection(2, "favorites")];
const files = [file(1, 1, 0), file(2, 1, 1), file(10, 2, 0)];
const cache = new FakeCache();
const pre = new Precache({ originalsDays: 7 });
pre.bind(cache);
pre.update(records(cols, files));
const before = pre.status();
expect(before.thumbnailsTotal).toBe(3);
expect(before.thumbnailsCached).toBe(0);
expect(before.originalsPinned).toBe(3); // files 1, 2, 10 all in window
expect(before.originalsCached).toBe(0);
pre.start();
await until(
() =>
cache.thumbFetched.length === 3 &&
cache.originalFetched.length === 3,
);
const after = pre.status();
expect(after.thumbnailsCached).toBe(3);
expect(after.originalsCached).toBe(3);
});
it("honours the disable flags", async () => {
const cols = [collection(1)];
const files = [file(1, 1, 0)];
const cache = new FakeCache();
const pre = new Precache({ thumbnails: false, originals: false });
pre.bind(cache);
pre.update(records(cols, files));
pre.start();
await new Promise((r) => setTimeout(r, 20));
expect(cache.thumbFetched).toEqual([]);
expect(cache.originalFetched).toEqual([]);
expect(pre.isPinned(1)).toBe(false);
expect(pre.status().thumbnailsTotal).toBe(0);
});
});
// ---- Integration through the real ContentCache and Library ----
const enteFile = (id: number, collectionID: number): EnteFile =>
file(id, collectionID, 0);
describe("Precache eviction integration", () => {
let root: string;
beforeEach(() => {
root = mkdtempSync(join(tmpdir(), "quak-precache-evict-"));
});
afterEach(() => {
if (root && existsSync(root))
rmSync(root, { recursive: true, force: true });
});
it("never evicts a pinned original the precache put in place", async () => {
const cacheDir = join(root, "cache");
// File 1 is the favorites album's only file (pinned regardless of age);
// files 2 and 3 sit outside the latest-week window, so only file 1 is
// pinned. The by-id map serves the bytes for each fetch.
const cols = [collection(1), collection(2, "favorites")];
const files = [file(1, 2, 30), file(2, 1, 40), file(3, 1, 50)];
const byID = new Map<number, EnteFile>(files.map((f) => [f.id, f]));
const source: ContentSource = {
original: async ({ destination }) => {
await writeFile(destination, Buffer.alloc(10, 1));
return { bytesWritten: 10 };
},
thumbnail: async ({ destination }) => {
await writeFile(destination, Buffer.alloc(10, 1));
return { bytesWritten: 10 };
},
};
const statfs: StatFsFn = async () => ({
bsize: 1,
bavail: 1_000_000_000,
});
const pre = new Precache({ originalsDays: 7 });
const cache = new ContentCache({
pools: new RequestPools(),
source,
cacheDirectory: cacheDir,
downloadDirectory: join(root, "photos"),
getFile: (id) => byID.get(id),
statfs,
cacheOriginalsMaxBytes: 25, // holds two 10-byte originals
freeBelowBytes: 0,
isPinned: (id) => pre.isPinned(id),
});
pre.bind(cache);
pre.update(records(cols, files));
expect(pre.isPinned(1)).toBe(true);
expect(pre.isPinned(2)).toBe(false);
await cache.open();
// Fill three originals; the 25-byte cap forces an eviction on the
// third, and the pinned file 1 must survive it even though it is the
// least-recently-used.
await cache.original(1);
utimesSync(join(cacheDir, "originals", "1.jpg"), 1000, 1000); // oldest
await cache.original(2);
utimesSync(join(cacheDir, "originals", "2.jpg"), 2000, 2000);
await cache.original(3);
expect(existsSync(join(cacheDir, "originals", "1.jpg"))).toBe(true);
expect(existsSync(join(cacheDir, "originals", "2.jpg"))).toBe(false);
expect(existsSync(join(cacheDir, "originals", "3.jpg"))).toBe(true);
});
});
describe("Precache preemption", () => {
let root: string;
beforeEach(() => {
root = mkdtempSync(join(tmpdir(), "quak-precache-preempt-"));
});
afterEach(() => {
if (root && existsSync(root))
rmSync(root, { recursive: true, force: true });
});
it("lets an on-demand original preempt the background originals fill", async () => {
const byID = new Map<number, EnteFile>([
[1, enteFile(1, 1)],
[2, enteFile(2, 1)],
[3, enteFile(3, 1)],
]);
const finished: number[] = [];
let openGate!: () => void;
const gate = new Promise<void>((r) => (openGate = r));
let sawFirst!: () => void;
const firstStarted = new Promise<void>((r) => (sawFirst = r));
let started = 0;
const source: ContentSource = {
original: async ({ file: f, destination }) => {
if (++started === 1) sawFirst();
await gate;
await writeFile(destination, Buffer.alloc(10, 1));
finished.push(f.id);
return { bytesWritten: 10 };
},
thumbnail: async ({ destination }) => {
await writeFile(destination, Buffer.alloc(10, 1));
return { bytesWritten: 10 };
},
};
// One content slot, so file 1 holds it while 2 and 3 wait.
const cache = new ContentCache({
pools: new RequestPools({ contentConcurrency: 1 }),
source,
cacheDirectory: join(root, "cache"),
downloadDirectory: join(root, "photos"),
getFile: (id) => byID.get(id),
statfs: async () => ({ bsize: 1, bavail: 1_000_000_000 }),
freeBelowBytes: 0,
});
await cache.open();
const pA = cache.ensureOriginals({ fileIDs: [1] }); // background
await firstStarted; // file 1 now holds the only slot
const pB = cache.original(2); // on-demand, queued behind file 1
const pC = cache.ensureOriginals({ fileIDs: [3] }); // background, queued
await new Promise((r) => setTimeout(r, 5)); // let both enqueue
openGate();
await Promise.all([pA, pB, pC]);
// On-demand file 2 was served before the background file 3.
expect(finished).toEqual([1, 2, 3]);
});
});
describe("Precache through Library.open", () => {
let root: string;
beforeEach(() => {
root = mkdtempSync(join(tmpdir(), "quak-precache-lib-"));
});
afterEach(() => {
if (root && existsSync(root))
rmSync(root, { recursive: true, force: true });
});
class MockClient {
served = false;
whoami(): { email: string; userID: number } {
return { email: "u@example.com", userID: 7 };
}
async collectionsSince(): Promise<CollectionsPage> {
if (this.served) return { collections: [], deleted: [], cursor: 1 };
this.served = true;
return {
collections: [collection(1), collection(2, "favorites")],
deleted: [],
cursor: 1,
};
}
async filesSince(args: { collectionID: number }): Promise<FilesPage> {
const files =
args.collectionID === 1
? [enteFile(1, 1), enteFile(2, 1)]
: [enteFile(3, 2)];
return { files, deleted: [], cursor: 1 };
}
}
it("starts both precaches from open() and reports them in status()", async () => {
const source: ContentSource = {
original: async ({ destination }) => {
await writeFile(destination, Buffer.alloc(10, 1));
return { bytesWritten: 10 };
},
thumbnail: async ({ destination }) => {
await writeFile(destination, Buffer.alloc(10, 1));
return { bytesWritten: 10 };
},
};
// Each fill reports "done" once the cache has recorded its files. The
// source returning is not enough: the cache records a file only after
// it has checked it on disk.
const finished = new Set<string>();
let bothFinished!: () => void;
const precached = new Promise<void>((r) => (bothFinished = r));
const lib = await Library.open({
client: new MockClient(),
cacheDirectory: join(root, "cache"),
contentSource: source,
refreshIntervalSeconds: 3600,
onProgress: (e) => {
if (
e.status === "done" &&
(e.operation === "precacheThumbnails" ||
e.operation === "precacheOriginals")
) {
finished.add(e.operation);
if (finished.size === 2) bothFinished();
}
},
});
// Every file's thumbnail is precached; the favorite (file 3) and the
// week's files (1, 2) all have their originals precached.
await precached;
const status = lib.status();
expect(status.thumbnailsTotal).toBe(3);
expect(status.thumbnailsCached).toBe(3);
expect(status.originalsPinned).toBe(3);
expect(status.originalsCached).toBe(3);
await lib.close();
});
it.each(["thumbnail", "original"] as const)(
"close() resolves only after a running %s precache fetch has written its file",
async (kind) => {
// Only the fill under test runs, and each of its fetches waits
// until the test releases it.
let release!: () => void;
const held = new Promise<void>((r) => (release = r));
let fetchStarted!: (destination: string) => void;
const started = new Promise<string>((r) => (fetchStarted = r));
const fetch = async ({ destination }: { destination: string }) => {
fetchStarted(destination);
await held;
await writeFile(destination, Buffer.alloc(10, 1));
return { bytesWritten: 10 };
};
const unused = async () => {
throw new Error("this fill is turned off");
};
const source: ContentSource =
kind === "thumbnail"
? { original: unused, thumbnail: fetch }
: { original: fetch, thumbnail: unused };
const lib = await Library.open({
client: new MockClient(),
cacheDirectory: join(root, "cache"),
contentSource: source,
refreshIntervalSeconds: 3600,
precacheThumbnails: kind === "thumbnail",
precacheOriginals: kind === "original",
});
try {
const destination = await started;
let closed = false;
const closing = lib.close().then(() => {
closed = true;
});
await new Promise((r) => setTimeout(r, 20));
expect(closed).toBe(false);
release();
await closing;
expect(existsSync(destination)).toBe(true);
} finally {
release();
await lib.close();
}
},
);
});
test/library/read.test.ts
+568
View File
@@ -0,0 +1,568 @@
/**
* Tests for the in-process read surface (issue #44).
*
* Phase 1 (#43) projected the decrypted store into plain `AlbumRecord` /
* `PhotoRecord` values. This phase adds the read API a CLI or in-process script
* uses, all served from RAM with no network:
*
* - `lib.albums` — `list` / `byName` / `byID`, returning thin `Album` wrappers.
* - `lib.photos` — `byID` (a `Photo` wrapper) and `records` (plain records).
* - `lib.timeline.groups` — photos bucketed by local day / week / month.
*
* Every call takes a single named-argument object; there are no positional
* arguments. The wrapper classes are for in-process callers only (they hold
* object identity, not JSON); the plain records remain the IPC-safe surface.
* Content-fetch methods (`Photo.original` / `thumbnail`) are a later unit and
* deliberately absent here — this surface is read-only.
*
* The detailed cases drive the API factories directly over a hand-built
* projection (`deriveRecords`), which keeps them free of disk and timers. A
* final section opens a real `Library` to prove the namespaces are wired to the
* live store and that a read never touches the client.
*/
import { describe, it, expect, beforeAll, afterAll } from "vitest";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import {
deriveRecords,
type DerivedRecords,
} from "../../src/library/records.js";
import {
Album,
Photo,
makeAlbumsAPI,
makePhotosAPI,
makeTimelineAPI,
type SavePathLookup,
type TimelineGroup,
} from "../../src/library/read.js";
import { Library } from "../../src/library/index.js";
import { MetadataStore } from "../../src/library/store.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
const OWNER = 42;
// Ente stores times in microseconds; records expose milliseconds. These
// helpers keep the fixtures readable: `ms(...)` picks an epoch-millisecond
// instant, `micros(...)` is what the fixture stores so the derived record's
// `takenAt` comes back as the same millisecond value.
const ms = (epochMillis: number): number => epochMillis;
const micros = (epochMillis: number): number => epochMillis * 1000;
const collection = (
id: number,
opts: Partial<Collection> = {},
): Collection => ({
id,
ownerID: OWNER,
key: new Uint8Array([id & 0xff, 1, 2, 3]),
name: `album-${id}`,
type: "album",
updationTime: micros(1_700_000_000_000),
isShared: false,
...opts,
});
const file = (
id: number,
collectionID: number,
opts: Partial<EnteFile> & {
creationTime?: number;
title?: string;
fileType?: EnteFile["metadata"]["fileType"];
latitude?: number;
longitude?: number;
} = {},
): EnteFile => {
const { creationTime, title, fileType, latitude, longitude, ...rest } =
opts;
const metadata: EnteFile["metadata"] = {
title: title ?? `file-${id}.jpg`,
fileType: fileType ?? "image",
creationTime: creationTime ?? micros(1_700_000_000_000),
modificationTime: micros(1_700_000_000_000),
};
if (latitude !== undefined) metadata.latitude = latitude;
if (longitude !== undefined) metadata.longitude = longitude;
return {
id,
collectionID,
ownerID: OWNER,
key: new Uint8Array([id & 0xff, 9, 8, 7]),
metadata,
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime: micros(1_700_000_000_000),
...rest,
};
};
// Build the three API objects over one fixed projection, the way `Library`
// wires them over its live store. Save paths are covered in
// content-library.test.ts; these tests never ask for one.
const apis = (records: DerivedRecords) => {
const derive = () => records;
const saves: SavePathLookup = {
savePath: () => {
throw new Error("no save paths in these tests");
},
isLocal: () => false,
};
return {
albums: makeAlbumsAPI(derive, saves),
photos: makePhotosAPI(derive, saves),
timeline: makeTimelineAPI(derive),
};
};
// Every file id present across all timeline groups, in group-then-member order.
const allFileIDs = (groups: TimelineGroup[]): number[] =>
groups.flatMap((g) => g.fileIDs);
describe("lib.albums", () => {
it("lists albums as Album wrappers, newest updated first", () => {
const records = deriveRecords(
[
collection(1, { updationTime: micros(1_700_000_000_000) }),
collection(2, { updationTime: micros(1_705_000_000_000) }),
],
[file(10, 1), file(20, 2)],
);
const albums = apis(records).albums.list();
expect(albums.every((a) => a instanceof Album)).toBe(true);
// Collection 2 updated later, so it sorts ahead of collection 1.
expect(albums.map((a) => a.collectionID)).toEqual([2, 1]);
});
it("exposes album fields and its photos newest first", () => {
const records = deriveRecords(
[
collection(7, {
name: "Trip",
type: "favorites",
isShared: true,
}),
],
[
file(1, 7, { creationTime: micros(1_600_000_000_000) }),
file(2, 7, { creationTime: micros(1_800_000_000_000) }),
file(3, 7, { creationTime: micros(1_700_000_000_000) }),
],
);
const album = apis(records).albums.byID({ collectionID: 7 })!;
expect(album.name).toBe("Trip");
expect(album.type).toBe("favorites");
expect(album.isShared).toBe(true);
expect(album.fileIDs).toEqual([2, 3, 1]);
const photos = album.photos.list();
expect(photos.every((p) => p instanceof Photo)).toBe(true);
expect(photos.map((p) => p.fileID)).toEqual([2, 3, 1]);
// record() hands back the plain, JSON-safe projection.
expect("key" in album.record()).toBe(false);
});
it("finds an album by exact name and returns undefined when absent", () => {
const records = deriveRecords(
[
collection(1, { name: "Berlin" }),
collection(2, { name: "Paris" }),
],
[file(10, 1), file(20, 2)],
);
const { albums } = apis(records);
expect(albums.byName({ albumName: "Paris" })?.collectionID).toBe(2);
expect(albums.byName({ albumName: "paris" })).toBeUndefined();
expect(albums.byName({ albumName: "Nowhere" })).toBeUndefined();
});
it("returns undefined for an unknown collection id", () => {
const records = deriveRecords([collection(1)], [file(10, 1)]);
expect(
apis(records).albums.byID({ collectionID: 999 }),
).toBeUndefined();
});
});
describe("lib.photos", () => {
it("byID returns a Photo wrapper carrying the mapped fields", () => {
const records = deriveRecords(
[collection(1)],
[
file(1001, 1, {
title: "IMG.jpg",
fileType: "video",
creationTime: micros(1_699_000_000_000),
latitude: 52.52,
longitude: 13.405,
pubMagicMetadata: { caption: "at the lake" },
}),
],
);
const photo = apis(records).photos.byID({ fileID: 1001 })!;
expect(photo).toBeInstanceOf(Photo);
expect(photo.title).toBe("IMG.jpg");
expect(photo.fileType).toBe("video");
expect(photo.takenAt).toBe(ms(1_699_000_000_000));
expect(photo.caption).toBe("at the lake");
expect(photo.latitude).toBeCloseTo(52.52);
expect(photo.isArchived).toBe(false);
expect(photo.isHidden).toBe(false);
// The wrapper hands back the plain record, IPC-safe.
expect("key" in photo.record()).toBe(false);
});
it("byID exposes modifiedAt, hash and the year taken", () => {
// Mid-July, so the year is 2021 in every time zone.
const takenAt = Date.UTC(2021, 6, 15, 12);
const records = deriveRecords(
[collection(1)],
[
file(1001, 1, {
metadata: {
title: "IMG.jpg",
fileType: "image",
creationTime: micros(takenAt),
modificationTime: micros(1_700_000_123_456),
hash: "aGFzaA==",
},
}),
],
);
const photo = apis(records).photos.byID({ fileID: 1001 })!;
expect(photo.modifiedAt).toBe(ms(1_700_000_123_456));
expect(photo.hash).toBe("aGFzaA==");
expect(photo.year).toBe(2021);
});
it("byID returns undefined for an unknown file id", () => {
const records = deriveRecords([collection(1)], [file(1, 1)]);
expect(apis(records).photos.byID({ fileID: 999 })).toBeUndefined();
});
it("records() returns plain records in requested order, deduped, skipping unknowns", () => {
const records = deriveRecords(
[collection(1)],
[file(1, 1), file(2, 1), file(3, 1)],
);
const out = apis(records).photos.records({
fileIDs: [3, 1, 3, 999, 2],
});
// Requested order preserved; the repeated 3 appears once; 999 is dropped.
expect(out.map((r) => r.fileID)).toEqual([3, 1, 2]);
// Plain records, not wrappers, and JSON round-trips whole.
expect(out[0]).not.toBeInstanceOf(Photo);
expect(JSON.parse(JSON.stringify(out[0]))).toEqual(out[0]);
});
it("emits one record for a file even when it belongs to several albums", () => {
// File 1001 is a member of collections 1 and 2.
const records = deriveRecords(
[collection(1), collection(2)],
[file(1001, 1), file(1001, 2)],
);
const out = apis(records).photos.records({ fileIDs: [1001, 1001] });
expect(out).toHaveLength(1);
expect(out[0]!.albumIDs).toEqual([1, 2]);
});
});
describe("lib.timeline grouping", () => {
// Group keys and `startsAt` are computed in local time. Pinning the zone to
// UTC makes the expected values exact and lets the fixtures use `Date.UTC`.
const savedTZ = process.env.TZ;
beforeAll(() => {
process.env.TZ = "UTC";
});
afterAll(() => {
if (savedTZ === undefined) delete process.env.TZ;
else process.env.TZ = savedTZ;
});
it("buckets by local day, newest group and newest member first", () => {
const records = deriveRecords(
[collection(1)],
[
file(1, 1, { creationTime: micros(Date.UTC(2024, 0, 15, 9)) }),
file(2, 1, { creationTime: micros(Date.UTC(2024, 0, 15, 18)) }),
file(3, 1, { creationTime: micros(Date.UTC(2024, 0, 16, 12)) }),
file(4, 1, { creationTime: micros(Date.UTC(2024, 1, 1, 12)) }),
],
);
const groups = apis(records).timeline.groups({ groupBy: "day" });
expect(groups.map((g) => g.key)).toEqual([
"2024-02-01",
"2024-01-16",
"2024-01-15",
]);
// Group start is local midnight of the day.
expect(groups[2]!.startsAt).toBe(Date.UTC(2024, 0, 15));
// Within the 2024-01-15 group, the later photo (id 2) is first.
expect(groups[2]!.fileIDs).toEqual([2, 1]);
});
it("buckets by week with weeks starting on Monday", () => {
// 2024-01-15 is a Monday; the week runs through Sunday 2024-01-21.
const records = deriveRecords(
[collection(1)],
[
file(1, 1, { creationTime: micros(Date.UTC(2024, 0, 15, 12)) }), // Mon
file(2, 1, { creationTime: micros(Date.UTC(2024, 0, 17, 12)) }), // Wed
file(3, 1, { creationTime: micros(Date.UTC(2024, 0, 21, 12)) }), // Sun
file(4, 1, { creationTime: micros(Date.UTC(2024, 0, 22, 12)) }), // next Mon
],
);
const groups = apis(records).timeline.groups({ groupBy: "week" });
// ISO week keys: 2024-01-15 is in 2024-W03, the next Monday in 2024-W04.
expect(groups.map((g) => g.key)).toEqual(["2024-W04", "2024-W03"]);
const first = groups.find((g) => g.key === "2024-W03")!;
expect(first.startsAt).toBe(Date.UTC(2024, 0, 15));
// The Sunday belongs to the Monday-started week, not the next one.
expect(first.fileIDs.sort((a, b) => a - b)).toEqual([1, 2, 3]);
});
it("assigns a Sunday to the preceding Monday's week across a month boundary", () => {
// 2024-01-14 is a Sunday; its week started Monday 2024-01-08.
const records = deriveRecords(
[collection(1)],
[file(1, 1, { creationTime: micros(Date.UTC(2024, 0, 14, 12)) })],
);
const groups = apis(records).timeline.groups({ groupBy: "week" });
expect(groups.map((g) => g.key)).toEqual(["2024-W02"]);
expect(groups[0]!.startsAt).toBe(Date.UTC(2024, 0, 8));
});
it("buckets by month", () => {
const records = deriveRecords(
[collection(1)],
[
file(1, 1, { creationTime: micros(Date.UTC(2024, 0, 3, 12)) }),
file(2, 1, { creationTime: micros(Date.UTC(2024, 0, 28, 12)) }),
file(3, 1, { creationTime: micros(Date.UTC(2024, 1, 9, 12)) }),
],
);
const groups = apis(records).timeline.groups({ groupBy: "month" });
expect(groups.map((g) => g.key)).toEqual(["2024-02", "2024-01"]);
expect(groups[1]!.startsAt).toBe(Date.UTC(2024, 0, 1));
expect(groups[1]!.fileIDs).toEqual([2, 1]);
});
it("lists each file once even when it belongs to several albums", () => {
// File 1001 is in collections 1 and 2 but must appear once in a group.
const records = deriveRecords(
[collection(1), collection(2)],
[
file(1001, 1, {
creationTime: micros(Date.UTC(2024, 0, 15, 12)),
}),
file(1001, 2, {
creationTime: micros(Date.UTC(2024, 0, 15, 12)),
}),
],
);
const groups = apis(records).timeline.groups({ groupBy: "day" });
expect(allFileIDs(groups)).toEqual([1001]);
});
});
describe("lib.timeline uses local time, not UTC", () => {
const savedTZ = process.env.TZ;
afterAll(() => {
if (savedTZ === undefined) delete process.env.TZ;
else process.env.TZ = savedTZ;
});
it("buckets by the viewer's local day", () => {
// Kolkata is UTC+5:30 with no DST. An instant at 2024-01-14T20:00Z is
// 2024-01-15 01:30 local, so it belongs to the local day 2024-01-15.
process.env.TZ = "Asia/Kolkata";
const records = deriveRecords(
[collection(1)],
[file(1, 1, { creationTime: micros(Date.UTC(2024, 0, 14, 20)) })],
);
const groups = apis(records).timeline.groups({ groupBy: "day" });
expect(groups[0]!.key).toBe("2024-01-15");
// Local midnight of 2024-01-15, which is 2024-01-14T18:30Z.
expect(groups[0]!.startsAt).toBe(new Date(2024, 0, 15).getTime());
expect(groups[0]!.startsAt).toBe(Date.UTC(2024, 0, 14, 18, 30));
});
});
describe("PhotoFilter", () => {
// A fixture spanning albums, file types, geotags, captions, and the two
// visibility states, all on the same local day so grouping is incidental.
const day = (h: number): number => micros(Date.UTC(2024, 2, 4, h));
const records = (): DerivedRecords =>
deriveRecords(
[
collection(1, { name: "Holidays" }),
collection(2, { name: "Work" }),
],
[
file(1, 1, {
title: "Beach sunset",
creationTime: day(1),
latitude: 1,
longitude: 2,
}),
file(2, 1, {
title: "clip.mov",
fileType: "video",
creationTime: day(2),
}),
file(3, 2, {
title: "invoice scan",
creationTime: day(3),
pubMagicMetadata: { caption: "SUNSET colours" },
}),
file(4, 2, {
title: "archived note",
creationTime: day(4),
magicMetadata: { visibility: 1 },
}),
file(5, 2, {
title: "secret",
creationTime: day(5),
magicMetadata: { visibility: 2 },
}),
],
);
const idsWith = (
filter: Parameters<
ReturnType<typeof apis>["timeline"]["groups"]
>[0]["filter"],
): number[] =>
allFileIDs(
apis(records()).timeline.groups({ groupBy: "day", filter }),
).sort((a, b) => a - b);
it("never includes hidden photos and excludes archived by default", () => {
// No filter: hidden (5) always gone, archived (4) gone unless asked for.
expect(idsWith(undefined)).toEqual([1, 2, 3]);
});
it("includes archived photos when includeArchived is set, hidden still never", () => {
expect(idsWith({ includeArchived: true })).toEqual([1, 2, 3, 4]);
});
it("filters by album membership", () => {
expect(idsWith({ albumID: 1 })).toEqual([1, 2]);
expect(idsWith({ albumID: 2 })).toEqual([3]);
});
it("filters by file type", () => {
expect(idsWith({ fileTypes: ["video"] })).toEqual([2]);
expect(idsWith({ fileTypes: ["image", "video"] })).toEqual([1, 2, 3]);
});
it("filters by presence or absence of location", () => {
expect(idsWith({ hasLocation: true })).toEqual([1]);
expect(idsWith({ hasLocation: false })).toEqual([2, 3]);
});
it("matches text case-insensitively against title, caption, and album name", () => {
// Title match (case-insensitive): "Beach sunset".
expect(idsWith({ text: "SUNSET" })).toEqual([1, 3]);
// Caption-only match: file 3's caption is "SUNSET colours".
expect(idsWith({ text: "colours" })).toEqual([3]);
// Album-name match: everything in "Holidays".
expect(idsWith({ text: "holiday" })).toEqual([1, 2]);
});
it("combines filters", () => {
// Images in album 1 with a location: only file 1.
expect(
idsWith({ albumID: 1, fileTypes: ["image"], hasLocation: true }),
).toEqual([1]);
});
});
/**
* A minimal mock `Client`, enough for `Library.open` to run its refresh loop.
* The queues are empty, so the background refresh over a seeded cache changes
* nothing; the counters prove that a read never calls the client.
*/
class MockClient {
userID = OWNER;
collectionsCalls = 0;
filesCalls = 0;
whoami(): { email: string; userID: number } {
return { email: "user@example.com", userID: this.userID };
}
async collectionsSince(args: {
sinceTime: number;
}): Promise<CollectionsPage> {
this.collectionsCalls++;
return { collections: [], deleted: [], cursor: args.sinceTime };
}
async filesSince(args: {
collectionID: number;
collectionKey: Uint8Array;
sinceTime: number;
}): Promise<FilesPage> {
this.filesCalls++;
return { files: [], deleted: [], cursor: args.sinceTime };
}
}
describe("Library exposes the read surface over its live store", () => {
let dir: string;
beforeAll(() => {
dir = mkdtempSync(join(tmpdir(), "quak-read-"));
});
afterAll(() => {
rmSync(dir, { recursive: true, force: true });
});
it("serves albums, photos, and timeline from RAM without calling the client", async () => {
const cacheDirectory = join(dir, "cache");
const path = join(cacheDirectory, "metadata.json");
// Seed a cache as a prior run left it, so open() serves it at once.
const seed = await MetadataStore.load(path);
seed.userID = OWNER;
seed.collectionsSinceTime = 100;
seed.putCollection(collection(1, { name: "Seeded" }));
seed.putFile(
file(1001, 1, { creationTime: micros(Date.UTC(2024, 5, 1, 12)) }),
);
await seed.save();
const client = new MockClient();
// A long interval keeps the background timer from firing during the test.
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: 3600,
});
try {
const collectionsBefore = client.collectionsCalls;
const filesBefore = client.filesCalls;
expect(lib.albums.list().map((a) => a.name)).toEqual(["Seeded"]);
expect(
lib.albums.byName({ albumName: "Seeded" })?.collectionID,
).toBe(1);
expect(lib.photos.byID({ fileID: 1001 })?.fileID).toBe(1001);
expect(
lib.photos.records({ fileIDs: [1001] }).map((r) => r.fileID),
).toEqual([1001]);
const groups = lib.timeline.groups({ groupBy: "month" });
expect(allFileIDs(groups)).toEqual([1001]);
// Reads are answered from RAM: no read called the client.
expect(client.collectionsCalls).toBe(collectionsBefore);
expect(client.filesCalls).toBe(filesBefore);
} finally {
await lib.close();
}
});
});
test/library/records.test.ts
+280
View File
@@ -0,0 +1,280 @@
/**
* Tests for the plain-record mapping in `src/library/records.ts` (issue #43).
*
* The library keeps decrypted `Collection`/`EnteFile` objects in RAM, but those
* carry binary keys and cannot cross the Electron IPC boundary. `deriveRecords`
* projects them into plain `AlbumRecord`/`PhotoRecord` values — no keys, no
* `Uint8Array`, JSON-safe — that the GUI process consumes. This file pins:
*
* 1. Field mapping from `metadata` and the two magic-metadata layers, using the
* real Ente field names confirmed against the fixtures in
* `test/cli/metadata-backup.test.ts` (`w`/`h`) and `test/library/store.test.ts`
* (`visibility`): title/takenAt precedence, caption, width/height, geo,
* visibility → isArchived/isHidden, fileType.
* 2. `takenAt` is milliseconds; Ente stores creationTime/editedTime in
* microseconds, so the record divides by 1000.
* 3. Deduplication: one `PhotoRecord` per fileID even when the file belongs to
* several collections, with every membership's collection id in `albumIDs`.
* 4. Ordering: photos and album `fileIDs` are newest first.
* 5. No key material survives the projection.
* 6. `diffRecords` reports exactly what changed between two derivations, and
* returns undefined when nothing changed.
*/
import { describe, it, expect } from "vitest";
import {
deriveRecords,
snapshotFrom,
diffRecords,
} from "../../src/library/records.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
const OWNER = 42;
// Microsecond epoch values, as Ente stores times. 1e15 ≈ 2001 in microseconds.
const T = (micros: number): number => micros;
const collection = (
id: number,
opts: Partial<Collection> = {},
): Collection => ({
id,
ownerID: OWNER,
key: new Uint8Array([id & 0xff, 1, 2, 3]),
name: `album-${id}`,
type: "album",
updationTime: T(1_700_000_000_000_000),
isShared: false,
...opts,
});
const file = (
id: number,
collectionID: number,
opts: Partial<EnteFile> & {
creationTime?: number;
title?: string;
} = {},
): EnteFile => {
const { creationTime, title, ...rest } = opts;
return {
id,
collectionID,
ownerID: OWNER,
key: new Uint8Array([id & 0xff, 9, 8, 7]),
metadata: {
title: title ?? `file-${id}.jpg`,
fileType: "image",
creationTime: creationTime ?? T(1_700_000_000_000_000),
modificationTime: T(1_700_000_000_000_000),
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime: T(1_700_000_000_000_000),
...rest,
};
};
describe("deriveRecords: photo mapping", () => {
it("projects a file into a plain PhotoRecord with no key material", () => {
const f = file(1001, 1, {
metadata: {
title: "IMG_1.jpg",
fileType: "image",
creationTime: T(1_699_000_000_000_000),
modificationTime: T(1_699_000_000_000_000),
latitude: 52.52,
longitude: 13.405,
},
});
const { photos } = deriveRecords([collection(1)], [f]);
const rec = photos.get(1001)!;
expect(rec.fileID).toBe(1001);
expect(rec.albumIDs).toEqual([1]);
expect(rec.title).toBe("IMG_1.jpg");
expect(rec.fileType).toBe("image");
expect(rec.latitude).toBeCloseTo(52.52);
expect(rec.longitude).toBeCloseTo(13.405);
expect(rec.isArchived).toBe(false);
expect(rec.isHidden).toBe(false);
// Safe to send over IPC: no key, no Uint8Array, JSON round-trips whole.
expect("key" in rec).toBe(false);
expect(JSON.parse(JSON.stringify(rec))).toEqual(rec);
});
it("takenAt is creationTime converted from microseconds to milliseconds", () => {
const f = file(1001, 1, { creationTime: T(1_699_000_000_000_000) });
const { photos } = deriveRecords([collection(1)], [f]);
expect(photos.get(1001)!.takenAt).toBe(1_699_000_000_000);
});
it("prefers pubMagicMetadata.editedName and editedTime over metadata", () => {
const f = file(1001, 1, {
title: "original.jpg",
creationTime: T(1_699_000_000_000_000),
pubMagicMetadata: {
editedName: "Sunset over the bay",
editedTime: T(1_650_000_000_000_000),
},
});
const { photos } = deriveRecords([collection(1)], [f]);
const rec = photos.get(1001)!;
expect(rec.title).toBe("Sunset over the bay");
expect(rec.takenAt).toBe(1_650_000_000_000);
});
it("falls back to metadata when the edited fields are empty or absent", () => {
const f = file(1001, 1, {
title: "original.jpg",
creationTime: T(1_699_000_000_000_000),
pubMagicMetadata: { editedName: "" },
});
const { photos } = deriveRecords([collection(1)], [f]);
const rec = photos.get(1001)!;
expect(rec.title).toBe("original.jpg");
expect(rec.takenAt).toBe(1_699_000_000_000);
});
it("maps caption and width/height from the public magic metadata", () => {
const f = file(1001, 1, {
pubMagicMetadata: {
caption: "at the beach",
w: 3000,
h: 2000,
},
});
const rec = deriveRecords([collection(1)], [f]).photos.get(1001)!;
expect(rec.caption).toBe("at the beach");
expect(rec.width).toBe(3000);
expect(rec.height).toBe(2000);
});
it("omits optional fields that are absent from the metadata", () => {
const rec = deriveRecords([collection(1)], [file(1001, 1)]).photos.get(
1001,
)!;
expect("caption" in rec).toBe(false);
expect("width" in rec).toBe(false);
expect("height" in rec).toBe(false);
expect("latitude" in rec).toBe(false);
expect("hash" in rec).toBe(false);
});
it("reads archived and hidden from private magicMetadata.visibility", () => {
const archived = file(1, 1, { magicMetadata: { visibility: 1 } });
const hidden = file(2, 1, { magicMetadata: { visibility: 2 } });
const visible = file(3, 1, { magicMetadata: { visibility: 0 } });
const { photos } = deriveRecords(
[collection(1)],
[archived, hidden, visible],
);
expect(photos.get(1)).toMatchObject({
isArchived: true,
isHidden: false,
});
expect(photos.get(2)).toMatchObject({
isArchived: false,
isHidden: true,
});
expect(photos.get(3)).toMatchObject({
isArchived: false,
isHidden: false,
});
});
});
describe("deriveRecords: dedup and ordering", () => {
it("emits one PhotoRecord per fileID across memberships, all albums listed", () => {
// File 1001 belongs to collections 1 and 2; 2002 only to 2.
const files = [
file(1001, 1, { creationTime: T(1_700_000_000_000_000) }),
file(1001, 2, { creationTime: T(1_700_000_000_000_000) }),
file(2002, 2, { creationTime: T(1_710_000_000_000_000) }),
];
const { photos } = deriveRecords([collection(1), collection(2)], files);
expect([...photos.keys()].sort((a, b) => a - b)).toEqual([1001, 2002]);
expect(photos.get(1001)!.albumIDs).toEqual([1, 2]);
expect(photos.get(2002)!.albumIDs).toEqual([2]);
});
it("orders snapshot photos newest first by takenAt", () => {
const files = [
file(1, 1, { creationTime: T(1_600_000_000_000_000) }),
file(2, 1, { creationTime: T(1_800_000_000_000_000) }),
file(3, 1, { creationTime: T(1_700_000_000_000_000) }),
];
const snap = snapshotFrom(deriveRecords([collection(1)], files), 123);
expect(snap.photos.map((p) => p.fileID)).toEqual([2, 3, 1]);
expect(snap.takenAt).toBe(123);
});
it("orders album fileIDs newest first", () => {
const files = [
file(1, 7, { creationTime: T(1_600_000_000_000_000) }),
file(2, 7, { creationTime: T(1_800_000_000_000_000) }),
file(3, 7, { creationTime: T(1_700_000_000_000_000) }),
];
const { albums } = deriveRecords([collection(7)], files);
expect(albums.get(7)!.fileIDs).toEqual([2, 3, 1]);
});
});
describe("deriveRecords: album mapping", () => {
it("carries collection identity, sharing, and the favorites type", () => {
const fav = collection(9, {
name: "Favorites",
type: "favorites",
isShared: true,
updationTime: T(1_705_000_000_000_000),
});
const rec = deriveRecords([fav], [file(1, 9)]).albums.get(9)!;
expect(rec).toMatchObject({
collectionID: 9,
name: "Favorites",
type: "favorites",
isShared: true,
updationTime: T(1_705_000_000_000_000),
});
expect("key" in rec).toBe(false);
expect(JSON.parse(JSON.stringify(rec))).toEqual(rec);
});
});
describe("diffRecords", () => {
const at = 999;
it("returns undefined when nothing changed", () => {
const a = deriveRecords([collection(1)], [file(1, 1)]);
const b = deriveRecords([collection(1)], [file(1, 1)]);
expect(diffRecords(a, b, at)).toBeUndefined();
});
it("reports added and changed albums and photos and removals", () => {
const before = deriveRecords(
[collection(1), collection(2)],
[file(1, 1), file(2, 2)],
);
// Collection 2 is gone (album + its only file removed). Collection 1 is
// renamed (changed album), gains file 3, and file 1 is retitled.
const after = deriveRecords(
[collection(1, { name: "renamed" })],
[
file(1, 1, {
pubMagicMetadata: { editedName: "new title" },
}),
file(3, 1),
],
);
const change = diffRecords(before, after, at)!;
expect(change.refreshedAt).toBe(at);
expect(change.albumIDsRemoved).toEqual([2]);
expect(change.fileIDsRemoved).toEqual([2]);
expect(change.albumsChanged.map((a) => a.collectionID)).toEqual([1]);
expect(change.albumsChanged[0]!.name).toBe("renamed");
expect(
change.photosChanged.map((p) => p.fileID).sort((x, y) => x - y),
).toEqual([1, 3]);
});
});
test/library/snapshot.test.ts
+304
View File
@@ -0,0 +1,304 @@
/**
* Tests for `Library.snapshot()` and `Library.subscribe()` (issue #43).
*
* These are the surface the GUI consumes across Electron IPC. `snapshot()` is
* synchronous — it reads the in-RAM store and projects it into plain records
* (no keys) — and `subscribe({ onChange })` delivers a `LibraryChange` whenever
* a refresh or `refreshCollection()` changes the derived records. The
* contracts:
*
* 1. `snapshot()` deduplicates a file across memberships into one record with
* every album id, orders photos newest first, and carries no key material.
* 2. `subscribe` fires on a refresh that changes something, with the exact
* changed and removed sets for both albums and photos.
* 3. A refresh that changes nothing (an empty diff) fires no change.
* 4. `unsubscribe()` stops further delivery.
*
* The client is the same scripted mock used by the refresh-loop tests: no
* crypto, no network. Interval tests use a short real interval and `vi.waitFor`.
*/
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { Library } from "../../src/library/index.js";
import type { LibraryChange } from "../../src/library/records.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
const USER_ID = 42;
const FAST_INTERVAL = 0.02;
const collection = (
id: number,
updationTime: number,
name = `album-${id}`,
): Collection => ({
id,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
name,
type: "album",
updationTime,
isShared: false,
});
const file = (
id: number,
collectionID: number,
creationTime: number,
): EnteFile => ({
id,
collectionID,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
metadata: {
title: `file-${id}.jpg`,
fileType: "image",
creationTime,
modificationTime: creationTime,
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime: creationTime,
});
class MockClient {
userID = USER_ID;
collectionsQueue: CollectionsPage[] = [];
filesByCollection = new Map<number, FilesPage[]>();
whoami(): { email: string; userID: number } {
return { email: "user@example.com", userID: this.userID };
}
async collectionsSince(args: {
sinceTime: number;
}): Promise<CollectionsPage> {
return (
this.collectionsQueue.shift() ?? {
collections: [],
deleted: [],
cursor: args.sinceTime,
}
);
}
async filesSince(args: {
collectionID: number;
collectionKey: Uint8Array;
sinceTime: number;
}): Promise<FilesPage> {
const queue = this.filesByCollection.get(args.collectionID);
return (
queue?.shift() ?? {
files: [],
deleted: [],
cursor: args.sinceTime,
}
);
}
filesFor(collectionID: number, ...pages: FilesPage[]): void {
this.filesByCollection.set(collectionID, pages);
}
}
describe("Library.snapshot and Library.subscribe", () => {
let dir: string;
let cacheDirectory: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), "quak-snapshot-"));
cacheDirectory = join(dir, "cache");
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
it("snapshot() dedupes across memberships, orders newest first, holds no keys", async () => {
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 100), collection(2, 100)],
deleted: [],
cursor: 100,
});
// File 1001 is in both collections; 2002 only in collection 2 and newer.
client.filesFor(1, {
files: [file(1001, 1, 1_600_000_000_000_000)],
deleted: [],
cursor: 1_600_000_000_000_000,
});
client.filesFor(2, {
files: [
file(1001, 2, 1_600_000_000_000_000),
file(2002, 2, 1_800_000_000_000_000),
],
deleted: [],
cursor: 1_800_000_000_000_000,
});
const lib = await Library.open({ client, cacheDirectory });
try {
const snap = lib.snapshot();
// One record per fileID, newest first, both albums on the shared file.
expect(snap.photos.map((p) => p.fileID)).toEqual([2002, 1001]);
const shared = snap.photos.find((p) => p.fileID === 1001)!;
expect(shared.albumIDs).toEqual([1, 2]);
expect(shared.takenAt).toBe(1_600_000_000_000);
expect(snap.albums.map((a) => a.collectionID).sort()).toEqual([
1, 2,
]);
// Nothing carries key material; the whole snapshot is JSON-safe.
expect(JSON.parse(JSON.stringify(snap))).toEqual(snap);
for (const p of snap.photos) expect("key" in p).toBe(false);
for (const a of snap.albums) expect("key" in a).toBe(false);
} finally {
await lib.close();
}
});
it("subscribe fires on a refresh change with the correct changed/removed sets", async () => {
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 100), collection(2, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 1_600_000_000_000_000)],
deleted: [],
cursor: 1_600_000_000_000_000,
});
client.filesFor(2, {
files: [file(2002, 2, 1_600_000_000_000_000)],
deleted: [],
cursor: 1_600_000_000_000_000,
});
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: FAST_INTERVAL,
});
const changes: LibraryChange[] = [];
const { unsubscribe } = lib.subscribe({
onChange: (c) => changes.push(c),
});
try {
// Next refresh: collection 2 (and its file) tombstoned; collection 1
// gains file 1003.
client.filesFor(1, {
files: [file(1003, 1, 1_700_000_000_000_000)],
deleted: [],
cursor: 1_700_000_000_000_000,
});
client.collectionsQueue.push({
collections: [collection(1, 200)],
deleted: [2],
cursor: 200,
});
await vi.waitFor(() => expect(changes.length).toBeGreaterThan(0), {
timeout: 2000,
interval: 5,
});
const change = changes[0]!;
expect(change.albumIDsRemoved).toEqual([2]);
expect(change.fileIDsRemoved).toEqual([2002]);
expect(change.photosChanged.map((p) => p.fileID)).toEqual([1003]);
expect(change.albumsChanged.map((a) => a.collectionID)).toEqual([
1,
]);
expect(change.refreshedAt).toBeGreaterThan(0);
} finally {
unsubscribe();
await lib.close();
}
});
it("a refresh that changes nothing fires no change", async () => {
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 1_600_000_000_000_000)],
deleted: [],
cursor: 1_600_000_000_000_000,
});
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: FAST_INTERVAL,
});
const changes: LibraryChange[] = [];
const { unsubscribe } = lib.subscribe({
onChange: (c) => changes.push(c),
});
try {
// Let several empty-diff ticks pass; none may deliver a change.
await new Promise((r) => setTimeout(r, FAST_INTERVAL * 1000 * 6));
expect(changes).toEqual([]);
} finally {
unsubscribe();
await lib.close();
}
});
it("unsubscribe stops further delivery", async () => {
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 1_600_000_000_000_000)],
deleted: [],
cursor: 1_600_000_000_000_000,
});
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: FAST_INTERVAL,
});
const changes: LibraryChange[] = [];
const { unsubscribe } = lib.subscribe({
onChange: (c) => changes.push(c),
});
unsubscribe();
try {
client.collectionsQueue.push({
collections: [collection(3, 300)],
deleted: [],
cursor: 300,
});
client.filesFor(3, {
files: [file(3003, 3, 1_700_000_000_000_000)],
deleted: [],
cursor: 1_700_000_000_000_000,
});
// The change lands in the store, but the cancelled subscriber sees
// nothing.
await vi.waitFor(
() => expect(lib.snapshot().albums.length).toBe(2),
{ timeout: 2000, interval: 5 },
);
expect(changes).toEqual([]);
} finally {
await lib.close();
}
});
});
test/library/store.test.ts
+288
View File
@@ -0,0 +1,288 @@
/**
* Tests for the on-disk JSON metadata store (`MetadataStore`).
*
* The store is the local cache the library keeps of the account's server
* state: one `metadata.json` file holding the user id, a schema version, the
* cursor for the incremental collections listing, and the decrypted
* collection and file records. The whole file is read into RAM on load and
* rewritten as a whole on save. A separate refresh unit (issue #42) is what
* populates it; this unit only stores.
*
* Four contracts are load-bearing and each is exercised below:
*
* 1. **Round-trip fidelity.** Everything put into the store — including the
* binary decryption keys, which JSON cannot hold directly and which the
* store base64-encodes — comes back byte-for-byte after a save and a fresh
* load. A cache that quietly dropped or mangled a field would hand the
* caller wrong keys or stale metadata.
*
* 2. **A missing or corrupt file loads as an empty store, never an error.**
* The file is only a cache: if it is absent (first run) or unreadable
* (interrupted write on an older build, disk corruption, hand-editing),
* the right answer is to start empty and let the refresh unit repopulate,
* not to crash the whole library.
*
* 3. **Writes are atomic and durable.** The store reuses the same
* fsync-before-rename atomic writer the download layer uses, so a reader
* never sees a half-written file and a crash cannot leave a truncated one.
* The observable consequence tested here is that a save leaves exactly the
* destination file behind — no temporary sibling — and that overwriting an
* existing store preserves a complete, re-loadable file.
*
* 4. **Permissions match `session.json`.** The directory is `0700` and the
* file is `0600`, because the records contain decrypted key material and
* must not be readable by other users on a shared machine.
*/
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import {
mkdtempSync,
rmSync,
readdirSync,
statSync,
writeFileSync,
mkdirSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import {
MetadataStore,
METADATA_SCHEMA_VERSION,
} from "../../src/library/store.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
// A representative decrypted collection, including a binary key and all three
// magic-metadata layers, so the round-trip test proves every field survives.
const sampleCollection = (): Collection => ({
id: 12345,
ownerID: 42,
key: new Uint8Array([1, 2, 3, 4, 5, 6, 7, 8]),
name: "Holiday 2026",
type: "album",
updationTime: 1_700_000_000_000_000,
isShared: true,
magicMetadata: { visibility: 0 },
pubMagicMetadata: { subType: 0, coverID: 999 },
sharedMagicMetadata: { note: "shared with a friend" },
});
// A representative decrypted file membership: metadata, both blob headers, a
// binary key, a content hash, and file/thumbnail sizes.
const sampleFile = (): EnteFile => ({
id: 67890,
collectionID: 12345,
ownerID: 42,
key: new Uint8Array([9, 8, 7, 6, 5, 4, 3, 2, 1]),
metadata: {
title: "IMG_0001.jpg",
fileType: "image",
creationTime: 1_699_000_000_000_000,
modificationTime: 1_699_000_500_000_000,
latitude: 52.52,
longitude: 13.405,
hash: "sha256:deadbeef",
},
magicMetadata: { editedName: "sunset" },
pubMagicMetadata: { editedTime: 1_699_000_600_000_000 },
file: { decryptionHeader: "ZmlsZUhlYWRlcg==", size: 4_194_304 },
thumbnail: { decryptionHeader: "dGh1bWJIZWFkZXI=", size: 8192 },
updationTime: 1_700_000_100_000_000,
});
describe("MetadataStore", () => {
let dir: string;
let path: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), "quak-store-"));
// Deliberately nest the store one level below the temp dir so save()
// has to create its own directory and set its mode.
path = join(dir, "cache", "metadata.json");
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
it("round-trips the whole model, keys and all", async () => {
const store = await MetadataStore.load(path);
store.userID = 42;
store.collectionsSinceTime = 1_700_000_000_000_000;
store.putCollection(sampleCollection());
store.putFile(sampleFile());
await store.save();
const reloaded = await MetadataStore.load(path);
expect(reloaded.userID).toBe(42);
expect(reloaded.collectionsSinceTime).toBe(1_700_000_000_000_000);
// The binary key must come back as the exact bytes, not a base64
// string or a plain object of numbered keys.
const collection = reloaded.getCollection(12345);
expect(collection).toEqual(sampleCollection());
expect(collection?.key).toBeInstanceOf(Uint8Array);
const file = reloaded.getFile(12345, 67890);
expect(file).toEqual(sampleFile());
expect(file?.key).toBeInstanceOf(Uint8Array);
expect(reloaded.listCollections()).toHaveLength(1);
expect(reloaded.listFiles(12345)).toHaveLength(1);
});
it("writes the declared schema version", async () => {
const store = await MetadataStore.load(path);
await store.save();
const reloaded = await MetadataStore.load(path);
expect(reloaded.schemaVersion).toBe(METADATA_SCHEMA_VERSION);
});
it("loads an empty store when the file is missing", async () => {
const store = await MetadataStore.load(path);
expect(store.userID).toBe(0);
expect(store.listCollections()).toEqual([]);
expect(store.getCollection(1)).toBeUndefined();
});
it("loads an empty store when the file is corrupt", async () => {
mkdirSync(join(dir, "cache"), { recursive: true });
writeFileSync(path, "{ this is not valid json ][");
const store = await MetadataStore.load(path);
expect(store.listCollections()).toEqual([]);
expect(store.listFiles(12345)).toEqual([]);
});
it("loads an empty store when the schema version does not match", async () => {
// A cache written by a future build with an incompatible schema is
// discarded rather than misread; the refresh unit repopulates it.
mkdirSync(join(dir, "cache"), { recursive: true });
writeFileSync(
path,
JSON.stringify({
schemaVersion: METADATA_SCHEMA_VERSION + 1,
userID: 42,
collectionsSinceTime: 0,
collections: [],
files: [],
}),
);
const store = await MetadataStore.load(path);
expect(store.userID).toBe(0);
expect(store.listCollections()).toEqual([]);
});
it("loads an empty store when the list of what did not decrypt is damaged", async () => {
// A backup reads that list on every run, so a damaged one that loaded
// would fail every backup until the cache was deleted. A string is
// damaged too, though JavaScript can loop over its characters.
mkdirSync(join(dir, "cache"), { recursive: true });
for (const undecryptable of [{}, "not a list", [1]]) {
writeFileSync(
path,
JSON.stringify({
schemaVersion: METADATA_SCHEMA_VERSION,
userID: 42,
collectionsSinceTime: 0,
collections: [],
files: [],
undecryptable,
}),
);
const store = await MetadataStore.load(path);
expect(store.userID).toBe(0);
expect(store.listUndecryptable()).toEqual([]);
}
});
it("creates the directory 0700 and the file 0600", async () => {
const store = await MetadataStore.load(path);
store.putCollection(sampleCollection());
await store.save();
// Directory 0700, file 0600: on a shared machine the decrypted keys
// in this file must be readable only by their owner. Mask to the
// permission bits; the file-type bits are not part of the assertion.
expect(statSync(join(dir, "cache")).mode & 0o777).toBe(0o700);
expect(statSync(path).mode & 0o777).toBe(0o600);
});
it("leaves exactly the destination behind, with no temp sibling", async () => {
const store = await MetadataStore.load(path);
store.putCollection(sampleCollection());
await store.save();
// The atomic writer stages a temporary file and renames it into
// place; on success nothing temporary is left in the directory.
expect(readdirSync(join(dir, "cache"))).toEqual(["metadata.json"]);
});
it("overwrites an existing store atomically and stays re-loadable", async () => {
const first = await MetadataStore.load(path);
first.userID = 1;
first.putCollection(sampleCollection());
await first.save();
const second = await MetadataStore.load(path);
second.userID = 2;
second.deleteCollection(12345);
await second.save();
const reloaded = await MetadataStore.load(path);
expect(reloaded.userID).toBe(2);
expect(reloaded.getCollection(12345)).toBeUndefined();
expect(readdirSync(join(dir, "cache"))).toEqual(["metadata.json"]);
});
it("deletes a collection together with its file memberships", async () => {
const store = await MetadataStore.load(path);
store.putCollection(sampleCollection());
store.putFile(sampleFile());
store.deleteCollection(12345);
expect(store.getCollection(12345)).toBeUndefined();
expect(store.getFile(12345, 67890)).toBeUndefined();
expect(store.listFiles(12345)).toEqual([]);
});
it("scopes file records to their collection membership", async () => {
// The same underlying file can be a member of two collections, each a
// separate record with its own key. Storing one must not touch the
// other, and lookups are per membership.
const store = await MetadataStore.load(path);
const inA = sampleFile();
const inB: EnteFile = {
...sampleFile(),
collectionID: 55555,
key: new Uint8Array([100, 101, 102]),
};
store.putFile(inA);
store.putFile(inB);
expect(store.getFile(12345, 67890)?.key).toEqual(inA.key);
expect(store.getFile(55555, 67890)?.key).toEqual(inB.key);
expect(store.listFiles(12345)).toHaveLength(1);
expect(store.listFiles(55555)).toHaveLength(1);
store.deleteFile(12345, 67890);
expect(store.getFile(12345, 67890)).toBeUndefined();
expect(store.getFile(55555, 67890)?.key).toEqual(inB.key);
});
// A backup reports these in this order. It must not depend on whether a
// refresh or `Library.refreshCollection()` fetched each one first.
it("lists what did not decrypt by collection ID, then file ID", async () => {
const store = await MetadataStore.load(path);
store.putUndecryptable({ collectionID: 2, fileID: 7, error: "a" });
store.putUndecryptable({ collectionID: 3, error: "b" });
store.putUndecryptable({ collectionID: 1, fileID: 9, error: "c" });
store.putUndecryptable({ collectionID: 2, fileID: 5, error: "d" });
expect(
store.listUndecryptable().map((r) => [r.collectionID, r.fileID]),
).toEqual([
[1, 9],
[2, 5],
[2, 7],
[3, undefined],
]);
});
});
test/live-photo.ts
+78
View File
@@ -0,0 +1,78 @@
/**
* Live photo fixtures for the download, content cache, backup and CLI tests.
*
* Ente stores a live photo as one ZIP holding its image and its video, which
* Ente's clients name `image.<ext>` and `video.<ext>`. The ZIP is built here
* with fflate from small fixed bytes and encrypted the way the server serves a
* file under 4 MiB, as one secretstream chunk. `cdnSource` serves it to the
* real download layer, so a test checks what quak stores for a real one.
*/
import { createHash } from "node:crypto";
import { zipSync } from "fflate";
import { ApiClient } from "../src/api/client.js";
import { encryptBlob, init, toBase64 } from "../src/crypto/index.js";
import {
makeDownloadContentSource,
type ContentSource,
} from "../src/library/content.js";
import type { EnteFile } from "../src/model/types.js";
export const IMAGE = new TextEncoder().encode("the still image");
export const VIDEO = new TextEncoder().encode("the few seconds of video");
// A live photo's ZIP; by default the entries an iPhone's live photo gets.
export const livePhotoZip = (
entries: Record<string, Uint8Array> = {
"image.heic": IMAGE,
"video.mov": VIDEO,
},
): Uint8Array => zipSync(entries);
// The content hash Ente's clients record for an original's bytes.
export const blake2b = (bytes: Uint8Array): string =>
createHash("blake2b512").update(bytes).digest("base64");
// The hash Ente's clients record for a live photo: the unkeyed BLAKE2b-512 of
// the image and of the video, each in standard base64, joined by a colon.
export const livePhotoHash = (image = IMAGE, video = VIDEO): string =>
`${blake2b(image)}:${blake2b(video)}`;
// `file` as a live photo whose original is `zip` and whose recorded hash is
// `hash`, and `body`, what the server serves for it: `zip` encrypted under the
// file's key and header.
export const asLivePhoto = async (
file: EnteFile,
zip = livePhotoZip(),
hash = livePhotoHash(),
): Promise<{ file: EnteFile; body: Uint8Array }> => {
await init();
const key = new Uint8Array(32).fill(file.id & 0xff);
const { header, ciphertext } = encryptBlob(zip, key);
return {
file: {
...file,
key,
metadata: { ...file.metadata, fileType: "livePhoto", hash },
file: { decryptionHeader: toBase64(header) },
},
body: ciphertext,
};
};
// A content source that downloads through the real download layer from a
// stand-in server, which serves `bodies` by file ID and a 404 for any other.
export const cdnSource = (bodies: Map<number, Uint8Array>): ContentSource =>
makeDownloadContentSource(
new ApiClient({
fetch: (async (url: string | URL) => {
const fileID = new URL(String(url)).searchParams.get("fileID");
const body = bodies.get(Number(fileID));
return body === undefined
? new Response("not found", { status: 404 })
: new Response(body);
}) as typeof globalThis.fetch,
retry: { attempts: 1 },
}),
);
test/mldata-fetch/mldata-fetch.test.ts
+54
View File
@@ -0,0 +1,54 @@
// `fetchMLDataBatch`, against a server that sends one file's ML data.
//
// ML data that decrypts and gunzips but is not a JSON object is reported with
// the reason, as ML data that does not parse is, and is never returned as a
// payload: the library would fail to store it.
import { gzipSync } from "node:zlib";
import sodium from "libsodium-wrappers-sumo";
import { beforeAll, describe, expect, it } from "vitest";
import { ApiClient } from "../../src/api/client.js";
import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
import { fetchMLDataBatch } from "../../src/mldata-fetch.js";
beforeAll(async () => {
await init();
await sodium.ready;
});
// Fetch the ML data of file 1 from a server that sends `json`, gzipped and
// encrypted with the file's key.
const fetchServing = async (json: string) => {
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptBlob(
new Uint8Array(gzipSync(json)),
key,
);
const data = [
{
fileID: 1,
encryptedData: toBase64(ciphertext),
decryptionHeader: toBase64(header),
},
];
const api = new ApiClient({
fetch: async () =>
new Response(JSON.stringify({ data }), {
headers: { "content-type": "application/json" },
}),
});
return fetchMLDataBatch(api, [1], new Map([[1, key]]));
};
describe("fetchMLDataBatch", () => {
it.each(["null", "5", "[]"])(
"gives a reason for ML data that is %s",
async (json) => {
const { payloads, errors } = await fetchServing(json);
expect(payloads.size).toBe(0);
expect(errors.get(1)).toEqual(new Error("not a JSON object"));
},
);
});
test/model/decrypt.test.ts
+122 -3
View File
@@ -145,7 +145,15 @@ const buildSharedRawCollection = (
const buildRawFile = (
collectionKey: Uint8Array,
opts?: { title?: string; fileType?: number; creationTime?: number },
opts?: {
// Any JSON value; `undefined` leaves the title out of the metadata.
title?: unknown;
fileType?: number;
creationTime?: number;
info?: { fileSize?: number; thumbSize?: number };
// Replaces the whole metadata JSON value.
metadata?: unknown;
},
): RawEnteFile => {
const fileKey = sodium.crypto_secretbox_keygen();
const { ciphertext: encFileKey, nonce: fileKeyNonce } = secretboxEncrypt(
@@ -153,8 +161,8 @@ const buildRawFile = (
collectionKey,
);
const metadata = {
title: opts?.title ?? "IMG_0001.jpg",
const defaultMetadata = {
title: opts && "title" in opts ? opts.title : "IMG_0001.jpg",
fileType: opts?.fileType ?? 0,
creationTime: opts?.creationTime ?? 1700000000000000,
modificationTime: 1700000000000000,
@@ -162,6 +170,8 @@ const buildRawFile = (
longitude: 2.3522,
hash: "abcdef1234567890",
};
const metadata =
opts && "metadata" in opts ? opts.metadata : defaultMetadata;
// File metadata is encrypted as a single-chunk secretstream blob
// (not secretbox). The decryptionHeader is the secretstream init header.
const metadataBytes = new TextEncoder().encode(JSON.stringify(metadata));
@@ -186,6 +196,7 @@ const buildRawFile = (
},
file: { decryptionHeader: toBase64(sodium.randombytes_buf(24)) },
thumbnail: { decryptionHeader: toBase64(sodium.randombytes_buf(24)) },
info: opts?.info,
updationTime: 1700000000000000,
};
};
@@ -315,6 +326,71 @@ describe("model.decryptFile", () => {
expect(file.metadata.longitude).toBeCloseTo(2.3522);
});
it("reads a missing or non-string title as an empty string", () => {
// The server controls the metadata JSON. A title that is not a
// string must not reach code that builds file names from it.
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
for (const title of [undefined, null, 42, ["a"], { x: "../y" }]) {
const file = decryptFile(
buildRawFile(collectionKey, { title }),
collectionKey,
);
expect(file.metadata.title).toBe("");
}
});
it("rejects metadata that is not a JSON object", () => {
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
for (const metadata of [null, "IMG_0001.jpg", 7, []]) {
const raw = buildRawFile(collectionKey, { metadata });
expect(() => decryptFile(raw, collectionKey)).toThrow(
"file 200: metadata is not a JSON object",
);
}
});
it("reads the recorded content hash, joining an older live photo's two parts", () => {
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
const hashOf = (metadata: Record<string, unknown>) =>
decryptFile(
buildRawFile(collectionKey, {
metadata: { title: "x", ...metadata },
}),
collectionKey,
).metadata.hash;
expect(hashOf({ fileType: 0, hash: "H" })).toBe("H");
expect(
hashOf({ fileType: 2, hash: "H", imageHash: "I", videoHash: "V" }),
).toBe("H");
expect(hashOf({ fileType: 2, imageHash: "I", videoHash: "V" })).toBe(
"I:V",
);
expect(hashOf({ fileType: 2, imageHash: "I" })).toBeUndefined();
expect(
hashOf({ fileType: 0, imageHash: "I", videoHash: "V" }),
).toBeUndefined();
expect(hashOf({ fileType: 0 })).toBeUndefined();
expect(hashOf({ fileType: 0, hash: 42 })).toBeUndefined();
expect(
hashOf({ fileType: 2, imageHash: "I", videoHash: 7 }),
).toBeUndefined();
// An empty string counts as absent, not as a hash to match.
expect(hashOf({ fileType: 0, hash: "" })).toBeUndefined();
expect(
hashOf({ fileType: 2, hash: "", imageHash: "I", videoHash: "V" }),
).toBe("I:V");
expect(
hashOf({ fileType: 2, imageHash: "", videoHash: "V" }),
).toBeUndefined();
expect(
hashOf({ fileType: 2, imageHash: "I", videoHash: "" }),
).toBeUndefined();
});
it("maps fileType numbers to FileType strings", () => {
// Ente uses: 0=image, 1=video, 2=livePhoto
const masterKey = sodium.crypto_secretbox_keygen();
@@ -357,4 +433,47 @@ describe("model.decryptFile", () => {
expect(() => decryptFile(raw, wrongKey)).toThrow();
});
it("carries the file and thumbnail byte sizes from info", () => {
// The server reports the encrypted-blob sizes in `info`; the cache
// needs them without a HEAD request, so decryptFile must copy them
// onto the file and thumbnail blobs.
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
const raw = buildRawFile(collectionKey, {
info: { fileSize: 4096, thumbSize: 512 },
});
const file = decryptFile(raw, collectionKey);
expect(file.file.size).toBe(4096);
expect(file.thumbnail.size).toBe(512);
});
it("leaves the sizes undefined when the server omits info", () => {
// Older files predate the info field; the sizes must stay undefined
// rather than become 0, so callers can tell "unknown" from "empty".
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
const raw = buildRawFile(collectionKey);
expect(raw.info).toBeUndefined();
const file = decryptFile(raw, collectionKey);
expect(file.file.size).toBeUndefined();
expect(file.thumbnail.size).toBeUndefined();
});
it("carries the deletion flag from the diff row", () => {
// The diff marks a deleted row with isDeleted; decryptFile copies it
// onto the file so a caller can tell a deleted row from a live one.
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
const raw = buildRawFile(collectionKey);
raw.isDeleted = true;
const file = decryptFile(raw, collectionKey);
expect(file.isDeleted).toBe(true);
});
});
test/packaging/build-context.test.ts
+42 -18
View File
@@ -2,17 +2,19 @@
// failures are silent.
//
// Excluding too little: a worktree left under `.claude/` is copied into the
// image, vitest globs its `test/` tree as well as the real one, and the
// containerised `make check` runs the whole suite twice over while reporting
// success. A compiled `bin/quak` is ~100 MB of context nobody needs.
// image, vitest globs its `test/` tree as well as the real one, and the test
// phase runs the whole suite twice over while reporting success. A compiled
// `bin/quak` is ~100 MB of context nobody needs.
//
// Excluding too much: Prettier 3 reads `.gitignore` as a default ignore file,
// so dropping it from the context silently changes which files
// `make fmt-check` looks at inside the image compared to the host.
// so dropping it from the context silently changes which files the lint
// phase's prettier check looks at compared to `make fmt-check` on the host.
// And without `.git`, a `docker build .` given no `VERSION` build arg cannot
// derive the version (`script/version`) and stamps `package.json`'s instead.
//
// Neither shows up as a build failure, so they are asserted here.
// None of these shows up as a build failure, so they are asserted here.
import { describe, expect, it } from "vitest";
import { readFileSync } from "node:fs";
import { existsSync, readFileSync } from "node:fs";
import { fileURLToPath } from "node:url";
import { join } from "node:path";
@@ -27,18 +29,19 @@ const patterns = (name: string): string[] =>
const dockerignore = patterns(".dockerignore");
describe(".dockerignore", () => {
// Everything here is either generated, enormous, or secret. `.claude/` is
// the correctness one: see the header comment and issue #25.
// Everything here is either generated, enormous, or secret. `.claude` is
// the correctness one: see the header comment and issue #25. The leading
// `/` anchors an entry at the root of the context.
it.each([
".claude/",
".quak/",
"bin/quak",
"node_modules",
"coverage",
"dist",
".vitest-cache/",
".nyc_output/",
"*.tsbuildinfo",
".claude",
"/.quak",
"/bin/quak",
"**/node_modules",
"/coverage",
"/dist",
"/.vitest-cache",
"/.nyc_output",
"/*.tsbuildinfo",
])("keeps %s out of the build context", (pattern) => {
expect(dockerignore).toContain(pattern);
});
@@ -46,4 +49,25 @@ describe(".dockerignore", () => {
it("leaves .gitignore in the build context for prettier", () => {
expect(dockerignore).not.toContain(".gitignore");
});
it("leaves .git in the build context for the version", () => {
expect(dockerignore).not.toContain(".git");
expect(dockerignore).not.toContain(".git/");
});
// The build stage is the final image, so a .git/config sent in would
// ship the clone's remote URL and any credential in it.
it("sends .git without its config", () => {
expect(dockerignore).toContain("**/.git/config");
});
// BuildKit lets a `Dockerfile.dockerignore` shadow the root one; such a
// file would silently give the build a different, unreviewed context —
// and eslint's flat config does not ignore dot-directories, so a stray
// `.claude/` worktree would be linted.
it("is not shadowed by a Dockerfile.dockerignore", () => {
expect(existsSync(join(repoRoot, "Dockerfile.dockerignore"))).toBe(
false,
);
});
});
test/packaging/entrypoints.test.ts
+1 -1
View File
@@ -1,6 +1,6 @@
// The package manifest promises three files that only exist after a build:
// `main`, `types`, and the `quak` binary. Nothing in the test suite used to
// look at them, and `make check` runs test, lint and fmt-check but never the
// look at them, and `make check` runs the test and lint phases but never the
// build, so `tsconfig.json` and `package.json` were free to drift apart. They
// did: `rootDir` was `./src` while `include` also pulled in `bin/**/*`, which
// is TS6059, and no build had succeeded for as long as that was true.

Some files were not shown because too many files have changed in this diff Show More