Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
e03e99b486 | ||
|
sneak
|
fd40714a6d |
No files matched your search
@@ -2,7 +2,7 @@
|
||||
|
||||
- branch from `next`
|
||||
- do the work in Next Step
|
||||
- delete Next Step from `TODO.md`
|
||||
- move Next Step to the top of Completed Steps
|
||||
- move the top item of Future Steps into Next Step
|
||||
- commit (`TODO.md` changes in the same commit as the work)
|
||||
- push
|
||||
@@ -17,13 +17,683 @@ pre-1.0
|
||||
|
||||
# Next Step
|
||||
|
||||
No bug is open, and no change sneak asked for is open. The check of quak against
|
||||
a live Ente account, https://git.eeqj.de/sneak/quak/issues/187, needs an account
|
||||
only sneak has, so it waits on him.
|
||||
No bug is open. Two changes sneak asked for are open:
|
||||
https://git.eeqj.de/sneak/quak/issues/211 and
|
||||
https://git.eeqj.de/sneak/quak/issues/212. The check of quak against a live Ente
|
||||
account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
|
||||
has, so it waits on him.
|
||||
|
||||
Tagging and releases are decided by sneak alone, and happen only when he
|
||||
declares one.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-18: Names quak builds from an album name or a title fit in a file name
|
||||
(https://git.eeqj.de/sneak/quak/issues/205). `shortenKeepingExtension` in
|
||||
`src/filename.ts` cuts a name short to a number of bytes, keeping its
|
||||
extension when it fits and always its first character, and can put text right
|
||||
before the extension; the backup's `withID` uses it for an ID.
|
||||
`defaultFileName` there gives the name a file is saved under when the caller
|
||||
gives no path: a prefix such as `thumb_` and the title with unsafe characters
|
||||
replaced, cut short with it to 255 bytes, and, for a live photo's image or
|
||||
video, with that part's own extension before the cut. `quak get`,
|
||||
`quak get-thumb`, `downloadFile` and `downloadThumbnail` use it without
|
||||
`--out` or `outPath`. `backup-metadata` cuts an album's name short so that its
|
||||
folder, `<id>-<name>`, fits in 255 bytes. `safeExtension` gives `.bin` for an
|
||||
extension of more than 16 letters and digits, so a save path's name always
|
||||
fits.
|
||||
|
||||
- 2026-10-18: `quak collections` no longer waits for the files of every
|
||||
collection (issue 210). On an empty cache it opened its library with a refresh
|
||||
that fetched and decrypted the files of each collection in turn, printing
|
||||
nothing and storing nothing until all were done, so on a large account it
|
||||
seemed to hang. It now reads the collection list alone through the new
|
||||
`lib.freshCollections()`, one request that stores nothing. The read commands
|
||||
open their library with the new `refreshOnOpen: false`, so each makes only the
|
||||
refresh it asks for. A refresh sends a `"progress"` event after each
|
||||
collection whose files it fetched, and the commands that wait on one print it
|
||||
on stderr, except with `--json`.
|
||||
|
||||
- 2026-10-16: ML data the server sends that does not decrypt, gunzip or parse is
|
||||
no longer taken for no ML data (issue 198). `fetchMLDataBatch` returns why for
|
||||
each such file beside the payloads; ML data that parses to something other
|
||||
than a JSON object, such as `null`, is one of them. The library keeps the
|
||||
reasons from its last ML data fetch that succeeded in memory, not in the
|
||||
cache, and gives them in `status().mlDataErrors` and in the `fetchMLData`
|
||||
event with status `"done"`; a fetch still running or one that fails does not
|
||||
change them. `quak backup` writes the reason as `mlDataError` in the file's
|
||||
JSON when no ML data is cached for it, and records the file in `failures.json`
|
||||
with an error starting `ML data: `; `backup-metadata` writes `mlDataError`
|
||||
too. Both exit 1. Tests serve ML data encrypted under a key that is not the
|
||||
file's.
|
||||
|
||||
- 2026-10-16: With `--verify`, an original whose hash does not match is no
|
||||
longer removed before its replacement is downloaded (issue 199).
|
||||
`replaceOriginal` in `src/library/content.ts` downloads the replacement to a
|
||||
temporary file beside it, a live photo's image and video each to its own,
|
||||
hashes it and renames it over the original only when it matches. When the
|
||||
download fails or does not match either, the original keeps its bytes and
|
||||
links, and the file goes into `failures.json`. The replacement is always
|
||||
downloaded, through the new `fetchOriginal` of the content cache, never copied
|
||||
from the cache.
|
||||
|
||||
- 2026-10-16: The `~/bin/quak` that `make install` builds retries a refused
|
||||
connection and a name that does not resolve, and the backup records either as
|
||||
transient (issue 206). Bun's `fetch` reports them with codes of its own,
|
||||
`ConnectionRefused` and `FailedToOpenSocket`, which `src/retry.ts` now counts
|
||||
as transport failures. A `POST` or `PUT` is not replayed on them, because bun
|
||||
also gives them when it resends a request on a new connection after a
|
||||
connection it kept open closed, and the server may already have the request.
|
||||
Bun already gave `ECONNRESET` for a reset and a `TimeoutError` for a deadline.
|
||||
`classify` in `src/backup.ts` now records any `TimeoutError` as transient, so
|
||||
a stalled download, whose error says only that it stalled, is no longer
|
||||
`unknown`. The shapes were found by running the binary against local
|
||||
listeners.
|
||||
|
||||
- 2026-10-15: An album whose folder or JSON `quak backup` cannot write no longer
|
||||
ends the run before the other albums and `failures.json` are written (issue
|
||||
197). The run prints `FAILED album <name>: <reason>`, lists the album's ID in
|
||||
`failedAlbums` in the result and in `--json`, goes on, and exits 1. An album
|
||||
whose folder would have the name of another album's JSON, as `x.json` beside
|
||||
`x` would, gets its ID added. A folder or link name over 255 bytes is cut
|
||||
short between two characters, by `shortenToBytes` in `src/filename.ts`, and
|
||||
gets the ID.
|
||||
|
||||
- 2026-10-15: A network failure is printed and recorded with its reason, not as
|
||||
just `fetch failed` (issue 196). `describeError` in `src/errors.ts` gives an
|
||||
error's message followed by the message of each error on its `cause` chain,
|
||||
such as `fetch failed: connect ECONNREFUSED 1.2.3.4:443`, and each address's
|
||||
reason when a host has several and none could be reached. `run` prints that,
|
||||
and the backup logs it and records it in `failures.json`, where `classify` now
|
||||
calls the failure transient when the chain holds an errno that
|
||||
`hasTransportCode` in `src/retry.ts` recognises, for a failed ML data fetch
|
||||
too. The walk along the chain moved from `src/retry.ts` to `src/errors.ts`,
|
||||
which both use. Tests refuse every connection of a backup, then only one
|
||||
file's download, and fail the ML data fetch.
|
||||
|
||||
- 2026-10-15: The README says that `make install` needs nix (issue 200), in
|
||||
Getting Started and in "Running the backup from cron": `make build-bin` runs
|
||||
bun through `nix-shell -p bun`. The `Makefile` is unchanged.
|
||||
|
||||
- 2026-10-14: `lib.backup()`, and so `quak backup`, works on up to the content
|
||||
pool's limit of originals at once, 5 by default, instead of one at a time
|
||||
(issue 182). That many workers take files from one shared list, each doing all
|
||||
the work for one original, its `--verify` hashing included, so no more
|
||||
originals than that are open at once. `BackupLibrary` says the limit as
|
||||
`contentConcurrency`. A test holds each download at the fake Ente server to
|
||||
count those in flight. Once a worker has thrown, as one does when `onProgress`
|
||||
throws, or the lock is lost, no worker takes another file; the backup fails
|
||||
once the originals under way are in place, and a lost lock stops it before it
|
||||
fetches ML data.
|
||||
|
||||
- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's
|
||||
review, though that issue closed on 2026-10-03 (issue 194). It now names the
|
||||
issues open on the tracker.
|
||||
|
||||
- 2026-10-14: A backup whose lock is lost no longer dies with a stack trace
|
||||
(issue 184). `lockBackupDirectory` gives proper-lockfile an `onCompromised`
|
||||
that records why the lock was lost instead of throwing from its timer, and
|
||||
returns the lock, which `quak backup` passes to `lib.backup()` as `lock`. The
|
||||
backup then stops before it starts another file with
|
||||
`lost the lock on DIR: REASON`, which `quak backup` prints as one line before
|
||||
it exits 1. Releasing a lost lock leaves `backup.lock` alone. Tests in
|
||||
`test/cli/backup-end-to-end.test.ts` remove the lock, or put another backup's
|
||||
in its place, while a download is held open.
|
||||
|
||||
- 2026-10-14: An album or file quak cannot decrypt, or whose metadata is not
|
||||
JSON, no longer fails the refresh, so one bad record no longer stops
|
||||
`quak backup` from storing anything (issue 181). The refresh leaves it out, an
|
||||
album with its files, and the library keeps it with the reason in
|
||||
`metadata.json` until the server sends a copy that decrypts or deletes it.
|
||||
`quak backup` stores every other file, records each such file in
|
||||
`failures.json` with an error starting `could not be decrypted:`, names each
|
||||
such album's ID in a printed line, or with `--json` in `undecryptableAlbums`,
|
||||
and exits 1 while any remain. A file whose copy in another album decrypts is
|
||||
not recorded: its original is stored from that copy, and the album where it
|
||||
does not decrypt neither links nor lists it. The read commands answer from
|
||||
what decrypted.
|
||||
|
||||
- 2026-10-14: `quak backup` no longer asks the server for changes every 3
|
||||
seconds while it runs (issue 183). It opens its library with
|
||||
`refreshIntervalSeconds: Infinity`, which the library now takes as no
|
||||
background timer, so the backup refreshes only as it starts and no timed
|
||||
refresh starts another ML data pass during the run. A test holds a backup's
|
||||
download open while two hours pass and counts its `/collections/v2` requests.
|
||||
|
||||
- 2026-10-14: `test/cli/backup-end-to-end.test.ts` runs `quak backup` as cron
|
||||
does (issue 186): `backupCommand` and `run`, the real `loadSession` reading a
|
||||
session saved after logging in, and every request answered by a fake of the
|
||||
Ente HTTP API serving data encrypted with libsodium. It covers the backup tree
|
||||
of an account with two albums of its own and one shared with it, a second run
|
||||
that downloads nothing, a download that succeeds after two 503s, a 404 that
|
||||
lands in `failures.json` with exit 1, a refresh answered 401 with exit 3, and
|
||||
`--verify` putting back an original changed on disk. The fake server and the
|
||||
account it logs in moved from the backup-metadata test into
|
||||
`test/ente-server.ts`, unchanged.
|
||||
|
||||
- 2026-10-14: `files --collection`, `get`, `get-thumb` and
|
||||
`helper fix-missing-thumbnails --file` take an ID only when it is written in
|
||||
decimal digits and is a safe integer (issue 185). Before, `Number()` read
|
||||
`0x10` as 16, `1e3` as 1000 and an empty string as 0, and `--file` dropped an
|
||||
ID it could not read without a word. Anything else now prints
|
||||
`Invalid file ID: <arg>` or `Invalid collection ID: <arg>` and exits 1 before
|
||||
any request; one bad `--file` ID fails the whole command. `parseID` in
|
||||
`src/cli-commands.ts` does the check for all four.
|
||||
|
||||
- 2026-10-06: The README says how to run `quak backup` from cron (issue 170):
|
||||
log in once as the job's user, a crontab with a backup every night and
|
||||
`--verify` on Sundays appending to a log file, and the `HOME` and
|
||||
`XDG_DATA_HOME` the job needs to find the saved session. A table gives each
|
||||
exit code: 0, the backup is complete; 1, files are in `failures.json` or
|
||||
another error stopped the run; 2, another backup of the directory is running;
|
||||
3, there is no usable session. The introduction lists everything the backup
|
||||
keeps for each file.
|
||||
|
||||
- 2026-10-06: `quak backup --verify` and `lib.backup({ verify: true })` hash
|
||||
each original already at its save path as the download check does, streamed, a
|
||||
live photo as `<imageHash>:<videoHash>` (issue 168). One that does not match
|
||||
the content hash its metadata records is logged, removed (both files of a live
|
||||
photo) and put back in the same run, downloaded or copied from the cache, and
|
||||
what is put back is hashed too. One that still does not match, or whose
|
||||
download fails, goes into `failures.json`. One with no recorded hash is left
|
||||
alone. The result, `--json` and the summary gain `verified`, `mismatched` and
|
||||
`unchecked`. Without `--verify` nothing is hashed.
|
||||
|
||||
- 2026-10-06: Two backups of the same directory never run at once (issue 169).
|
||||
`lib.backup()` takes a lock, `backup.lock` in its download directory, made
|
||||
with `proper-lockfile`, before its refresh, and removes it when it ends,
|
||||
whether it succeeds or fails. A second backup of the directory, from another
|
||||
process or the same one, fails at once with an error naming the directory.
|
||||
`quak backup` takes the lock before it opens its library, so a refused run
|
||||
sends no request; it prints the error as one line and exits 2. A lock that has
|
||||
gone 10 seconds untouched, left by a run that could not remove it, is taken
|
||||
over by the next run.
|
||||
|
||||
- 2026-10-06: `quak backup` writes each original's EXIF, XMP and dimensions into
|
||||
the file's JSON as `imageMetadata`, what `backup-metadata --exif` records
|
||||
(issue 167): for a live photo from its image, for a video nothing, and `{}`
|
||||
for an original with none of them. A failed read puts the reason in
|
||||
`imageMetadataError` and fails neither the file nor the run. An original is
|
||||
read when the run stores it or when its JSON has neither field; otherwise the
|
||||
field is taken from that JSON. The hand-built JPEGs moved to
|
||||
`test/exif-jpeg.ts`, beside the HEIC.
|
||||
|
||||
- 2026-10-06: When the server answers HTTP 401 and that ends a command that
|
||||
loads the saved session, because the server no longer accepts its token, the
|
||||
command prints one line,
|
||||
`quak: the saved session is no longer valid; run "quak login"`, and exits 3
|
||||
(issue 164). `quak backup` meets that 401 on the refresh it starts with,
|
||||
before it touches any file. For the same commands, a missing or corrupt
|
||||
session file keeps its message and also exits 3, so a cron job can tell that
|
||||
the user must log in again. `quak logout` is unchanged. `run` in
|
||||
`src/cli-run.ts` recognises the 401, which reaches it unchanged.
|
||||
|
||||
- 2026-10-06: `quak backup` retries a failed request for longer than the other
|
||||
commands do (issue 165). `src/retry.ts` exports `UNATTENDED_RETRY_OPTIONS`
|
||||
beside the unchanged default: 10 attempts, a 1 s base delay and a 60 s cap, so
|
||||
a request that keeps failing waits at most 243 s before it gives up.
|
||||
`bin/quak.ts` loads the backup's session with them, so its refresh, ML data
|
||||
and downloads all use them. What is retried and the backoff formula are
|
||||
unchanged.
|
||||
|
||||
- 2026-10-06: The files this repository copies from `sneak/prompts` are copied
|
||||
again from its commit `dd4027b` (issue 171). `.gitignore` and `.dockerignore`
|
||||
keep out more secret files, and this repository's build artifacts follow the
|
||||
copied content. `script/check` runs `script/fmt-check` again, `script/cibuild`
|
||||
runs `script/bootstrap` and `script/check` before the image build,
|
||||
`script/fmt` and `script/fmt-check` find the pinned yarn under nvm, and the
|
||||
image's last stage marks `/app` safe for git.
|
||||
|
||||
- 2026-10-06: `quak backup` writes the account and album records
|
||||
`backup-metadata` writes (issue 166): `account.json` with the account's
|
||||
`email` and `userID`, and in each album's JSON its `ownerID`, `isShared`,
|
||||
`updationTime` and, when present, its three layers of magic metadata. Each
|
||||
file's JSON gains its `updationTime`.
|
||||
|
||||
- 2026-10-05: `quak backup` writes each file's ML data (its faces and CLIP
|
||||
embedding) into the file's JSON as `mlData`, the payload
|
||||
`lib.mldata.forFile()` returns (issue 163). `lib.backup()` waits for an ML
|
||||
data fetch before it writes the JSON files. When that fetch fails, each file
|
||||
whose ML data is not cached gets the reason in `mlDataError` and counts as
|
||||
failed, and the next run fetches it again.
|
||||
|
||||
- 2026-10-03: The download-albums example test no longer fails on a disk with
|
||||
under 50 GiB free (issue 160). It opens its libraries with
|
||||
`freeBelowBytes: 0`, so the free space of the disk it runs on cannot shrink
|
||||
the cache, evict the original it cached and add a fetch to the count. No other
|
||||
test's result depends on that free space. The 50 GiB default is unchanged.
|
||||
|
||||
- 2026-10-02: `docker build .` stamps the commit's tag or short commit, not
|
||||
`dev` (issue 154). `script/build` writes the version `script/version` prints
|
||||
into `dist/package.json`: the `VERSION` environment variable or build arg when
|
||||
one is given, otherwise `git describe --tags --always`, otherwise the version
|
||||
`package.json` declares. `.dockerignore` sends `.git`, and a checkout with
|
||||
`.git` whose version comes out empty, `dev` or `unknown` fails the build.
|
||||
`make build-bin` bundles the built `dist/`, so the single binary reports the
|
||||
same version.
|
||||
|
||||
- 2026-10-02: `photo.exif()` returns every EXIF tag in the file as `ExifTags`,
|
||||
keyed by tag name, each as exifreader decodes it, not only the thirteen common
|
||||
fields (issue 156). The embedded thumbnail's tags are under `Thumbnail`,
|
||||
without the thumbnail image. The thirteen typed methods stay, each picking its
|
||||
field from the tags `exif()` returns, typed as in `PhotoExif`. The example
|
||||
script's JSON files now carry every tag.
|
||||
|
||||
- 2026-10-01: The content cache no longer looks for a live photo that an earlier
|
||||
version cached as one ZIP (issue 151). When the cache opens, a live photo's
|
||||
file that no JSON file names is now always left alone.
|
||||
|
||||
- 2026-10-01: `examples/download-albums.ts` logs in, opens the library, and for
|
||||
every album downloads each photo to its save path, writes the photo's record
|
||||
and EXIF fields to a JSON file beside it, and writes the album's photos to
|
||||
`albums/<collectionID>.json` (issue 144). The build compiles it to
|
||||
`dist/examples/`; the README's "Examples" section says how to run it.
|
||||
|
||||
- 2026-10-01: A `Photo` has one async method for each field of `exif()`, named
|
||||
and typed as in `PhotoExif`: `make()`, `model()`, `lensModel()`,
|
||||
`dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`,
|
||||
`iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()`
|
||||
and `gpsAltitude()` (issue 148). Each calls `exif()` and returns its one
|
||||
field, or undefined when the file lacks it. `Photo` implements a type with one
|
||||
method per `PhotoExif` field, so the build's type check fails when a field has
|
||||
no method. A test checks, on the JPEG and the HEIC, that each method gives the
|
||||
same value as `exif()`.
|
||||
|
||||
- 2026-10-01: Each original's save path is
|
||||
`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
|
||||
download directory, which defaults to `photos` in the working directory (issue
|
||||
143). The date is the photo's `takenAt` in the machine's time zone.
|
||||
`photo.savePath` is always a string, with or without a content cache, and
|
||||
`isLocal` is true only when the original is at its save path.
|
||||
`photo.download()` puts the original there, copied from the cache when the
|
||||
cache holds it and fetched otherwise. `lib.backup()` does the same for each
|
||||
file, writes each file's JSON beside its original, and links `collections/` to
|
||||
the save paths.
|
||||
|
||||
- 2026-10-01: `photo.exif()` and `backup-metadata --exif` read EXIF from
|
||||
HEIC/HEIF originals, a live photo's HEIC image included, as well as JPEG and
|
||||
the other image formats `exifreader` reads (issue 145). `exifreader` replaces
|
||||
`exif-reader` and the JPEG segment scan; `PhotoExif` is unchanged. The `exif`
|
||||
field of `backup-metadata --exif` is now exifreader's tag output, and
|
||||
`exifRaw` holds the whole EXIF block it could not read. The tests use a real
|
||||
HEIC, `test/exif.heic`.
|
||||
|
||||
- 2026-10-01: A `Photo` has `savePath`, `isLocal`, `content()`, `exif()`,
|
||||
`modifiedAt`, `hash` and `year` (issue 141). `savePath` is where
|
||||
`lib.backup()` writes the original under the library's download directory; for
|
||||
a live photo not yet stored, it carries the title's extension, and the backup
|
||||
may store the image under a different one. `isLocal` says whether the whole
|
||||
original is there. Both look only at the disk. `content()` returns the
|
||||
original's bytes and `exif()` the common EXIF fields of a JPEG; both may
|
||||
download the original, and `exif()` downloads no video. `PhotoRecord` gains
|
||||
`modifiedAt` and `hash`, and the JPEG EXIF scan moved to `src/exif.ts`.
|
||||
|
||||
- 2026-09-29: The "Workflow" list at the top of this file now says to branch
|
||||
from `next` and open a pull request that targets `next`, that the repository
|
||||
manager squash-merges a reviewed pull request into `next`, and that only sneak
|
||||
merges `next` into `main`, as the README does (issue 137).
|
||||
|
||||
- 2026-09-29: The README's "Development workflow" and "For LLMs" sections now
|
||||
say work branches from `next`, every pull request targets `next`, the
|
||||
repository manager squash-merges reviewed pull requests into `next`, and only
|
||||
sneak merges `next` into `main` (issue 135).
|
||||
|
||||
- 2026-09-29: Brought the README and this file in line with `next` after the
|
||||
milestone merge (issue 132). The Next Step says no implementation work is open
|
||||
and the cache design (issue 36) waits on sneak's review. README corrections:
|
||||
the Getting Started comments say when the library refreshes; most, not all,
|
||||
Makefile targets call a script; `script/lint` and `script/test` have no host
|
||||
path, though `yarn test` does; the SRP handshake uses `fast-srp-hap`, outside
|
||||
`crypto/`, and a thumbnail upload's MD5 uses `node:crypto`; the SRP password
|
||||
is the first 16 bytes of a 32-byte subkey; the key attributes and token come
|
||||
after SRP, after the TOTP code SRP may ask for, or after the email OTP that
|
||||
replaces SRP when the account has email MFA on, and quak cannot answer a
|
||||
passkey; the auth token is sent as URL-safe base64 with padding; every
|
||||
`TypeError` is retried; each download attempt writes its own temporary files,
|
||||
two for a live photo, and only the attempt that completes renames them into
|
||||
place; `runMetadataBackup` records a failed download in the file's JSON; a
|
||||
second `client.logout()` does not throw; `quak logout` with no session exits
|
||||
0, and names the cache directory only when it knows it; `login`, `whoami` and
|
||||
`logout` open no library; `--exif` records XMP and, for a JPEG, EXIF, and no
|
||||
IPTC; which commands take `--json`; a failed ML data request may not have been
|
||||
retried; the thumbnail fixer is not limited to baseline JPEG; `quak backup`
|
||||
still fetches ML data; a backup's JSON holds the basic metadata fields quak
|
||||
keeps and the private and public magic metadata, not every decrypted field,
|
||||
and the README no longer lists what the magic metadata holds; the default
|
||||
cache directory is the per-user one, not an XDG path on macOS; pinned
|
||||
originals can exceed `cacheOriginalsMaxBytes`; which tests cover which
|
||||
operations; a default read is only as current as the last refresh whose
|
||||
requests all succeeded; `fresh()` and `lib.backup()` join a refresh already
|
||||
running; a `Photo` has no `thumbnailPath` or `originalPath`; and `408` and
|
||||
`429` are retried.
|
||||
|
||||
- 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks
|
||||
that the image's and the video's temp files are fsynced before either is
|
||||
renamed into place, and the directory after both renames, as the `writeAtomic`
|
||||
test does for one file.
|
||||
|
||||
- 2026-09-23: Live photos are stored as their image and their video (issue 107).
|
||||
A live photo, which Ente stores as one ZIP, is unpacked as it downloads into
|
||||
`<fileID>.<ext>` for the image and for the video, each with its extension from
|
||||
the ZIP, beside `<fileID>.livephoto.json`, which names the two. Both are
|
||||
checked against the recorded hash and renamed into place only when both are
|
||||
complete. A ZIP holding a second image or video, or whose two files come to
|
||||
more than 20 times its size plus 16 MiB, is refused. The backup and the
|
||||
content cache count the live photo as stored only with both files, the album
|
||||
folder links both, `quak get` writes both, and the content result gives the
|
||||
video as `videoPath`. A ZIP an earlier version stored is replaced on the next
|
||||
backup run, and removed from the cache when the library opens it.
|
||||
|
||||
- 2026-09-23: Settled the package metadata (issue 6). quak is not published, so
|
||||
`package.json` is marked `"private": true` and the `files` field is gone.
|
||||
`engines.node` is `>=22`, the major version `script/bootstrap` and the
|
||||
`Dockerfile` use. An `exports` map makes `.` and `./package.json` the only
|
||||
importable paths; `runMetadataBackup`, the thumbnail helpers and their types
|
||||
stay internal to the CLI.
|
||||
|
||||
- 2026-09-23: Brought the README and this file in line with the tree (issue
|
||||
111). The layout lists `src/library/` and the other source files, the backup
|
||||
layout names `failures.json` and the optional `thumbnails/`, "Opening a
|
||||
library" says what happens when the first refresh fails, the Testing section
|
||||
gives the 60-second hard cap and 20-second target for `make test` and names
|
||||
the 90-second `timeout` in the `Dockerfile` as the backstop for a hung test,
|
||||
and "Tag v1.0.0" is no longer listed as the next step.
|
||||
|
||||
- 2026-09-23: Tested `quak login` and `backup-metadata --exif` (issue 110).
|
||||
`loginCommand` takes its login function and its prompts from `CliContext`, and
|
||||
`bin/quak.ts` passes `Client.login` and the terminal prompts. Tests cover a
|
||||
login from `QUAK_EMAIL` and `QUAK_PASSWORD` with no prompt, the TOTP prompt, a
|
||||
failed login, and the saved session's modes, and show that `--exif` and
|
||||
`--all` each turn on EXIF extraction and that it is off without them.
|
||||
|
||||
- 2026-09-23: `quak backup` writes each original once and no longer fills the
|
||||
cache (issue 106). An original fetched for a backup is written by the download
|
||||
writer straight into the backup's `originals/`, and the content cache records
|
||||
it there instead of keeping its own copy; one the cache already held is still
|
||||
copied. `quak backup` opens its library with the thumbnail and originals
|
||||
precache off.
|
||||
|
||||
- 2026-09-23: `backup-metadata`, `helper list-missing-thumbnails` and
|
||||
`helper fix-missing-thumbnails` refresh before they answer (issue 100). Each
|
||||
awaits `lib.fresh()` before reading, so a file added since the cache was
|
||||
written is included, and a failed refresh prints one line and exits 1 instead
|
||||
of answering from a stale or empty cache. The README lists them among the
|
||||
commands that refresh first.
|
||||
|
||||
- 2026-09-23: `quak backup` waits for the server refresh and fails when it fails
|
||||
(issue 99). `lib.backup()` joins a refresh already running or starts one, as
|
||||
`fresh()` does, and rejects before touching any file when it fails, leaving
|
||||
`failures.json` as it was, so `quak backup` prints the error as one line and
|
||||
exits 1 instead of backing up the previous run's file list, or nothing, and
|
||||
exiting 0.
|
||||
|
||||
- 2026-09-23: CLI errors print a message instead of a stack trace (issue 102).
|
||||
An error a command throws is printed as one `quak: MESSAGE` line on stderr and
|
||||
the CLI exits 1 once output has drained. The wrapper that does this moved from
|
||||
`bin/quak.ts` to `src/cli-run.ts`, and `bin/quak.ts` now awaits
|
||||
`program.parseAsync()`.
|
||||
|
||||
- 2026-09-23: Opening a library no longer deletes another process's download in
|
||||
progress (issue 105). The download writer's temp files are named
|
||||
`.quak-<pid>-<random>.tmp`, and `removeLeftoverTempFiles`, moved from the
|
||||
backup into the download module, deletes a `.quak-*.tmp` file only when the
|
||||
process ID in its name is no longer running. The content cache calls it at
|
||||
`open()` for `originals/` and `thumbnails/`, the backup as before.
|
||||
|
||||
- 2026-09-23: Re-vendored the lint and test setup from the template (issue 96).
|
||||
Linting and testing are the `lint` and `test` phases of the `Dockerfile`;
|
||||
`script/lint` and `script/test` each build one with `--no-cache`, and the last
|
||||
stage compiles and depends on both, so `script/cibuild` is one build.
|
||||
`Dockerfile.lint`, `CHECK_EPOCH`, `LINT_EPOCH` and the tests that checked them
|
||||
are gone; `REPO_POLICIES.md` is re-copied.
|
||||
|
||||
- 2026-09-23: Stopped `helper fix-missing-thumbnails` retrying files the server
|
||||
always refuses (issue 109). Both thumbnail helpers skip a file another account
|
||||
owns without fetching it. The fixer skips a file whose recorded thumbnail size
|
||||
is 0 or unknown before downloading it, and otherwise tries smaller encodings
|
||||
(720 px quality 50 down to 160 px quality 20) until the encrypted thumbnail is
|
||||
no larger than that size, skipping the file if none fits.
|
||||
|
||||
- 2026-09-23: Tested the live-photo hash check's error paths (issue 117). Tests
|
||||
download a live photo whose ZIP names an unknown compression method, one whose
|
||||
ZIP has no image entry and one with no video entry, and check that nothing is
|
||||
stored and the error names the file ID; the unreadable one is not retried.
|
||||
|
||||
- 2026-09-23: `quak logout` ends the session on the server (issue 108). It calls
|
||||
`POST /users/logout` through the new `Client.logoutOnServer()`, then deletes
|
||||
`session.json` even when that call fails, says so and exits 1. It prints the
|
||||
account's cache directory and says it still holds decrypted data. The default
|
||||
cache path is now `defaultCacheDirectory()` in the library, shared with
|
||||
`Library.open`.
|
||||
|
||||
- 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files
|
||||
in one collection with the same title, and two collections with the same name,
|
||||
each get their ID added to the name (`IMG_0001 (12345).JPG`, `Trip (10)/`), so
|
||||
none replaces another's symlink or JSON. Each run removes symlinks into
|
||||
`originals/` for files no longer in the collection, and the folders of deleted
|
||||
or renamed collections, leaving anything else in `collections/` alone. The
|
||||
README backup layout states the naming rule.
|
||||
|
||||
- 2026-09-23: Checked downloaded originals against their recorded content hash
|
||||
(issue 68). `downloadFile`, which `quak get`, the content cache and backup all
|
||||
use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and
|
||||
stores nothing on a mismatch, failing with an error naming the file ID. A live
|
||||
photo ZIP is unpacked as it streams with `fflate` and its image and video
|
||||
hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older
|
||||
clients' `imageHash` and `videoHash` fields for live photos. A file with no
|
||||
recorded hash is stored unchecked.
|
||||
|
||||
- 2026-09-23: Kept one account's cache from mixing with another's (issue 104).
|
||||
When `metadata.json` in the cache directory was written for a different,
|
||||
non-zero user ID than the client's, `Library.open` deletes it and `mldata/`
|
||||
and starts empty, so the first refresh enumerates from 0. This only happens
|
||||
with `--cache-dir` or an explicit `cacheDirectory`; the default path already
|
||||
includes the user ID. A test opens one account's cache as another account.
|
||||
|
||||
- 2026-09-23: `backup-metadata` no longer stops on one failed ML data request
|
||||
(issue 101). Each request of up to 200 files is tried on its own; a failed one
|
||||
is logged, its files are written with the reason in `mlDataError`, and the
|
||||
command exits 1 once the dump is complete. `fetchMLData`, which only this
|
||||
command used, is gone; the command calls `fetchMLDataBatch` per batch.
|
||||
- 2026-09-23: Single-sourced the version string (issue 5). `package.json` is the
|
||||
only place it is written: `src/index.ts` imports it for `VERSION` and
|
||||
`bin/quak.ts` passes `VERSION` to commander. tsc copies `package.json` to
|
||||
`dist/package.json`, so the import resolves from the built output too, and
|
||||
`script/build` runs the built CLI with `--version` to prove it. A test checks
|
||||
that `VERSION` and `quak --version` both equal the `package.json` version.
|
||||
- 2026-09-23: Tested that `Library.close()` waits for the originals precache
|
||||
(issue 93). The test that holds a precache fetch open while `close()` runs now
|
||||
runs once with only the thumbnail fill and once with only the originals fill,
|
||||
so dropping either wait from `Precache.close()` fails a test.
|
||||
- 2026-09-23: Fixed two intermittently failing library tests (issue 90).
|
||||
`Library.close()` now returns a promise that resolves once an in-flight
|
||||
refresh (including its cache write), the ML data fetch and running precache
|
||||
sweeps have finished; the library tests await it, so `afterEach` no longer
|
||||
removes the cache directory while something is still writing into it. The
|
||||
precache test waits for both fills to report "done" instead of for its stub
|
||||
source to be called, which happened before the cache recorded the file.
|
||||
- 2026-09-23: Pinned three guards reviewers found untested (issue 89). The
|
||||
download idle deadline's timer is unref'd, so it can never keep the process
|
||||
alive, and a test checks no timer is left after a download completes or fails.
|
||||
A test covers the rejection of `#` in a request path. The EXIF scan compares
|
||||
the `Exif` header only in an APP1 segment of length 8 or more, so it never
|
||||
reads the next segment's bytes, with a test for a short one.
|
||||
- 2026-09-23: Made the download deadline an idle deadline (issue 24).
|
||||
`downloadTimeoutMs` now aborts a file or thumbnail download only after no
|
||||
bytes have arrived for that long, default 60 seconds, instead of bounding the
|
||||
whole transfer at 10 minutes, so a slow download that keeps making progress
|
||||
completes. A download that fails before reading the whole body cancels it, so
|
||||
a failed file no longer holds its connection.
|
||||
- 2026-09-23: Every `ApiClient` request URL is now built by one function next to
|
||||
the class (issue 18), so a self-hosted `apiOrigin` with a base path keeps it
|
||||
on every request, a path works with or without a leading slash, and query
|
||||
parameters are percent-encoded. A path containing `?` or `#` is rejected with
|
||||
an error instead of being silently cut.
|
||||
- 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies
|
||||
moved from `bin/quak.ts` into `src/cli-commands.ts` as functions that take
|
||||
their options and a context (output streams, session directory, cache
|
||||
directory, session loader) and return an exit code; `bin/quak.ts` only wires
|
||||
them to commander and exits with the code once stdout and stderr have drained,
|
||||
so nothing below it calls `process.exit`. `test/cli/commands.test.ts` drives
|
||||
them with a fake client: session file modes, logout, the missing and corrupt
|
||||
session paths, and the output and exit code of `whoami`, `collections`,
|
||||
`files`, `get`, `get-thumb`, `backup` and `helper list-missing-thumbnails`.
|
||||
- 2026-09-23: Hardened the backup tree's atomic copy (issue 22). `copyAtomic`
|
||||
fsyncs its temp file before the rename and the directory after it, through the
|
||||
download writer's `fsyncPath`; each backup run deletes `.quak-backup-*.tmp`
|
||||
files whose process is no longer running. The README backup layout names the
|
||||
temp files and states that the rename replaces a symlink and takes the temp
|
||||
file's permissions. Added tests for a missing and an unwritable destination
|
||||
directory for `downloadFile` and `downloadThumbnail`.
|
||||
- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue
|
||||
11). Every segment length is checked against the remaining bytes and lengths
|
||||
under 2 stop the scan, so a truncated or corrupt original can neither throw
|
||||
nor loop. A malformed or unparseable EXIF segment is recorded as
|
||||
`imageMetadata.exifError`, and a failure to read the original as
|
||||
`imageMetadataError` in the per-file JSON, instead of the field being left
|
||||
out.
|
||||
- 2026-09-22: Hardened the retry classifier (issue 80). A `POST` or `PUT` is
|
||||
replayed only when every errno in the cause chain is a connect errno, and it
|
||||
no longer follows redirects. `getRetryOptions()` returns a copy. Tests pin
|
||||
every errno the classifier names, the cause-chain depth limit, cycle
|
||||
termination, and a fresh deadline per attempt for every retrying entry point.
|
||||
The README's endpoint list is the one place that names the requests the replay
|
||||
rule covers.
|
||||
- 2026-09-22: Stopped `make test` collecting tests from checkouts nested under
|
||||
`.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a
|
||||
nested checkout ran the whole suite again; `vitest.config.ts` now adds
|
||||
`.claude/**` to vitest's default excludes, and
|
||||
`test/packaging/nested-checkout.test.ts` plants a nested checkout in a temp
|
||||
directory and fails if vitest would collect it.
|
||||
- 2026-09-22: Dropped the deprecated `@types/libsodium-wrappers-sumo` stub from
|
||||
`devDependencies` (issue 27). It shipped no declarations; the types come from
|
||||
`libsodium-wrappers-sumo` itself. `yarn.lock` regenerated by `yarn remove`.
|
||||
- 2026-09-22: Hardened the client session lifecycle (issue 10).
|
||||
`Client.fromJSON` checks every snapshot field and each key's decoded length
|
||||
and names the bad field; `toJSON` reads the token through
|
||||
`ApiClient.getAuthToken` and throws when there is none; `logout` zeroes the
|
||||
key buffers, and `collectionsSince` re-checks for logout after its request so
|
||||
it never decrypts with zeroed keys. The CLI reports a corrupt session file
|
||||
separately from a missing one (`src/cli-session.ts`).
|
||||
- 2026-09-22: Sanitized file names taken from server metadata (issue 9). A new
|
||||
`src/filename.ts` holds the one sanitizer, used by `quak get`/`get-thumb`
|
||||
without `--out`, `downloadFile`/`downloadThumbnail` without `outPath`, and the
|
||||
backup and metadata backup trees; it removes separators, control characters,
|
||||
leading dots and Windows device names, and falls back to a name built from the
|
||||
ID for an empty title. Originals-cache extensions are letters and digits only,
|
||||
else `.bin`. A user-supplied path is used as is. `decryptFile` reads a missing
|
||||
or non-string title as "" and rejects metadata that is not a JSON object.
|
||||
- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
|
||||
snippets) to match the shipped cache/API library on `next` (issue 53, issue
|
||||
13). Documented `Library.open` and its options, the default-read vs `fresh()`
|
||||
distinction (and that the CLI's read commands are fresh), the record types and
|
||||
`snapshot()`/`subscribe()`, the `albums`/`photos`/`timeline` read surface,
|
||||
`Photo` content methods, `thumbnails.ensure`, the `mldata` search surface,
|
||||
`backup()`, the three request pools (10/5/25), and the on-disk cache layout;
|
||||
noted the deferred content-hash integrity check (issue 68). Docs-only; no code
|
||||
changed.
|
||||
- 2026-09-22: Added resumable, deletion-aware enumeration to `Client` (issue 38,
|
||||
closes issue 7). `collectionsSince`/`filesSince` take a starting cursor,
|
||||
decrypt live records, surface tombstoned ids in a separate `deleted` list (a
|
||||
tombstone has nothing to decrypt, so it is a bare id, not a hollow record),
|
||||
and return the max `updationTime` seen as the cursor to resume from.
|
||||
`filesSince` refuses to loop when the diff reports `hasMore` without advancing
|
||||
the cursor (issue 7). `listCollections`/`listFiles` are now thin wrappers that
|
||||
enumerate from `sinceTime: 0` and drop deletions, so existing callers are
|
||||
unaffected.
|
||||
- 2026-09-22: Carried file size, thumbnail size, and the deletion flag through
|
||||
`decryptFile` (issue 37, foundation for the cache/API design). Live files now
|
||||
populate `file.size`/`thumbnail.size` from the server's `info` (left
|
||||
`undefined` when the server omits it), and `isDeleted` is carried from the
|
||||
diff row onto `EnteFile`. No caller change: `listFiles` still filters deleted
|
||||
rows before decrypting. Surfacing a tombstone through decryption belongs to
|
||||
the enumeration unit (issue 38).
|
||||
- 2026-08-10: Made `lint-once.test.ts` enforce what its header claims. It walked
|
||||
`make check` only, so it never read `Dockerfile` — the image CI builds through
|
||||
`script/cibuild` — and a second `prettier --check .` could be added there with
|
||||
the suite staying green. The walk now also starts at
|
||||
`.gitea/workflows/check.yml` and follows its `run:` steps, so the graph under
|
||||
test is the one CI executes rather than the one someone assumed it executes.
|
||||
The lockfile assertion was a substring check against the whole of
|
||||
`script/bootstrap`, which has two install sites and so reported the branch the
|
||||
containers never take; the two branches are now resolved separately and every
|
||||
`yarn install` in each is required to be `--frozen-lockfile`. Prettier is
|
||||
counted per occurrence instead of per line, so two invocations chained with
|
||||
`&&` no longer read as one, and edges are followed on counted lines instead of
|
||||
being skipped. Every way for the walk to reach nothing — an unknown target, an
|
||||
unknown script, a missing file, a node with no commands, an unknown node kind
|
||||
— is a thrown error rather than a quiet zero. Every assertion in the file was
|
||||
mutation-tested individually.
|
||||
- 2026-08-10: Stopped `make check` running `prettier --check .` twice. Since
|
||||
linting moved into Docker, the duplicate was one container pass and one host
|
||||
pass of the same check: `script/lint` builds `Dockerfile.lint`, which runs
|
||||
prettier as a build step, and `script/check` then called `script/fmt-check` as
|
||||
well. The host call is gone from `script/check` and from `script/precommit`;
|
||||
the container keeps checking formatting, because a successful
|
||||
`Dockerfile.lint` build is what CI treats as proof of a clean tree, and it is
|
||||
also what still fails the pre-commit hook on a badly formatted tree.
|
||||
`script/fmt-check` survives as a standalone entrypoint, whose verdict cannot
|
||||
drift from the container's. A test walks the invocation graph from each
|
||||
entrypoint — through the Makefile shims, the `script/` calls and the
|
||||
`docker build` — and asserts the prettier count, so the duplication cannot
|
||||
come back unnoticed.
|
||||
- 2026-08-10: Moved all linting into Docker. `script/lint` builds a new root
|
||||
`Dockerfile.lint`, which copies the repo into the digest-pinned node image and
|
||||
runs eslint and prettier as build steps, so a successful build is a clean
|
||||
lint; no host lint path remains and `yarn lint` is gone from `package.json`. A
|
||||
fail-closed `LINT_EPOCH` guard stops Docker serving the linter layers from
|
||||
cache, which is how a lint build returns success in under a second having
|
||||
linted nothing. The lint stage inside `Dockerfile` and its `COPY --from=lint`
|
||||
ordering hack are gone: that image now runs `make test` and `make build` only,
|
||||
because `script/check` calls `script/lint` and running it in a container would
|
||||
mean docker inside docker. `script/cibuild` builds the lint image first, then
|
||||
the test and build image.
|
||||
- 2026-08-09: Made `make docker` green and policy-conformant. Multi-stage
|
||||
Dockerfile: a lint stage runs `make fmt-check` and `make lint`, and the check
|
||||
stage takes a `COPY --from=lint` dependency on it before running `make check`
|
||||
and `make build`. `CHECK_EPOCH` and a fail-closed guard stop Docker serving
|
||||
those two layers from cache, which is what let a build report success without
|
||||
running the suite. `script/projectname` says `quak`, so the image is tagged
|
||||
`quak`; `script/bootstrap` updates apt lists before installing, so a Debian
|
||||
base works; `.dockerignore` no longer ships the compiled binary, the caches or
|
||||
agent worktrees into the build context, and keeps `.gitignore` in it for
|
||||
prettier.
|
||||
- 2026-08-09: Fixed the TypeScript build. `rootDir` is the repo root, so `bin/`
|
||||
compiles alongside `src/` instead of failing with TS6059; output is
|
||||
`dist/src/` and `dist/bin/`, which is where `main`, `types` and `bin.quak` now
|
||||
point. `script/build` verifies the declared entrypoints exist after the
|
||||
compiler runs and makes the CLI executable, the Dockerfile runs `make build`
|
||||
as well as `make check`, and a `quak` script makes the README's
|
||||
`yarn quak <command>` examples work.
|
||||
- 2026-08-09: Retry policy: no retry on 4xx (except `408` and `429`),
|
||||
exponential backoff with full jitter on 5xx, transport failures and truncated
|
||||
transfers, under per-attempt deadlines that cover the response body as well as
|
||||
the request. Downloads retry request, stream consumption and decryption as one
|
||||
unit; `postJSON` and `putJSON` are replayed only when the connection was never
|
||||
established.
|
||||
- 2026-08-09: Downloads verify the secretstream terminated on `TAG_FINAL` and
|
||||
write output atomically: a truncated body is rejected instead of landing on
|
||||
disk as a short file, and plaintext is staged in a sibling temp file and
|
||||
renamed into place, so a failed download leaves the destination untouched.
|
||||
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
|
||||
shims, README Entrypoints section
|
||||
- 2026-06-10: Decrypted collections shared by other users (sealed-box keys);
|
||||
listCollections drops deleted-collection tombstones.
|
||||
- 2026-06-10: Login hardening: dual-2FA empty-string fields handled, TOTP
|
||||
preferred when a passkey is also enrolled, interactive input via
|
||||
@inquirer/prompts.
|
||||
- 2026-06-10: Replaced sharp with pure JS (jpeg-js + exif-reader); added
|
||||
single-binary bun build and make install.
|
||||
- 2026-06-09: Added backup-metadata command (ML data always included, --exif
|
||||
opt-in); rewrote README to match the implementation; added thumbnail helper
|
||||
tests.
|
||||
- 2026-05-13: Full CLI surface: login, backup with dedup symlink layout,
|
||||
collections, files, get, get-thumb, thumbnail repair helpers.
|
||||
- 2026-05-13: Client OO API with literate usage tests; file download and
|
||||
decryption; all three metadata layers decrypted and persisted; renamed quack
|
||||
to quak.
|
||||
- 2026-05-11: SRP login flow (email OTP + TOTP) and ApiClient.
|
||||
|
||||
# Future Steps
|
||||
|
||||
- Future desktop client, separate repo:
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
#!/bin/sh
|
||||
# Writes src/clip-keyword-vectors.ts and test/clip-keywords-sunset.json by
|
||||
# running bin/clip-keyword-vectors.py in a Python image pinned by digest, with
|
||||
# its packages pinned by hash. Run it by hand from a checkout, then `make fmt`;
|
||||
# no gate runs it. Besides the Python image, it downloads about 265 MB, most of
|
||||
# it Ente's two CLIP models.
|
||||
set -eu
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
|
||||
# python:3.13-slim (Python 3.13.17 on Debian 13.7), 2026-10-21. The wheels
|
||||
# below are the ones for this image on linux/amd64.
|
||||
IMAGE=python@sha256:57b29a7c050c4e679e4bea572e14b3965cf4b5839de2da49d55e49ffc4024f39
|
||||
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
docker run --rm -i --platform linux/amd64 \
|
||||
--user "$(id -u):$(id -g)" -e HOME=/tmp \
|
||||
-v "$ROOT:/repo" -w /repo "$IMAGE" sh -c '
|
||||
python -m venv /tmp/venv &&
|
||||
/tmp/venv/bin/pip install --quiet --disable-pip-version-check \
|
||||
--no-cache-dir --no-deps --require-hashes -r /dev/stdin &&
|
||||
/tmp/venv/bin/python bin/clip-keyword-vectors.py' <<'EOF'
|
||||
numpy==2.5.3 --hash=sha256:a5fa86b80fd24bcd1aff83ad23be44ea323de3f787be8f8b15d4a65621e25321
|
||||
onnxruntime==1.30.0 --hash=sha256:86f940afc801ea9681a4da8af84fbe95e1d9ea7d80903952cc1bfad54faad38f
|
||||
pillow==12.3.0 --hash=sha256:0847a763afefb695bc912d7c131e7e0632d4edc1d8698f58ddabec8e46b8b6d3
|
||||
tokenizers==0.23.2 --hash=sha256:41c2f84d172449b4dadb9cdc508e3e364076613c35b16e76ecfe47a60d1e3305
|
||||
EOF
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -1,158 +0,0 @@
|
||||
"""Write src/clip-keyword-vectors.ts and test/clip-keywords-sunset.json.
|
||||
|
||||
bin/clip-keyword-vectors runs this in a pinned Python image. It downloads
|
||||
Ente's Discover list, Ente's CLIP text and image models, the CLIP tokenizer
|
||||
and a public-domain photo of a sunset, checks each against its sha256, and
|
||||
writes:
|
||||
|
||||
- src/clip-keyword-vectors.ts: each Discover category of Ente's photos app,
|
||||
with the text embedding Ente's CLIP text model gives its query;
|
||||
- test/clip-keywords-sunset.json: the CLIP embedding Ente's image model gives
|
||||
the photo, prepared as Ente's apps prepare a photo.
|
||||
|
||||
Both embeddings are scaled to length 1, as Ente's apps scale them.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import urllib.request
|
||||
|
||||
import numpy as np
|
||||
import onnxruntime
|
||||
from PIL import Image, ImageOps
|
||||
from tokenizers import Tokenizer
|
||||
|
||||
ENTE_COMMIT = "f5ad70a860ed25d0c8057b0efb4bd6eaefe4b0c5"
|
||||
DISCOVER_PATH = "mobile/apps/photos/assets/discover.json"
|
||||
DISCOVER_URL = (
|
||||
f"https://raw.githubusercontent.com/ente-io/ente/{ENTE_COMMIT}/{DISCOVER_PATH}"
|
||||
)
|
||||
DISCOVER_SHA256 = "cf70ac7116b592f7921ca4cf67cdbb421a8e6b287fbe44b2be678b8d198e5c24"
|
||||
|
||||
TEXT_MODEL = "mobileclip_s2_text_opset18_quant.onnx"
|
||||
TEXT_MODEL_SHA256 = "d92f33dfcff83077fc2e0d3414250710efbb51795dfd89767bdbefb5fdc47322"
|
||||
IMAGE_MODEL = "mobileclip_s2_image_gelu_opset20.onnx"
|
||||
IMAGE_MODEL_SHA256 = "205a430af825e501c5138e5bb9abea942482a7a4fd4a680e98e47cf0830dce7e"
|
||||
|
||||
# The CLIP tokenizer Hugging Face publishes for OpenAI's CLIP. Its vocabulary
|
||||
# and merges are those of bpe_simple_vocab_16e6.txt, the vocabulary Ente's
|
||||
# apps tokenize with.
|
||||
TOKENIZER_URL = (
|
||||
"https://huggingface.co/openai/clip-vit-base-patch32/resolve/"
|
||||
"3d74acf9a28c67741b2f4f2ea7635f0aaf6f0268/tokenizer.json"
|
||||
)
|
||||
TOKENIZER_SHA256 = "b556ac8c99757ffb677208af34bc8c6721572114111a6e0aaf5fa69ff0b8d842"
|
||||
|
||||
# "Sunset off the Massachusetts coast", a NOAA photo in the public domain:
|
||||
# https://commons.wikimedia.org/wiki/File:Sunset_off_the_Massachusetts_coast_-_NOAA.jpg
|
||||
PHOTO_URL = (
|
||||
"https://upload.wikimedia.org/wikipedia/commons/d/db/"
|
||||
"Sunset_off_the_Massachusetts_coast_-_NOAA.jpg"
|
||||
)
|
||||
PHOTO_SHA256 = "be62f3289c9bf7b75af68cd6978021d54e8a531d5d4cc493f6e9fbf5e1ab74d3"
|
||||
|
||||
# Ente's text model takes 77 token ids, padded with 0; its image model takes
|
||||
# a 256x256 RGB image.
|
||||
TOKEN_COUNT = 77
|
||||
IMAGE_SIZE = 256
|
||||
|
||||
|
||||
def download(url, sha256):
|
||||
# Wikimedia refuses requests without a User-Agent.
|
||||
request = urllib.request.Request(url, headers={"User-Agent": "quak"})
|
||||
with urllib.request.urlopen(request) as response:
|
||||
data = response.read()
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
if digest != sha256:
|
||||
raise SystemExit(f"{url} has sha256 {digest}, not {sha256}")
|
||||
return data
|
||||
|
||||
|
||||
def ente_model(name, sha256):
|
||||
return onnxruntime.InferenceSession(
|
||||
download(f"https://models.ente.com/{name}", sha256),
|
||||
providers=["CPUExecutionProvider"],
|
||||
)
|
||||
|
||||
|
||||
def run(session, tensor):
|
||||
(output,) = session.run(None, {session.get_inputs()[0].name: tensor})
|
||||
embedding = output[0]
|
||||
return embedding / np.linalg.norm(embedding)
|
||||
|
||||
|
||||
def text_embedding(session, tokenizer, query):
|
||||
ids = tokenizer.encode(query).ids
|
||||
ids += [0] * (TOKEN_COUNT - len(ids))
|
||||
return run(session, np.array([ids], dtype=np.int32))
|
||||
|
||||
|
||||
def image_embedding(session, photo):
|
||||
image = ImageOps.exif_transpose(Image.open(io.BytesIO(photo))).convert("RGB")
|
||||
# Scale to cover IMAGE_SIZE x IMAGE_SIZE and crop the centre.
|
||||
square = ImageOps.fit(image, (IMAGE_SIZE, IMAGE_SIZE), Image.Resampling.BILINEAR)
|
||||
pixels = np.asarray(square, dtype=np.float32) / 255
|
||||
return run(session, pixels.transpose(2, 0, 1)[np.newaxis])
|
||||
|
||||
|
||||
def numbers(embedding):
|
||||
return ", ".join(str(value) for value in embedding)
|
||||
|
||||
|
||||
def write_categories(prompts, embeddings):
|
||||
lines = [
|
||||
"// Written by bin/clip-keyword-vectors; do not edit by hand.",
|
||||
"//",
|
||||
"// The Discover categories of Ente's photos app, in their order, from",
|
||||
f"// {DISCOVER_PATH} in https://github.com/ente-io/ente",
|
||||
f"// at commit {ENTE_COMMIT}. Each has the text",
|
||||
f"// embedding Ente's CLIP text model ({TEXT_MODEL})",
|
||||
"// gives its query, scaled to length 1. `showVideo` is false for a category",
|
||||
"// that leaves out videos.",
|
||||
"",
|
||||
"export const DISCOVER_CATEGORIES = [",
|
||||
]
|
||||
for prompt, embedding in zip(prompts, embeddings):
|
||||
lines += [
|
||||
" {",
|
||||
f" title: {json.dumps(prompt['title'])},",
|
||||
f" query: {json.dumps(prompt['query'])},",
|
||||
f" minScore: {json.dumps(prompt['minScore'])},",
|
||||
f" showVideo: {json.dumps(prompt.get('showVideo', True))},",
|
||||
f" textEmbedding: [{numbers(embedding)}],",
|
||||
" },",
|
||||
]
|
||||
lines.append("];")
|
||||
with open("src/clip-keyword-vectors.ts", "w", encoding="utf-8") as file:
|
||||
file.write("\n".join(lines) + "\n")
|
||||
|
||||
|
||||
def write_fixture(embedding):
|
||||
with open("test/clip-keywords-sunset.json", "w", encoding="utf-8") as file:
|
||||
file.write(
|
||||
"{\n"
|
||||
f' "photo": {json.dumps(PHOTO_URL)},\n'
|
||||
f' "sha256": {json.dumps(PHOTO_SHA256)},\n'
|
||||
f' "embedding": [{numbers(embedding)}]\n'
|
||||
"}\n"
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
prompts = json.loads(download(DISCOVER_URL, DISCOVER_SHA256))["prompts"]
|
||||
tokenizer = Tokenizer.from_str(
|
||||
download(TOKENIZER_URL, TOKENIZER_SHA256).decode("utf-8")
|
||||
)
|
||||
text_model = ente_model(TEXT_MODEL, TEXT_MODEL_SHA256)
|
||||
embeddings = [
|
||||
text_embedding(text_model, tokenizer, prompt["query"]) for prompt in prompts
|
||||
]
|
||||
write_categories(prompts, embeddings)
|
||||
|
||||
image_model = ente_model(IMAGE_MODEL, IMAGE_MODEL_SHA256)
|
||||
write_fixture(image_embedding(image_model, download(PHOTO_URL, PHOTO_SHA256)))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+16
-47
@@ -2,9 +2,8 @@
|
||||
|
||||
import { stdout, stderr } from "node:process";
|
||||
import { input, password } from "@inquirer/prompts";
|
||||
import { Command, Option } from "commander";
|
||||
import { Command } from "commander";
|
||||
import envPaths from "env-paths";
|
||||
import { logLevel, logRequests, stderrLog, type Log } from "../src/cli-log.js";
|
||||
import { init } from "../src/crypto/index.js";
|
||||
import {
|
||||
type CliContext,
|
||||
@@ -30,9 +29,6 @@ const paths = envPaths("quak", { suffix: "" });
|
||||
|
||||
const program = new Command();
|
||||
|
||||
// Program options, so they are taken before or after any command's name.
|
||||
// `-v` counts its repeats, the way commander's documentation counts a repeated
|
||||
// option.
|
||||
program
|
||||
.name("quak")
|
||||
.description("CLI for the Ente end-to-end encrypted photo service")
|
||||
@@ -41,46 +37,21 @@ program
|
||||
"--cache-dir <path>",
|
||||
"Directory for the local metadata/content cache " +
|
||||
"(default: the per-user cache directory)",
|
||||
)
|
||||
.option(
|
||||
"-v, --verbose",
|
||||
"Log more on stderr: debug lines with -v, trace lines too with -vv",
|
||||
(_value: string, previous: number) => previous + 1,
|
||||
0,
|
||||
)
|
||||
.addOption(
|
||||
new Option(
|
||||
"-q, --quiet",
|
||||
"Print nothing on stderr but the error a failing command exits with",
|
||||
).conflicts("verbose"),
|
||||
)
|
||||
.configureHelp({ showGlobalOptions: true });
|
||||
|
||||
interface ProgramOptions {
|
||||
cacheDir?: string;
|
||||
verbose: number;
|
||||
quiet?: boolean;
|
||||
}
|
||||
|
||||
// The log for this run, at the level `-v` and `-q` ask for.
|
||||
const log = (): Log => {
|
||||
const { verbose, quiet } = program.opts<ProgramOptions>();
|
||||
return stderrLog(stderr, logLevel(verbose, quiet ?? false));
|
||||
};
|
||||
);
|
||||
|
||||
const context = (): CliContext => ({
|
||||
stdout,
|
||||
log: log(),
|
||||
stderr,
|
||||
sessionDir: paths.data,
|
||||
cacheDir: program.opts<ProgramOptions>().cacheDir,
|
||||
loadSession: (path) => loadSession(path, logRequests(log())),
|
||||
login: (opts) => Client.login({ ...opts, apiOptions: logRequests(log()) }),
|
||||
cacheDir: program.opts<{ cacheDir?: string }>().cacheDir,
|
||||
loadSession,
|
||||
login: (opts) => Client.login(opts),
|
||||
prompt: (message) => input({ message }),
|
||||
promptSecret: (message) => password({ message, mask: true }),
|
||||
});
|
||||
|
||||
const run = (command: Promise<number>): Promise<void> =>
|
||||
runCommand(command, stdout, stderr, log(), (code) => process.exit(code));
|
||||
runCommand(command, stdout, stderr, (code) => process.exit(code));
|
||||
|
||||
program
|
||||
.command("login")
|
||||
@@ -107,15 +78,14 @@ program
|
||||
|
||||
program
|
||||
.command("files")
|
||||
.description("List the files in a collection, or every file in the account")
|
||||
.argument(
|
||||
"[collectionID]",
|
||||
"Collection ID (from `quak collections`); without it, every file " +
|
||||
"is listed once",
|
||||
.description("List files in a collection")
|
||||
.requiredOption(
|
||||
"--collection <id>",
|
||||
"Collection ID (from `quak collections`)",
|
||||
)
|
||||
.option("--json", "Output as JSON array")
|
||||
.action((collectionID: string | undefined, opts: { json?: boolean }) =>
|
||||
run(filesCommand(context(), collectionID, opts)),
|
||||
.action((opts: { collection: string; json?: boolean }) =>
|
||||
run(filesCommand(context(), opts)),
|
||||
);
|
||||
|
||||
program
|
||||
@@ -123,6 +93,7 @@ program
|
||||
.description("Download and decrypt a single file")
|
||||
.argument("<fileID>", "File ID (from `quak files`)")
|
||||
.option("--out <path>", "Output file path")
|
||||
.option("--collection <id>", "Accepted for compatibility; ignored")
|
||||
.action((fileID: string, opts: { out?: string }) =>
|
||||
run(getCommand(context(), fileID, opts)),
|
||||
);
|
||||
@@ -132,6 +103,7 @@ program
|
||||
.description("Download and decrypt a thumbnail")
|
||||
.argument("<fileID>", "File ID (from `quak files`)")
|
||||
.option("--out <path>", "Output file path")
|
||||
.option("--collection <id>", "Accepted for compatibility; ignored")
|
||||
.action((fileID: string, opts: { out?: string }) =>
|
||||
run(getThumbCommand(context(), fileID, opts)),
|
||||
);
|
||||
@@ -170,10 +142,7 @@ program
|
||||
{
|
||||
...context(),
|
||||
loadSession: (path) =>
|
||||
loadSession(
|
||||
path,
|
||||
logRequests(log(), UNATTENDED_RETRY_OPTIONS),
|
||||
),
|
||||
loadSession(path, { retry: UNATTENDED_RETRY_OPTIONS }),
|
||||
},
|
||||
dir,
|
||||
opts,
|
||||
|
||||
+12
-57
@@ -36,15 +36,6 @@ export interface ApiClientOptions {
|
||||
retry?: RetryOptions;
|
||||
requestTimeoutMs?: number;
|
||||
downloadTimeoutMs?: number;
|
||||
// Called once for each request sent, with the status of its response or
|
||||
// the error it failed with, for a caller that logs requests. A request
|
||||
// that is retried is reported once for each attempt.
|
||||
onRequest?: (request: {
|
||||
method: string;
|
||||
url: string;
|
||||
status?: number;
|
||||
error?: unknown;
|
||||
}) => void;
|
||||
}
|
||||
|
||||
export interface StreamOptions {
|
||||
@@ -168,7 +159,6 @@ export class ApiClient {
|
||||
private readonly retry: ResolvedRetryOptions;
|
||||
private readonly requestTimeoutMs: number;
|
||||
private readonly downloadTimeoutMs: number;
|
||||
private readonly onRequest: NonNullable<ApiClientOptions["onRequest"]>;
|
||||
private token: string | undefined;
|
||||
|
||||
constructor(opts?: ApiClientOptions) {
|
||||
@@ -190,7 +180,6 @@ export class ApiClient {
|
||||
opts?.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS;
|
||||
this.downloadTimeoutMs =
|
||||
opts?.downloadTimeoutMs ?? DEFAULT_DOWNLOAD_TIMEOUT_MS;
|
||||
this.onRequest = opts?.onRequest ?? (() => {});
|
||||
this.token = opts?.authToken;
|
||||
}
|
||||
|
||||
@@ -225,33 +214,6 @@ export class ApiClient {
|
||||
return h;
|
||||
}
|
||||
|
||||
// Send one request and report it to `onRequest`, with the status of its
|
||||
// response or the error it failed with. `reportedURL` is the URL given to
|
||||
// `onRequest`, when that must not be the one sent.
|
||||
private async send(
|
||||
url: string,
|
||||
init: RequestInit & { method: string },
|
||||
reportedURL = url,
|
||||
): Promise<Response> {
|
||||
let resp: Response;
|
||||
try {
|
||||
resp = await this._fetch(url, init);
|
||||
} catch (err) {
|
||||
this.onRequest({
|
||||
method: init.method,
|
||||
url: reportedURL,
|
||||
error: err,
|
||||
});
|
||||
throw err;
|
||||
}
|
||||
this.onRequest({
|
||||
method: init.method,
|
||||
url: reportedURL,
|
||||
status: resp.status,
|
||||
});
|
||||
return resp;
|
||||
}
|
||||
|
||||
private async throwIfError(resp: Response): Promise<void> {
|
||||
if (resp.ok) return;
|
||||
const requestID = resp.headers.get("x-request-id") ?? undefined;
|
||||
@@ -295,7 +257,7 @@ export class ApiClient {
|
||||
const url = buildURL(this.apiOrigin, path, query);
|
||||
// A GET changes nothing, so it is retried under the full policy.
|
||||
return withRetry(async () => {
|
||||
const resp = await this.send(url, {
|
||||
const resp = await this._fetch(url, {
|
||||
method: "GET",
|
||||
headers: this.headers(),
|
||||
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
||||
@@ -318,7 +280,7 @@ export class ApiClient {
|
||||
// 3xx status, which is not retried.
|
||||
return withRetry(
|
||||
async () => {
|
||||
const resp = await this.send(url, {
|
||||
const resp = await this._fetch(url, {
|
||||
method: "POST",
|
||||
headers: this.headers({
|
||||
"Content-Type": "application/json",
|
||||
@@ -359,23 +321,16 @@ export class ApiClient {
|
||||
// object at one key in one request, so replaying it either overwrites
|
||||
// the same bytes or lands them for the first time. There is no partial
|
||||
// state to protect, hence the full policy rather than the POST rule.
|
||||
// It is reported without its query, which holds the upload's
|
||||
// signature: a log keeps what it is given.
|
||||
const { origin, pathname } = new URL(presignedURL);
|
||||
await withRetry(async () => {
|
||||
const resp = await this.send(
|
||||
presignedURL,
|
||||
{
|
||||
method: "PUT",
|
||||
headers: {
|
||||
"Content-Type": "application/octet-stream",
|
||||
"Content-Length": String(data.length),
|
||||
},
|
||||
body: data,
|
||||
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
||||
const resp = await this._fetch(presignedURL, {
|
||||
method: "PUT",
|
||||
headers: {
|
||||
"Content-Type": "application/octet-stream",
|
||||
"Content-Length": String(data.length),
|
||||
},
|
||||
origin + pathname,
|
||||
);
|
||||
body: data,
|
||||
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
||||
});
|
||||
if (!resp.ok) {
|
||||
// An ApiError, not a bare Error: without the status on the
|
||||
// error the upload path cannot be classified at all, and a
|
||||
@@ -393,7 +348,7 @@ export class ApiClient {
|
||||
// Same replay and redirect rules as `postJSON`, for the same reasons.
|
||||
return withRetry(
|
||||
async () => {
|
||||
const resp = await this.send(url, {
|
||||
const resp = await this._fetch(url, {
|
||||
method: "PUT",
|
||||
headers: this.headers({
|
||||
"Content-Type": "application/json",
|
||||
@@ -439,7 +394,7 @@ export class ApiClient {
|
||||
// headers, when no bytes have arrived either.
|
||||
const deadline = idleDeadline(this.downloadTimeoutMs);
|
||||
try {
|
||||
const resp = await this.send(url, {
|
||||
const resp = await this._fetch(url, {
|
||||
method: "GET",
|
||||
headers: this.headers(),
|
||||
signal: deadline.signal,
|
||||
|
||||
+11
-29
@@ -14,9 +14,9 @@
|
||||
// YYYY/YYYY-MM/YYYY-MM-DD/
|
||||
// YYYY-MM-DD.<fileID>.<ext> the decrypted bytes (the save path)
|
||||
// YYYY-MM-DD.<fileID>.json per-file metadata sidecar, with
|
||||
// the file's ML data, its
|
||||
// keywords, and its original's
|
||||
// EXIF, XMP and dimensions
|
||||
// the file's ML data and its
|
||||
// original's EXIF, XMP and
|
||||
// dimensions
|
||||
// collections/<name>/<title> symlink to the original
|
||||
// collections/<name>.json per-collection metadata
|
||||
// account.json the account's email and user ID
|
||||
@@ -77,7 +77,6 @@ import { readFile } from "node:fs/promises";
|
||||
import { dirname, extname, join, relative, resolve } from "node:path";
|
||||
import lockfile from "proper-lockfile";
|
||||
|
||||
import { clipKeywords } from "./clip-keywords.js";
|
||||
import { removeLeftoverTempFiles } from "./download/index.js";
|
||||
import { describeError } from "./errors.js";
|
||||
import {
|
||||
@@ -102,10 +101,7 @@ import type { MLData } from "./mldata-fetch.js";
|
||||
import type { Collection, EnteFile, FileMetadata } from "./model/types.js";
|
||||
import { hasTransportCode } from "./retry.js";
|
||||
|
||||
// Reports one line of the backup's progress. `level` is "error" when the line
|
||||
// reports something that failed, such as a file or an album, and is left out
|
||||
// for every other line.
|
||||
export type ProgressCallback = (message: string, level?: "error") => void;
|
||||
export type ProgressCallback = (message: string) => void;
|
||||
|
||||
// The lock in a backup directory, as `lockBackupDirectory` returns it.
|
||||
export interface BackupLock {
|
||||
@@ -498,9 +494,8 @@ const imageMetadataFor = async (
|
||||
}
|
||||
};
|
||||
|
||||
// The file's JSON: its basic fields, its magic metadata, its ML data without
|
||||
// the CLIP embedding and the keywords that embedding gives, or the reason the
|
||||
// ML data is missing, and its image metadata.
|
||||
// The file's JSON: its basic fields, its magic metadata, its ML data or the
|
||||
// reason the ML data is missing, and its image metadata.
|
||||
const writeSidecar = (
|
||||
path: string,
|
||||
file: EnteFile,
|
||||
@@ -516,13 +511,7 @@ const writeSidecar = (
|
||||
};
|
||||
if (file.magicMetadata) meta.magicMetadata = file.magicMetadata;
|
||||
if (file.pubMagicMetadata) meta.pubMagicMetadata = file.pubMagicMetadata;
|
||||
if (ml.mlData) {
|
||||
const { clip: _clip, ...mlDataWithoutClip } = ml.mlData;
|
||||
meta.mlData = mlDataWithoutClip;
|
||||
const isVideo = file.metadata.fileType === "video";
|
||||
const keywords = clipKeywords(ml.mlData, isVideo);
|
||||
if (keywords) meta.keywords = keywords;
|
||||
}
|
||||
if (ml.mlData) meta.mlData = ml.mlData;
|
||||
if (ml.mlDataError) meta.mlDataError = ml.mlDataError;
|
||||
if (image.imageMetadata) meta.imageMetadata = image.imageMetadata;
|
||||
if (image.imageMetadataError) {
|
||||
@@ -665,7 +654,7 @@ const runLockedBackup = async (
|
||||
for (const record of lib.undecryptable()) {
|
||||
const error = `could not be decrypted: ${record.error}`;
|
||||
if (record.fileID === undefined) {
|
||||
log(`FAILED album ${record.collectionID}: ${error}`, "error");
|
||||
log(`FAILED album ${record.collectionID}: ${error}`);
|
||||
undecryptableAlbums.push(record.collectionID);
|
||||
} else if (scope.has(record.collectionID)) {
|
||||
const copies = memberships.get(record.fileID);
|
||||
@@ -710,7 +699,6 @@ const runLockedBackup = async (
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED verifying original ${file.metadata.title}: ${describeError(err)}`,
|
||||
"error",
|
||||
);
|
||||
recordFailure(
|
||||
file.id,
|
||||
@@ -744,7 +732,6 @@ const runLockedBackup = async (
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED original ${file.metadata.title}: ${describeError(err)}`,
|
||||
"error",
|
||||
);
|
||||
recordFailure(
|
||||
file.id,
|
||||
@@ -820,7 +807,7 @@ const runLockedBackup = async (
|
||||
await lib.fetchMLData();
|
||||
} catch (err) {
|
||||
mlDataFailure = err;
|
||||
log(`FAILED ML data: ${describeError(err)}`, "error");
|
||||
log(`FAILED ML data: ${describeError(err)}`);
|
||||
}
|
||||
for (const file of distinct.values()) {
|
||||
if (lock.lost) throw lock.lost;
|
||||
@@ -881,10 +868,7 @@ const runLockedBackup = async (
|
||||
downloadDirectory,
|
||||
);
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED removing old album directories: ${describeError(err)}`,
|
||||
"error",
|
||||
);
|
||||
log(`FAILED removing old album directories: ${describeError(err)}`);
|
||||
}
|
||||
|
||||
// An album whose folder or JSON cannot be written, such as one whose
|
||||
@@ -913,7 +897,6 @@ const runLockedBackup = async (
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED removing old links in ${c.name}: ${describeError(err)}`,
|
||||
"error",
|
||||
);
|
||||
}
|
||||
|
||||
@@ -932,7 +915,6 @@ const runLockedBackup = async (
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED symlink ${c.name}/${linkName}: ${describeError(err)}`,
|
||||
"error",
|
||||
);
|
||||
recordFailure(
|
||||
link.file.id,
|
||||
@@ -949,7 +931,7 @@ const runLockedBackup = async (
|
||||
metaFiles,
|
||||
);
|
||||
} catch (err) {
|
||||
log(`FAILED album ${c.name}: ${describeError(err)}`, "error");
|
||||
log(`FAILED album ${c.name}: ${describeError(err)}`);
|
||||
failedAlbums.push(c.id);
|
||||
}
|
||||
}
|
||||
|
||||
+127
-150
@@ -4,10 +4,7 @@
|
||||
// code; a thrown error is left to the caller. Nothing here calls
|
||||
// `process.exit`: `bin/quak.ts` wires these to the command line, and `run` in
|
||||
// `cli-run.ts` prints a thrown error as one line and exits once output has
|
||||
// drained. Output must stay byte-identical (see `cli-output.ts`). Each line on
|
||||
// stderr goes through `ctx.log` at its level (see `cli-log.ts`). Every command
|
||||
// that exits non-zero logs a fatal line saying why; one that exits 1 because
|
||||
// files or albums failed logs it last, saying how many and where to look.
|
||||
// drained. Output must stay byte-identical (see `cli-output.ts`).
|
||||
|
||||
import {
|
||||
copyFileSync,
|
||||
@@ -36,23 +33,14 @@ import {
|
||||
originalName,
|
||||
thumbnailName,
|
||||
} from "./cli-output.js";
|
||||
import { type Log, logRefresh } from "./cli-log.js";
|
||||
import {
|
||||
cachedFilesByID,
|
||||
freshCollections,
|
||||
freshFiles,
|
||||
freshFile,
|
||||
freshFilesByID,
|
||||
} from "./cli-read.js";
|
||||
import { freshCollections, freshFiles, freshFile } from "./cli-read.js";
|
||||
import { withExtension } from "./filename.js";
|
||||
import { runMetadataBackup } from "./metadata-backup.js";
|
||||
import type { Collection, EnteFile } from "./model/types.js";
|
||||
import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js";
|
||||
|
||||
export interface CliContext {
|
||||
stdout: { write(text: string): unknown };
|
||||
// Writes a line on stderr at a log level, if the run's level prints it.
|
||||
log: Log;
|
||||
stderr: { write(text: string): unknown };
|
||||
// Directory holding `session.json`.
|
||||
sessionDir: string;
|
||||
// The `--cache-dir` global, or undefined to let the library pick its
|
||||
@@ -94,16 +82,16 @@ const requireSession = (ctx: CliContext): Client | undefined => {
|
||||
try {
|
||||
client = ctx.loadSession(sessionPath(ctx));
|
||||
} catch (err) {
|
||||
ctx.log("fatal", err instanceof Error ? err.message : String(err));
|
||||
ctx.log(
|
||||
"fatal",
|
||||
`Run "quak logout" and then "quak login" to replace it.`,
|
||||
ctx.stderr.write(
|
||||
`${err instanceof Error ? err.message : err}\n` +
|
||||
`Run "quak logout" and then "quak login" to replace it.\n`,
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
if (!client) {
|
||||
ctx.log("fatal", `Not logged in. Run "quak login" first.`);
|
||||
ctx.log("fatal", `Session file: ${sessionPath(ctx)}`);
|
||||
ctx.stderr.write(
|
||||
`Not logged in. Run "quak login" first.\nSession file: ${sessionPath(ctx)}\n`,
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
return client;
|
||||
@@ -119,7 +107,7 @@ const parseID = (
|
||||
): number | undefined => {
|
||||
const id = Number(text);
|
||||
if (/^[0-9]+$/.test(text) && Number.isSafeInteger(id)) return id;
|
||||
ctx.log("fatal", `Invalid ${kind} ID: ${text}`);
|
||||
ctx.stderr.write(`Invalid ${kind} ID: ${text}\n`);
|
||||
return undefined;
|
||||
};
|
||||
|
||||
@@ -135,16 +123,18 @@ const readLibraryClient = (client: Client): LibraryClient => ({
|
||||
contentSource: () => client.contentSource(),
|
||||
});
|
||||
|
||||
// Open a library for a single point command. The aggressive background
|
||||
// precache (issue #48) is off, because a one-shot `collections` or `get` must
|
||||
// not start downloading the whole account. No refresh starts as the library
|
||||
// opens or on a timer. Each command asks for the refresh it needs.
|
||||
// `collections` and `files` with a collection ID need none (issues #210 and
|
||||
// #217). Nor do `get` and `get-thumb` on an empty cache (issue #222). Nor does
|
||||
// `helper fix-missing-thumbnails --file` on an empty cache (issue #223). While
|
||||
// a refresh runs, each collection whose files it has fetched is logged, so a
|
||||
// large account shows progress rather than nothing.
|
||||
const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
|
||||
// Open a library for a single point command: the aggressive background precache
|
||||
// (issue #48) is off — a one-shot `collections` or `get` must not start
|
||||
// downloading the whole account — and no refresh starts as it opens or on a
|
||||
// timer: each command asks for the one it needs, and `collections` needs none
|
||||
// (issue #210). While a refresh runs, each collection whose files it has
|
||||
// fetched is counted on stderr, unless `json` is set, so a large account shows
|
||||
// progress rather than nothing.
|
||||
const openReadLibrary = (
|
||||
ctx: CliContext,
|
||||
client: Client,
|
||||
opts: { json?: boolean } = {},
|
||||
): Promise<Library> =>
|
||||
Library.open({
|
||||
client: readLibraryClient(client),
|
||||
cacheDirectory: ctx.cacheDir,
|
||||
@@ -152,7 +142,12 @@ const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
|
||||
refreshIntervalSeconds: Infinity,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
onProgress: logRefresh(ctx.log),
|
||||
onProgress: (event) => {
|
||||
if (opts.json || event.status !== "progress") return;
|
||||
ctx.stderr.write(
|
||||
`Fetched the files of ${event.fetched} of ${event.total} collections\n`,
|
||||
);
|
||||
},
|
||||
});
|
||||
|
||||
export const loginCommand = async (ctx: CliContext): Promise<number> => {
|
||||
@@ -161,7 +156,7 @@ export const loginCommand = async (ctx: CliContext): Promise<number> => {
|
||||
const password =
|
||||
process.env.QUAK_PASSWORD ?? (await ctx.promptSecret("Password"));
|
||||
|
||||
ctx.log("info", "Authenticating...");
|
||||
ctx.stderr.write("Authenticating...\n");
|
||||
try {
|
||||
const client = await ctx.login({
|
||||
email,
|
||||
@@ -172,12 +167,11 @@ export const loginCommand = async (ctx: CliContext): Promise<number> => {
|
||||
|
||||
saveSession(ctx.sessionDir, client.toJSON());
|
||||
const info = client.whoami();
|
||||
ctx.log("info", `Logged in as ${info.email} (user ${info.userID})`);
|
||||
ctx.log("info", `Session saved to ${sessionPath(ctx)}`);
|
||||
ctx.stderr.write(`Logged in as ${info.email} (user ${info.userID})\n`);
|
||||
ctx.stderr.write(`Session saved to ${sessionPath(ctx)}\n`);
|
||||
} catch (err) {
|
||||
ctx.log(
|
||||
"fatal",
|
||||
`Login failed: ${err instanceof Error ? err.message : err}`,
|
||||
ctx.stderr.write(
|
||||
`Login failed: ${err instanceof Error ? err.message : err}\n`,
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
@@ -199,7 +193,7 @@ export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
|
||||
export const logoutCommand = async (ctx: CliContext): Promise<number> => {
|
||||
const path = sessionPath(ctx);
|
||||
if (!existsSync(path)) {
|
||||
ctx.log("info", "No session found.");
|
||||
ctx.stderr.write("No session found.\n");
|
||||
return 0;
|
||||
}
|
||||
await init();
|
||||
@@ -217,15 +211,16 @@ export const logoutCommand = async (ctx: CliContext): Promise<number> => {
|
||||
}
|
||||
unlinkSync(path);
|
||||
if (failure === undefined) {
|
||||
ctx.log("info", "Session ended on the server.");
|
||||
ctx.stderr.write("Session ended on the server.\n");
|
||||
} else {
|
||||
ctx.log("fatal", `Could not end the session on the server: ${failure}`);
|
||||
ctx.stderr.write(
|
||||
`Could not end the session on the server: ${failure}\n`,
|
||||
);
|
||||
}
|
||||
ctx.log("info", "Session deleted.");
|
||||
ctx.stderr.write("Session deleted.\n");
|
||||
if (cacheDir !== undefined) {
|
||||
ctx.log(
|
||||
"warning",
|
||||
`Cache directory ${cacheDir} still holds decrypted data; delete it to remove that data.`,
|
||||
ctx.stderr.write(
|
||||
`Cache directory ${cacheDir} still holds decrypted data; delete it to remove that data.\n`,
|
||||
);
|
||||
}
|
||||
return failure === undefined ? 0 : 1;
|
||||
@@ -274,32 +269,26 @@ export const collectionsCommand = async (
|
||||
}
|
||||
};
|
||||
|
||||
// Lists the files of the collection `collectionIDStr` names, or without it every
|
||||
// file in the account, each once (issue #211).
|
||||
export const filesCommand = async (
|
||||
ctx: CliContext,
|
||||
collectionIDStr: string | undefined,
|
||||
opts: { json?: boolean },
|
||||
opts: { collection: string; json?: boolean },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 3;
|
||||
const collectionID =
|
||||
collectionIDStr === undefined
|
||||
? undefined
|
||||
: parseID(ctx, "collection", collectionIDStr);
|
||||
if (collectionIDStr !== undefined && collectionID === undefined) return 1;
|
||||
const collectionID = parseID(ctx, "collection", opts.collection);
|
||||
if (collectionID === undefined) return 1;
|
||||
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
const lib = await openReadLibrary(ctx, client, opts);
|
||||
try {
|
||||
// Ask the server for current state (issue #36 amendment, issue #52).
|
||||
// Each file prints from its own decrypted metadata (raw title,
|
||||
// microsecond creationTime) via cli-output. The files print in the
|
||||
// order `freshFiles` returns them, not the projection's newest-first
|
||||
// order.
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52). Each file prints from its own decrypted
|
||||
// metadata (raw title, microsecond creationTime) via cli-output, and in
|
||||
// the pre-library CLI's enumeration order, not the projection's
|
||||
// newest-first order.
|
||||
const files = await freshFiles(lib, collectionID);
|
||||
if (!files) {
|
||||
ctx.log("fatal", `Collection ${collectionID} not found`);
|
||||
ctx.stderr.write(`Collection ${collectionID} not found\n`);
|
||||
return 1;
|
||||
}
|
||||
|
||||
@@ -333,19 +322,20 @@ export const getCommand = async (
|
||||
try {
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const file = await freshFile(lib, fileID);
|
||||
if (!file) {
|
||||
ctx.log("fatal", `File ${fileID} not found`);
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
ctx.stderr.write(`File ${fileID} not found\n`);
|
||||
return 1;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await lib.originalOf(file);
|
||||
const result = await photo.original();
|
||||
// Default name is the file's own title, as the pre-library CLI used
|
||||
// (not the editedName-preferring projection title) (issue #52).
|
||||
if (result.videoPath === undefined) {
|
||||
const outPath = opts.out ?? originalName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
ctx.log("info", `${result.bytes} bytes -> ${outPath}`);
|
||||
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
return 0;
|
||||
}
|
||||
// A live photo is written as its image and its video, each named after
|
||||
@@ -359,16 +349,17 @@ export const getCommand = async (
|
||||
? originalName(file, videoExt)
|
||||
: withExtension(opts.out, videoExt);
|
||||
if (imageOut.toLowerCase() === videoOut.toLowerCase()) {
|
||||
ctx.log(
|
||||
"fatal",
|
||||
`File ${fileID} is a live photo, and its video would also be written to ${imageOut}`,
|
||||
ctx.stderr.write(
|
||||
`File ${fileID} is a live photo, and its video would also be written to ${imageOut}\n`,
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
copyFileSync(result.path, imageOut);
|
||||
copyFileSync(result.videoPath, videoOut);
|
||||
ctx.log("info", `${result.bytes} bytes -> ${imageOut}`);
|
||||
ctx.log("info", `${statSync(videoOut).size} bytes -> ${videoOut}`);
|
||||
ctx.stderr.write(
|
||||
`${result.bytes} bytes -> ${imageOut}\n` +
|
||||
`${statSync(videoOut).size} bytes -> ${videoOut}\n`,
|
||||
);
|
||||
return 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
@@ -390,18 +381,19 @@ export const getThumbCommand = async (
|
||||
try {
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const file = await freshFile(lib, fileID);
|
||||
if (!file) {
|
||||
ctx.log("fatal", `File ${fileID} not found`);
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
ctx.stderr.write(`File ${fileID} not found\n`);
|
||||
return 1;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await lib.thumbnailOf(file);
|
||||
const result = await photo.thumbnail();
|
||||
// Default name is thumb_<file's own title>, as the pre-library CLI
|
||||
// used (not the projection title) (issue #52).
|
||||
const outPath = opts.out ?? thumbnailName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
ctx.log("info", `${result.bytes} bytes -> ${outPath}`);
|
||||
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
return 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
@@ -423,14 +415,9 @@ export const backupMetadataCommand = async (
|
||||
await lib.fresh();
|
||||
const result = await runMetadataBackup(lib, client, dir, {
|
||||
exif: opts.exif || opts.all,
|
||||
onProgress: (msg, level) => ctx.log(level ?? "info", msg),
|
||||
onProgress: (msg) => ctx.stderr.write(msg + "\n"),
|
||||
});
|
||||
if (result.filesWithMLDataError === 0) return 0;
|
||||
ctx.log(
|
||||
"fatal",
|
||||
`quak: the ML data of ${result.filesWithMLDataError} file(s) could not be read; see mlDataError in their JSON in ${dir}`,
|
||||
);
|
||||
return 1;
|
||||
return result.filesWithMLDataError > 0 ? 1 : 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
@@ -445,7 +432,7 @@ export const backupCommand = async (
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 3;
|
||||
|
||||
ctx.log("info", "Starting backup...");
|
||||
ctx.stderr.write("Starting backup...\n");
|
||||
// The lock is taken before the library opens and starts its refresh, so a
|
||||
// run refused while another backup of `dir` runs sends no request.
|
||||
let lock: BackupLock;
|
||||
@@ -453,7 +440,7 @@ export const backupCommand = async (
|
||||
lock = await lockBackupDirectory(dir);
|
||||
} catch (err) {
|
||||
if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;
|
||||
ctx.log("fatal", `quak: ${(err as Error).message}`);
|
||||
ctx.stderr.write(`quak: ${(err as Error).message}\n`);
|
||||
return 2;
|
||||
}
|
||||
try {
|
||||
@@ -468,48 +455,45 @@ export const backupCommand = async (
|
||||
refreshIntervalSeconds: Infinity,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
onProgress: logRefresh(ctx.log),
|
||||
});
|
||||
try {
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: dir,
|
||||
lock,
|
||||
verify: opts.verify,
|
||||
onProgress: (msg, level) => ctx.log(level ?? "info", msg),
|
||||
onProgress: (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
},
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(JSON.stringify(result, null, 2) + "\n");
|
||||
} else {
|
||||
ctx.log("info", "\n--- Backup complete ---");
|
||||
ctx.log("info", ` Total files: ${result.totalFiles}`);
|
||||
ctx.log("info", ` Downloaded: ${result.downloaded}`);
|
||||
ctx.log("info", ` Skipped: ${result.skipped}`);
|
||||
ctx.stderr.write("\n--- Backup complete ---\n");
|
||||
ctx.stderr.write(` Total files: ${result.totalFiles}\n`);
|
||||
ctx.stderr.write(` Downloaded: ${result.downloaded}\n`);
|
||||
ctx.stderr.write(` Skipped: ${result.skipped}\n`);
|
||||
if (opts.verify) {
|
||||
ctx.log("info", ` Verified: ${result.verified}`);
|
||||
ctx.log("info", ` Mismatched: ${result.mismatched}`);
|
||||
ctx.log("info", ` Unchecked: ${result.unchecked}`);
|
||||
ctx.stderr.write(` Verified: ${result.verified}\n`);
|
||||
ctx.stderr.write(` Mismatched: ${result.mismatched}\n`);
|
||||
ctx.stderr.write(` Unchecked: ${result.unchecked}\n`);
|
||||
}
|
||||
ctx.log("info", ` Failed: ${result.failed}`);
|
||||
ctx.stderr.write(` Failed: ${result.failed}\n`);
|
||||
if (result.errors.length > 0) {
|
||||
ctx.log("error", "\nFailed files:");
|
||||
ctx.stderr.write("\nFailed files:\n");
|
||||
for (const e of result.errors) {
|
||||
ctx.log(
|
||||
"error",
|
||||
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}`,
|
||||
ctx.stderr.write(
|
||||
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const albums =
|
||||
result.undecryptableAlbums.length + result.failedAlbums.length;
|
||||
if (result.failed === 0 && albums === 0) return 0;
|
||||
ctx.log(
|
||||
"fatal",
|
||||
`quak: ${result.failed} file(s) and ${albums} album(s) failed; see ${join(dir, "failures.json")} and the FAILED lines`,
|
||||
);
|
||||
return 1;
|
||||
return result.failed > 0 ||
|
||||
result.undecryptableAlbums.length > 0 ||
|
||||
result.failedAlbums.length > 0
|
||||
? 1
|
||||
: 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
@@ -525,24 +509,23 @@ export const listMissingThumbnailsCommand = async (
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 3;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
const lib = await openReadLibrary(ctx, client, opts);
|
||||
try {
|
||||
// Refresh first so files added since the cache was written are
|
||||
// checked; a failed refresh throws.
|
||||
await lib.fresh();
|
||||
const missing = await listMissingThumbnails(lib, client, (msg, level) =>
|
||||
ctx.log(level ?? "info", msg),
|
||||
);
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(JSON.stringify(missing, null, 2) + "\n");
|
||||
} else {
|
||||
if (missing.length === 0) {
|
||||
ctx.log("info", "No missing thumbnails found.");
|
||||
ctx.stderr.write("No missing thumbnails found.\n");
|
||||
} else {
|
||||
ctx.log(
|
||||
"info",
|
||||
`\n${missing.length} file(s) with missing thumbnails:`,
|
||||
ctx.stderr.write(
|
||||
`\n${missing.length} file(s) with missing thumbnails:\n`,
|
||||
);
|
||||
for (const m of missing) {
|
||||
ctx.stdout.write(
|
||||
@@ -571,39 +554,33 @@ export const fixMissingThumbnailsCommand = async (
|
||||
if (fileID === undefined) return 1;
|
||||
fileIDs.push(fileID);
|
||||
}
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
const lib = await openReadLibrary(ctx, client, opts);
|
||||
try {
|
||||
let found: Map<number, { file: EnteFile; collection: Collection }>;
|
||||
if (fileIDs.length > 0) {
|
||||
found = await freshFilesByID(lib, fileIDs);
|
||||
} else {
|
||||
// Refresh first so files added since the cache was written are
|
||||
// checked; a failed refresh throws.
|
||||
await lib.fresh();
|
||||
ctx.log("info", "Scanning for missing thumbnails...");
|
||||
const missing = await listMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
(msg, level) => ctx.log(level ?? "info", msg),
|
||||
);
|
||||
// Refresh first so files added since the cache was written are found;
|
||||
// a failed refresh throws.
|
||||
await lib.fresh();
|
||||
if (fileIDs.length === 0) {
|
||||
ctx.stderr.write("Scanning for missing thumbnails...\n");
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
});
|
||||
fileIDs = missing.map((m) => m.fileID);
|
||||
if (fileIDs.length === 0) {
|
||||
ctx.log("info", "No missing thumbnails found.");
|
||||
ctx.stderr.write("No missing thumbnails found.\n");
|
||||
return 0;
|
||||
}
|
||||
ctx.log("info", `Found ${fileIDs.length} file(s) to fix.`);
|
||||
found = cachedFilesByID(lib, fileIDs);
|
||||
ctx.stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
|
||||
}
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
fileIDs,
|
||||
found,
|
||||
(msg, level) => ctx.log(level ?? "info", msg),
|
||||
(msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
},
|
||||
);
|
||||
|
||||
const failed = results.filter((r) => r.status === "failed").length;
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(JSON.stringify(results, null, 2) + "\n");
|
||||
} else {
|
||||
@@ -611,30 +588,30 @@ export const fixMissingThumbnailsCommand = async (
|
||||
const skipped = results.filter(
|
||||
(r) => r.status === "skipped",
|
||||
).length;
|
||||
ctx.log("info", `\n--- Done ---`);
|
||||
ctx.log("info", ` Fixed: ${fixed}`);
|
||||
ctx.log("info", ` Skipped: ${skipped}`);
|
||||
ctx.log("info", ` Failed: ${failed}`);
|
||||
const failed = results.filter((r) => r.status === "failed").length;
|
||||
ctx.stderr.write(`\n--- Done ---\n`);
|
||||
ctx.stderr.write(` Fixed: ${fixed}\n`);
|
||||
ctx.stderr.write(` Skipped: ${skipped}\n`);
|
||||
ctx.stderr.write(` Failed: ${failed}\n`);
|
||||
if (skipped > 0) {
|
||||
ctx.log("info", "\nSkipped:");
|
||||
ctx.stderr.write("\nSkipped:\n");
|
||||
for (const r of results.filter((r) => r.status === "skipped")) {
|
||||
ctx.log("info", ` ${r.fileID}\t${r.title}\t${r.reason}`);
|
||||
ctx.stderr.write(
|
||||
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
if (failed > 0) {
|
||||
ctx.log("error", "\nFailed files:");
|
||||
ctx.stderr.write("\nFailed files:\n");
|
||||
for (const r of results.filter((r) => r.status === "failed")) {
|
||||
ctx.log("error", ` ${r.fileID}\t${r.title}\t${r.reason}`);
|
||||
ctx.stderr.write(
|
||||
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (failed === 0) return 0;
|
||||
ctx.log(
|
||||
"fatal",
|
||||
`quak: ${failed} file(s) failed; see the "Failed files:" list, or the JSON with --json`,
|
||||
);
|
||||
return 1;
|
||||
return results.some((r) => r.status === "failed") ? 1 : 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
|
||||
-102
@@ -1,102 +0,0 @@
|
||||
// The CLI's log level, and the one function the commands write stderr lines
|
||||
// through.
|
||||
//
|
||||
// Every line quak logs goes to stderr, at one of six levels, from the
|
||||
// quietest: fatal, error, warning, info, debug, trace. A run prints the lines
|
||||
// at its own level and at every quieter one. Its level is info unless `-v` or
|
||||
// `-q` asks for another (see "What quak prints" in the README). A command that
|
||||
// runs and exits non-zero logs a fatal line saying why, so even `-q`, which
|
||||
// prints only fatal lines, never leaves such an exit unexplained. An error
|
||||
// line reports something that failed, such as a file or an album, and a
|
||||
// warning something that did not fail but needs the user's attention. stdout
|
||||
// carries a command's output and the prompts of `login`, and is not a log.
|
||||
// Two things on stderr are not log lines: the `error:` message for a
|
||||
// command line that quak cannot parse, and the usage printed when no command
|
||||
// is given.
|
||||
//
|
||||
// The library itself writes no lines: its layers report through callbacks, and
|
||||
// the CLI logs what they report. `logRequests` gives the API client options
|
||||
// that log each request and each retry, and `logRefresh` the library's
|
||||
// `onProgress` that logs what a refresh and an ML data fetch report.
|
||||
|
||||
import type { ApiClientOptions } from "./api/client.js";
|
||||
import { describeError } from "./errors.js";
|
||||
import type { RefreshEvent } from "./library/index.js";
|
||||
import type { RetryOptions } from "./retry.js";
|
||||
|
||||
const LEVELS = ["fatal", "error", "warning", "info", "debug", "trace"] as const;
|
||||
|
||||
export type LogLevel = (typeof LEVELS)[number];
|
||||
|
||||
// Writes `line` on stderr when the run's level prints lines at `level`.
|
||||
export type Log = (level: LogLevel, line: string) => void;
|
||||
|
||||
// The level asked for by `-v` given `verbose` times, or by `-q`: info without
|
||||
// either, debug for `-v`, trace for `-vv` and more, fatal for `-q`.
|
||||
// `bin/quak.ts` refuses `-q` and `-v` together.
|
||||
export const logLevel = (verbose: number, quiet: boolean): LogLevel => {
|
||||
if (quiet) return "fatal";
|
||||
if (verbose === 0) return "info";
|
||||
return verbose === 1 ? "debug" : "trace";
|
||||
};
|
||||
|
||||
// The log that writes each line at `level` or quieter to `stderr`, ending it
|
||||
// with a newline.
|
||||
export const stderrLog =
|
||||
(stderr: { write(text: string): unknown }, level: LogLevel): Log =>
|
||||
(lineLevel, line) => {
|
||||
if (LEVELS.indexOf(lineLevel) <= LEVELS.indexOf(level)) {
|
||||
stderr.write(line + "\n");
|
||||
}
|
||||
};
|
||||
|
||||
// API client options that log each request at trace, with the status of its
|
||||
// response or the error it failed with, and each retry at debug, with why and
|
||||
// how long it waits. `retry` is the retry policy, the default when not given.
|
||||
export const logRequests = (
|
||||
log: Log,
|
||||
retry?: RetryOptions,
|
||||
): ApiClientOptions => ({
|
||||
retry: {
|
||||
...retry,
|
||||
onRetry: (err, attempt, delayMs) =>
|
||||
log(
|
||||
"debug",
|
||||
`Attempt ${attempt} failed, retrying in ${Math.round(delayMs)} ms: ${describeError(err)}`,
|
||||
),
|
||||
},
|
||||
onRequest: ({ method, url, status, error }) =>
|
||||
log(
|
||||
"trace",
|
||||
`${method} ${url}: ${error === undefined ? `HTTP ${status}` : describeError(error)}`,
|
||||
),
|
||||
});
|
||||
|
||||
// How a log line names each operation the library reports the start and end
|
||||
// of.
|
||||
const OPERATION_NAMES: Record<RefreshEvent["operation"], string> = {
|
||||
refresh: "Refresh",
|
||||
fetchMLData: "ML data fetch",
|
||||
precacheThumbnails: "Thumbnail precache",
|
||||
precacheOriginals: "Original precache",
|
||||
};
|
||||
|
||||
// A library `onProgress` that logs, at info, each collection whose files a
|
||||
// refresh has fetched, and at debug the start and end of each refresh and ML
|
||||
// data fetch, with the error of one that failed.
|
||||
export const logRefresh =
|
||||
(log: Log) =>
|
||||
(event: RefreshEvent): void => {
|
||||
if (event.status === "progress") {
|
||||
log(
|
||||
"info",
|
||||
`Fetched the files of ${event.fetched} of ${event.total} collections`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
const error = event.error === undefined ? "" : `: ${event.error}`;
|
||||
log(
|
||||
"debug",
|
||||
`${OPERATION_NAMES[event.operation]} ${event.status}${error}`,
|
||||
);
|
||||
};
|
||||
+33
-118
@@ -1,41 +1,35 @@
|
||||
// How the CLI's read commands obtain current data.
|
||||
//
|
||||
// `collections`, `files`, `get`, and `get-thumb` must answer for
|
||||
// `collections`, `files --collection`, `get`, and `get-thumb` must answer for
|
||||
// the account's state at the moment the command runs, not for whatever the
|
||||
// local cache last happened to hold (owner amendment, issue #36). `get`,
|
||||
// `get-thumb` and `files` without a collection ID therefore force a server
|
||||
// round-trip through `Library.fresh()` and only then read, so a change made
|
||||
// elsewhere is visible. `collections` asks the library for the collection list
|
||||
// alone through `Library.freshCollections()` (issue #210): a refresh would also
|
||||
// fetch the files of every collection, which on a large account takes long and
|
||||
// which `collections` does not print. `files` with a collection ID reads that
|
||||
// list too, then that one collection's files through `Library.freshFiles()`
|
||||
// (issue #217). On an empty cache, `get` and `get-thumb` read that list, then
|
||||
// one collection's files at a time, until they find the file (issue #222).
|
||||
// `helper fix-missing-thumbnails --file` does the same until it finds every
|
||||
// file it names (issue #223).
|
||||
// local cache last happened to hold (owner amendment, issue #36). `files`,
|
||||
// `get` and `get-thumb` therefore force a server round-trip through
|
||||
// `Library.fresh()` and only then read — so a collection, file, or metadata
|
||||
// change made elsewhere is visible. `collections` asks the library for the
|
||||
// collection list alone through `Library.freshCollections()` (issue #210): a
|
||||
// refresh would also fetch the files of every collection, which on a large
|
||||
// account takes long and which `collections` does not print.
|
||||
//
|
||||
// `collections` and `files` with a collection ID list in the server's order.
|
||||
// `files` without one lists in the library's own enumeration order
|
||||
// (`listFiles()`). Neither command uses the `albums`/`photos` projection's
|
||||
// newest-first order, which re-sorts the rows. The field values still come
|
||||
// from each record's raw metadata via `cli-output.ts`. `files` without a
|
||||
// collection ID lists every file in the account once (issue #211): the library
|
||||
// keeps a record of a file for each collection that holds it, and only the
|
||||
// first collection's record is listed.
|
||||
// `collections` lists in the server's order and `files` in the library's own
|
||||
// enumeration order (`listFiles()`), the orders the pre-library CLI printed,
|
||||
// rather than the `albums`/`photos` projection's newest-first order, which
|
||||
// re-sorts the rows. The field values still come from each record's raw
|
||||
// metadata via `cli-output.ts`.
|
||||
|
||||
import type { Collection, EnteFile } from "./model/types.js";
|
||||
import type { Photo, PhotosAPI } from "./library/index.js";
|
||||
|
||||
// The slice of `Library` these helpers read. `Library` satisfies it
|
||||
// structurally; a test can drive them with a stand-in that records each fresh
|
||||
// read and serves records in a known enumeration order.
|
||||
// structurally; a test can drive them with a stand-in that records the
|
||||
// `fresh()` and `freshCollections()` calls and serves records in a known
|
||||
// enumeration order.
|
||||
export interface FreshReadLibrary {
|
||||
fresh(): Promise<unknown>;
|
||||
freshCollections(): Promise<Collection[]>;
|
||||
freshFiles(collection: Collection): Promise<EnteFile[]>;
|
||||
listCollections(): Collection[];
|
||||
getCollection(id: number): Collection | undefined;
|
||||
listFiles(collectionID: number): EnteFile[];
|
||||
getFileByID(fileID: number): EnteFile | undefined;
|
||||
photos: Pick<PhotosAPI, "byID">;
|
||||
}
|
||||
|
||||
// Every live collection, current as of one collections request, in the
|
||||
@@ -44,107 +38,28 @@ export const freshCollections = (
|
||||
lib: FreshReadLibrary,
|
||||
): Promise<Collection[]> => lib.freshCollections();
|
||||
|
||||
// The files of one collection, as the server holds them now, in the server's
|
||||
// The files of one collection, current as of a forced refresh, in enumeration
|
||||
// order. `undefined` (not an empty list) when the collection does not exist, so
|
||||
// the caller can tell "no such collection" from "an empty collection". Without
|
||||
// a collection ID, every file in the account as of a forced refresh, each once,
|
||||
// as held by the first collection that holds it.
|
||||
// the caller can tell "no such collection" from "an empty collection".
|
||||
export const freshFiles = async (
|
||||
lib: FreshReadLibrary,
|
||||
collectionID?: number,
|
||||
collectionID: number,
|
||||
): Promise<EnteFile[] | undefined> => {
|
||||
if (collectionID !== undefined) {
|
||||
const collections = await lib.freshCollections();
|
||||
const collection = collections.find((c) => c.id === collectionID);
|
||||
return collection ? lib.freshFiles(collection) : undefined;
|
||||
}
|
||||
await lib.fresh();
|
||||
const listed = new Set<number>();
|
||||
const files: EnteFile[] = [];
|
||||
for (const collection of lib.listCollections()) {
|
||||
for (const file of lib.listFiles(collection.id)) {
|
||||
if (listed.has(file.id)) continue;
|
||||
listed.add(file.id);
|
||||
files.push(file);
|
||||
}
|
||||
}
|
||||
return files;
|
||||
if (!lib.getCollection(collectionID)) return undefined;
|
||||
return lib.listFiles(collectionID);
|
||||
};
|
||||
|
||||
// One file as the server holds it now, or `undefined` when no collection holds
|
||||
// it. When the cache holds the account's collections, it refreshes and looks
|
||||
// the file up in the cache. On an empty cache, it reads the collection list,
|
||||
// then one collection's files at a time, and stops at the first collection
|
||||
// that holds the file. That search stores nothing.
|
||||
// One file, current as of a forced refresh, resolved to both its content
|
||||
// handle (`Photo`, for fetching bytes) and its raw record (`EnteFile`, for the
|
||||
// default output name and field values). `undefined` when the file is unknown.
|
||||
export const freshFile = async (
|
||||
lib: FreshReadLibrary,
|
||||
fileID: number,
|
||||
): Promise<EnteFile | undefined> => {
|
||||
// Only a refresh stores collections, so a cache holding any holds them all.
|
||||
if (lib.listCollections().length > 0) {
|
||||
await lib.fresh();
|
||||
return lib.getFileByID(fileID);
|
||||
}
|
||||
for (const collection of await lib.freshCollections()) {
|
||||
const files = await lib.freshFiles(collection);
|
||||
const file = files.find((f) => f.id === fileID);
|
||||
if (file) return file;
|
||||
}
|
||||
return undefined;
|
||||
};
|
||||
|
||||
// The most recently updated collection first, and the higher ID first on a
|
||||
// tie. `Library.albums.list()` lists albums in this order.
|
||||
const byNewestCollection = (a: Collection, b: Collection): number =>
|
||||
b.updationTime - a.updationTime || b.id - a.id;
|
||||
|
||||
// Each file named by `fileIDs` that the cache holds, mapped to its record and
|
||||
// the collection it was found in. Collections are searched in
|
||||
// `byNewestCollection` order, and a file is found in the first that holds it.
|
||||
// So `helper fix-missing-thumbnails` names a file after the same album as
|
||||
// `helper list-missing-thumbnails`. It reads the cache alone, so the caller
|
||||
// refreshes first.
|
||||
export const cachedFilesByID = (
|
||||
lib: FreshReadLibrary,
|
||||
fileIDs: number[],
|
||||
): Map<number, { file: EnteFile; collection: Collection }> => {
|
||||
const wanted = new Set(fileIDs);
|
||||
const found = new Map<number, { file: EnteFile; collection: Collection }>();
|
||||
const collections = [...lib.listCollections()].sort(byNewestCollection);
|
||||
for (const collection of collections) {
|
||||
for (const file of lib.listFiles(collection.id)) {
|
||||
if (wanted.has(file.id) && !found.has(file.id)) {
|
||||
found.set(file.id, { file, collection });
|
||||
}
|
||||
}
|
||||
}
|
||||
return found;
|
||||
};
|
||||
|
||||
// Each file named by `fileIDs` as the server holds it now, mapped to its record
|
||||
// and collection as `cachedFilesByID` maps it. When the cache holds the
|
||||
// account's collections, it refreshes and looks the files up in the cache. On
|
||||
// an empty cache, it reads the collection list, then one collection's files at
|
||||
// a time, and stops once it has found every file. That search stores nothing.
|
||||
export const freshFilesByID = async (
|
||||
lib: FreshReadLibrary,
|
||||
fileIDs: number[],
|
||||
): Promise<Map<number, { file: EnteFile; collection: Collection }>> => {
|
||||
// Only a refresh stores collections, so a cache holding any holds them all.
|
||||
if (lib.listCollections().length > 0) {
|
||||
await lib.fresh();
|
||||
return cachedFilesByID(lib, fileIDs);
|
||||
}
|
||||
const wanted = new Set(fileIDs);
|
||||
const found = new Map<number, { file: EnteFile; collection: Collection }>();
|
||||
const collections = await lib.freshCollections();
|
||||
for (const collection of [...collections].sort(byNewestCollection)) {
|
||||
if (found.size === wanted.size) break;
|
||||
for (const file of await lib.freshFiles(collection)) {
|
||||
if (wanted.has(file.id) && !found.has(file.id)) {
|
||||
found.set(file.id, { file, collection });
|
||||
}
|
||||
}
|
||||
}
|
||||
return found;
|
||||
): Promise<{ photo: Photo; file: EnteFile } | undefined> => {
|
||||
await lib.fresh();
|
||||
const photo = lib.photos.byID({ fileID });
|
||||
const file = lib.getFileByID(fileID);
|
||||
if (!photo || !file) return undefined;
|
||||
return { photo, file };
|
||||
};
|
||||
+9
-12
@@ -2,23 +2,21 @@
|
||||
|
||||
import type { Writable } from "node:stream";
|
||||
import { ApiError } from "./api/client.js";
|
||||
import type { Log } from "./cli-log.js";
|
||||
import { describeError } from "./errors.js";
|
||||
|
||||
// Run a command and exit with its code once stdout/stderr have drained.
|
||||
// Exiting before the drain can truncate piped output, and the library can keep
|
||||
// the event loop alive after a command returns, so a plain return could hang.
|
||||
// An error the command throws is logged as one fatal `quak: MESSAGE` line,
|
||||
// without the stack trace, and exits 1. MESSAGE is the error as
|
||||
// `describeError` gives it, so it carries the reason Node's `fetch` puts on
|
||||
// `cause`. A 401 from the server means it no longer accepts the saved session:
|
||||
// that logs one fatal line saying to log in again and exits 3, as a missing or
|
||||
// corrupt session file does.
|
||||
// An error the command throws is printed as one `quak: MESSAGE` line, without
|
||||
// the stack trace, and exits 1. MESSAGE is the error as `describeError` gives
|
||||
// it, so it carries the reason Node's `fetch` puts on `cause`. A 401 from the
|
||||
// server means it no longer accepts the saved session: that prints one line
|
||||
// saying to log in again and exits 3, as a missing or corrupt session file
|
||||
// does.
|
||||
export const run = async (
|
||||
command: Promise<number>,
|
||||
stdout: Writable,
|
||||
stderr: Writable,
|
||||
log: Log,
|
||||
exit: (code: number) => void,
|
||||
): Promise<void> => {
|
||||
let code: number;
|
||||
@@ -26,13 +24,12 @@ export const run = async (
|
||||
code = await command;
|
||||
} catch (err) {
|
||||
if (err instanceof ApiError && err.status === 401) {
|
||||
log(
|
||||
"fatal",
|
||||
`quak: the saved session is no longer valid; run "quak login"`,
|
||||
stderr.write(
|
||||
`quak: the saved session is no longer valid; run "quak login"\n`,
|
||||
);
|
||||
code = 3;
|
||||
} else {
|
||||
log("fatal", `quak: ${describeError(err)}`);
|
||||
stderr.write(`quak: ${describeError(err)}\n`);
|
||||
code = 1;
|
||||
}
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,57 +0,0 @@
|
||||
// Keywords for a file from its CLIP embedding.
|
||||
//
|
||||
// Ente stores no keywords. A file's ML data holds a CLIP embedding, the 512
|
||||
// numbers Ente's image model gave the photo, and Ente's photos app sorts files
|
||||
// into the categories of its Discover section with it: a file falls in a
|
||||
// category when the cosine similarity of its CLIP embedding and the text
|
||||
// embedding of the category's query is at least the category's `minScore`,
|
||||
// unless the file is a video and the category leaves out videos. A file's
|
||||
// keywords are the titles of the categories it falls in. The categories and
|
||||
// their text embeddings are in `clip-keyword-vectors.ts`.
|
||||
|
||||
import { DISCOVER_CATEGORIES } from "./clip-keyword-vectors.js";
|
||||
import { clipEmbedding } from "./library/mldata.js";
|
||||
import type { MLData } from "./mldata-fetch.js";
|
||||
|
||||
// How many numbers a CLIP embedding from Ente's image model has.
|
||||
const EMBEDDING_LENGTH = 512;
|
||||
|
||||
const norm = (vector: number[]): number => Math.hypot(...vector);
|
||||
|
||||
const cosineSimilarity = (a: number[], b: number[]): number => {
|
||||
let dot = 0;
|
||||
a.forEach((value, i) => {
|
||||
dot += value * (b[i] ?? 0);
|
||||
});
|
||||
return dot / (norm(a) * norm(b));
|
||||
};
|
||||
|
||||
// The titles of the Discover categories a file falls in, highest score first,
|
||||
// or undefined when its ML data has no CLIP embedding of 512 finite numbers
|
||||
// that are not all zero. Tests pass their own `categories`.
|
||||
export const clipKeywords = (
|
||||
mlData: MLData,
|
||||
isVideo: boolean,
|
||||
categories = DISCOVER_CATEGORIES,
|
||||
): string[] | undefined => {
|
||||
const embedding = clipEmbedding(mlData);
|
||||
if (
|
||||
!embedding ||
|
||||
embedding.length !== EMBEDDING_LENGTH ||
|
||||
norm(embedding) === 0
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const matches: { title: string; score: number }[] = [];
|
||||
for (const category of categories) {
|
||||
if (isVideo && !category.showVideo) continue;
|
||||
const score = cosineSimilarity(embedding, category.textEmbedding);
|
||||
if (score >= category.minScore) {
|
||||
matches.push({ title: category.title, score });
|
||||
}
|
||||
}
|
||||
return matches
|
||||
.sort((a, b) => b.score - a.score)
|
||||
.map((match) => match.title);
|
||||
};
|
||||
+8
-8
@@ -92,10 +92,7 @@ export const shortenKeepingExtension = (
|
||||
// short so that the whole fits in a file name, keeping the title's extension
|
||||
// whenever it fits. `ext` (".mov"), when given, takes the place of the
|
||||
// title's extension before the name is cut short; a live photo's image and
|
||||
// video are each named with their own. The cut can leave a device name
|
||||
// (`CONSOLE.<long extension>` becomes `CON.<long extension>`); that name gets
|
||||
// a leading `_`, as `sanitizeFileName` gives one, and is cut a byte shorter to
|
||||
// leave room for it.
|
||||
// video are each named with their own.
|
||||
export const defaultFileName = (
|
||||
title: string,
|
||||
fileID: number,
|
||||
@@ -104,8 +101,11 @@ export const defaultFileName = (
|
||||
const { prefix = "", ext } = opts;
|
||||
const safe = sanitizeFileName(title, `file-${fileID}`);
|
||||
const name = ext === undefined ? safe : withExtension(safe, ext);
|
||||
const room = MAX_NAME_BYTES - Buffer.byteLength(prefix);
|
||||
const cut = prefix + shortenKeepingExtension(name, room);
|
||||
if (!RESERVED_DEVICE_NAME.test(cut)) return cut;
|
||||
return `_${prefix}${shortenKeepingExtension(name, room - 1)}`;
|
||||
return (
|
||||
prefix +
|
||||
shortenKeepingExtension(
|
||||
name,
|
||||
MAX_NAME_BYTES - Buffer.byteLength(prefix),
|
||||
)
|
||||
);
|
||||
};
|
||||
+11
-38
@@ -581,34 +581,14 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
fileID: number,
|
||||
opts?: ContentOptions,
|
||||
): Promise<ContentResult> {
|
||||
const file = this.fileOrThrow(fileID);
|
||||
return this.get(file, "original", "on-demand", opts?.onProgress);
|
||||
return this.get(fileID, "original", "on-demand", opts?.onProgress);
|
||||
}
|
||||
|
||||
async thumbnail(
|
||||
fileID: number,
|
||||
opts?: ContentOptions,
|
||||
): Promise<ContentResult> {
|
||||
const file = this.fileOrThrow(fileID);
|
||||
return this.get(file, "thumbnail", "on-demand", opts?.onProgress);
|
||||
}
|
||||
|
||||
// The original of `file`, fetched and cached as `original` does. `file` is
|
||||
// used as given, so `getFile` need not know it.
|
||||
async originalOf(
|
||||
file: EnteFile,
|
||||
opts?: ContentOptions,
|
||||
): Promise<ContentResult> {
|
||||
return this.get(file, "original", "on-demand", opts?.onProgress);
|
||||
}
|
||||
|
||||
// The thumbnail of `file`, fetched and cached as `thumbnail` does. `file`
|
||||
// is used as given, so `getFile` need not know it.
|
||||
async thumbnailOf(
|
||||
file: EnteFile,
|
||||
opts?: ContentOptions,
|
||||
): Promise<ContentResult> {
|
||||
return this.get(file, "thumbnail", "on-demand", opts?.onProgress);
|
||||
return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
|
||||
}
|
||||
|
||||
// Put the original at the save path of `file` under the download directory
|
||||
@@ -635,7 +615,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
destination: string,
|
||||
): Promise<ContentResult> {
|
||||
const result = await this.acquire(
|
||||
this.fileOrThrow(fileID),
|
||||
fileID,
|
||||
"original",
|
||||
"on-demand",
|
||||
undefined,
|
||||
@@ -658,7 +638,8 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
fileID: number,
|
||||
destination: string,
|
||||
): Promise<{ path: string; videoPath?: string }> {
|
||||
const file = this.fileOrThrow(fileID);
|
||||
const file = this.getFile(fileID);
|
||||
if (!file) throw new Error(`content cache: unknown file ${fileID}`);
|
||||
return this.pools.content.run(
|
||||
async () => {
|
||||
const fetched = await this.fetchInto(
|
||||
@@ -740,8 +721,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
onProgress: ((event: EnsureEvent) => void) | undefined,
|
||||
): Promise<EnsureResult> {
|
||||
try {
|
||||
const file = this.fileOrThrow(fileID);
|
||||
const result = await this.acquire(file, kind, priority, signal);
|
||||
const result = await this.acquire(fileID, kind, priority, signal);
|
||||
const status = result.cached ? "skipped" : "done";
|
||||
onProgress?.({ fileID, status, path: result.path });
|
||||
return { fileID, path: result.path };
|
||||
@@ -756,16 +736,8 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
}
|
||||
}
|
||||
|
||||
// The record `getFile` gives for `fileID`. It throws for a file `getFile`
|
||||
// does not know.
|
||||
private fileOrThrow(fileID: number): EnteFile {
|
||||
const file = this.getFile(fileID);
|
||||
if (!file) throw new Error(`content cache: unknown file ${fileID}`);
|
||||
return file;
|
||||
}
|
||||
|
||||
private async get(
|
||||
file: EnteFile,
|
||||
fileID: number,
|
||||
kind: Kind,
|
||||
priority: Priority,
|
||||
onProgress: ((event: ContentEvent) => void) | undefined,
|
||||
@@ -773,7 +745,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
const onByte: ProgressCallback | undefined = onProgress
|
||||
? (bytesDone) => onProgress({ status: "downloading", bytesDone })
|
||||
: undefined;
|
||||
const result = await this.acquire(file, kind, priority, undefined, {
|
||||
const result = await this.acquire(fileID, kind, priority, undefined, {
|
||||
onByte,
|
||||
});
|
||||
onProgress?.(
|
||||
@@ -795,7 +767,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
// photo's original is present only with its video, and is returned with
|
||||
// it.
|
||||
private async acquire(
|
||||
file: EnteFile,
|
||||
fileID: number,
|
||||
kind: Kind,
|
||||
priority: Priority,
|
||||
signal: AbortSignal | undefined,
|
||||
@@ -806,7 +778,8 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
videoPath?: string;
|
||||
cached: boolean;
|
||||
}> {
|
||||
const fileID = file.id;
|
||||
const file = this.getFile(fileID);
|
||||
if (!file) throw new Error(`content cache: unknown file ${fileID}`);
|
||||
const isLivePhoto =
|
||||
kind === "original" && file.metadata.fileType === "livePhoto";
|
||||
|
||||
|
||||
+3
-54
@@ -20,10 +20,9 @@
|
||||
//
|
||||
// `freshCollections()` is the fresh read of the collection list alone (issue
|
||||
// #210): one collections request, no collection's files, and nothing stored,
|
||||
// for a caller such as `quak collections` that needs no more. `freshFiles()`
|
||||
// is the fresh read of one collection's files, and stores nothing either
|
||||
// (issue #217). A caller that asks for each refresh it needs can also open the
|
||||
// library with `refreshOnOpen: false`, so that `open()` starts none.
|
||||
// for a caller such as `quak collections` that needs no more. A caller that
|
||||
// asks for each refresh it needs can also open the library with
|
||||
// `refreshOnOpen: false`, so that `open()` starts none.
|
||||
//
|
||||
// A refresh reports through `onProgress` each collection whose files it has
|
||||
// fetched, so a large account, whose first refresh fetches the files of every
|
||||
@@ -66,8 +65,6 @@ import {
|
||||
ContentCache,
|
||||
savePath,
|
||||
storedAtSavePath,
|
||||
type ContentOptions,
|
||||
type ContentResult,
|
||||
type ContentSource,
|
||||
type ThumbnailsAPI,
|
||||
type EnsureOptions,
|
||||
@@ -628,54 +625,6 @@ export class Library {
|
||||
return collections;
|
||||
}
|
||||
|
||||
// The fresh read of one collection's files (issue #217): every live file
|
||||
// in it that decrypts, in the server's order. `collection` is a record
|
||||
// `freshCollections()` returned, whose key decrypts the files. It fetches
|
||||
// no other collection's files. Like `freshCollections()`, it stores
|
||||
// nothing. It returns each file once, as the latest copy the server sent,
|
||||
// where the first copy was.
|
||||
async freshFiles(collection: Collection): Promise<EnteFile[]> {
|
||||
if (this.closed) throw new Error("the library is closed");
|
||||
const { files } = await this.client.filesSince({
|
||||
collectionID: collection.id,
|
||||
collectionKey: collection.key,
|
||||
sinceTime: 0,
|
||||
});
|
||||
const byID = new Map<number, EnteFile>();
|
||||
for (const file of files) byID.set(file.id, file);
|
||||
return [...byID.values()];
|
||||
}
|
||||
|
||||
// The original of `file`, fetched and cached as `Photo.original()` does.
|
||||
// `file` is used as given, so the store need not hold it. On an empty
|
||||
// cache, `quak get` passes a file that `freshFiles()` returned.
|
||||
async originalOf(
|
||||
file: EnteFile,
|
||||
opts?: ContentOptions,
|
||||
): Promise<ContentResult> {
|
||||
if (!this.cache) {
|
||||
throw new Error(
|
||||
"originalOf requires a library opened with a content cache",
|
||||
);
|
||||
}
|
||||
return this.cache.originalOf(file, opts);
|
||||
}
|
||||
|
||||
// The thumbnail of `file`, fetched and cached as `Photo.thumbnail()` does.
|
||||
// `file` is used as given, so the store need not hold it. On an empty
|
||||
// cache, `quak get-thumb` passes a file that `freshFiles()` returned.
|
||||
async thumbnailOf(
|
||||
file: EnteFile,
|
||||
opts?: ContentOptions,
|
||||
): Promise<ContentResult> {
|
||||
if (!this.cache) {
|
||||
throw new Error(
|
||||
"thumbnailOf requires a library opened with a content cache",
|
||||
);
|
||||
}
|
||||
return this.cache.thumbnailOf(file, opts);
|
||||
}
|
||||
|
||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||
// library's, each original at its save path, with a durable failure
|
||||
// ledger (issue #51). Takes the lock in that directory first, unless the
|
||||
|
||||
@@ -61,7 +61,7 @@ export interface MLIndex {
|
||||
// Pull the CLIP embedding out of a payload, or undefined when it is absent or
|
||||
// misshapen. Kept strict so a bad payload is skipped rather than corrupting the
|
||||
// packed index.
|
||||
export const clipEmbedding = (payload: MLData): number[] | undefined => {
|
||||
const clipEmbedding = (payload: MLData): number[] | undefined => {
|
||||
const clip = payload.clip;
|
||||
if (typeof clip !== "object" || clip === null) return undefined;
|
||||
const embedding = (clip as { embedding?: unknown }).embedding;
|
||||
|
||||
+4
-21
@@ -1,7 +1,6 @@
|
||||
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import * as jpeg from "jpeg-js";
|
||||
import { clipKeywords } from "./clip-keywords.js";
|
||||
import type { Client } from "./client.js";
|
||||
import { describeError } from "./errors.js";
|
||||
import { readExifTags } from "./exif.js";
|
||||
@@ -18,10 +17,7 @@ import {
|
||||
} from "./mldata-fetch.js";
|
||||
import type { EnteFile } from "./model/types.js";
|
||||
|
||||
// Reports one line of the dump's progress. `level` is "error" when the line
|
||||
// reports ML data that could not be read, and is left out for every other
|
||||
// line.
|
||||
export type ProgressCallback = (message: string, level?: "error") => void;
|
||||
export type ProgressCallback = (message: string) => void;
|
||||
|
||||
export interface MetadataBackupOptions {
|
||||
exif?: boolean;
|
||||
@@ -98,8 +94,7 @@ const extractExif = async (
|
||||
|
||||
// Dump every decrypted metadata layer the account holds into a directory tree
|
||||
// of plain JSON: account, per-collection, and per-file records including the
|
||||
// private and public magic metadata and the ML data, with the keywords of its
|
||||
// CLIP embedding in place of the embedding (see clip-keywords.ts). Collections
|
||||
// private and public magic metadata and (by default) the ML data. Collections
|
||||
// and files are enumerated from the library's cache, which the caller refreshes
|
||||
// first. Returns how many files have no ML data because their ML data request
|
||||
// failed or their ML data did not decrypt, gunzip or parse; they are still
|
||||
@@ -200,17 +195,13 @@ export const runMetadataBackup = async (
|
||||
for (const [id, payload] of payloads) mlDataMap.set(id, payload);
|
||||
for (const [id, err] of errors) {
|
||||
const reason = describeError(err);
|
||||
log(
|
||||
`ML data of file ${id} could not be read: ${reason}`,
|
||||
"error",
|
||||
);
|
||||
log(`ML data of file ${id} could not be read: ${reason}`);
|
||||
mlDataErrors.set(id, reason);
|
||||
}
|
||||
} catch (err) {
|
||||
const reason = err instanceof Error ? err.message : String(err);
|
||||
log(
|
||||
`ML data request for ${batch.length} file(s) failed: ${reason}`,
|
||||
"error",
|
||||
);
|
||||
for (const id of batch) mlDataErrors.set(id, reason);
|
||||
}
|
||||
@@ -232,16 +223,8 @@ export const runMetadataBackup = async (
|
||||
if (file.pubMagicMetadata)
|
||||
fileMeta.pubMagicMetadata = file.pubMagicMetadata;
|
||||
|
||||
// The ML data without the CLIP embedding, and the keywords that
|
||||
// embedding gives.
|
||||
const ml = mlDataMap.get(file.id);
|
||||
if (ml) {
|
||||
const { clip: _clip, ...mlDataWithoutClip } = ml;
|
||||
fileMeta.mlData = mlDataWithoutClip;
|
||||
const isVideo = file.metadata.fileType === "video";
|
||||
const keywords = clipKeywords(ml, isVideo);
|
||||
if (keywords) fileMeta.keywords = keywords;
|
||||
}
|
||||
if (ml) fileMeta.mlData = ml;
|
||||
const mlError = mlDataErrors.get(file.id);
|
||||
if (mlError) fileMeta.mlDataError = mlError;
|
||||
|
||||
|
||||
+1
-8
@@ -24,9 +24,6 @@ export interface RetryOptions {
|
||||
sleep?: (ms: number) => Promise<void>;
|
||||
// Injected so the jitter is reproducible under test.
|
||||
random?: () => number;
|
||||
// Called before each wait with the error that failed attempt number
|
||||
// `attempt` and how long the wait is, for a caller that logs retries.
|
||||
onRetry?: (err: unknown, attempt: number, delayMs: number) => void;
|
||||
}
|
||||
|
||||
export type ResolvedRetryOptions = Required<RetryOptions>;
|
||||
@@ -38,7 +35,6 @@ export const DEFAULT_RETRY_OPTIONS: ResolvedRetryOptions = {
|
||||
sleep: (ms: number): Promise<void> =>
|
||||
new Promise((resolve) => setTimeout(resolve, ms)),
|
||||
random: Math.random,
|
||||
onRetry: () => {},
|
||||
};
|
||||
|
||||
// For a run nobody is watching, such as `quak backup` from cron. A request that
|
||||
@@ -59,7 +55,6 @@ export const resolveRetryOptions = (
|
||||
maxDelayMs: opts?.maxDelayMs ?? DEFAULT_RETRY_OPTIONS.maxDelayMs,
|
||||
sleep: opts?.sleep ?? DEFAULT_RETRY_OPTIONS.sleep,
|
||||
random: opts?.random ?? DEFAULT_RETRY_OPTIONS.random,
|
||||
onRetry: opts?.onRetry ?? DEFAULT_RETRY_OPTIONS.onRetry,
|
||||
});
|
||||
|
||||
// Transport failures: the request did not complete, for reasons below HTTP.
|
||||
@@ -239,9 +234,7 @@ export const withRetry = async <T>(
|
||||
// and its status intact.
|
||||
throw err;
|
||||
}
|
||||
const delayMs = backoffMs(attempt, policy);
|
||||
policy.onRetry(err, attempt, delayMs);
|
||||
await policy.sleep(delayMs);
|
||||
await policy.sleep(backoffMs(attempt, policy));
|
||||
}
|
||||
}
|
||||
};
|
||||
+37
-17
@@ -5,7 +5,7 @@ import type { Client } from "./client.js";
|
||||
import type { Library } from "./library/index.js";
|
||||
import { ApiError } from "./api/client.js";
|
||||
import { encryptBlob, toBase64 } from "./crypto/index.js";
|
||||
import type { Collection, EnteFile } from "./model/types.js";
|
||||
import type { EnteFile } from "./model/types.js";
|
||||
|
||||
// The server refuses a thumbnail larger than the one it already records for the
|
||||
// file (`thumbnail.size`, the encrypted size), so these encodings are tried
|
||||
@@ -49,10 +49,7 @@ export interface ThumbnailFixResult {
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
// Reports one line of progress. `level` is "error" when the line reports a
|
||||
// file that could not be checked or fixed, and is left out for every other
|
||||
// line.
|
||||
export type ProgressCallback = (message: string, level?: "error") => void;
|
||||
export type ProgressCallback = (message: string) => void;
|
||||
|
||||
// Enumerate every file the library knows about, newest album first, each file
|
||||
// once, and report those whose server-side thumbnail is missing. "Missing" is
|
||||
@@ -114,7 +111,6 @@ export const listMissingThumbnails = async (
|
||||
} else {
|
||||
log(
|
||||
`[${album.name}] Could not check ${photo.title}: ${err instanceof Error ? err.message : String(err)} (not reported as missing)`,
|
||||
"error",
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -226,18 +222,16 @@ const encryptThumbnailWithin = (
|
||||
return undefined;
|
||||
};
|
||||
|
||||
// Regenerate and upload a thumbnail for each requested file. `found` holds the
|
||||
// record of each requested file that a collection holds, and the collection it
|
||||
// was found in. Each original is read through the library's content cache, by
|
||||
// `lib.originalOf(file)`. The new thumbnail is a JPEG, encrypted under the
|
||||
// file's own key. It is uploaded, then registered with the server. Each file is
|
||||
// Regenerate and upload a thumbnail for each requested file. Originals are read
|
||||
// through the library's content cache (`photo.original()`); the generated
|
||||
// thumbnail is JPEG-encoded, encrypted under the file's own key, and registered
|
||||
// with the server — the encrypt-and-upload path is unchanged. Each file is
|
||||
// resolved to one outcome (fixed / skipped / failed) and a failure on one file
|
||||
// never stops the others.
|
||||
export const fixMissingThumbnails = async (
|
||||
lib: Library,
|
||||
client: Client,
|
||||
fileIDs: number[],
|
||||
found: Map<number, { file: EnteFile; collection: Collection }>,
|
||||
onProgress?: ProgressCallback,
|
||||
): Promise<ThumbnailFixResult[]> => {
|
||||
const log = onProgress ?? (() => {});
|
||||
@@ -245,8 +239,31 @@ export const fixMissingThumbnails = async (
|
||||
const api = client.getApiClient();
|
||||
const { userID } = client.whoami();
|
||||
|
||||
// Resolve each requested fileID to its file record and owning album by
|
||||
// enumerating the library, each file taken from the first album that holds
|
||||
// it. The raw `EnteFile` carries the per-file key the thumbnail is
|
||||
// encrypted under, which the projected records deliberately do not.
|
||||
const wanted = new Set(fileIDs);
|
||||
const fileMap = new Map<
|
||||
number,
|
||||
{ file: EnteFile; collectionName: string }
|
||||
>();
|
||||
for (const album of lib.albums.list()) {
|
||||
for (const photo of album.photos.list()) {
|
||||
if (!wanted.has(photo.fileID) || fileMap.has(photo.fileID))
|
||||
continue;
|
||||
const file = lib.getFile(album.collectionID, photo.fileID);
|
||||
if (file) {
|
||||
fileMap.set(photo.fileID, {
|
||||
file,
|
||||
collectionName: album.name,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (const fileID of fileIDs) {
|
||||
const entry = found.get(fileID);
|
||||
const entry = fileMap.get(fileID);
|
||||
if (!entry) {
|
||||
results.push({
|
||||
fileID,
|
||||
@@ -258,8 +275,7 @@ export const fixMissingThumbnails = async (
|
||||
continue;
|
||||
}
|
||||
|
||||
const { file } = entry;
|
||||
const collectionName = entry.collection.name;
|
||||
const { file, collectionName } = entry;
|
||||
const title = file.metadata.title;
|
||||
|
||||
const skipReason = reasonToSkip(file, userID);
|
||||
@@ -277,10 +293,15 @@ export const fixMissingThumbnails = async (
|
||||
const maxSize = file.thumbnail.size!;
|
||||
|
||||
try {
|
||||
const photo = lib.photos.byID({ fileID });
|
||||
if (!photo) {
|
||||
throw new Error("file not present in the library cache");
|
||||
}
|
||||
|
||||
log(
|
||||
`[${collectionName}] Downloading ${title} for thumbnail generation...`,
|
||||
);
|
||||
const { path } = await lib.originalOf(file);
|
||||
const { path } = await photo.original();
|
||||
const fileBytes = new Uint8Array(readFileSync(path));
|
||||
|
||||
if (!isJpeg(fileBytes)) {
|
||||
@@ -338,7 +359,6 @@ export const fixMissingThumbnails = async (
|
||||
} catch (err) {
|
||||
log(
|
||||
`[${collectionName}] FAILED ${title}: ${err instanceof Error ? err.message : err}`,
|
||||
"error",
|
||||
);
|
||||
results.push({
|
||||
fileID,
|
||||
|
||||
@@ -1222,65 +1222,3 @@ describe("ApiClient non-idempotent requests", () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// `onRequest` hears of each request the client sends, as its response arrives
|
||||
// or as it fails. The CLI logs these with `-vv`. A retried request is reported
|
||||
// once for each attempt.
|
||||
describe("ApiClient onRequest", () => {
|
||||
it("reports each request with its status, or with the error it failed with", async () => {
|
||||
const refused = new TypeError("fetch failed");
|
||||
const { fetch } = scriptedFetch(
|
||||
refused,
|
||||
textResponse("unavailable", 503),
|
||||
jsonResponse({ collections: [] }),
|
||||
jsonResponse({}),
|
||||
streamResponse(new Uint8Array([1, 2, 3])),
|
||||
);
|
||||
const requests: unknown[] = [];
|
||||
const client = new ApiClient({
|
||||
fetch,
|
||||
retry: noWait,
|
||||
onRequest: (request) => requests.push(request),
|
||||
});
|
||||
|
||||
await client.getJSON("/collections/v2", { sinceTime: 0 });
|
||||
await client.postJSON("/users/logout", {});
|
||||
await readAll(await client.getFileStream(7));
|
||||
|
||||
const collections = "https://api.ente.io/collections/v2?sinceTime=0";
|
||||
expect(requests).toEqual([
|
||||
{ method: "GET", url: collections, error: refused },
|
||||
{ method: "GET", url: collections, status: 503 },
|
||||
{ method: "GET", url: collections, status: 200 },
|
||||
{
|
||||
method: "POST",
|
||||
url: "https://api.ente.io/users/logout",
|
||||
status: 200,
|
||||
},
|
||||
{
|
||||
method: "GET",
|
||||
url: "https://files.ente.io/?fileID=7",
|
||||
status: 200,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("reports a presigned upload without its query, which holds the upload's signature", async () => {
|
||||
const { fetch } = scriptedFetch(new Response(null, { status: 200 }));
|
||||
const requests: unknown[] = [];
|
||||
const client = new ApiClient({
|
||||
fetch,
|
||||
retry: noWait,
|
||||
onRequest: (request) => requests.push(request),
|
||||
});
|
||||
|
||||
await client.putFile(
|
||||
"https://s3.example/obj?X-Amz-Signature=abc",
|
||||
new Uint8Array([1, 2]),
|
||||
);
|
||||
|
||||
expect(requests).toEqual([
|
||||
{ method: "PUT", url: "https://s3.example/obj", status: 200 },
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -54,7 +54,6 @@ import {
|
||||
saveSession,
|
||||
type CliContext,
|
||||
} from "../../src/cli-commands.js";
|
||||
import { stderrLog } from "../../src/cli-log.js";
|
||||
import { run } from "../../src/cli-run.js";
|
||||
import { loadSession } from "../../src/cli-session.js";
|
||||
import { Client } from "../../src/client.js";
|
||||
@@ -86,17 +85,10 @@ const DAY = "2024/2024-03/2024-03-01";
|
||||
const TOAST = new TextEncoder().encode("a few seconds of the wedding toast");
|
||||
|
||||
// Ente's ML data for beach.jpg: a face it found, and the photo's CLIP
|
||||
// embedding, here that of a photo of a sunset (see clip-keywords.test.ts).
|
||||
// embedding.
|
||||
const ML_DATA = {
|
||||
face: { faces: [{ faceID: "100_1", score: 0.98 }] },
|
||||
clip: {
|
||||
embedding: JSON.parse(
|
||||
readFileSync(
|
||||
new URL("../clip-keywords-sunset.json", import.meta.url),
|
||||
"utf-8",
|
||||
),
|
||||
).embedding,
|
||||
},
|
||||
clip: { embedding: [0.5, 0.6, 0.7] },
|
||||
};
|
||||
|
||||
// Each file's ID, its owner, its original as uploaded, and the metadata its
|
||||
@@ -311,10 +303,9 @@ const backup = async (
|
||||
const prompted = async (): Promise<never> => {
|
||||
throw new Error("quak backup prompted");
|
||||
};
|
||||
const log = stderrLog(stderr, "info");
|
||||
const ctx: CliContext = {
|
||||
stdout,
|
||||
log,
|
||||
stderr,
|
||||
sessionDir,
|
||||
cacheDir: `${dir}-cache`,
|
||||
loadSession: (path) =>
|
||||
@@ -328,7 +319,7 @@ const backup = async (
|
||||
promptSecret: prompted,
|
||||
};
|
||||
const code = await new Promise<number>((resolve) => {
|
||||
void run(backupCommand(ctx, dir, opts), stdout, stderr, log, resolve);
|
||||
void run(backupCommand(ctx, dir, opts), stdout, stderr, resolve);
|
||||
});
|
||||
return { code, stderr: printed, stdout: output };
|
||||
};
|
||||
@@ -418,7 +409,7 @@ describe("the first backup of the account", () => {
|
||||
expect(downloads.sort()).toEqual([100, 200, 300]);
|
||||
});
|
||||
|
||||
// The tree under "The backup directory" in the README. Each original is at
|
||||
// The tree "Backup layout" in the README describes. Each original is at
|
||||
// its save path, `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, with
|
||||
// the extension of its title; the live photo is its image and its video,
|
||||
// each with the extension it had inside the ZIP, and a `.livephoto.json`
|
||||
@@ -493,10 +484,7 @@ describe("the first backup of the account", () => {
|
||||
},
|
||||
updationTime: TAKEN,
|
||||
});
|
||||
// The JSON holds the ML data without its CLIP embedding, and the
|
||||
// keywords the embedding gives in its place.
|
||||
expect(beach.mlData).toEqual({ face: ML_DATA.face });
|
||||
expect(beach.keywords).toEqual(["Sunset"]);
|
||||
expect(beach.mlData).toEqual(ML_DATA);
|
||||
expect(beach.imageMetadata.exif.Make.description).toBe("Canon");
|
||||
expect(beach.imageMetadata.exif.Model.description).toBe("EOS R5");
|
||||
|
||||
@@ -507,7 +495,6 @@ describe("the first backup of the account", () => {
|
||||
metadata: { title: "toast.mp4", fileType: "video" },
|
||||
});
|
||||
expect(toast).not.toHaveProperty("mlData");
|
||||
expect(toast).not.toHaveProperty("keywords");
|
||||
expect(toast).not.toHaveProperty("imageMetadata");
|
||||
expect(toast).not.toHaveProperty("imageMetadataError");
|
||||
|
||||
@@ -1117,9 +1104,6 @@ describe("a backup whose lock is lost", () => {
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toBe(
|
||||
"Starting backup...\n" +
|
||||
"Fetched the files of 1 of 3 collections\n" +
|
||||
"Fetched the files of 2 of 3 collections\n" +
|
||||
"Fetched the files of 3 of 3 collections\n" +
|
||||
"Refreshing library...\n" +
|
||||
"Fetching original beach.jpg (100)...\n" +
|
||||
"Fetching original IMG_0002.HEIC (300)...\n" +
|
||||
|
||||
@@ -42,7 +42,6 @@ import {
|
||||
saveSession,
|
||||
type CliContext,
|
||||
} from "../../src/cli-commands.js";
|
||||
import { stderrLog } from "../../src/cli-log.js";
|
||||
import { loadSession } from "../../src/cli-session.js";
|
||||
import { Client, type ClientSnapshot } from "../../src/client.js";
|
||||
import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
|
||||
@@ -305,7 +304,7 @@ const context = (ente: FakeEnte): CliContext => {
|
||||
saveSession(join(root, "session"), ente.snapshot());
|
||||
return {
|
||||
stdout,
|
||||
log: stderrLog(stderr, "info"),
|
||||
stderr,
|
||||
sessionDir: join(root, "session"),
|
||||
cacheDir: join(root, "cache"),
|
||||
loadSession: (path) =>
|
||||
@@ -371,24 +370,6 @@ describe("quak backup with records it cannot decrypt", () => {
|
||||
expect(stderr.text).toContain(" Failed: 2\n");
|
||||
});
|
||||
|
||||
// The album is an error line. The last line is fatal and counts the files
|
||||
// and the album, so `-q`, which prints fatal lines only, still says why
|
||||
// the backup exits 1.
|
||||
it("logs the album at error, and ends with one fatal line counting the files and the album", async () => {
|
||||
const ctx = {
|
||||
...context(brokenAccount()),
|
||||
log: stderrLog(stderr, "error"),
|
||||
};
|
||||
expect(await backupCommand(ctx, backupDir(), {})).toBe(1);
|
||||
|
||||
expect(stderr.text).toMatch(
|
||||
/^FAILED album 2: could not be decrypted: /,
|
||||
);
|
||||
expect(stderr.text.split("\n").at(-2)).toBe(
|
||||
`quak: 2 file(s) and 1 album(s) failed; see ${join(backupDir(), "failures.json")} and the FAILED lines`,
|
||||
);
|
||||
});
|
||||
|
||||
it("names each album it cannot decrypt in the --json result", async () => {
|
||||
expect(await backup(brokenAccount(), { json: true })).toBe(1);
|
||||
|
||||
@@ -567,7 +548,7 @@ describe("the read commands with records quak cannot decrypt", () => {
|
||||
expect(await collectionsCommand(context(ente), {})).toBe(0);
|
||||
expect(stdout.text).toBe("1\talbum\tTrip\n");
|
||||
|
||||
expect(await filesCommand(context(ente), "1", {})).toBe(0);
|
||||
expect(await filesCommand(context(ente), { collection: "1" })).toBe(0);
|
||||
expect(stdout.text).toBe("10\timage\tphoto-10.jpg\n");
|
||||
});
|
||||
});
|
||||
+7
-101
@@ -52,7 +52,6 @@ import { tmpdir } from "node:os";
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
|
||||
import { runBackup, type BackupLibrary } from "../../src/backup.js";
|
||||
import { DISCOVER_CATEGORIES } from "../../src/clip-keyword-vectors.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import type { ContentSource } from "../../src/library/content.js";
|
||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||
@@ -947,19 +946,9 @@ describe("the backup lock", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// Also serves ML data: file 100 has `ML_PAYLOAD`, whose CLIP embedding is that
|
||||
// of a photo of a sunset (see clip-keywords.test.ts), and the other two have
|
||||
// none. While `mlError` is set, every ML data request fails with it.
|
||||
const SUNSET_EMBEDDING: number[] = JSON.parse(
|
||||
readFileSync(
|
||||
new URL("../clip-keywords-sunset.json", import.meta.url),
|
||||
"utf-8",
|
||||
),
|
||||
).embedding;
|
||||
const ML_PAYLOAD = {
|
||||
face: { faces: [] },
|
||||
clip: { embedding: SUNSET_EMBEDDING },
|
||||
};
|
||||
// Also serves ML data: file 100 has `ML_PAYLOAD`, the other two have none.
|
||||
// While `mlError` is set, every ML data request fails with it.
|
||||
const ML_PAYLOAD = { face: { faces: [] }, clip: { embedding: [0.5, 0.25] } };
|
||||
|
||||
class MLClient extends MockClient {
|
||||
mlError?: string;
|
||||
@@ -979,99 +968,16 @@ const fileJSON = (outDir: string, fileID: number): Record<string, unknown> =>
|
||||
JSON.parse(readFileSync(saved(outDir, `${fileID}.json`), "utf-8"));
|
||||
|
||||
describe("ML data in each file's JSON", () => {
|
||||
// The JSON holds the ML data without its CLIP embedding, and in its place
|
||||
// the keywords the embedding gives.
|
||||
it("writes a file's ML data and keywords, and no ML field for a file that has none", async () => {
|
||||
it("writes a file's ML data, and no ML field for a file that has none", async () => {
|
||||
const lib = await openLibrary(stubSource(), new MLClient());
|
||||
const outDir = join(root, "backup");
|
||||
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } });
|
||||
expect(fileJSON(outDir, 100).keywords).toEqual(["Sunset"]);
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
|
||||
expect(fileJSON(outDir, 101)).not.toHaveProperty("mlData");
|
||||
expect(fileJSON(outDir, 101)).not.toHaveProperty("mlDataError");
|
||||
expect(fileJSON(outDir, 101)).not.toHaveProperty("keywords");
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
// A file's ML data can hold faces and no CLIP embedding. Its JSON then has
|
||||
// the ML data and no keywords.
|
||||
it("writes no keywords for ML data without a CLIP embedding", async () => {
|
||||
const faces = { face: { faces: [{ faceID: "101_1", score: 0.9 }] } };
|
||||
|
||||
// Gives file 101 ML data holding only faces.
|
||||
class FacesClient extends MockClient {
|
||||
async fetchMLData(args: { fileIDs: number[] }): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}> {
|
||||
const payloads = new Map<number, MLData>();
|
||||
if (args.fileIDs.includes(101)) payloads.set(101, faces);
|
||||
return { payloads, errors: new Map() };
|
||||
}
|
||||
}
|
||||
const lib = await openLibrary(stubSource(), new FacesClient());
|
||||
const outDir = join(root, "backup");
|
||||
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(fileJSON(outDir, 101).mlData).toEqual(faces);
|
||||
expect(fileJSON(outDir, 101)).not.toHaveProperty("keywords");
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
// Ente's photos app keeps videos out of some Discover categories, Identity
|
||||
// among them. Here a photo and a video share one CLIP embedding, Identity's
|
||||
// own text embedding, so the photo gets the keyword Identity and the video
|
||||
// does not.
|
||||
it("keeps a video out of a category that leaves out videos", async () => {
|
||||
const identity = DISCOVER_CATEGORIES.find(
|
||||
(c) => c.title === "Identity",
|
||||
)!;
|
||||
expect(identity.showVideo).toBe(false);
|
||||
const toast = file(102, 1, "toast.mp4");
|
||||
const video = {
|
||||
...toast,
|
||||
metadata: { ...toast.metadata, fileType: "video" as const },
|
||||
};
|
||||
|
||||
// Serves beach.jpg and the video in the Vacation album, and gives
|
||||
// each ML data holding only that embedding.
|
||||
class VideoClient extends MockClient {
|
||||
override async filesSince(args: {
|
||||
collectionID: number;
|
||||
}): Promise<FilesPage> {
|
||||
const files =
|
||||
args.collectionID === 1
|
||||
? [file(100, 1, "beach.jpg"), video]
|
||||
: [];
|
||||
return { files, deleted: [], cursor: 1 };
|
||||
}
|
||||
async fetchMLData(args: { fileIDs: number[] }): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}> {
|
||||
const payloads = new Map<number, MLData>();
|
||||
for (const id of args.fileIDs) {
|
||||
payloads.set(id, {
|
||||
clip: { embedding: identity.textEmbedding },
|
||||
});
|
||||
}
|
||||
return { payloads, errors: new Map() };
|
||||
}
|
||||
}
|
||||
const lib = await openLibrary(stubSource(), new VideoClient());
|
||||
const outDir = join(root, "backup");
|
||||
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(fileJSON(outDir, 100).keywords).toContain("Identity");
|
||||
expect(fileJSON(outDir, 102)).toHaveProperty("keywords");
|
||||
expect(fileJSON(outDir, 102).keywords).not.toContain("Identity");
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
@@ -1087,7 +993,7 @@ describe("ML data in each file's JSON", () => {
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
// File 100's ML data was cached by the first run and is kept.
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } });
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
|
||||
expect(fileJSON(outDir, 100)).not.toHaveProperty("mlDataError");
|
||||
for (const fileID of [101, 200]) {
|
||||
expect(fileJSON(outDir, fileID).mlDataError).toBe(
|
||||
@@ -1120,7 +1026,7 @@ describe("ML data in each file's JSON", () => {
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(result.skipped).toBe(3);
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } });
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
|
||||
for (const fileID of [100, 101, 200]) {
|
||||
expect(fileJSON(outDir, fileID)).not.toHaveProperty("mlDataError");
|
||||
}
|
||||
|
||||
+8
-56
@@ -1,18 +1,10 @@
|
||||
/**
|
||||
* Tests for the retry options `bin/quak.ts` loads the saved session with
|
||||
* (`quak backup` gets the unattended ones, every other command the default),
|
||||
* each with the callbacks that log its requests and retries, for `quak files`
|
||||
* taking its collection ID as an optional argument, and for `quak get` and
|
||||
* `quak get-thumb` refusing `--collection`.
|
||||
* Tests for the retry options `bin/quak.ts` loads the saved session with:
|
||||
* `quak backup` gets the unattended ones, every other command the default.
|
||||
*
|
||||
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
|
||||
* replaced by one that records the options it is given and reports no session,
|
||||
* so the command stops before it makes any request.
|
||||
*
|
||||
* The first load of the sources `bin/quak.ts` imports compiles them, and the
|
||||
* first `init()` starts libsodium. On a loaded host that takes longer than a
|
||||
* test's timeout, so both happen here, before the tests run. Each test's
|
||||
* `vi.resetModules()` then only runs the compiled sources again.
|
||||
*/
|
||||
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
@@ -20,13 +12,8 @@ import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterAll, afterEach, describe, expect, it, vi } from "vitest";
|
||||
import type { ApiClientOptions } from "../../src/api/client.js";
|
||||
import "../../src/cli-commands.js";
|
||||
import { init } from "../../src/crypto/index.js";
|
||||
import "../../src/index.js";
|
||||
import { UNATTENDED_RETRY_OPTIONS } from "../../src/retry.js";
|
||||
|
||||
await init();
|
||||
|
||||
const loaded = vi.hoisted(() => [] as (ApiClientOptions | undefined)[]);
|
||||
|
||||
vi.mock("../../src/cli-session.js", () => ({
|
||||
@@ -39,19 +26,14 @@ vi.mock("../../src/cli-session.js", () => ({
|
||||
const argv = process.argv;
|
||||
const dir = mkdtempSync(join(tmpdir(), "quak-bin-test-"));
|
||||
|
||||
// Run `quak <args>` to completion and return what it wrote on stderr, which is
|
||||
// kept off the terminal. The exit is swallowed.
|
||||
const quak = async (...args: string[]): Promise<string> => {
|
||||
let written = "";
|
||||
vi.spyOn(process.stderr, "write").mockImplementation((text) => {
|
||||
written += String(text);
|
||||
return true;
|
||||
});
|
||||
// Run `quak <args>` to completion. The "Not logged in" message and the exit
|
||||
// are swallowed.
|
||||
const quak = async (...args: string[]): Promise<void> => {
|
||||
vi.spyOn(process.stderr, "write").mockImplementation(() => true);
|
||||
vi.spyOn(process, "exit").mockImplementation(() => undefined as never);
|
||||
process.argv = ["node", "quak", ...args];
|
||||
vi.resetModules();
|
||||
await import("../../bin/quak.js");
|
||||
return written;
|
||||
};
|
||||
|
||||
afterEach(() => {
|
||||
@@ -77,43 +59,13 @@ describe("bin/quak.ts session loading", () => {
|
||||
attempts,
|
||||
baseDelayMs,
|
||||
maxDelayMs,
|
||||
onRetry: expect.any(Function),
|
||||
}),
|
||||
onRequest: expect.any(Function),
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("loads the session for another command with the default retry options", async () => {
|
||||
it("loads the session for another command with the default options", async () => {
|
||||
await quak("collections");
|
||||
expect(loaded).toEqual([
|
||||
{
|
||||
retry: { onRetry: expect.any(Function) },
|
||||
onRequest: expect.any(Function),
|
||||
},
|
||||
]);
|
||||
expect(loaded).toEqual([undefined]);
|
||||
});
|
||||
});
|
||||
|
||||
// The command line takes `files` with or without a collection ID, and the
|
||||
// command runs as far as its session check either way (issue #211).
|
||||
describe("bin/quak.ts files", () => {
|
||||
it("runs without a collection ID", async () => {
|
||||
expect(await quak("files")).toMatch(/^Not logged in/);
|
||||
});
|
||||
|
||||
it("runs with a collection ID", async () => {
|
||||
expect(await quak("files", "7")).toMatch(/^Not logged in/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("bin/quak.ts get and get-thumb", () => {
|
||||
it.each(["get", "get-thumb"])(
|
||||
"%s refuses --collection as an unknown option",
|
||||
async (command) => {
|
||||
expect(await quak(command, "1", "--collection", "2")).toMatch(
|
||||
/^error: unknown option '--collection'/,
|
||||
);
|
||||
},
|
||||
);
|
||||
});
|
||||
+131
-482
@@ -51,7 +51,6 @@ import {
|
||||
listMissingThumbnailsCommand,
|
||||
fixMissingThumbnailsCommand,
|
||||
} from "../../src/cli-commands.js";
|
||||
import { stderrLog } from "../../src/cli-log.js";
|
||||
import { run } from "../../src/cli-run.js";
|
||||
import { loadSession } from "../../src/cli-session.js";
|
||||
import {
|
||||
@@ -122,31 +121,25 @@ const FILES: Record<number, EnteFile[]> = {
|
||||
// An original is 7 bytes and a thumbnail 3. `failID` makes that file's
|
||||
// original fail; `emptyThumbID` makes the server report that file's
|
||||
// thumbnail as empty. `withNewFile` adds new.jpg (102) to Vacation, advancing
|
||||
// the collection's updationTime as the server does. `beachInWork` adds
|
||||
// beach.jpg (100) to Work as well, so that one file is in both albums.
|
||||
// `refreshError` makes listing collections fail with that message.
|
||||
// the collection's updationTime as the server does, and `refreshError` makes
|
||||
// listing collections fail with that message.
|
||||
const fakeClient = (
|
||||
opts: {
|
||||
failID?: number;
|
||||
emptyThumbID?: number;
|
||||
withNewFile?: boolean;
|
||||
beachInWork?: boolean;
|
||||
refreshError?: string;
|
||||
} = {},
|
||||
) => {
|
||||
const collections = opts.withNewFile
|
||||
? [{ ...COLLECTIONS[0], updationTime: 2 }, COLLECTIONS[1]]
|
||||
: COLLECTIONS;
|
||||
let files = FILES;
|
||||
if (opts.withNewFile) {
|
||||
files = {
|
||||
...files,
|
||||
1: [...files[1], { ...file(102, 1, "new.jpg"), updationTime: 2 }],
|
||||
};
|
||||
}
|
||||
if (opts.beachInWork) {
|
||||
files = { ...files, 2: [...files[2], file(100, 2, "beach.jpg")] };
|
||||
}
|
||||
const files = opts.withNewFile
|
||||
? {
|
||||
...FILES,
|
||||
1: [...FILES[1], { ...file(102, 1, "new.jpg"), updationTime: 2 }],
|
||||
}
|
||||
: FILES;
|
||||
const source: ContentSource = {
|
||||
original: async ({ file: f, destination }) => {
|
||||
if (f.id === opts.failID) throw new Error("HTTP 500 from server");
|
||||
@@ -194,6 +187,9 @@ const FETCHED_BOTH_ALBUMS =
|
||||
"Fetched the files of 1 of 2 collections\n" +
|
||||
"Fetched the files of 2 of 2 collections\n";
|
||||
|
||||
// The same for an account with one album.
|
||||
const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n";
|
||||
|
||||
// Collects everything written to it.
|
||||
class Output {
|
||||
text = "";
|
||||
@@ -206,11 +202,9 @@ let root: string;
|
||||
let stdout: Output;
|
||||
let stderr: Output;
|
||||
|
||||
// The commands log at the default level, info, as `quak` without `-v` or `-q`
|
||||
// does; the levels have their own tests, in `log.test.ts`.
|
||||
const context = (client: Client | null = fakeClient()): CliContext => ({
|
||||
stdout,
|
||||
log: stderrLog(stderr, "info"),
|
||||
stderr,
|
||||
sessionDir: join(root, "session"),
|
||||
cacheDir: join(root, "cache"),
|
||||
loadSession: () => client,
|
||||
@@ -263,7 +257,7 @@ describe("session file", () => {
|
||||
const dir = join(root, "backup");
|
||||
expect(await whoamiCommand(ctx)).toBe(3);
|
||||
expect(await collectionsCommand(ctx, {})).toBe(3);
|
||||
expect(await filesCommand(ctx, "1", {})).toBe(3);
|
||||
expect(await filesCommand(ctx, { collection: "1" })).toBe(3);
|
||||
expect(await getCommand(ctx, "100", {})).toBe(3);
|
||||
expect(await getThumbCommand(ctx, "100", {})).toBe(3);
|
||||
expect(await backupMetadataCommand(ctx, dir, {})).toBe(3);
|
||||
@@ -510,14 +504,17 @@ describe("collections", () => {
|
||||
|
||||
describe("files", () => {
|
||||
it("prints one tab-separated line per file", async () => {
|
||||
expect(await filesCommand(context(), "1", {})).toBe(0);
|
||||
expect(await filesCommand(context(), { collection: "1" })).toBe(0);
|
||||
expect(stdout.text).toBe(
|
||||
"100\timage\tbeach.jpg\n" + "101\timage\tsunset.jpg\n",
|
||||
);
|
||||
});
|
||||
|
||||
it("prints a JSON array with --json", async () => {
|
||||
const code = await filesCommand(context(), "2", { json: true });
|
||||
const code = await filesCommand(context(), {
|
||||
collection: "2",
|
||||
json: true,
|
||||
});
|
||||
expect(code).toBe(0);
|
||||
expect(JSON.parse(stdout.text)).toEqual([
|
||||
{
|
||||
@@ -531,69 +528,39 @@ describe("files", () => {
|
||||
});
|
||||
|
||||
it("exits 1 for an unknown collection", async () => {
|
||||
expect(await filesCommand(context(), "9", {})).toBe(1);
|
||||
expect(stderr.text).toBe("Collection 9 not found\n");
|
||||
});
|
||||
|
||||
// Without a collection ID it lists every file in the account once (issue
|
||||
// #211). beach.jpg is in Vacation and Work, and Vacation comes first, so it
|
||||
// is listed once, as Vacation's.
|
||||
it("without a collection ID prints every file once", async () => {
|
||||
const ctx = context(fakeClient({ beachInWork: true }));
|
||||
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||
expect(stdout.text).toBe(
|
||||
"100\timage\tbeach.jpg\n" +
|
||||
"101\timage\tsunset.jpg\n" +
|
||||
"200\timage\tdiagram.png\n",
|
||||
expect(await filesCommand(context(), { collection: "9" })).toBe(1);
|
||||
expect(stderr.text).toBe(
|
||||
FETCHED_BOTH_ALBUMS + "Collection 9 not found\n",
|
||||
);
|
||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS);
|
||||
});
|
||||
|
||||
it("without a collection ID gives each file the first album holding it with --json", async () => {
|
||||
const ctx = context(fakeClient({ beachInWork: true }));
|
||||
expect(await filesCommand(ctx, undefined, { json: true })).toBe(0);
|
||||
expect(
|
||||
JSON.parse(stdout.text).map(
|
||||
(row: { id: number; collectionID: number }) => [
|
||||
row.id,
|
||||
row.collectionID,
|
||||
],
|
||||
),
|
||||
).toEqual([
|
||||
[100, 1],
|
||||
[101, 1],
|
||||
[200, 2],
|
||||
]);
|
||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS);
|
||||
});
|
||||
});
|
||||
|
||||
describe("get and get-thumb", () => {
|
||||
it("get finds a file in any album", async () => {
|
||||
it("get finds a file in any album without --collection", async () => {
|
||||
const out = join(root, "diagram.png");
|
||||
expect(await getCommand(context(), "200", { out })).toBe(0);
|
||||
expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
|
||||
expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
|
||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + `7 bytes -> ${out}\n`);
|
||||
});
|
||||
|
||||
it("get-thumb finds a file in any album", async () => {
|
||||
it("get-thumb finds a file in any album without --collection", async () => {
|
||||
const out = join(root, "thumb.jpg");
|
||||
expect(await getThumbCommand(context(), "200", { out })).toBe(0);
|
||||
expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
|
||||
expect(stderr.text).toBe(`3 bytes -> ${out}\n`);
|
||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + `3 bytes -> ${out}\n`);
|
||||
});
|
||||
|
||||
it("get exits 1 when no album has the file", async () => {
|
||||
const out = join(root, "x");
|
||||
expect(await getCommand(context(), "999", { out })).toBe(1);
|
||||
expect(stderr.text).toBe("File 999 not found\n");
|
||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + "File 999 not found\n");
|
||||
expect(existsSync(out)).toBe(false);
|
||||
});
|
||||
|
||||
it("get-thumb exits 1 when no album has the file", async () => {
|
||||
const out = join(root, "x");
|
||||
expect(await getThumbCommand(context(), "999", { out })).toBe(1);
|
||||
expect(stderr.text).toBe("File 999 not found\n");
|
||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + "File 999 not found\n");
|
||||
expect(existsSync(out)).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -641,7 +608,7 @@ describe("names from an album name or a title over 255 bytes", () => {
|
||||
expect(Buffer.byteLength(name)).toBe(255);
|
||||
expect(code).toBe(0);
|
||||
expect(readdirSync(dir)).toEqual([name]);
|
||||
expect(stderr.text).toBe(`7 bytes -> ${name}\n`);
|
||||
expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `7 bytes -> ${name}\n`);
|
||||
});
|
||||
|
||||
it("get-thumb cuts the title short so that thumb_ and it fit in 255 bytes, and exits 0", async () => {
|
||||
@@ -653,7 +620,7 @@ describe("names from an album name or a title over 255 bytes", () => {
|
||||
expect(Buffer.byteLength(name)).toBe(255);
|
||||
expect(code).toBe(0);
|
||||
expect(readdirSync(dir)).toEqual([name]);
|
||||
expect(stderr.text).toBe(`3 bytes -> ${name}\n`);
|
||||
expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `3 bytes -> ${name}\n`);
|
||||
});
|
||||
|
||||
it("backup-metadata cuts the album name short so that its folder's name fits in 255 bytes, and exits 0", async () => {
|
||||
@@ -742,7 +709,8 @@ describe("a live photo", () => {
|
||||
Buffer.from(VIDEO),
|
||||
);
|
||||
expect(stderr.text).toBe(
|
||||
`${IMAGE.length} bytes -> IMG_0300.heic\n` +
|
||||
FETCHED_ONE_ALBUM +
|
||||
`${IMAGE.length} bytes -> IMG_0300.heic\n` +
|
||||
`${VIDEO.length} bytes -> IMG_0300.mov\n`,
|
||||
);
|
||||
});
|
||||
@@ -796,7 +764,8 @@ describe("a live photo", () => {
|
||||
expect(readFileSync(out)).toEqual(Buffer.from(IMAGE));
|
||||
expect(readFileSync(video)).toEqual(Buffer.from(VIDEO));
|
||||
expect(stderr.text).toBe(
|
||||
`${IMAGE.length} bytes -> ${out}\n${VIDEO.length} bytes -> ${video}\n`,
|
||||
FETCHED_ONE_ALBUM +
|
||||
`${IMAGE.length} bytes -> ${out}\n${VIDEO.length} bytes -> ${video}\n`,
|
||||
);
|
||||
});
|
||||
|
||||
@@ -810,7 +779,8 @@ describe("a live photo", () => {
|
||||
expect(existsSync(out)).toBe(false);
|
||||
expect(existsSync(join(root, "photo.mov"))).toBe(false);
|
||||
expect(stderr.text).toBe(
|
||||
`File 300 is a live photo, and its video would also be written to ${out}\n`,
|
||||
FETCHED_ONE_ALBUM +
|
||||
`File 300 is a live photo, and its video would also be written to ${out}\n`,
|
||||
);
|
||||
});
|
||||
|
||||
@@ -905,40 +875,6 @@ describe("backup", () => {
|
||||
);
|
||||
});
|
||||
|
||||
// A failed file is an error line both where the backup fails it and in
|
||||
// the list at the end. The last line is fatal, so `-q`, which prints
|
||||
// fatal lines only, still says why the backup exits 1.
|
||||
it("logs a failed file at error, and ends with one fatal line counting the failures", async () => {
|
||||
const dir = join(root, "backup");
|
||||
const last = `quak: 1 file(s) and 0 album(s) failed; see ${join(dir, "failures.json")} and the FAILED lines\n`;
|
||||
const ctx = context(fakeClient({ failID: 101 }));
|
||||
expect(
|
||||
await backupCommand(
|
||||
{ ...ctx, log: stderrLog(stderr, "error") },
|
||||
dir,
|
||||
{},
|
||||
),
|
||||
).toBe(1);
|
||||
expect(stderr.text).toBe(
|
||||
"FAILED original sunset.jpg: HTTP 500 from server\n" +
|
||||
"\nFailed files:\n" +
|
||||
" [Vacation] sunset.jpg (id 101): HTTP 500 from server\n" +
|
||||
last,
|
||||
);
|
||||
|
||||
stderr.text = "";
|
||||
expect(
|
||||
await backupCommand(
|
||||
{ ...ctx, log: stderrLog(stderr, "fatal") },
|
||||
dir,
|
||||
{ json: true },
|
||||
),
|
||||
).toBe(1);
|
||||
expect(stderr.text).toBe(last);
|
||||
});
|
||||
|
||||
// `--json` shapes stdout only: the log lines stay on stderr, and the JSON
|
||||
// takes the place of the summary.
|
||||
it("prints the result as JSON with --json, still exiting 1 on a failure", async () => {
|
||||
const ctx = context(fakeClient({ failID: 101 }));
|
||||
const code = await backupCommand(ctx, join(root, "backup"), {
|
||||
@@ -953,31 +889,7 @@ describe("backup", () => {
|
||||
failed: 1,
|
||||
});
|
||||
expect(result.errors[0].fileID).toBe(101);
|
||||
expect(stderr.text).toContain(
|
||||
"Fetching original sunset.jpg (101)...\n",
|
||||
);
|
||||
expect(stderr.text).not.toContain("Backup complete");
|
||||
});
|
||||
|
||||
// On an empty cache the library's first refresh fetches the files of
|
||||
// every album before the backup can start, which on a large account takes
|
||||
// a while; each album it has fetched the files of is counted on stderr,
|
||||
// with `--json` too.
|
||||
it("counts on stderr each album whose files its first refresh fetched, with --json too", async () => {
|
||||
const start =
|
||||
"Starting backup...\n" +
|
||||
FETCHED_BOTH_ALBUMS +
|
||||
"Refreshing library...\n";
|
||||
for (const json of [false, true]) {
|
||||
stderr.text = "";
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
const ctx = {
|
||||
...context(),
|
||||
cacheDir: mkdtempSync(join(root, "cache-")),
|
||||
};
|
||||
expect(await backupCommand(ctx, dir, { json })).toBe(0);
|
||||
expect(stderr.text.slice(0, start.length)).toBe(start);
|
||||
}
|
||||
expect(stderr.text).toBe("Starting backup...\n");
|
||||
});
|
||||
|
||||
it("--verify downloads again an original that does not match its hash, prints the counts, and exits 0", async () => {
|
||||
@@ -1067,7 +979,6 @@ describe("backup", () => {
|
||||
backupCommand(ctx, dir, {}),
|
||||
new PassThrough(),
|
||||
runStderr,
|
||||
stderrLog(runStderr, "info"),
|
||||
resolve,
|
||||
);
|
||||
});
|
||||
@@ -1308,9 +1219,7 @@ describe("helper list-missing-thumbnails", () => {
|
||||
expect(stderr.text).toContain("No missing thumbnails found.\n");
|
||||
});
|
||||
|
||||
// `--json` shapes stdout only: the progress stays on stderr, and the JSON
|
||||
// takes the place of the count and the list.
|
||||
it("prints a JSON array with --json, and its progress on stderr", async () => {
|
||||
it("prints a JSON array with --json and no progress", async () => {
|
||||
const ctx = context(fakeClient({ emptyThumbID: 200 }));
|
||||
expect(await listMissingThumbnailsCommand(ctx, { json: true })).toBe(0);
|
||||
expect(JSON.parse(stdout.text)).toEqual([
|
||||
@@ -1321,54 +1230,7 @@ describe("helper list-missing-thumbnails", () => {
|
||||
reason: "empty thumbnail (0 bytes)",
|
||||
},
|
||||
]);
|
||||
expect(stderr.text).toContain(FETCHED_BOTH_ALBUMS);
|
||||
expect(stderr.text).toContain("[Work] Checking thumbnails...\n");
|
||||
expect(stderr.text).not.toContain("with missing thumbnails");
|
||||
});
|
||||
});
|
||||
|
||||
describe("helper fix-missing-thumbnails", () => {
|
||||
// File 999 is in no album, so fixing it fails. The list of failed files
|
||||
// is at error, and the last line is fatal, so `-q`, which prints fatal
|
||||
// lines only, still says why the command exits 1.
|
||||
it("lists a failed file at error, and ends with one fatal line counting the failures", async () => {
|
||||
const last = `quak: 1 file(s) failed; see the "Failed files:" list, or the JSON with --json\n`;
|
||||
const ctx = { ...context(), log: stderrLog(stderr, "error") };
|
||||
expect(await fixMissingThumbnailsCommand(ctx, { file: ["999"] })).toBe(
|
||||
1,
|
||||
);
|
||||
expect(stderr.text).toBe(
|
||||
"\nFailed files:\n" +
|
||||
" 999\tunknown\tfile not found in any collection\n" +
|
||||
last,
|
||||
);
|
||||
|
||||
stderr.text = "";
|
||||
const quiet = { ...context(), log: stderrLog(stderr, "fatal") };
|
||||
expect(
|
||||
await fixMissingThumbnailsCommand(quiet, {
|
||||
file: ["999"],
|
||||
json: true,
|
||||
}),
|
||||
).toBe(1);
|
||||
expect(stderr.text).toBe(last);
|
||||
});
|
||||
|
||||
// beach.jpg (100) is in Vacation (1) and Work (2), both updated at the same
|
||||
// time. The fixer names a file after the first album holding it in the
|
||||
// order of `Library.albums.list()`. On a tie, the album with the higher ID
|
||||
// comes first, so beach.jpg is named after Work.
|
||||
it("names a photo in two albums after the same album on an empty cache as on a cache that holds the account", async () => {
|
||||
const ctx = context(fakeClient({ beachInWork: true }));
|
||||
const opts = { file: ["100"], json: true };
|
||||
const inWork = [{ fileID: 100, collection: "Work", status: "skipped" }];
|
||||
expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
|
||||
expect(JSON.parse(stdout.text)).toMatchObject(inWork);
|
||||
|
||||
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||
stdout.text = "";
|
||||
expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
|
||||
expect(JSON.parse(stdout.text)).toMatchObject(inWork);
|
||||
expect(stderr.text).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
@@ -1399,11 +1261,11 @@ describe("backup-metadata --exif", () => {
|
||||
});
|
||||
|
||||
// Each test first runs `files` so the cache holds the account as it was, then
|
||||
// changes the server under it. (`collections` and `files` with a collection ID
|
||||
// would not do: they store nothing.)
|
||||
// changes the server under it. (`collections` would not do: it reads the
|
||||
// collection list alone and stores nothing.)
|
||||
describe("backup-metadata and the thumbnail helpers refresh first", () => {
|
||||
beforeEach(async () => {
|
||||
expect(await filesCommand(context(), undefined, {})).toBe(0);
|
||||
expect(await filesCommand(context(), { collection: "1" })).toBe(0);
|
||||
stdout.text = "";
|
||||
stderr.text = "";
|
||||
});
|
||||
@@ -1462,10 +1324,10 @@ describe("backup-metadata and the thumbnail helpers refresh first", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// `files`, `get`, `get-thumb` and `helper fix-missing-thumbnails --file` take
|
||||
// an ID only when it is written in decimal digits and is a safe integer.
|
||||
// `Number()` alone reads `0x10` as 16, `1e3` as 1000 and an empty string as 0,
|
||||
// so the command would act on a file the user never named.
|
||||
// `files --collection`, `get`, `get-thumb` and `helper fix-missing-thumbnails
|
||||
// --file` take an ID only when it is written in decimal digits and is a safe
|
||||
// integer. `Number()` alone reads `0x10` as 16, `1e3` as 1000 and an empty
|
||||
// string as 0, so the command would act on a file the user never named.
|
||||
// Anything else is refused with one line naming it, and exit 1, before any
|
||||
// request.
|
||||
//
|
||||
@@ -1519,18 +1381,21 @@ describe("an ID given on the command line", () => {
|
||||
|
||||
it("files looks up a decimal collection ID", async () => {
|
||||
const ctx = emptyAccount();
|
||||
expect(await filesCommand(ctx, "1", {})).toBe(1);
|
||||
expect(await filesCommand(ctx, { collection: "1" })).toBe(1);
|
||||
expect(stderr.text).toBe("Collection 1 not found\n");
|
||||
expect(requests).not.toHaveLength(0);
|
||||
});
|
||||
|
||||
it.each(NOT_IDS)("files refuses %j before any request", async (arg) => {
|
||||
const ctx = emptyAccount();
|
||||
expect(await filesCommand(ctx, arg, {})).toBe(1);
|
||||
expect(stderr.text).toBe(`Invalid collection ID: ${arg}\n`);
|
||||
expect(stdout.text).toBe("");
|
||||
expect(requests).toHaveLength(0);
|
||||
});
|
||||
it.each(NOT_IDS)(
|
||||
"files refuses --collection %j before any request",
|
||||
async (arg) => {
|
||||
const ctx = emptyAccount();
|
||||
expect(await filesCommand(ctx, { collection: arg })).toBe(1);
|
||||
expect(stderr.text).toBe(`Invalid collection ID: ${arg}\n`);
|
||||
expect(stdout.text).toBe("");
|
||||
expect(requests).toHaveLength(0);
|
||||
},
|
||||
);
|
||||
|
||||
it("get looks up a decimal file ID", async () => {
|
||||
expect(await getCommand(emptyAccount(), "100", {})).toBe(1);
|
||||
@@ -1586,83 +1451,20 @@ describe("an ID given on the command line", () => {
|
||||
);
|
||||
});
|
||||
|
||||
// An account with many albums (issues #210, #217 and #222). On an empty cache
|
||||
// `collections`, `files` with a collection ID and `get` used to wait for a
|
||||
// refresh that fetched and decrypted the files of every album, one album after
|
||||
// another. So did `helper fix-missing-thumbnails --file` (issue #223). On such
|
||||
// an account they seemed to hang. The account here has 40 albums of one photo
|
||||
// each, encrypted as Ente's clients encrypt them, on the fake Ente server of
|
||||
// `test/ente-server.ts`. The tests record the path and query of each request
|
||||
// quak sends. The session is the one logging in to that server saves.
|
||||
// An account with many albums (issue #210). On an empty cache `collections`
|
||||
// used to wait for a refresh that fetched and decrypted the files of every
|
||||
// album, one album after another, before it printed anything, so on such an
|
||||
// account it seemed to hang. The account here has 40 albums of one photo each,
|
||||
// encrypted as Ente's clients encrypt them, on the fake Ente server of
|
||||
// `test/ente-server.ts`, which also records the path of each request quak
|
||||
// sends. The session is the one logging in to that server saves.
|
||||
describe("an account with many albums", () => {
|
||||
const ALBUMS = 40;
|
||||
const PASSWORD = "many-albums";
|
||||
const requests: string[] = [];
|
||||
const paths: string[] = [];
|
||||
let server: typeof globalThis.fetch;
|
||||
let snapshot: ClientSnapshot;
|
||||
|
||||
// What `files` without a collection ID prints on an empty cache: every
|
||||
// album's photo on stdout, and on stderr each album whose files its
|
||||
// refresh fetched.
|
||||
let everyPhoto = "";
|
||||
let progress = "";
|
||||
for (let n = 1; n <= ALBUMS; n++) {
|
||||
everyPhoto += `${n * 100}\timage\tphoto-${n}.jpg\n`;
|
||||
progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`;
|
||||
}
|
||||
|
||||
// The key of each album, by album ID.
|
||||
const albumKeys = new Map<number, Uint8Array>();
|
||||
|
||||
// A photo in album `collectionID` as the server sends it, and its original
|
||||
// encrypted as the server stores it. The original's bytes are the title.
|
||||
// These tests fetch no thumbnail. The server records a thumbnail size, so
|
||||
// `helper fix-missing-thumbnails` downloads the original before it skips
|
||||
// the photo as not a JPEG.
|
||||
const photoRow = (photo: {
|
||||
id: number;
|
||||
collectionID: number;
|
||||
title: string;
|
||||
updationTime: number;
|
||||
}): { row: Record<string, unknown>; original: Uint8Array } => {
|
||||
const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const photoKeyBox = encryptSecretbox(
|
||||
photoKey,
|
||||
albumKeys.get(photo.collectionID)!,
|
||||
);
|
||||
const metadata = encryptStreamBlob(
|
||||
new TextEncoder().encode(
|
||||
JSON.stringify({
|
||||
title: photo.title,
|
||||
fileType: 0,
|
||||
creationTime: 1000,
|
||||
modificationTime: 1000,
|
||||
}),
|
||||
),
|
||||
photoKey,
|
||||
);
|
||||
const original = encryptStreamBlob(
|
||||
new TextEncoder().encode(photo.title),
|
||||
photoKey,
|
||||
);
|
||||
const row = {
|
||||
id: photo.id,
|
||||
collectionID: photo.collectionID,
|
||||
ownerID: USER_ID,
|
||||
encryptedKey: toBase64(photoKeyBox.ciphertext),
|
||||
keyDecryptionNonce: toBase64(photoKeyBox.nonce),
|
||||
metadata: {
|
||||
encryptedData: toBase64(metadata.ciphertext),
|
||||
decryptionHeader: toBase64(metadata.header),
|
||||
},
|
||||
file: { decryptionHeader: toBase64(original.header) },
|
||||
thumbnail: { decryptionHeader: "" },
|
||||
info: { thumbSize: 1000 },
|
||||
updationTime: photo.updationTime,
|
||||
};
|
||||
return { row, original: original.ciphertext };
|
||||
};
|
||||
|
||||
// Album n is named `Album n` and holds one photo, n * 100, titled
|
||||
// `photo-n.jpg`.
|
||||
beforeAll(async () => {
|
||||
@@ -1670,10 +1472,8 @@ describe("an account with many albums", () => {
|
||||
const account = await buildAccount(PASSWORD);
|
||||
const collections: Record<string, unknown>[] = [];
|
||||
const filesByCollection: Record<number, Record<string, unknown>[]> = {};
|
||||
const fileCiphertexts: Record<number, Uint8Array> = {};
|
||||
for (let id = 1; id <= ALBUMS; id++) {
|
||||
const albumKey = sodium.crypto_secretbox_keygen();
|
||||
albumKeys.set(id, albumKey);
|
||||
const albumKeyBox = encryptSecretbox(albumKey, account.masterKey);
|
||||
const name = encryptSecretbox(
|
||||
new TextEncoder().encode(`Album ${id}`),
|
||||
@@ -1689,14 +1489,37 @@ describe("an account with many albums", () => {
|
||||
type: "album",
|
||||
updationTime: 1,
|
||||
});
|
||||
const { row, original } = photoRow({
|
||||
id: id * 100,
|
||||
collectionID: id,
|
||||
title: `photo-${id}.jpg`,
|
||||
updationTime: 1,
|
||||
});
|
||||
filesByCollection[id] = [row];
|
||||
fileCiphertexts[id * 100] = original;
|
||||
const photoKey =
|
||||
sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const photoKeyBox = encryptSecretbox(photoKey, albumKey);
|
||||
const metadata = encryptStreamBlob(
|
||||
new TextEncoder().encode(
|
||||
JSON.stringify({
|
||||
title: `photo-${id}.jpg`,
|
||||
fileType: 0,
|
||||
creationTime: 1000,
|
||||
modificationTime: 1000,
|
||||
}),
|
||||
),
|
||||
photoKey,
|
||||
);
|
||||
filesByCollection[id] = [
|
||||
{
|
||||
id: id * 100,
|
||||
collectionID: id,
|
||||
ownerID: USER_ID,
|
||||
encryptedKey: toBase64(photoKeyBox.ciphertext),
|
||||
keyDecryptionNonce: toBase64(photoKeyBox.nonce),
|
||||
metadata: {
|
||||
encryptedData: toBase64(metadata.ciphertext),
|
||||
decryptionHeader: toBase64(metadata.header),
|
||||
},
|
||||
// These tests fetch no original and no thumbnail.
|
||||
file: { decryptionHeader: "" },
|
||||
thumbnail: { decryptionHeader: "" },
|
||||
updationTime: 1,
|
||||
},
|
||||
];
|
||||
}
|
||||
const ente = buildMetaFetch({
|
||||
verifier: account.verifier,
|
||||
@@ -1706,7 +1529,7 @@ describe("an account with many albums", () => {
|
||||
collections,
|
||||
filesByCollection,
|
||||
encryptedMLData: {},
|
||||
fileCiphertexts,
|
||||
fileCiphertexts: {},
|
||||
});
|
||||
server = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||
const url =
|
||||
@@ -1715,8 +1538,7 @@ describe("an account with many albums", () => {
|
||||
: input instanceof URL
|
||||
? input.href
|
||||
: input.url;
|
||||
const { pathname, search } = new URL(url);
|
||||
requests.push(pathname + search);
|
||||
paths.push(new URL(url).pathname);
|
||||
return ente(input, init);
|
||||
}) as typeof globalThis.fetch;
|
||||
const client = await Client.login({
|
||||
@@ -1728,17 +1550,16 @@ describe("an account with many albums", () => {
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
requests.length = 0;
|
||||
paths.length = 0;
|
||||
});
|
||||
|
||||
// A context whose saved session sends its requests to `fetch`, by default
|
||||
// the fake server.
|
||||
const accountContext = (fetch = server): CliContext => {
|
||||
// A context whose saved session sends its requests to the fake server.
|
||||
const accountContext = (): CliContext => {
|
||||
const ctx = context();
|
||||
saveSession(ctx.sessionDir, snapshot);
|
||||
return {
|
||||
...ctx,
|
||||
loadSession: (path) => loadSession(path, { fetch }),
|
||||
loadSession: (path) => loadSession(path, { fetch: server }),
|
||||
};
|
||||
};
|
||||
|
||||
@@ -1751,219 +1572,47 @@ describe("an account with many albums", () => {
|
||||
albums += `${id}\talbum\tAlbum ${id}\n`;
|
||||
expect(stdout.text).toBe(albums);
|
||||
expect(stderr.text).toBe("");
|
||||
expect(requests).toEqual(["/collections/v2?sinceTime=0"]);
|
||||
expect(paths).toEqual(["/collections/v2"]);
|
||||
|
||||
// It stored nothing that would make a refresh skip an album's files:
|
||||
// `files` after it still lists every album's photo.
|
||||
// `files` after it still finds the album's photo.
|
||||
stdout.text = "";
|
||||
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||
expect(stdout.text).toBe(everyPhoto);
|
||||
});
|
||||
|
||||
it("files with a collection ID asks for the album list and that album's files alone", async () => {
|
||||
expect(await filesCommand(accountContext(), "7", {})).toBe(0);
|
||||
|
||||
expect(await filesCommand(ctx, { collection: "7" })).toBe(0);
|
||||
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
||||
expect(stderr.text).toBe("");
|
||||
expect(requests).toEqual([
|
||||
"/collections/v2?sinceTime=0",
|
||||
"/collections/v2/diff?collectionID=7&sinceTime=0",
|
||||
]);
|
||||
});
|
||||
|
||||
// Here the server sends album 7's files in two pages. Photo 700 changed
|
||||
// between them, so both pages hold a copy of it.
|
||||
it("files with a collection ID lists only the later copy of a photo sent in two pages", async () => {
|
||||
const firstPage = [
|
||||
photoRow({
|
||||
id: 700,
|
||||
collectionID: 7,
|
||||
title: "photo-7.jpg",
|
||||
updationTime: 1,
|
||||
}).row,
|
||||
photoRow({
|
||||
id: 701,
|
||||
collectionID: 7,
|
||||
title: "photo-701.jpg",
|
||||
updationTime: 1,
|
||||
}).row,
|
||||
];
|
||||
const secondPage = [
|
||||
photoRow({
|
||||
id: 700,
|
||||
collectionID: 7,
|
||||
title: "photo-7-edited.jpg",
|
||||
updationTime: 2,
|
||||
}).row,
|
||||
];
|
||||
const paged = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||
const url = new URL(input instanceof Request ? input.url : input);
|
||||
if (
|
||||
url.pathname !== "/collections/v2/diff" ||
|
||||
url.searchParams.get("collectionID") !== "7"
|
||||
) {
|
||||
return server(input, init);
|
||||
}
|
||||
const first = url.searchParams.get("sinceTime") === "0";
|
||||
return new Response(
|
||||
JSON.stringify({
|
||||
diff: first ? firstPage : secondPage,
|
||||
hasMore: first,
|
||||
}),
|
||||
);
|
||||
}) as typeof globalThis.fetch;
|
||||
|
||||
expect(await filesCommand(accountContext(paged), "7", {})).toBe(0);
|
||||
|
||||
// Photo 700 is listed once, with the later title, before photo 701.
|
||||
expect(stdout.text).toBe(
|
||||
"700\timage\tphoto-7-edited.jpg\n701\timage\tphoto-701.jpg\n",
|
||||
);
|
||||
});
|
||||
|
||||
// The cache stays empty, so a later `files` without a collection ID still
|
||||
// fetches the files of all 40 albums.
|
||||
it("files with a collection ID stores nothing in the cache", async () => {
|
||||
const ctx = accountContext();
|
||||
expect(await filesCommand(ctx, "7", {})).toBe(0);
|
||||
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
|
||||
|
||||
stdout.text = "";
|
||||
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||
expect(stdout.text).toBe(everyPhoto);
|
||||
expect(stderr.text).toBe(progress);
|
||||
});
|
||||
|
||||
// Photo 200 is in album 2, so `get` asks for no file of album 3 or later.
|
||||
// The last request downloads the photo's original.
|
||||
it("get on an empty cache asks for the albums' files only until one holds the photo", async () => {
|
||||
const out = join(root, "photo.jpg");
|
||||
expect(await getCommand(accountContext(), "200", { out })).toBe(0);
|
||||
|
||||
expect(readFileSync(out, "utf-8")).toBe("photo-2.jpg");
|
||||
expect(stderr.text).toBe(`11 bytes -> ${out}\n`);
|
||||
expect(requests).toEqual([
|
||||
"/collections/v2?sinceTime=0",
|
||||
"/collections/v2/diff?collectionID=1&sinceTime=0",
|
||||
"/collections/v2/diff?collectionID=2&sinceTime=0",
|
||||
"/?fileID=200",
|
||||
]);
|
||||
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
|
||||
});
|
||||
|
||||
// `files` first fills the cache, so `get` waits for a refresh. No album has
|
||||
// changed since, so the refresh asks for no album's files.
|
||||
it("get on a cache that holds the account asks for no unchanged album's files, also for a photo in no album", async () => {
|
||||
const ctx = accountContext();
|
||||
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||
requests.length = 0;
|
||||
stderr.text = "";
|
||||
|
||||
const out = join(root, "photo.jpg");
|
||||
expect(await getCommand(ctx, "4000", { out })).toBe(0);
|
||||
expect(readFileSync(out, "utf-8")).toBe("photo-40.jpg");
|
||||
expect(stderr.text).toBe(`12 bytes -> ${out}\n`);
|
||||
expect(requests).toEqual([
|
||||
"/collections/v2?sinceTime=1",
|
||||
"/?fileID=4000",
|
||||
]);
|
||||
|
||||
requests.length = 0;
|
||||
stderr.text = "";
|
||||
expect(await getCommand(ctx, "999", { out })).toBe(1);
|
||||
expect(stderr.text).toBe("File 999 not found\n");
|
||||
expect(requests).toEqual(["/collections/v2?sinceTime=1"]);
|
||||
});
|
||||
|
||||
// The helper searches the most recently updated album first. All 40 were
|
||||
// updated at the same time, and on a tie the higher ID comes first. Photo
|
||||
// 3900 is in album 39, so the helper asks for no file of album 38 or
|
||||
// earlier. The last request downloads the photo's original.
|
||||
it("fix-missing-thumbnails --file on an empty cache asks for the albums' files only until one holds the photo", async () => {
|
||||
const opts = { file: ["3900"], json: true };
|
||||
expect(await fixMissingThumbnailsCommand(accountContext(), opts)).toBe(
|
||||
0,
|
||||
);
|
||||
|
||||
const reason =
|
||||
"unsupported image format (only baseline JPEG can be regenerated)";
|
||||
expect(JSON.parse(stdout.text)).toEqual([
|
||||
{
|
||||
fileID: 3900,
|
||||
title: "photo-39.jpg",
|
||||
collection: "Album 39",
|
||||
status: "skipped",
|
||||
reason,
|
||||
},
|
||||
]);
|
||||
expect(stderr.text).toBe(
|
||||
"[Album 39] Downloading photo-39.jpg for thumbnail generation...\n" +
|
||||
`[Album 39] Skipping photo-39.jpg: ${reason}\n`,
|
||||
);
|
||||
expect(requests).toEqual([
|
||||
"/collections/v2?sinceTime=0",
|
||||
"/collections/v2/diff?collectionID=40&sinceTime=0",
|
||||
"/collections/v2/diff?collectionID=39&sinceTime=0",
|
||||
"/?fileID=3900",
|
||||
]);
|
||||
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
|
||||
});
|
||||
|
||||
// `files` first fills the cache, so the helper waits for a refresh. No
|
||||
// album has changed since, so the refresh asks for no album's files.
|
||||
it("fix-missing-thumbnails --file on a cache that holds the account asks for no unchanged album's files, also for a photo in no album", async () => {
|
||||
const ctx = accountContext();
|
||||
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||
requests.length = 0;
|
||||
stdout.text = "";
|
||||
stderr.text = "";
|
||||
|
||||
const opts = { file: ["3900"], json: true };
|
||||
expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
|
||||
const reason =
|
||||
"unsupported image format (only baseline JPEG can be regenerated)";
|
||||
expect(JSON.parse(stdout.text)).toEqual([
|
||||
{
|
||||
fileID: 3900,
|
||||
title: "photo-39.jpg",
|
||||
collection: "Album 39",
|
||||
status: "skipped",
|
||||
reason,
|
||||
},
|
||||
]);
|
||||
expect(stderr.text).toBe(
|
||||
"[Album 39] Downloading photo-39.jpg for thumbnail generation...\n" +
|
||||
`[Album 39] Skipping photo-39.jpg: ${reason}\n`,
|
||||
);
|
||||
expect(requests).toEqual([
|
||||
"/collections/v2?sinceTime=1",
|
||||
"/?fileID=3900",
|
||||
]);
|
||||
|
||||
requests.length = 0;
|
||||
stdout.text = "";
|
||||
const notFound = { file: ["999"], json: true };
|
||||
expect(await fixMissingThumbnailsCommand(ctx, notFound)).toBe(1);
|
||||
expect(JSON.parse(stdout.text)).toMatchObject([
|
||||
{ fileID: 999, reason: "file not found in any collection" },
|
||||
]);
|
||||
expect(requests).toEqual(["/collections/v2?sinceTime=1"]);
|
||||
});
|
||||
|
||||
it("files counts on stderr each album whose files its refresh fetched", async () => {
|
||||
expect(await filesCommand(accountContext(), undefined, {})).toBe(0);
|
||||
expect(await filesCommand(accountContext(), { collection: "7" })).toBe(
|
||||
0,
|
||||
);
|
||||
|
||||
expect(stdout.text).toBe(everyPhoto);
|
||||
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
||||
let progress = "";
|
||||
for (let n = 1; n <= ALBUMS; n++) {
|
||||
progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`;
|
||||
}
|
||||
expect(stderr.text).toBe(progress);
|
||||
});
|
||||
|
||||
// `--json` shapes stdout only, so the refresh's progress is still counted
|
||||
// on stderr.
|
||||
it("files --json counts on stderr each album whose files its refresh fetched", async () => {
|
||||
const opts = { json: true };
|
||||
expect(await filesCommand(accountContext(), undefined, opts)).toBe(0);
|
||||
// With `--json` the refresh's progress is not printed, as `backup` and
|
||||
// the thumbnail helpers print none of theirs.
|
||||
it("files --json prints no progress", async () => {
|
||||
expect(
|
||||
await filesCommand(accountContext(), {
|
||||
collection: "7",
|
||||
json: true,
|
||||
}),
|
||||
).toBe(0);
|
||||
|
||||
expect(JSON.parse(stdout.text)).toHaveLength(ALBUMS);
|
||||
expect(stderr.text).toBe(progress);
|
||||
expect(JSON.parse(stdout.text)).toEqual([
|
||||
{
|
||||
id: 700,
|
||||
title: "photo-7.jpg",
|
||||
fileType: "image",
|
||||
creationTime: 1000,
|
||||
collectionID: 7,
|
||||
},
|
||||
]);
|
||||
expect(stderr.text).toBe("");
|
||||
});
|
||||
});
|
||||
@@ -1,359 +0,0 @@
|
||||
/**
|
||||
* Tests for the CLI's log levels, set with `-v`/`--verbose` and `-q`/`--quiet`
|
||||
* (`src/cli-log.ts`, issue #212).
|
||||
*
|
||||
* Every line quak prints on stderr is a log line at one of six levels, from
|
||||
* the quietest: fatal, error, warning, info, debug, trace. A run prints the
|
||||
* lines at its own level and at every quieter one. The level is info unless
|
||||
* asked otherwise: `-v` raises it to debug, `-vv` to trace, and more repeats
|
||||
* stay at trace. `-q` lowers it to fatal, so a run prints only the line a
|
||||
* command prints as it fails with a non-zero exit code. stdout holds the
|
||||
* command's output at every level.
|
||||
*
|
||||
* Most tests run `bin/quak.ts`, as the smoke test does, against an account of
|
||||
* one album holding one photo: the saved session is in a temporary directory,
|
||||
* and `fetch` is a fake of the Ente server that serves the album and the photo
|
||||
* encrypted as Ente's clients encrypt them. Each run has a cache directory of
|
||||
* its own, so its refresh fetches the album's files, as on a first run.
|
||||
*/
|
||||
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import sodium from "libsodium-wrappers-sumo";
|
||||
import {
|
||||
afterAll,
|
||||
afterEach,
|
||||
beforeAll,
|
||||
describe,
|
||||
expect,
|
||||
it,
|
||||
vi,
|
||||
} from "vitest";
|
||||
|
||||
import { ApiClient } from "../../src/api/client.js";
|
||||
import { saveSession } from "../../src/cli-commands.js";
|
||||
import { logRefresh, logRequests, type Log } from "../../src/cli-log.js";
|
||||
import { init, toBase64 } from "../../src/crypto/index.js";
|
||||
import { encryptSecretbox, encryptStreamBlob } from "../ente-server.js";
|
||||
|
||||
// `bin/quak.ts` reads the saved session from the data directory env-paths
|
||||
// gives it, which here is a temporary directory made before the first run.
|
||||
const paths = vi.hoisted(() => ({ data: "" }));
|
||||
vi.mock("env-paths", () => ({ default: () => paths }));
|
||||
|
||||
const USER_ID = 42;
|
||||
const root = mkdtempSync(join(tmpdir(), "quak-log-test-"));
|
||||
const argv = process.argv;
|
||||
|
||||
class ExitCalled extends Error {}
|
||||
|
||||
// Run `quak <args>` to its exit, with a cache of its own, and return its exit
|
||||
// code and what it wrote on stdout and on stderr, which are kept off the
|
||||
// terminal. `process.exit` is stubbed to throw, so nothing runs after it, as
|
||||
// nothing does after a real exit.
|
||||
const quak = async (
|
||||
...args: string[]
|
||||
): Promise<{ code: number; stdout: string; stderr: string }> => {
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
let code = NaN;
|
||||
vi.spyOn(process.stdout, "write").mockImplementation((text) => {
|
||||
stdout += String(text);
|
||||
return true;
|
||||
});
|
||||
vi.spyOn(process.stderr, "write").mockImplementation((text) => {
|
||||
stderr += String(text);
|
||||
return true;
|
||||
});
|
||||
vi.spyOn(process, "exit").mockImplementation((exitCode) => {
|
||||
code = Number(exitCode ?? 0);
|
||||
throw new ExitCalled();
|
||||
});
|
||||
const cache = mkdtempSync(join(root, "cache-"));
|
||||
process.argv = ["node", "quak", "--cache-dir", cache, ...args];
|
||||
vi.resetModules();
|
||||
await expect(import("../../bin/quak.js")).rejects.toBeInstanceOf(
|
||||
ExitCalled,
|
||||
);
|
||||
return { code, stdout, stderr };
|
||||
};
|
||||
|
||||
// Album 1, "Trips", of user 42, holding photo 100, "beach.jpg". The album's
|
||||
// key is encrypted with the account's master key, the photo's key with the
|
||||
// album's, and the photo's metadata with its own key. Only the session and
|
||||
// these two records are needed: the tests never log in.
|
||||
beforeAll(async () => {
|
||||
await init();
|
||||
await sodium.ready;
|
||||
const masterKey = sodium.crypto_secretbox_keygen();
|
||||
const keyPair = sodium.crypto_box_keypair();
|
||||
paths.data = join(root, "session");
|
||||
saveSession(paths.data, {
|
||||
email: "log@example.com",
|
||||
userID: USER_ID,
|
||||
token: "token",
|
||||
masterKey: toBase64(masterKey),
|
||||
secretKey: toBase64(keyPair.privateKey),
|
||||
publicKey: toBase64(keyPair.publicKey),
|
||||
});
|
||||
|
||||
const albumKey = sodium.crypto_secretbox_keygen();
|
||||
const albumKeyBox = encryptSecretbox(albumKey, masterKey);
|
||||
const name = encryptSecretbox(new TextEncoder().encode("Trips"), albumKey);
|
||||
const album = {
|
||||
id: 1,
|
||||
owner: { id: USER_ID },
|
||||
encryptedKey: toBase64(albumKeyBox.ciphertext),
|
||||
keyDecryptionNonce: toBase64(albumKeyBox.nonce),
|
||||
encryptedName: toBase64(name.ciphertext),
|
||||
nameDecryptionNonce: toBase64(name.nonce),
|
||||
type: "album",
|
||||
updationTime: 1,
|
||||
};
|
||||
const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const photoKeyBox = encryptSecretbox(photoKey, albumKey);
|
||||
const metadata = encryptStreamBlob(
|
||||
new TextEncoder().encode(
|
||||
JSON.stringify({
|
||||
title: "beach.jpg",
|
||||
fileType: 0,
|
||||
creationTime: 1000,
|
||||
modificationTime: 1000,
|
||||
}),
|
||||
),
|
||||
photoKey,
|
||||
);
|
||||
const photo = {
|
||||
id: 100,
|
||||
collectionID: 1,
|
||||
ownerID: USER_ID,
|
||||
encryptedKey: toBase64(photoKeyBox.ciphertext),
|
||||
keyDecryptionNonce: toBase64(photoKeyBox.nonce),
|
||||
metadata: {
|
||||
encryptedData: toBase64(metadata.ciphertext),
|
||||
decryptionHeader: toBase64(metadata.header),
|
||||
},
|
||||
// These tests fetch no original and no thumbnail.
|
||||
file: { decryptionHeader: "" },
|
||||
thumbnail: { decryptionHeader: "" },
|
||||
updationTime: 1,
|
||||
};
|
||||
|
||||
const json = (body: unknown): Response =>
|
||||
new Response(JSON.stringify(body), {
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
vi.stubGlobal("fetch", async (input: string) => {
|
||||
const { pathname } = new URL(input);
|
||||
if (pathname === "/collections/v2") {
|
||||
return json({ collections: [album] });
|
||||
}
|
||||
if (pathname === "/collections/v2/diff") {
|
||||
return json({ diff: [photo], hasMore: false });
|
||||
}
|
||||
return new Response("not found", { status: 404 });
|
||||
});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
process.argv = argv;
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
vi.unstubAllGlobals();
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// What `quak files` writes on stdout at every level.
|
||||
const FILES = "100\timage\tbeach.jpg\n";
|
||||
|
||||
// What it writes on stderr at each level. Info is the refresh's progress, one
|
||||
// line for the one album whose files it fetched. Debug adds the start and end
|
||||
// of the refresh, and trace each request with the status of its answer.
|
||||
const INFO = "Fetched the files of 1 of 1 collections\n";
|
||||
const DEBUG = "Refresh started\n" + INFO + "Refresh done\n";
|
||||
const TRACE =
|
||||
"Refresh started\n" +
|
||||
"GET https://api.ente.io/collections/v2?sinceTime=0: HTTP 200\n" +
|
||||
"GET https://api.ente.io/collections/v2/diff?collectionID=1&sinceTime=0: HTTP 200\n" +
|
||||
INFO +
|
||||
"Refresh done\n";
|
||||
|
||||
describe("the log level", () => {
|
||||
it("is info by default", async () => {
|
||||
expect(await quak("files")).toEqual({
|
||||
code: 0,
|
||||
stdout: FILES,
|
||||
stderr: INFO,
|
||||
});
|
||||
});
|
||||
|
||||
it("is debug with -v", async () => {
|
||||
expect(await quak("-v", "files")).toEqual({
|
||||
code: 0,
|
||||
stdout: FILES,
|
||||
stderr: DEBUG,
|
||||
});
|
||||
});
|
||||
|
||||
it("is trace with -vv", async () => {
|
||||
expect(await quak("-vv", "files")).toEqual({
|
||||
code: 0,
|
||||
stdout: FILES,
|
||||
stderr: TRACE,
|
||||
});
|
||||
});
|
||||
|
||||
it("stays at trace with -vvv", async () => {
|
||||
expect((await quak("-vvv", "files")).stderr).toBe(TRACE);
|
||||
});
|
||||
|
||||
it("counts --verbose --verbose as -vv", async () => {
|
||||
expect((await quak("--verbose", "--verbose", "files")).stderr).toBe(
|
||||
TRACE,
|
||||
);
|
||||
});
|
||||
|
||||
// Both are options of the program, taken before or after the command's
|
||||
// name, as `--cache-dir` is.
|
||||
it("is set by an option after the command's name too", async () => {
|
||||
expect((await quak("files", "-v")).stderr).toBe(DEBUG);
|
||||
expect((await quak("files", "--json", "-vv")).stderr).toBe(TRACE);
|
||||
expect((await quak("files", "-q")).stderr).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
describe("-q", () => {
|
||||
it("prints nothing on stderr when the command works", async () => {
|
||||
expect(await quak("-q", "files")).toEqual({
|
||||
code: 0,
|
||||
stdout: FILES,
|
||||
stderr: "",
|
||||
});
|
||||
});
|
||||
|
||||
// No album holds file 999, so fixing it fails. The summary from
|
||||
// `--- Done ---` to the end of the `Failed files:` list is left out. The
|
||||
// line saying why the command failed is the fatal one, and it is printed.
|
||||
it("prints only the line a failing command exits with", async () => {
|
||||
const fix = ["helper", "fix-missing-thumbnails", "--file", "999"];
|
||||
expect(await quak("-q", ...fix)).toEqual({
|
||||
code: 1,
|
||||
stdout: "",
|
||||
stderr: `quak: 1 file(s) failed; see the "Failed files:" list, or the JSON with --json\n`,
|
||||
});
|
||||
});
|
||||
|
||||
it("is refused with -v, in one line, before the command runs", async () => {
|
||||
for (const args of [
|
||||
["-q", "-v", "files"],
|
||||
["files", "--verbose", "--quiet"],
|
||||
]) {
|
||||
const { code, stdout, stderr } = await quak(...args);
|
||||
expect(code).toBe(1);
|
||||
expect(stdout).toBe("");
|
||||
expect(stderr).toBe(
|
||||
"error: option '-q, --quiet' cannot be used with option '-v, --verbose'\n",
|
||||
);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// `--help` lists both options, and each command's help lists them too, since
|
||||
// each command takes them.
|
||||
describe("--help", () => {
|
||||
it("lists -v and -q for the program and for each command", async () => {
|
||||
for (const args of [["--help"], ["files", "--help"]]) {
|
||||
const { stdout } = await quak(...args);
|
||||
expect(stdout).toContain("-v, --verbose");
|
||||
expect(stdout).toContain("-q, --quiet");
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// The library writes no lines itself. Its layers report through callbacks,
|
||||
// and the CLI logs what they report.
|
||||
describe("what the library reports", () => {
|
||||
const collect = (): { log: Log; lines: string[] } => {
|
||||
const lines: string[] = [];
|
||||
return { log: (level, line) => lines.push(`${level}: ${line}`), lines };
|
||||
};
|
||||
|
||||
// A request answered 503 and retried: each attempt is a request, logged
|
||||
// at trace with its status, and the retry is logged at debug with how
|
||||
// long it waits and why.
|
||||
it("logs each request at trace and each retry at debug", async () => {
|
||||
const { log, lines } = collect();
|
||||
const answers = [
|
||||
new Response("unavailable", { status: 503 }),
|
||||
new Response("{}", {
|
||||
headers: { "content-type": "application/json" },
|
||||
}),
|
||||
];
|
||||
const client = new ApiClient({
|
||||
...logRequests(log, {
|
||||
sleep: async () => {},
|
||||
random: () => 0.5,
|
||||
}),
|
||||
fetch: async () => answers.shift()!,
|
||||
});
|
||||
|
||||
await client.getJSON("/collections/v2", { sinceTime: 0 });
|
||||
|
||||
const url = "https://api.ente.io/collections/v2?sinceTime=0";
|
||||
expect(lines).toEqual([
|
||||
`trace: GET ${url}: HTTP 503`,
|
||||
"debug: Attempt 1 failed, retrying in 250 ms: HTTP 503",
|
||||
`trace: GET ${url}: HTTP 200`,
|
||||
]);
|
||||
});
|
||||
|
||||
it("logs a request that failed with its error", async () => {
|
||||
const { log, lines } = collect();
|
||||
const client = new ApiClient({
|
||||
...logRequests(log, { attempts: 1 }),
|
||||
fetch: async () => {
|
||||
throw new TypeError("fetch failed", {
|
||||
cause: new Error("connect ECONNREFUSED 1.2.3.4:443"),
|
||||
});
|
||||
},
|
||||
});
|
||||
|
||||
await expect(client.getJSON("/collections/v2")).rejects.toThrow();
|
||||
|
||||
expect(lines).toEqual([
|
||||
"trace: GET https://api.ente.io/collections/v2: fetch failed: connect ECONNREFUSED 1.2.3.4:443",
|
||||
]);
|
||||
});
|
||||
|
||||
// Each operation is named in plain words.
|
||||
it("logs a refresh's progress at info, and its start and end at debug", () => {
|
||||
const { log, lines } = collect();
|
||||
const onProgress = logRefresh(log);
|
||||
|
||||
onProgress({ operation: "refresh", status: "started" });
|
||||
onProgress({
|
||||
operation: "refresh",
|
||||
status: "progress",
|
||||
fetched: 1,
|
||||
total: 2,
|
||||
});
|
||||
onProgress({
|
||||
operation: "refresh",
|
||||
status: "failed",
|
||||
error: "HTTP 503",
|
||||
});
|
||||
onProgress({ operation: "fetchMLData", status: "started" });
|
||||
onProgress({ operation: "fetchMLData", status: "done", fetched: 3 });
|
||||
|
||||
expect(lines).toEqual([
|
||||
"debug: Refresh started",
|
||||
"info: Fetched the files of 1 of 2 collections",
|
||||
"debug: Refresh failed: HTTP 503",
|
||||
"debug: ML data fetch started",
|
||||
"debug: ML data fetch done",
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -41,14 +41,12 @@ import { beforeAll, afterAll, describe, expect, it, vi } from "vitest";
|
||||
import { init, toBase64, encryptBlob } from "../../src/crypto/index.js";
|
||||
import * as jpegJs from "jpeg-js";
|
||||
import { Client } from "../../src/client.js";
|
||||
import { DISCOVER_CATEGORIES } from "../../src/clip-keyword-vectors.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import {
|
||||
runMetadataBackup,
|
||||
type MetadataBackupOptions,
|
||||
} from "../../src/metadata-backup.js";
|
||||
import { backupMetadataCommand } from "../../src/cli-commands.js";
|
||||
import { stderrLog, type LogLevel } from "../../src/cli-log.js";
|
||||
import {
|
||||
buildAccount,
|
||||
buildMetaFetch,
|
||||
@@ -57,8 +55,8 @@ import {
|
||||
type MetaMockState,
|
||||
} from "../ente-server.js";
|
||||
|
||||
// One file per ML data request, so each file of the mock account is fetched in
|
||||
// a request of its own and one of them can fail on its own.
|
||||
// One file per ML data request, so the two files of the mock account are
|
||||
// fetched in two requests and one of them can fail on its own.
|
||||
vi.mock("../../src/mldata-fetch.js", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("../../src/mldata-fetch.js")>()),
|
||||
MLDATA_BATCH_SIZE: 1,
|
||||
@@ -67,21 +65,6 @@ vi.mock("../../src/mldata-fetch.js", async (importOriginal) => ({
|
||||
const TEST_EMAIL = "metabackup@example.com";
|
||||
const TEST_PASSWORD = "metapass";
|
||||
|
||||
// The CLIP embedding of a photo of a sunset (see clip-keywords.test.ts), which
|
||||
// the ML data of beach.jpg holds.
|
||||
const SUNSET_EMBEDDING: number[] = JSON.parse(
|
||||
readFileSync(
|
||||
new URL("../clip-keywords-sunset.json", import.meta.url),
|
||||
"utf-8",
|
||||
),
|
||||
).embedding;
|
||||
|
||||
// Identity, a Discover category of Ente's photos app that leaves out videos.
|
||||
const IDENTITY = DISCOVER_CATEGORIES.find((c) => c.title === "Identity")!;
|
||||
|
||||
// ML data holding faces and no CLIP embedding, which file 302 has.
|
||||
const FACES_ONLY = { face: { faces: [{ faceID: "face-def", score: 0.9 }] } };
|
||||
|
||||
let mock: MetaMockState;
|
||||
let testDir: string;
|
||||
|
||||
@@ -220,65 +203,6 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
|
||||
updationTime: 1710000000000000,
|
||||
};
|
||||
|
||||
// A file in coll 10 with the ML data given. `fileType` is Ente's number: 0
|
||||
// an image, 1 a video.
|
||||
const vacationFile = (
|
||||
id: number,
|
||||
title: string,
|
||||
fileType: number,
|
||||
mlData: object,
|
||||
) => {
|
||||
const fileKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const { ciphertext: encKey, nonce } = encryptSecretbox(fileKey, ck1);
|
||||
const meta = encryptStreamBlob(
|
||||
new TextEncoder().encode(
|
||||
JSON.stringify({
|
||||
title,
|
||||
fileType,
|
||||
creationTime: 1700000000000000,
|
||||
modificationTime: 1700000000000000,
|
||||
}),
|
||||
),
|
||||
fileKey,
|
||||
);
|
||||
const ml = encryptBlob(
|
||||
new Uint8Array(gzipSync(Buffer.from(JSON.stringify(mlData)))),
|
||||
fileKey,
|
||||
);
|
||||
return {
|
||||
raw: {
|
||||
id,
|
||||
collectionID: 10,
|
||||
ownerID: 42,
|
||||
encryptedKey: toBase64(encKey),
|
||||
keyDecryptionNonce: toBase64(nonce),
|
||||
metadata: {
|
||||
encryptedData: toBase64(meta.ciphertext),
|
||||
decryptionHeader: toBase64(meta.header),
|
||||
},
|
||||
file: {
|
||||
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
|
||||
},
|
||||
thumbnail: {
|
||||
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
|
||||
},
|
||||
updationTime: 1700000000000000,
|
||||
},
|
||||
mlData: {
|
||||
encryptedData: toBase64(ml.ciphertext),
|
||||
decryptionHeader: toBase64(ml.header),
|
||||
},
|
||||
};
|
||||
};
|
||||
// Files 300 and 301 in coll 10: a photo and a video whose ML data holds
|
||||
// only a CLIP embedding, Identity's own text embedding.
|
||||
const identityML = { clip: { embedding: IDENTITY.textEmbedding } };
|
||||
const identityPhoto = vacationFile(300, "passport.jpg", 0, identityML);
|
||||
const identityVideo = vacationFile(301, "passport.mp4", 1, identityML);
|
||||
// File 302 in coll 10: a photo whose ML data holds faces and no CLIP
|
||||
// embedding.
|
||||
const facesPhoto = vacationFile(302, "party.jpg", 0, FACES_ONLY);
|
||||
|
||||
// Encrypt ML data for file 100 (gzipped JSON, encrypted with file key)
|
||||
const mlPayload = JSON.stringify({
|
||||
face: {
|
||||
@@ -305,7 +229,7 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
|
||||
clip: {
|
||||
version: 1,
|
||||
client: "test",
|
||||
embedding: SUNSET_EMBEDDING,
|
||||
embedding: [0.5, 0.6, 0.7],
|
||||
},
|
||||
});
|
||||
const gzipped = gzipSync(Buffer.from(mlPayload));
|
||||
@@ -345,23 +269,12 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
|
||||
keyAttributes: account.keyAttributes,
|
||||
encryptedToken: account.encryptedToken,
|
||||
collections: [rawColl1, rawColl2],
|
||||
filesByCollection: {
|
||||
10: [
|
||||
rawFile1,
|
||||
identityPhoto.raw,
|
||||
identityVideo.raw,
|
||||
facesPhoto.raw,
|
||||
],
|
||||
20: [rawFile2],
|
||||
},
|
||||
filesByCollection: { 10: [rawFile1], 20: [rawFile2] },
|
||||
encryptedMLData: {
|
||||
100: {
|
||||
encryptedData: toBase64(mlCiphertext),
|
||||
decryptionHeader: toBase64(mlHeader),
|
||||
},
|
||||
300: identityPhoto.mlData,
|
||||
301: identityVideo.mlData,
|
||||
302: facesPhoto.mlData,
|
||||
},
|
||||
fileCiphertexts: { 100: encFileBody1 },
|
||||
};
|
||||
@@ -518,9 +431,7 @@ describe("quak backup-metadata", () => {
|
||||
expect(account.email).toBe(TEST_EMAIL);
|
||||
});
|
||||
|
||||
// The CLIP embedding is left out of the ML data, and the keywords it gives
|
||||
// are written in its place.
|
||||
it("fetches and decrypts ML data by default, with keywords for the CLIP embedding", async () => {
|
||||
it("fetches and decrypts ML data by default", async () => {
|
||||
const outDir = join(testDir, "ml-data");
|
||||
await runBackup(outDir);
|
||||
|
||||
@@ -540,41 +451,8 @@ describe("quak backup-metadata", () => {
|
||||
expect(fileMeta.mlData.face.faces[0].faceID).toBe("face-abc");
|
||||
expect(fileMeta.mlData.face.faces[0].score).toBeCloseTo(0.98);
|
||||
expect(fileMeta.mlData.face.faces[0].detection.box.x).toBeCloseTo(0.1);
|
||||
expect(fileMeta.mlData).not.toHaveProperty("clip");
|
||||
expect(fileMeta.keywords).toEqual(["Sunset"]);
|
||||
});
|
||||
|
||||
// Ente's photos app keeps videos out of some Discover categories, Identity
|
||||
// among them. Files 300 and 301, a photo and a video, share one CLIP
|
||||
// embedding, Identity's own text embedding, so the photo gets the keyword
|
||||
// Identity and the video does not.
|
||||
it("keeps a video out of a category that leaves out videos", async () => {
|
||||
expect(IDENTITY.showVideo).toBe(false);
|
||||
const outDir = join(testDir, "video-keywords");
|
||||
await runBackup(outDir);
|
||||
|
||||
const vacation = join(outDir, "collections", "10-Vacation");
|
||||
const fileJSON = (fileID: number) =>
|
||||
JSON.parse(readFileSync(join(vacation, `${fileID}.json`), "utf-8"));
|
||||
expect(fileJSON(300).keywords).toContain("Identity");
|
||||
expect(fileJSON(301)).toHaveProperty("keywords");
|
||||
expect(fileJSON(301).keywords).not.toContain("Identity");
|
||||
});
|
||||
|
||||
// A file's ML data can hold faces and no CLIP embedding. File 302's JSON
|
||||
// then has the ML data and no keywords.
|
||||
it("writes no keywords for ML data without a CLIP embedding", async () => {
|
||||
const outDir = join(testDir, "no-clip");
|
||||
await runBackup(outDir);
|
||||
|
||||
const fileMeta = JSON.parse(
|
||||
readFileSync(
|
||||
join(outDir, "collections", "10-Vacation", "302.json"),
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
expect(fileMeta.mlData).toEqual(FACES_ONLY);
|
||||
expect(fileMeta).not.toHaveProperty("keywords");
|
||||
expect(fileMeta.mlData.clip).toBeDefined();
|
||||
expect(fileMeta.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]);
|
||||
});
|
||||
|
||||
it("extracts EXIF from downloaded files when --exif is set", async () => {
|
||||
@@ -612,12 +490,9 @@ describe("quak backup-metadata", () => {
|
||||
});
|
||||
|
||||
describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
// Run the CLI command against `fetch`, the mock unless given, logging at
|
||||
// `level`, and return its exit code, stderr and output directory.
|
||||
const runCommand = async (
|
||||
fetch = buildMetaFetch(mock),
|
||||
level: LogLevel = "info",
|
||||
) => {
|
||||
// Run the CLI command against `fetch`, the mock unless given, and return
|
||||
// its exit code, stderr and output directory.
|
||||
const runCommand = async (fetch = buildMetaFetch(mock)) => {
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
@@ -631,10 +506,7 @@ describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
const code = await backupMetadataCommand(
|
||||
{
|
||||
stdout: { write: () => true },
|
||||
log: stderrLog(
|
||||
{ write: (text: string) => (stderr += text) },
|
||||
level,
|
||||
),
|
||||
stderr: { write: (text: string) => (stderr += text) },
|
||||
sessionDir: testDir,
|
||||
cacheDir: mkdtempSync(join(testDir, "cache-")),
|
||||
loadSession: () => client,
|
||||
@@ -645,16 +517,13 @@ describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
return { code, stderr, outDir };
|
||||
};
|
||||
|
||||
// The failed request is logged at error, so it is the first line printed
|
||||
// at that level.
|
||||
it("writes every file, marks the failed batch's files, and exits 1", async () => {
|
||||
const { code, stderr, outDir } = await runCommand(
|
||||
buildMetaFetch(mock, 200),
|
||||
"error",
|
||||
);
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toMatch(/^ML data request for 1 file\(s\) failed: /);
|
||||
expect(stderr).toContain("ML data request for 1 file(s) failed");
|
||||
|
||||
const ok = JSON.parse(
|
||||
readFileSync(
|
||||
@@ -662,7 +531,7 @@ describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
expect(ok.keywords).toEqual(["Sunset"]);
|
||||
expect(ok.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]);
|
||||
expect(ok.mlDataError).toBeUndefined();
|
||||
|
||||
const failed = JSON.parse(
|
||||
@@ -677,9 +546,7 @@ describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
});
|
||||
|
||||
// The server sends ML data for diagram.png encrypted under a key that is
|
||||
// not the file's, so quak cannot decrypt it. That is logged at error, and
|
||||
// the last line is fatal, so `-q`, which prints fatal lines only, still
|
||||
// says why the command exits 1.
|
||||
// not the file's, so quak cannot decrypt it.
|
||||
it("writes the reason for ML data that does not decrypt, and exits 1", async () => {
|
||||
const otherKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const { header, ciphertext } = encryptBlob(
|
||||
@@ -697,14 +564,12 @@ describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
},
|
||||
},
|
||||
}),
|
||||
"error",
|
||||
);
|
||||
|
||||
expect(code).toBe(1);
|
||||
const reason = "secretstream chunk authentication failed";
|
||||
expect(stderr).toBe(
|
||||
`ML data of file 200 could not be read: ${reason}\n` +
|
||||
`quak: the ML data of 1 file(s) could not be read; see mlDataError in their JSON in ${outDir}\n`,
|
||||
expect(stderr).toContain(
|
||||
`ML data of file 200 could not be read: ${reason}\n`,
|
||||
);
|
||||
const failed = JSON.parse(
|
||||
readFileSync(
|
||||
@@ -720,7 +585,7 @@ describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
expect(ok.keywords).toEqual(["Sunset"]);
|
||||
expect(ok.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]);
|
||||
});
|
||||
|
||||
it("exits 0 when every ML data request succeeds", async () => {
|
||||
|
||||
+49
-171
@@ -3,25 +3,20 @@
|
||||
* issue #36, issue #52).
|
||||
*
|
||||
* The `collections`, `files`, `get`, and `get-thumb` commands must answer for
|
||||
* current server state, not the local cache. `get`, `get-thumb` and `files`
|
||||
* without a collection ID force a `Library.fresh()` round-trip before they
|
||||
* read. The stand-in library below serves no cached record until `fresh()` has
|
||||
* been awaited, so a helper that read without refreshing would come back empty
|
||||
* and fail here. `collections` asks the library for the collection list alone
|
||||
* (`freshCollections()`, issue #210). `files` with a collection ID asks for
|
||||
* that list, then for that one collection's files (`freshFiles()`, issue
|
||||
* #217). Neither may wait for a refresh, which would fetch the files of every
|
||||
* collection. On an empty cache, `get` and `get-thumb` ask for that list, then
|
||||
* for one collection's files at a time, until they find the file (issue #222).
|
||||
* `helper fix-missing-thumbnails --file` does the same until it finds every
|
||||
* file it names, reading the most recently updated collection first (issue
|
||||
* #223).
|
||||
* current server state, not the local cache. `files`, `get` and `get-thumb`
|
||||
* force a `Library.fresh()` round-trip before they read; the stand-in library
|
||||
* below serves nothing until `fresh()` has been awaited, so a helper that read
|
||||
* without refreshing would come back empty and fail here. `collections` asks
|
||||
* the library for the collection list alone (`freshCollections()`, issue
|
||||
* #210), and must not wait for a refresh, which would fetch the files of every
|
||||
* collection.
|
||||
*
|
||||
* `collections` and `files` also list in the order the library gives them, not
|
||||
* `collections` and `files` also list in the library's enumeration order
|
||||
* (`listCollections`/`listFiles`) — the order the pre-library CLI printed — not
|
||||
* the albums/photos projection's newest-first order. The fixtures are seeded in
|
||||
* an order that a newest-first sort would rearrange, so a regression to the
|
||||
* projection order would fail here too. Field values still come from the raw
|
||||
* metadata via `cli-output.ts`.
|
||||
* an enumeration order that a newest-first sort would rearrange, so a
|
||||
* regression to the projection order would fail here too. Field values still
|
||||
* come from the raw metadata via `cli-output.ts`.
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
@@ -30,10 +25,10 @@ import {
|
||||
freshCollections,
|
||||
freshFiles,
|
||||
freshFile,
|
||||
freshFilesByID,
|
||||
type FreshReadLibrary,
|
||||
} from "../../src/cli-read.js";
|
||||
import { fileListRow } from "../../src/cli-output.js";
|
||||
import type { Photo } from "../../src/library/index.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
|
||||
const collection = (id: number, updationTime: number): Collection => ({
|
||||
@@ -67,15 +62,14 @@ const file = (
|
||||
updationTime: creationTime,
|
||||
});
|
||||
|
||||
// A library that reveals its cached records only after `fresh()` has been
|
||||
// awaited, and serves them in the enumeration order it was given.
|
||||
// `freshCollections()` and `freshFiles()` answer in that order without a
|
||||
// refresh.
|
||||
// A library that reveals its records only after `fresh()` has been awaited, and
|
||||
// serves them in the enumeration order it was given. `freshCollections()`
|
||||
// answers the collection list in that order without a refresh. `photos.byID`
|
||||
// returns a stand-in `Photo` carrying just the fileID the helper passes
|
||||
// through.
|
||||
class FakeLibrary implements FreshReadLibrary {
|
||||
freshCalls = 0;
|
||||
freshCollectionsCalls = 0;
|
||||
// The ID of each collection `freshFiles()` was asked for.
|
||||
freshFilesCalls: number[] = [];
|
||||
private refreshed = false;
|
||||
|
||||
constructor(
|
||||
@@ -94,13 +88,9 @@ class FakeLibrary implements FreshReadLibrary {
|
||||
return this.collections;
|
||||
}
|
||||
|
||||
async freshFiles(collection: Collection): Promise<EnteFile[]> {
|
||||
this.freshFilesCalls.push(collection.id);
|
||||
return this.files.filter((f) => f.collectionID === collection.id);
|
||||
}
|
||||
|
||||
listCollections(): Collection[] {
|
||||
return this.refreshed ? this.collections : [];
|
||||
getCollection(id: number): Collection | undefined {
|
||||
if (!this.refreshed) return undefined;
|
||||
return this.collections.find((c) => c.id === id);
|
||||
}
|
||||
|
||||
listFiles(collectionID: number): EnteFile[] {
|
||||
@@ -113,6 +103,14 @@ class FakeLibrary implements FreshReadLibrary {
|
||||
if (!this.refreshed) return undefined;
|
||||
return this.files.find((f) => f.id === fileID);
|
||||
}
|
||||
|
||||
photos = {
|
||||
byID: ({ fileID }: { fileID: number }): Photo | undefined => {
|
||||
if (!this.refreshed) return undefined;
|
||||
if (!this.files.some((f) => f.id === fileID)) return undefined;
|
||||
return { fileID } as unknown as Photo;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
||||
@@ -136,24 +134,18 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
||||
expect(rows.map((c) => c.id)).not.toEqual(newestFirst);
|
||||
});
|
||||
|
||||
it("freshFiles with a collection ID reads that collection's files alone and keeps their order and raw fields", async () => {
|
||||
// Given in the order 10, 11, 12; creationTimes ascending, so a
|
||||
it("freshFiles refreshes first, lists in enumeration order, keeps raw fields", async () => {
|
||||
// Enumeration order by id 10, 11, 12; creationTimes ascending, so a
|
||||
// newest-first sort would reverse them.
|
||||
const files = [
|
||||
file(10, 1, 1_700_000_000_000_000),
|
||||
file(11, 1, 1_700_000_000_000_001),
|
||||
file(12, 1, 1_700_000_000_000_002),
|
||||
file(20, 2, 1_700_000_000_000_003),
|
||||
];
|
||||
const lib = new FakeLibrary(
|
||||
[collection(1, 100), collection(2, 200)],
|
||||
files,
|
||||
);
|
||||
const lib = new FakeLibrary([collection(1, 100)], files);
|
||||
|
||||
const rows = await freshFiles(lib, 1);
|
||||
expect(lib.freshCalls).toBe(0);
|
||||
expect(lib.freshCollectionsCalls).toBe(1);
|
||||
expect(lib.freshFilesCalls).toEqual([1]);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(rows?.map((f) => f.id)).toEqual([10, 11, 12]);
|
||||
|
||||
// Field values come from raw metadata: microsecond creationTime and the
|
||||
@@ -183,144 +175,30 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
// Without a collection ID, `files` lists every file in the account once
|
||||
// (issue #211). File 11 is in collections 1 and 2; collection 2 is
|
||||
// enumerated first, so 11 is listed once, with collection 2's record.
|
||||
it("freshFiles without a collection ID lists every file once, from the first collection holding it", async () => {
|
||||
const files = [
|
||||
file(10, 1, 1_700_000_000_000_000),
|
||||
file(11, 1, 1_700_000_000_000_001),
|
||||
file(20, 2, 1_700_000_000_000_002),
|
||||
file(11, 2, 1_700_000_000_000_001),
|
||||
];
|
||||
const lib = new FakeLibrary(
|
||||
[collection(2, 100), collection(1, 200)],
|
||||
files,
|
||||
);
|
||||
|
||||
const rows = await freshFiles(lib);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(rows?.map((f) => [f.id, f.collectionID])).toEqual([
|
||||
[20, 2],
|
||||
[11, 2],
|
||||
[10, 1],
|
||||
]);
|
||||
});
|
||||
|
||||
it("freshFiles returns undefined for an unknown collection", async () => {
|
||||
const lib = new FakeLibrary([collection(1, 100)], []);
|
||||
const rows = await freshFiles(lib, 999);
|
||||
expect(lib.freshCalls).toBe(0);
|
||||
expect(lib.freshFilesCalls).toEqual([]);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(rows).toBeUndefined();
|
||||
});
|
||||
|
||||
// Three collections, each holding one file.
|
||||
const threeCollections = (): FakeLibrary =>
|
||||
new FakeLibrary(
|
||||
[collection(1, 100), collection(2, 200), collection(3, 300)],
|
||||
[
|
||||
file(10, 1, 1_700_000_000_000_000),
|
||||
file(20, 2, 1_700_000_000_000_001),
|
||||
file(30, 3, 1_700_000_000_000_002),
|
||||
],
|
||||
it("freshFile refreshes first, then resolves the photo and its raw record", async () => {
|
||||
const f = file(10, 1, 1_700_000_000_000_000);
|
||||
const lib = new FakeLibrary([collection(1, 100)], [f]);
|
||||
|
||||
const resolved = await freshFile(lib, 10);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(resolved?.photo.fileID).toBe(10);
|
||||
expect(resolved?.file.metadata.title).toBe("file-10.jpg");
|
||||
expect(resolved?.file.metadata.creationTime).toBe(
|
||||
1_700_000_000_000_000,
|
||||
);
|
||||
|
||||
it("freshFile on an empty cache reads each collection's files in the server's order until one holds the file", async () => {
|
||||
const lib = threeCollections();
|
||||
|
||||
const found = await freshFile(lib, 20);
|
||||
expect(lib.freshCalls).toBe(0);
|
||||
expect(lib.freshCollectionsCalls).toBe(1);
|
||||
expect(lib.freshFilesCalls).toEqual([1, 2]);
|
||||
expect(found?.metadata.title).toBe("file-20.jpg");
|
||||
expect(found?.metadata.creationTime).toBe(1_700_000_000_000_001);
|
||||
});
|
||||
|
||||
// The stand-in's cache holds what the server holds once `fresh()` has run.
|
||||
it("freshFile on a cache that holds the collections refreshes, then looks the file up in the cache", async () => {
|
||||
const lib = threeCollections();
|
||||
await lib.fresh();
|
||||
|
||||
const found = await freshFile(lib, 30);
|
||||
expect(lib.freshCalls).toBe(2);
|
||||
expect(lib.freshCollectionsCalls).toBe(0);
|
||||
expect(lib.freshFilesCalls).toEqual([]);
|
||||
expect(found?.collectionID).toBe(3);
|
||||
});
|
||||
|
||||
it("freshFile on an empty cache returns undefined for an unknown file, after reading every collection's files", async () => {
|
||||
const lib = threeCollections();
|
||||
|
||||
const found = await freshFile(lib, 404);
|
||||
expect(lib.freshCalls).toBe(0);
|
||||
expect(lib.freshFilesCalls).toEqual([1, 2, 3]);
|
||||
expect(found).toBeUndefined();
|
||||
});
|
||||
|
||||
// The server lists collections 1, 2 and 3, and collection 3 was updated
|
||||
// most recently. File 11 is in collections 1 and 2.
|
||||
const fileInTwoCollections = (): FakeLibrary =>
|
||||
new FakeLibrary(
|
||||
[collection(1, 100), collection(2, 200), collection(3, 300)],
|
||||
[
|
||||
file(10, 1, 1_700_000_000_000_000),
|
||||
file(11, 1, 1_700_000_000_000_001),
|
||||
file(11, 2, 1_700_000_000_000_001),
|
||||
file(20, 2, 1_700_000_000_000_002),
|
||||
file(30, 3, 1_700_000_000_000_003),
|
||||
],
|
||||
);
|
||||
|
||||
it("freshFilesByID on an empty cache reads the most recently updated collection's files first, until it has found every file", async () => {
|
||||
const lib = fileInTwoCollections();
|
||||
|
||||
const found = await freshFilesByID(lib, [11, 20]);
|
||||
expect(lib.freshCalls).toBe(0);
|
||||
expect(lib.freshCollectionsCalls).toBe(1);
|
||||
expect(lib.freshFilesCalls).toEqual([3, 2]);
|
||||
expect(found.get(11)?.collection.id).toBe(2);
|
||||
expect(found.get(11)?.file.collectionID).toBe(2);
|
||||
expect(found.get(20)?.collection.id).toBe(2);
|
||||
});
|
||||
|
||||
// Collection 1 was updated last, though collections 2 and 3 have higher
|
||||
// IDs. File 11 is in collections 1 and 2.
|
||||
it("freshFilesByID on an empty cache reads the most recently updated collection first, though its ID is the lowest", async () => {
|
||||
const lib = new FakeLibrary(
|
||||
[collection(1, 400), collection(2, 200), collection(3, 300)],
|
||||
[
|
||||
file(11, 1, 1_700_000_000_000_001),
|
||||
file(11, 2, 1_700_000_000_000_001),
|
||||
],
|
||||
);
|
||||
|
||||
const found = await freshFilesByID(lib, [11]);
|
||||
expect(lib.freshFilesCalls).toEqual([1]);
|
||||
expect(found.get(11)?.collection.id).toBe(1);
|
||||
expect(found.get(11)?.file.collectionID).toBe(1);
|
||||
});
|
||||
|
||||
// The stand-in's cache holds what the server holds once `fresh()` has run.
|
||||
it("freshFilesByID on a cache that holds the collections refreshes, then finds each file in the most recently updated collection that holds it", async () => {
|
||||
const lib = fileInTwoCollections();
|
||||
await lib.fresh();
|
||||
|
||||
const found = await freshFilesByID(lib, [11, 404]);
|
||||
expect(lib.freshCalls).toBe(2);
|
||||
expect(lib.freshCollectionsCalls).toBe(0);
|
||||
expect(lib.freshFilesCalls).toEqual([]);
|
||||
expect(found.get(11)?.collection.id).toBe(2);
|
||||
expect(found.get(11)?.file.collectionID).toBe(2);
|
||||
expect(found.has(404)).toBe(false);
|
||||
});
|
||||
|
||||
it("freshFilesByID on an empty cache reads every collection's files when a file is in none of them", async () => {
|
||||
const lib = fileInTwoCollections();
|
||||
|
||||
const found = await freshFilesByID(lib, [10, 404]);
|
||||
expect(lib.freshFilesCalls).toEqual([3, 2, 1]);
|
||||
expect(found.get(10)?.collection.id).toBe(1);
|
||||
expect(found.has(404)).toBe(false);
|
||||
it("freshFile returns undefined for an unknown file", async () => {
|
||||
const lib = new FakeLibrary([collection(1, 100)], []);
|
||||
const resolved = await freshFile(lib, 404);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(resolved).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -6,7 +6,6 @@ import { PassThrough } from "node:stream";
|
||||
import { describe, it, expect } from "vitest";
|
||||
|
||||
import { ApiError } from "../../src/api/client.js";
|
||||
import { stderrLog } from "../../src/cli-log.js";
|
||||
import { run } from "../../src/cli-run.js";
|
||||
|
||||
// A stream whose written text is kept in `text`; writes finish at once, so
|
||||
@@ -24,13 +23,7 @@ const runToExit = async (
|
||||
const stdout = collector();
|
||||
const stderr = collector();
|
||||
const code = await new Promise<number>((resolve) => {
|
||||
void run(
|
||||
command,
|
||||
stdout.stream,
|
||||
stderr.stream,
|
||||
stderrLog(stderr.stream, "info"),
|
||||
resolve,
|
||||
);
|
||||
void run(command, stdout.stream, stderr.stream, resolve);
|
||||
});
|
||||
return { code, stdout: stdout.text(), stderr: stderr.text() };
|
||||
};
|
||||
|
||||
@@ -1,109 +0,0 @@
|
||||
{
|
||||
"photo": "https://upload.wikimedia.org/wikipedia/commons/d/db/Sunset_off_the_Massachusetts_coast_-_NOAA.jpg",
|
||||
"sha256": "be62f3289c9bf7b75af68cd6978021d54e8a531d5d4cc493f6e9fbf5e1ab74d3",
|
||||
"embedding": [
|
||||
-0.0021123015, -0.04437615, -0.062354308, 0.006249992, 0.022141185,
|
||||
0.011159019, 0.01227392, -0.00030532264, 0.03186437, 0.09196922,
|
||||
0.039111234, -0.037084445, -0.032962628, -0.019820206, -0.013141825,
|
||||
0.0047183544, 0.017137334, 0.046581306, 0.0047622235, 0.04982521,
|
||||
0.023128154, 0.032080866, 0.037709303, 0.019241406, 0.0032100384,
|
||||
-0.014275843, 0.049337693, 0.025953151, 0.014263995, -0.0062328726,
|
||||
-0.040504098, -0.015083303, -0.008123875, -0.006233639, -0.05352683,
|
||||
-0.0048083956, 0.0763486, -0.013155067, 0.025114346, 0.022016674,
|
||||
0.0001588023, -0.010341572, -0.0037518279, -0.019181008, 0.04549337,
|
||||
-0.0010725886, 0.004636866, 0.017177915, -0.0117317885, 0.0026097358,
|
||||
0.030738998, 0.013158572, -0.08856403, -0.0121391015, 0.011477229,
|
||||
-0.0033104012, 0.035524756, -0.018607879, -0.0071739843, -0.031906743,
|
||||
0.0056713386, -0.0044195866, 0.00292471, 0.00061686034, -0.043075223,
|
||||
0.09040427, 0.001522537, -0.011445652, 0.009609886, -0.0053201183,
|
||||
-0.0057452377, 0.00972853, -0.0024397348, -0.021255083, 0.015893811,
|
||||
-0.006020548, -0.09428023, 0.014277583, -0.017975299, -0.011878315,
|
||||
0.011825521, 0.00014203975, -0.0020437292, -0.05305343, -0.024840152,
|
||||
-0.011336872, -0.002796541, -0.00036171873, 0.004530499, 0.044486605,
|
||||
0.021602718, -0.01292857, 0.024530916, 0.019559948, -0.039928272,
|
||||
-0.0681174, 0.049101, 0.02461532, 0.0365531, -0.04070358, -0.011011939,
|
||||
-0.0027188526, -0.020370876, 0.012904856, 0.043157287, -0.051658224,
|
||||
-0.3726012, 0.019549074, -0.0499252, -0.04800559, -0.011493859,
|
||||
0.02893256, 0.060341954, 0.0013929688, -0.051283866, -0.024286803,
|
||||
0.044124287, -0.005995806, -0.062817656, 0.007369614, -0.021470634,
|
||||
-0.028147545, 0.00411659, 0.04234773, 0.03787647, 0.008579682,
|
||||
-0.038568925, -0.027498025, -0.028416378, 0.0022012168, 0.010906191,
|
||||
-0.013745787, 0.018267838, 0.013531097, -0.061559845, 0.004682517,
|
||||
-0.005993303, 0.0028113858, -0.0028286583, 0.0026029332, -0.06693403,
|
||||
-0.0072381413, -0.02419584, 0.0037009937, -0.0011296495, -0.030310456,
|
||||
0.0005999609, 0.009945354, 0.024915686, 0.038672045, 0.023009432,
|
||||
-0.05093051, -0.09101409, -0.06686167, -0.06211481, -0.022600474,
|
||||
0.0024902367, -0.0067171026, -0.016769778, -0.0032803628, 0.008623083,
|
||||
-0.0016110836, 0.014925073, 0.033811048, 0.012829449, -0.052575592,
|
||||
0.03728281, -0.0051201633, 0.021238064, -0.031086605, -0.0043402985,
|
||||
0.25054494, -0.028396713, -0.010725523, -0.023445267, 0.021751683,
|
||||
0.003130971, 0.012270548, -0.014836475, 0.014466885, 0.0021478906,
|
||||
-0.088153616, 0.015356315, 0.039639276, -0.040283255, 0.014923924,
|
||||
-0.029303031, -0.004475201, 0.047299225, -0.06726505, -0.012491676,
|
||||
-0.002080582, 0.03506341, 0.04146718, 0.0074114986, -0.009582633,
|
||||
0.006623749, -0.05454898, -0.0034387247, -0.014780821, 0.03698328,
|
||||
-0.034441486, 0.22032379, -0.021792699, 0.008001907, 0.0341019,
|
||||
0.0025784625, -0.0071689896, -0.014151225, -0.050547987, -0.030372301,
|
||||
-0.03288388, -0.042987447, -0.0033656165, 0.021502906, -0.0030374376,
|
||||
-0.024413, -0.03506603, 0.030891743, 0.0069994032, 0.004887366,
|
||||
0.030707076, 0.0065348223, 0.0008240555, 0.016809713, -0.024151647,
|
||||
-0.0040458064, 0.044567548, 0.03908212, -0.061782435, 0.0026002831,
|
||||
0.02003242, -0.0115451, -0.045135964, -0.0663542, 0.005240975,
|
||||
0.0010777109, -0.009276002, -0.011919081, 0.027654119, -0.013634551,
|
||||
0.0129154315, -0.013195512, 0.040571272, -0.099290505, -0.04656693,
|
||||
-0.20979854, 0.04181004, -0.06769166, -0.04121946, 0.018260444,
|
||||
0.005617506, -0.15694222, 0.032472793, 0.014189598, 0.008345103,
|
||||
-0.041118417, 0.0074527003, -0.011494104, 0.004906474, -0.0027100458,
|
||||
0.0009994594, -0.041061398, -0.045274835, 0.015187793, 0.034275036,
|
||||
-0.02813289, 0.02098019, -0.015043198, 0.03716667, 0.013542642,
|
||||
-0.008104835, 0.014965486, -0.027978558, 0.021163663, -0.006942737,
|
||||
-0.034969814, -0.038961593, 0.016802894, -0.012318175, -0.047821593,
|
||||
-0.01933216, 0.0064432374, 0.0322358, -0.020437784, 0.02659447,
|
||||
-0.03022022, -0.019443465, -0.094178624, -0.017009107, 0.03860671,
|
||||
-0.00023695534, -0.033089854, -0.014677462, 0.012356399, 0.018040499,
|
||||
-0.19360113, 0.011710817, -0.019871691, 0.020124774, 0.024948105,
|
||||
-0.024573864, -0.028349072, -0.041731454, 0.032646254, 0.015098165,
|
||||
0.0018447012, -0.01129152, -0.02481687, 0.036048, 0.20797811,
|
||||
0.04317757, -0.017239755, 0.033431605, -0.011384422, 0.0099174455,
|
||||
-0.19739223, -0.008901062, -0.056426458, 0.014591864, 0.0044474266,
|
||||
0.008209326, -0.012241819, 0.042553645, -0.021552453, 0.008611734,
|
||||
0.048944127, -0.013872049, 0.019179989, 0.008183264, -0.053223096,
|
||||
-0.027327595, -0.0033542586, -0.012353858, -0.010941209, 0.010212546,
|
||||
0.029714134, -0.011385593, -0.0013529479, 0.014505385, 0.01148663,
|
||||
0.012613636, -0.0074430504, -0.014008101, -0.028146038, -0.042778917,
|
||||
0.02143165, 0.016812252, -0.06286215, 0.02085532, -0.035128277,
|
||||
0.0241282, 0.01397908, -0.034992814, 0.013153766, 0.0068236147,
|
||||
0.013594184, -0.06859393, 0.051849406, 0.029675119, -0.0037699796,
|
||||
-0.00013216575, 0.02916647, 0.031859048, -0.015006051, 0.0001439966,
|
||||
0.0026584554, -0.0046948595, 0.039754897, 0.034873154, 0.0012870439,
|
||||
-0.011516082, -0.074089095, 0.019241294, 0.00076195505, 0.0027334464,
|
||||
-0.036206458, -0.0011109262, -0.048838835, -0.00204685, 0.01179991,
|
||||
0.0068136845, 0.009852495, 0.017308988, 0.03159545, 0.058803353,
|
||||
0.038116634, 0.008836143, -0.010460637, -0.010994099, -0.0012926942,
|
||||
-0.048370842, 0.037696566, -0.08093239, -0.0041242186, -0.005866373,
|
||||
0.02048607, -0.12549639, 0.011705739, -0.0021425558, 0.018078592,
|
||||
-0.038228087, 0.004245971, -0.11560507, 0.01381203, -0.02734893,
|
||||
0.029595908, 0.101082794, -0.042842254, -0.08397301, -0.1546237,
|
||||
-0.037624564, 0.0069606556, -0.008020479, 0.0043268185, 0.016504597,
|
||||
0.044554908, -0.05240209, 0.020519335, -0.050994243, -0.019861093,
|
||||
0.022066997, -0.0188911, 0.012919696, -0.022872692, -0.012961374,
|
||||
-0.020071112, 0.0074553066, 0.06402014, 0.04065211, 0.014139608,
|
||||
0.06221546, -0.02574805, 0.010114545, 0.020619567, 0.032264773,
|
||||
-0.009148828, 0.022849955, -0.016593235, -0.032456633, -0.006737303,
|
||||
-0.0037493357, -0.065912865, -0.0024181998, -0.0037048983, -0.019304488,
|
||||
0.009085037, 0.057529982, -0.0378338, 0.01792042, 0.01885012,
|
||||
-0.0043600523, 0.017382368, -0.027792772, 0.006749118, -0.0053297766,
|
||||
-0.011628587, 0.015736576, 0.024801968, 0.011543913, -0.01159873,
|
||||
-0.000759976, 0.042183932, 0.05764386, -0.06422416, 0.010577374,
|
||||
-0.02261249, 0.02282793, -0.05141075, -0.11287078, -0.020670144,
|
||||
-0.020953743, 0.030885734, -0.012597732, -0.121472664, 0.028447151,
|
||||
-0.0415664, -0.010061708, 0.06441338, 0.0271207, -0.0016693748,
|
||||
-0.0045694243, 0.020342957, 0.030528478, 0.007129517, -0.0029600926,
|
||||
-0.007061397, 0.06036486, -0.004736018, -0.0090786135, -0.07601116,
|
||||
-0.039556272, -0.051269546, -0.019473685, 0.029858151, 0.028464556,
|
||||
0.012424476, 0.013742254, 0.008742927, 0.006065751, -0.009892663,
|
||||
0.07699999, -0.003443197, 0.0027863425, 0.029197995, 0.0211013,
|
||||
0.01098379, 0.016684106, -0.028917195, 0.008122037, 0.0265241,
|
||||
-0.04461223
|
||||
]
|
||||
}
|
||||
@@ -1,129 +0,0 @@
|
||||
// `clipKeywords`: the keywords quak writes for a file in place of its CLIP
|
||||
// embedding.
|
||||
//
|
||||
// Ente's ML data for a file holds a CLIP embedding, 512 numbers its image
|
||||
// model gave the photo. Ente's photos app puts a file in a category of its
|
||||
// Discover section when the cosine similarity of that embedding and the text
|
||||
// embedding of the category's query is at least the category's minimum score,
|
||||
// and keeps videos out of some categories. `clipKeywords` applies that rule
|
||||
// and returns the titles of the categories a file falls in, highest score
|
||||
// first; `quak backup` and `quak backup-metadata` write them as `keywords`.
|
||||
|
||||
import { readFileSync } from "node:fs";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { DISCOVER_CATEGORIES } from "../src/clip-keyword-vectors.js";
|
||||
import { clipKeywords } from "../src/clip-keywords.js";
|
||||
|
||||
// 512 numbers: the ones given, then zeros.
|
||||
const vector = (...start: number[]): number[] => [
|
||||
...start,
|
||||
...new Array<number>(512 - start.length).fill(0),
|
||||
];
|
||||
|
||||
const withEmbedding = (embedding: unknown) => ({
|
||||
face: { faces: [] },
|
||||
clip: { embedding },
|
||||
});
|
||||
|
||||
// Two made-up categories, each with a text embedding along one axis. The
|
||||
// embedding [3, 4] has a cosine similarity of 3/5 with "Across" and 4/5 with
|
||||
// "Up". Videos are kept out of "Across".
|
||||
const ACROSS = {
|
||||
title: "Across",
|
||||
query: "across",
|
||||
minScore: 0.6,
|
||||
showVideo: false,
|
||||
textEmbedding: vector(1),
|
||||
};
|
||||
const UP = {
|
||||
title: "Up",
|
||||
query: "up",
|
||||
minScore: 0.5,
|
||||
showVideo: true,
|
||||
textEmbedding: vector(0, 1),
|
||||
};
|
||||
|
||||
describe("clipKeywords", () => {
|
||||
it("gives each category whose minimum score the similarity reaches, highest score first", () => {
|
||||
expect(
|
||||
clipKeywords(withEmbedding(vector(3, 4)), false, [ACROSS, UP]),
|
||||
).toEqual(["Up", "Across"]);
|
||||
});
|
||||
|
||||
it("leaves out a category when the similarity is just below its minimum score", () => {
|
||||
const stricter = { ...ACROSS, minScore: 0.6 + 1e-9 };
|
||||
|
||||
expect(
|
||||
clipKeywords(withEmbedding(vector(3, 4)), false, [stricter, UP]),
|
||||
).toEqual(["Up"]);
|
||||
});
|
||||
|
||||
it("keeps a video out of a category that leaves out videos", () => {
|
||||
expect(
|
||||
clipKeywords(withEmbedding(vector(3, 4)), true, [ACROSS, UP]),
|
||||
).toEqual(["Up"]);
|
||||
});
|
||||
|
||||
it("gives an empty list for an embedding that falls in no category", () => {
|
||||
expect(
|
||||
clipKeywords(withEmbedding(vector(0, 0, 1)), false, [ACROSS, UP]),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["no CLIP embedding", { face: { faces: [] } }],
|
||||
["a CLIP part that is not an object", { clip: null }],
|
||||
["a CLIP part without an embedding", { clip: {} }],
|
||||
["an embedding of 511 numbers", withEmbedding(vector(3, 4).slice(1))],
|
||||
["an embedding with NaN in it", withEmbedding(vector(3, NaN))],
|
||||
["an embedding with Infinity in it", withEmbedding(vector(Infinity))],
|
||||
["an embedding of strings", withEmbedding(vector(3, 4).map(String))],
|
||||
["an embedding of zeros", withEmbedding(vector())],
|
||||
])("gives no keywords for ML data with %s", (_, mlData) => {
|
||||
expect(clipKeywords(mlData, false, [ACROSS, UP])).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
// The CLIP embedding Ente's image model gives a public-domain NOAA photo of a
|
||||
// sunset, made by bin/clip-keyword-vectors from the photo at `photo`.
|
||||
const SUNSET: { photo: string; sha256: string; embedding: number[] } =
|
||||
JSON.parse(
|
||||
readFileSync(
|
||||
new URL("clip-keywords-sunset.json", import.meta.url),
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
|
||||
describe("Ente's Discover categories", () => {
|
||||
it("are the 16 of Ente's photos app, each with a text embedding of norm 1", () => {
|
||||
expect(DISCOVER_CATEGORIES.map((c) => c.title)).toEqual([
|
||||
"Identity",
|
||||
"Screenshots",
|
||||
"QR Codes",
|
||||
"Receipts",
|
||||
"Notes",
|
||||
"Memes",
|
||||
"Visiting Cards",
|
||||
"Babies",
|
||||
"Pets",
|
||||
"Selfies",
|
||||
"Wallpapers",
|
||||
"Food",
|
||||
"Celebrations",
|
||||
"Sunset",
|
||||
"Hills",
|
||||
"Greenery",
|
||||
]);
|
||||
for (const category of DISCOVER_CATEGORIES) {
|
||||
expect(category.textEmbedding).toHaveLength(512);
|
||||
expect(Math.hypot(...category.textEmbedding)).toBeCloseTo(1, 5);
|
||||
}
|
||||
});
|
||||
|
||||
it("give a photo of a sunset the keyword Sunset and no other", () => {
|
||||
expect(
|
||||
clipKeywords({ clip: { embedding: SUNSET.embedding } }, false),
|
||||
).toEqual(["Sunset"]);
|
||||
});
|
||||
});
|
||||
+2
-5
@@ -19,11 +19,8 @@ import { SRP, SrpServer } from "fast-srp-hap";
|
||||
import { toBase64, deriveKEK, deriveLoginSubkey } from "../src/crypto/index.js";
|
||||
import type { KeyAttributes } from "../src/auth/types.js";
|
||||
|
||||
// The account's password is stretched at libsodium's lowest Argon2id cost.
|
||||
// Every login stretches it again, and at a realistic cost that takes seconds on
|
||||
// a loaded host.
|
||||
const TEST_OPS = 1;
|
||||
const TEST_MEM = 8192;
|
||||
const TEST_OPS = 2;
|
||||
const TEST_MEM = 64 * 1024 * 1024;
|
||||
|
||||
export interface MetaMockState {
|
||||
verifier: Buffer;
|
||||
|
||||
@@ -216,13 +216,4 @@ describe("defaultFileName", () => {
|
||||
`${"t".repeat(250)}.heic`,
|
||||
);
|
||||
});
|
||||
|
||||
// Cutting `CONSOLE` or `NULL` short before a long extension leaves `CON`
|
||||
// or `NUL`, so the name gets the `_` `sanitizeFileName` gives a device
|
||||
// name, and the part before the extension gives up a byte for it.
|
||||
it("never returns a Windows device name, even once cut short", () => {
|
||||
const ext = `.${"e".repeat(251)}`;
|
||||
expect(defaultFileName(`CONSOLE${ext}`, 42)).toBe(`_CO${ext}`);
|
||||
expect(defaultFileName(`NULL${ext}`, 42)).toBe(`_NU${ext}`);
|
||||
});
|
||||
});
|
||||
@@ -253,12 +253,6 @@ describe("Library content wiring", () => {
|
||||
await expect(
|
||||
lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
|
||||
).rejects.toThrow(/content cache/i);
|
||||
await expect(lib.originalOf(file(1, 1))).rejects.toThrow(
|
||||
/content cache/i,
|
||||
);
|
||||
await expect(lib.thumbnailOf(file(1, 1))).rejects.toThrow(
|
||||
/content cache/i,
|
||||
);
|
||||
await lib.close();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -375,25 +375,6 @@ describe("ContentCache.original / thumbnail", () => {
|
||||
await cache.open();
|
||||
await expect(cache.original(999)).rejects.toThrow(/unknown file/i);
|
||||
});
|
||||
|
||||
// `getFile` knows no file here, so the cache can fetch only the file it is
|
||||
// given.
|
||||
it("originalOf and thumbnailOf fetch and cache the file they are given", async () => {
|
||||
const { cache, source } = buildCache({ files: [] });
|
||||
await cache.open();
|
||||
|
||||
const original = await cache.originalOf(file(5));
|
||||
const thumbnail = await cache.thumbnailOf(file(5));
|
||||
|
||||
expect(source.originalCalls).toEqual([5]);
|
||||
expect(source.thumbnailCalls).toEqual([5]);
|
||||
expect(original.path).toBe(join(cacheDir, "originals", "5.jpg"));
|
||||
expect(thumbnail.path).toBe(join(cacheDir, "thumbnails", "5.jpg"));
|
||||
expect(cache.pathsFor(5)).toEqual({
|
||||
originalPath: original.path,
|
||||
thumbnailPath: thumbnail.path,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("ContentCache.ensureThumbnails", () => {
|
||||
|
||||
@@ -555,36 +555,6 @@ describe("withRetry", () => {
|
||||
expect(delays).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("tells `onRetry` of each attempt it retries, before the wait", async () => {
|
||||
// A caller that logs retries, as the CLI does with `-v`, learns why
|
||||
// each attempt failed and how long the next one is put off. The last
|
||||
// attempt is not retried, so it is not reported.
|
||||
const { sleep, delays } = recordingSleep();
|
||||
const retries: [string, number, number][] = [];
|
||||
let calls = 0;
|
||||
const failure = withRetry(
|
||||
() => {
|
||||
calls++;
|
||||
return Promise.reject(new ApiError(`attempt ${calls}`, 503));
|
||||
},
|
||||
{
|
||||
attempts: 3,
|
||||
sleep,
|
||||
random: () => 0.5,
|
||||
onRetry: (err, attempt, delayMs) => {
|
||||
retries.push([(err as Error).message, attempt, delayMs]);
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
await expect(failure).rejects.toThrow("attempt 3");
|
||||
expect(retries).toEqual([
|
||||
["attempt 1", 1, 250],
|
||||
["attempt 2", 2, 500],
|
||||
]);
|
||||
expect(delays).toEqual([250, 500]);
|
||||
});
|
||||
|
||||
it("does not retry at all when attempts is 1", async () => {
|
||||
const { sleep, delays } = recordingSleep();
|
||||
let calls = 0;
|
||||
@@ -738,7 +708,6 @@ describe("retry defaults", () => {
|
||||
expect(resolved.maxDelayMs).toBe(DEFAULT_RETRY_OPTIONS.maxDelayMs);
|
||||
expect(typeof resolved.sleep).toBe("function");
|
||||
expect(typeof resolved.random).toBe("function");
|
||||
expect(typeof resolved.onRetry).toBe("function");
|
||||
});
|
||||
|
||||
it("resolves to the defaults when given nothing", () => {
|
||||
|
||||
@@ -6,12 +6,10 @@
|
||||
* working thumbnails, others return 404 or empty bodies. The tests
|
||||
* verify that the detection and repair logic handles each case correctly.
|
||||
*
|
||||
* `listMissingThumbnails` takes an open `Library` to enumerate the files. Both
|
||||
* helpers take the `Client` for the API operations: the thumbnail existence
|
||||
* check, and the encrypt-and-upload path. `fixMissingThumbnails` takes the
|
||||
* record of each file, which these tests look up with `cachedFilesByID`. It
|
||||
* reads each original through the library's content cache, by
|
||||
* `lib.originalOf(file)`.
|
||||
* As of issue #52 both helpers take an open `Library` for enumeration and the
|
||||
* `Client` for the API operations that stay unchanged (the thumbnail existence
|
||||
* check, and the encrypt-and-upload path). `fixMissingThumbnails` reads each
|
||||
* original through the library's content cache (`photo.original()`).
|
||||
*
|
||||
* `fixMissingThumbnails` is the most complex function in quak: it
|
||||
* downloads the original file, generates a JPEG thumbnail with jpeg-js,
|
||||
@@ -43,7 +41,6 @@ import {
|
||||
import { SRP, SrpServer } from "fast-srp-hap";
|
||||
import { Client } from "../../src/client.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import { cachedFilesByID } from "../../src/cli-read.js";
|
||||
import {
|
||||
listMissingThumbnails,
|
||||
fixMissingThumbnails,
|
||||
@@ -57,11 +54,8 @@ import type { RetryOptions } from "../../src/retry.js";
|
||||
|
||||
const TEST_EMAIL = "thumb@example.com";
|
||||
const TEST_PASSWORD = "thumbpass";
|
||||
// The password is stretched at libsodium's lowest Argon2id cost. Each test
|
||||
// builds an account and logs in, and at a realistic cost that takes seconds on
|
||||
// a loaded host.
|
||||
const TEST_OPS = 1;
|
||||
const TEST_MEM = 8192;
|
||||
const TEST_OPS = 2;
|
||||
const TEST_MEM = 64 * 1024 * 1024;
|
||||
const TEST_TIME = 1700000000000000;
|
||||
|
||||
interface ThumbMockState {
|
||||
@@ -550,22 +544,11 @@ describe("listMissingThumbnails", () => {
|
||||
const client = await login(counted.fetch, { ...noWait });
|
||||
const lib = await openLib(client);
|
||||
|
||||
const errors: string[] = [];
|
||||
const missing = await listMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
(msg, level) => {
|
||||
if (level === "error") errors.push(msg);
|
||||
},
|
||||
);
|
||||
const missing = await listMissingThumbnails(lib, client);
|
||||
lib.close();
|
||||
|
||||
// Only the genuinely empty thumbnail is reported. The file that
|
||||
// could not be checked is reported as an error.
|
||||
// Only the genuinely empty thumbnail is reported.
|
||||
expect(missing.map((m) => m.fileID)).toEqual([101]);
|
||||
expect(errors).toEqual([
|
||||
expect.stringMatching(/^\[.*\] Could not check file-102\.jpg: /),
|
||||
]);
|
||||
// And the 500 was retried rather than accepted as an answer: four
|
||||
// attempts is the library default.
|
||||
expect(counted.matched()).toBe(4);
|
||||
@@ -662,12 +645,7 @@ describe("fixMissingThumbnails", () => {
|
||||
const client = await login(buildThumbFetch(fixMock));
|
||||
const lib = await openLib(client);
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
[101],
|
||||
cachedFilesByID(lib, [101]),
|
||||
);
|
||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
||||
lib.close();
|
||||
|
||||
expect(results.length).toBe(1);
|
||||
@@ -704,12 +682,7 @@ describe("fixMissingThumbnails", () => {
|
||||
const client = await login(buildThumbFetch(fixMock));
|
||||
const lib = await openLib(client);
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
[999],
|
||||
cachedFilesByID(lib, [999]),
|
||||
);
|
||||
const results = await fixMissingThumbnails(lib, client, [999]);
|
||||
lib.close();
|
||||
|
||||
expect(results.length).toBe(1);
|
||||
@@ -726,16 +699,7 @@ describe("fixMissingThumbnails", () => {
|
||||
const client = await login(buildThumbFetch(fixMock));
|
||||
const lib = await openLib(client);
|
||||
|
||||
const errors: string[] = [];
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
[101, 102],
|
||||
cachedFilesByID(lib, [101, 102]),
|
||||
(msg, level) => {
|
||||
if (level === "error") errors.push(msg);
|
||||
},
|
||||
);
|
||||
const results = await fixMissingThumbnails(lib, client, [101, 102]);
|
||||
lib.close();
|
||||
|
||||
expect(results.length).toBe(2);
|
||||
@@ -743,10 +707,6 @@ describe("fixMissingThumbnails", () => {
|
||||
const failure = results.find((r) => r.fileID === 102)!;
|
||||
expect(success.status).toBe("fixed");
|
||||
expect(failure.status).toBe("failed");
|
||||
// The failed file, and only it, is reported as an error.
|
||||
expect(errors).toEqual([
|
||||
expect.stringMatching(/^\[.*\] FAILED file-102\.jpg: /),
|
||||
]);
|
||||
});
|
||||
|
||||
it("skips a non-JPEG image and a video as unsupported, not failed (issue #17)", async () => {
|
||||
@@ -762,7 +722,6 @@ describe("fixMissingThumbnails", () => {
|
||||
lib,
|
||||
client,
|
||||
[101, 103, 104],
|
||||
cachedFilesByID(lib, [101, 103, 104]),
|
||||
);
|
||||
lib.close();
|
||||
|
||||
@@ -797,12 +756,7 @@ describe("fixMissingThumbnails", () => {
|
||||
const client = await login(counted.fetch);
|
||||
const lib = await openLib(client);
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
[101],
|
||||
cachedFilesByID(lib, [101]),
|
||||
);
|
||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
||||
lib.close();
|
||||
|
||||
expect(results[0]!.status).toBe("skipped");
|
||||
@@ -823,12 +777,7 @@ describe("fixMissingThumbnails", () => {
|
||||
const client = await login(counted.fetch);
|
||||
const lib = await openLib(client);
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
[101],
|
||||
cachedFilesByID(lib, [101]),
|
||||
);
|
||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
||||
lib.close();
|
||||
|
||||
expect(results[0]!.status).toBe("skipped");
|
||||
@@ -869,12 +818,7 @@ describe("fixMissingThumbnails", () => {
|
||||
const client = await login(buildThumbFetch(fixMock));
|
||||
const lib = await openLib(client);
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
[101],
|
||||
cachedFilesByID(lib, [101]),
|
||||
);
|
||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
||||
lib.close();
|
||||
|
||||
expect(results[0]!.status).toBe("fixed");
|
||||
|
||||
@@ -3,15 +3,8 @@ import { configDefaults, defineConfig } from "vitest/config";
|
||||
// vitest does not read .gitignore when looking for tests. A checkout nested
|
||||
// under .claude/ has its own test/ tree, and without this exclude the suite
|
||||
// runs once per nested checkout and still reports success.
|
||||
//
|
||||
// vitest starts one worker per CPU core by default, and a shared host has
|
||||
// dozens. Four keep the suite well inside the Dockerfile's time limit. vitest
|
||||
// 2 rejects a maxWorkers below its default minWorkers, so minWorkers is set
|
||||
// too.
|
||||
export default defineConfig({
|
||||
test: {
|
||||
exclude: [...configDefaults.exclude, ".claude/**"],
|
||||
minWorkers: 1,
|
||||
maxWorkers: 4,
|
||||
},
|
||||
});
|
||||
Reference in new issue
Block a user