Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
3cabeba249 |
No files matched your search
@@ -26,8 +26,10 @@ check:
|
||||
build:
|
||||
@script/build
|
||||
|
||||
build-bin:
|
||||
@script/build-bin
|
||||
# Bundles the built dist/, so the binary reports the version script/build
|
||||
# stamped.
|
||||
build-bin: build
|
||||
nix-shell -p bun --run "bun build dist/bin/quak.js --compile --outfile bin/quak"
|
||||
|
||||
install: build-bin
|
||||
mkdir -p ~/bin
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
- branch from `next`
|
||||
- do the work in Next Step
|
||||
- delete Next Step from `TODO.md`
|
||||
- move Next Step to the top of Completed Steps
|
||||
- move the top item of Future Steps into Next Step
|
||||
- commit (`TODO.md` changes in the same commit as the work)
|
||||
- push
|
||||
@@ -17,13 +17,622 @@ pre-1.0
|
||||
|
||||
# Next Step
|
||||
|
||||
No bug is open, and no change sneak asked for is open. The check of quak against
|
||||
a live Ente account, https://git.eeqj.de/sneak/quak/issues/187, needs an account
|
||||
only sneak has, so it waits on him.
|
||||
Four bugs are open: https://git.eeqj.de/sneak/quak/issues/198,
|
||||
https://git.eeqj.de/sneak/quak/issues/199,
|
||||
https://git.eeqj.de/sneak/quak/issues/205 and
|
||||
https://git.eeqj.de/sneak/quak/issues/206. The check of quak against a live Ente
|
||||
account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
|
||||
has, so it waits on him.
|
||||
|
||||
Tagging and releases are decided by sneak alone, and happen only when he
|
||||
declares one.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-15: An album whose folder or JSON `quak backup` cannot write no longer
|
||||
ends the run before the other albums and `failures.json` are written (issue
|
||||
197). The run prints `FAILED album <name>: <reason>`, lists the album's ID in
|
||||
`failedAlbums` in the result and in `--json`, goes on, and exits 1. An album
|
||||
whose folder would have the name of another album's JSON, as `x.json` beside
|
||||
`x` would, gets its ID added. A folder or link name over 255 bytes is cut
|
||||
short between two characters, by `shortenToBytes` in `src/filename.ts`, and
|
||||
gets the ID.
|
||||
|
||||
- 2026-10-15: A network failure is printed and recorded with its reason, not as
|
||||
just `fetch failed` (issue 196). `describeError` in `src/errors.ts` gives an
|
||||
error's message followed by the message of each error on its `cause` chain,
|
||||
such as `fetch failed: connect ECONNREFUSED 1.2.3.4:443`, and each address's
|
||||
reason when a host has several and none could be reached. `run` prints that,
|
||||
and the backup logs it and records it in `failures.json`, where `classify` now
|
||||
calls the failure transient when the chain holds an errno that
|
||||
`hasTransportCode` in `src/retry.ts` recognises, for a failed ML data fetch
|
||||
too. The walk along the chain moved from `src/retry.ts` to `src/errors.ts`,
|
||||
which both use. Tests refuse every connection of a backup, then only one
|
||||
file's download, and fail the ML data fetch.
|
||||
|
||||
- 2026-10-15: The README says that `make install` needs nix (issue 200), in
|
||||
Getting Started and in "Running the backup from cron": `make build-bin` runs
|
||||
bun through `nix-shell -p bun`. The `Makefile` is unchanged.
|
||||
|
||||
- 2026-10-14: `lib.backup()`, and so `quak backup`, works on up to the content
|
||||
pool's limit of originals at once, 5 by default, instead of one at a time
|
||||
(issue 182). That many workers take files from one shared list, each doing all
|
||||
the work for one original, its `--verify` hashing included, so no more
|
||||
originals than that are open at once. `BackupLibrary` says the limit as
|
||||
`contentConcurrency`. A test holds each download at the fake Ente server to
|
||||
count those in flight. Once a worker has thrown, as one does when `onProgress`
|
||||
throws, or the lock is lost, no worker takes another file; the backup fails
|
||||
once the originals under way are in place, and a lost lock stops it before it
|
||||
fetches ML data.
|
||||
|
||||
- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's
|
||||
review, though that issue closed on 2026-10-03 (issue 194). It now names the
|
||||
issues open on the tracker.
|
||||
|
||||
- 2026-10-14: A backup whose lock is lost no longer dies with a stack trace
|
||||
(issue 184). `lockBackupDirectory` gives proper-lockfile an `onCompromised`
|
||||
that records why the lock was lost instead of throwing from its timer, and
|
||||
returns the lock, which `quak backup` passes to `lib.backup()` as `lock`. The
|
||||
backup then stops before it starts another file with
|
||||
`lost the lock on DIR: REASON`, which `quak backup` prints as one line before
|
||||
it exits 1. Releasing a lost lock leaves `backup.lock` alone. Tests in
|
||||
`test/cli/backup-end-to-end.test.ts` remove the lock, or put another backup's
|
||||
in its place, while a download is held open.
|
||||
|
||||
- 2026-10-14: An album or file quak cannot decrypt, or whose metadata is not
|
||||
JSON, no longer fails the refresh, so one bad record no longer stops
|
||||
`quak backup` from storing anything (issue 181). The refresh leaves it out, an
|
||||
album with its files, and the library keeps it with the reason in
|
||||
`metadata.json` until the server sends a copy that decrypts or deletes it.
|
||||
`quak backup` stores every other file, records each such file in
|
||||
`failures.json` with an error starting `could not be decrypted:`, names each
|
||||
such album's ID in a printed line, or with `--json` in `undecryptableAlbums`,
|
||||
and exits 1 while any remain. A file whose copy in another album decrypts is
|
||||
not recorded: its original is stored from that copy, and the album where it
|
||||
does not decrypt neither links nor lists it. The read commands answer from
|
||||
what decrypted.
|
||||
|
||||
- 2026-10-14: `quak backup` no longer asks the server for changes every 3
|
||||
seconds while it runs (issue 183). It opens its library with
|
||||
`refreshIntervalSeconds: Infinity`, which the library now takes as no
|
||||
background timer, so the backup refreshes only as it starts and no timed
|
||||
refresh starts another ML data pass during the run. A test holds a backup's
|
||||
download open while two hours pass and counts its `/collections/v2` requests.
|
||||
|
||||
- 2026-10-14: `test/cli/backup-end-to-end.test.ts` runs `quak backup` as cron
|
||||
does (issue 186): `backupCommand` and `run`, the real `loadSession` reading a
|
||||
session saved after logging in, and every request answered by a fake of the
|
||||
Ente HTTP API serving data encrypted with libsodium. It covers the backup tree
|
||||
of an account with two albums of its own and one shared with it, a second run
|
||||
that downloads nothing, a download that succeeds after two 503s, a 404 that
|
||||
lands in `failures.json` with exit 1, a refresh answered 401 with exit 3, and
|
||||
`--verify` putting back an original changed on disk. The fake server and the
|
||||
account it logs in moved from the backup-metadata test into
|
||||
`test/ente-server.ts`, unchanged.
|
||||
|
||||
- 2026-10-14: `files --collection`, `get`, `get-thumb` and
|
||||
`helper fix-missing-thumbnails --file` take an ID only when it is written in
|
||||
decimal digits and is a safe integer (issue 185). Before, `Number()` read
|
||||
`0x10` as 16, `1e3` as 1000 and an empty string as 0, and `--file` dropped an
|
||||
ID it could not read without a word. Anything else now prints
|
||||
`Invalid file ID: <arg>` or `Invalid collection ID: <arg>` and exits 1 before
|
||||
any request; one bad `--file` ID fails the whole command. `parseID` in
|
||||
`src/cli-commands.ts` does the check for all four.
|
||||
|
||||
- 2026-10-06: The README says how to run `quak backup` from cron (issue 170):
|
||||
log in once as the job's user, a crontab with a backup every night and
|
||||
`--verify` on Sundays appending to a log file, and the `HOME` and
|
||||
`XDG_DATA_HOME` the job needs to find the saved session. A table gives each
|
||||
exit code: 0, the backup is complete; 1, files are in `failures.json` or
|
||||
another error stopped the run; 2, another backup of the directory is running;
|
||||
3, there is no usable session. The introduction lists everything the backup
|
||||
keeps for each file.
|
||||
|
||||
- 2026-10-06: `quak backup --verify` and `lib.backup({ verify: true })` hash
|
||||
each original already at its save path as the download check does, streamed, a
|
||||
live photo as `<imageHash>:<videoHash>` (issue 168). One that does not match
|
||||
the content hash its metadata records is logged, removed (both files of a live
|
||||
photo) and put back in the same run, downloaded or copied from the cache, and
|
||||
what is put back is hashed too. One that still does not match, or whose
|
||||
download fails, goes into `failures.json`. One with no recorded hash is left
|
||||
alone. The result, `--json` and the summary gain `verified`, `mismatched` and
|
||||
`unchecked`. Without `--verify` nothing is hashed.
|
||||
|
||||
- 2026-10-06: Two backups of the same directory never run at once (issue 169).
|
||||
`lib.backup()` takes a lock, `backup.lock` in its download directory, made
|
||||
with `proper-lockfile`, before its refresh, and removes it when it ends,
|
||||
whether it succeeds or fails. A second backup of the directory, from another
|
||||
process or the same one, fails at once with an error naming the directory.
|
||||
`quak backup` takes the lock before it opens its library, so a refused run
|
||||
sends no request; it prints the error as one line and exits 2. A lock that has
|
||||
gone 10 seconds untouched, left by a run that could not remove it, is taken
|
||||
over by the next run.
|
||||
|
||||
- 2026-10-06: `quak backup` writes each original's EXIF, XMP and dimensions into
|
||||
the file's JSON as `imageMetadata`, what `backup-metadata --exif` records
|
||||
(issue 167): for a live photo from its image, for a video nothing, and `{}`
|
||||
for an original with none of them. A failed read puts the reason in
|
||||
`imageMetadataError` and fails neither the file nor the run. An original is
|
||||
read when the run stores it or when its JSON has neither field; otherwise the
|
||||
field is taken from that JSON. The hand-built JPEGs moved to
|
||||
`test/exif-jpeg.ts`, beside the HEIC.
|
||||
|
||||
- 2026-10-06: When the server answers HTTP 401 and that ends a command that
|
||||
loads the saved session, because the server no longer accepts its token, the
|
||||
command prints one line,
|
||||
`quak: the saved session is no longer valid; run "quak login"`, and exits 3
|
||||
(issue 164). `quak backup` meets that 401 on the refresh it starts with,
|
||||
before it touches any file. For the same commands, a missing or corrupt
|
||||
session file keeps its message and also exits 3, so a cron job can tell that
|
||||
the user must log in again. `quak logout` is unchanged. `run` in
|
||||
`src/cli-run.ts` recognises the 401, which reaches it unchanged.
|
||||
|
||||
- 2026-10-06: `quak backup` retries a failed request for longer than the other
|
||||
commands do (issue 165). `src/retry.ts` exports `UNATTENDED_RETRY_OPTIONS`
|
||||
beside the unchanged default: 10 attempts, a 1 s base delay and a 60 s cap, so
|
||||
a request that keeps failing waits at most 243 s before it gives up.
|
||||
`bin/quak.ts` loads the backup's session with them, so its refresh, ML data
|
||||
and downloads all use them. What is retried and the backoff formula are
|
||||
unchanged.
|
||||
|
||||
- 2026-10-06: The files this repository copies from `sneak/prompts` are copied
|
||||
again from its commit `dd4027b` (issue 171). `.gitignore` and `.dockerignore`
|
||||
keep out more secret files, and this repository's build artifacts follow the
|
||||
copied content. `script/check` runs `script/fmt-check` again, `script/cibuild`
|
||||
runs `script/bootstrap` and `script/check` before the image build,
|
||||
`script/fmt` and `script/fmt-check` find the pinned yarn under nvm, and the
|
||||
image's last stage marks `/app` safe for git.
|
||||
|
||||
- 2026-10-06: `quak backup` writes the account and album records
|
||||
`backup-metadata` writes (issue 166): `account.json` with the account's
|
||||
`email` and `userID`, and in each album's JSON its `ownerID`, `isShared`,
|
||||
`updationTime` and, when present, its three layers of magic metadata. Each
|
||||
file's JSON gains its `updationTime`.
|
||||
|
||||
- 2026-10-05: `quak backup` writes each file's ML data (its faces and CLIP
|
||||
embedding) into the file's JSON as `mlData`, the payload
|
||||
`lib.mldata.forFile()` returns (issue 163). `lib.backup()` waits for an ML
|
||||
data fetch before it writes the JSON files. When that fetch fails, each file
|
||||
whose ML data is not cached gets the reason in `mlDataError` and counts as
|
||||
failed, and the next run fetches it again.
|
||||
|
||||
- 2026-10-03: The download-albums example test no longer fails on a disk with
|
||||
under 50 GiB free (issue 160). It opens its libraries with
|
||||
`freeBelowBytes: 0`, so the free space of the disk it runs on cannot shrink
|
||||
the cache, evict the original it cached and add a fetch to the count. No other
|
||||
test's result depends on that free space. The 50 GiB default is unchanged.
|
||||
|
||||
- 2026-10-02: `docker build .` stamps the commit's tag or short commit, not
|
||||
`dev` (issue 154). `script/build` writes the version `script/version` prints
|
||||
into `dist/package.json`: the `VERSION` environment variable or build arg when
|
||||
one is given, otherwise `git describe --tags --always`, otherwise the version
|
||||
`package.json` declares. `.dockerignore` sends `.git`, and a checkout with
|
||||
`.git` whose version comes out empty, `dev` or `unknown` fails the build.
|
||||
`make build-bin` bundles the built `dist/`, so the single binary reports the
|
||||
same version.
|
||||
|
||||
- 2026-10-02: `photo.exif()` returns every EXIF tag in the file as `ExifTags`,
|
||||
keyed by tag name, each as exifreader decodes it, not only the thirteen common
|
||||
fields (issue 156). The embedded thumbnail's tags are under `Thumbnail`,
|
||||
without the thumbnail image. The thirteen typed methods stay, each picking its
|
||||
field from the tags `exif()` returns, typed as in `PhotoExif`. The example
|
||||
script's JSON files now carry every tag.
|
||||
|
||||
- 2026-10-01: The content cache no longer looks for a live photo that an earlier
|
||||
version cached as one ZIP (issue 151). When the cache opens, a live photo's
|
||||
file that no JSON file names is now always left alone.
|
||||
|
||||
- 2026-10-01: `examples/download-albums.ts` logs in, opens the library, and for
|
||||
every album downloads each photo to its save path, writes the photo's record
|
||||
and EXIF fields to a JSON file beside it, and writes the album's photos to
|
||||
`albums/<collectionID>.json` (issue 144). The build compiles it to
|
||||
`dist/examples/`; the README's "Examples" section says how to run it.
|
||||
|
||||
- 2026-10-01: A `Photo` has one async method for each field of `exif()`, named
|
||||
and typed as in `PhotoExif`: `make()`, `model()`, `lensModel()`,
|
||||
`dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`,
|
||||
`iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()`
|
||||
and `gpsAltitude()` (issue 148). Each calls `exif()` and returns its one
|
||||
field, or undefined when the file lacks it. `Photo` implements a type with one
|
||||
method per `PhotoExif` field, so the build's type check fails when a field has
|
||||
no method. A test checks, on the JPEG and the HEIC, that each method gives the
|
||||
same value as `exif()`.
|
||||
|
||||
- 2026-10-01: Each original's save path is
|
||||
`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
|
||||
download directory, which defaults to `photos` in the working directory (issue
|
||||
143). The date is the photo's `takenAt` in the machine's time zone.
|
||||
`photo.savePath` is always a string, with or without a content cache, and
|
||||
`isLocal` is true only when the original is at its save path.
|
||||
`photo.download()` puts the original there, copied from the cache when the
|
||||
cache holds it and fetched otherwise. `lib.backup()` does the same for each
|
||||
file, writes each file's JSON beside its original, and links `collections/` to
|
||||
the save paths.
|
||||
|
||||
- 2026-10-01: `photo.exif()` and `backup-metadata --exif` read EXIF from
|
||||
HEIC/HEIF originals, a live photo's HEIC image included, as well as JPEG and
|
||||
the other image formats `exifreader` reads (issue 145). `exifreader` replaces
|
||||
`exif-reader` and the JPEG segment scan; `PhotoExif` is unchanged. The `exif`
|
||||
field of `backup-metadata --exif` is now exifreader's tag output, and
|
||||
`exifRaw` holds the whole EXIF block it could not read. The tests use a real
|
||||
HEIC, `test/exif.heic`.
|
||||
|
||||
- 2026-10-01: A `Photo` has `savePath`, `isLocal`, `content()`, `exif()`,
|
||||
`modifiedAt`, `hash` and `year` (issue 141). `savePath` is where
|
||||
`lib.backup()` writes the original under the library's download directory; for
|
||||
a live photo not yet stored, it carries the title's extension, and the backup
|
||||
may store the image under a different one. `isLocal` says whether the whole
|
||||
original is there. Both look only at the disk. `content()` returns the
|
||||
original's bytes and `exif()` the common EXIF fields of a JPEG; both may
|
||||
download the original, and `exif()` downloads no video. `PhotoRecord` gains
|
||||
`modifiedAt` and `hash`, and the JPEG EXIF scan moved to `src/exif.ts`.
|
||||
|
||||
- 2026-09-29: The "Workflow" list at the top of this file now says to branch
|
||||
from `next` and open a pull request that targets `next`, that the repository
|
||||
manager squash-merges a reviewed pull request into `next`, and that only sneak
|
||||
merges `next` into `main`, as the README does (issue 137).
|
||||
|
||||
- 2026-09-29: The README's "Development workflow" and "For LLMs" sections now
|
||||
say work branches from `next`, every pull request targets `next`, the
|
||||
repository manager squash-merges reviewed pull requests into `next`, and only
|
||||
sneak merges `next` into `main` (issue 135).
|
||||
|
||||
- 2026-09-29: Brought the README and this file in line with `next` after the
|
||||
milestone merge (issue 132). The Next Step says no implementation work is open
|
||||
and the cache design (issue 36) waits on sneak's review. README corrections:
|
||||
the Getting Started comments say when the library refreshes; most, not all,
|
||||
Makefile targets call a script; `script/lint` and `script/test` have no host
|
||||
path, though `yarn test` does; the SRP handshake uses `fast-srp-hap`, outside
|
||||
`crypto/`, and a thumbnail upload's MD5 uses `node:crypto`; the SRP password
|
||||
is the first 16 bytes of a 32-byte subkey; the key attributes and token come
|
||||
after SRP, after the TOTP code SRP may ask for, or after the email OTP that
|
||||
replaces SRP when the account has email MFA on, and quak cannot answer a
|
||||
passkey; the auth token is sent as URL-safe base64 with padding; every
|
||||
`TypeError` is retried; each download attempt writes its own temporary files,
|
||||
two for a live photo, and only the attempt that completes renames them into
|
||||
place; `runMetadataBackup` records a failed download in the file's JSON; a
|
||||
second `client.logout()` does not throw; `quak logout` with no session exits
|
||||
0, and names the cache directory only when it knows it; `login`, `whoami` and
|
||||
`logout` open no library; `--exif` records XMP and, for a JPEG, EXIF, and no
|
||||
IPTC; which commands take `--json`; a failed ML data request may not have been
|
||||
retried; the thumbnail fixer is not limited to baseline JPEG; `quak backup`
|
||||
still fetches ML data; a backup's JSON holds the basic metadata fields quak
|
||||
keeps and the private and public magic metadata, not every decrypted field,
|
||||
and the README no longer lists what the magic metadata holds; the default
|
||||
cache directory is the per-user one, not an XDG path on macOS; pinned
|
||||
originals can exceed `cacheOriginalsMaxBytes`; which tests cover which
|
||||
operations; a default read is only as current as the last refresh whose
|
||||
requests all succeeded; `fresh()` and `lib.backup()` join a refresh already
|
||||
running; a `Photo` has no `thumbnailPath` or `originalPath`; and `408` and
|
||||
`429` are retried.
|
||||
|
||||
- 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks
|
||||
that the image's and the video's temp files are fsynced before either is
|
||||
renamed into place, and the directory after both renames, as the `writeAtomic`
|
||||
test does for one file.
|
||||
|
||||
- 2026-09-23: Live photos are stored as their image and their video (issue 107).
|
||||
A live photo, which Ente stores as one ZIP, is unpacked as it downloads into
|
||||
`<fileID>.<ext>` for the image and for the video, each with its extension from
|
||||
the ZIP, beside `<fileID>.livephoto.json`, which names the two. Both are
|
||||
checked against the recorded hash and renamed into place only when both are
|
||||
complete. A ZIP holding a second image or video, or whose two files come to
|
||||
more than 20 times its size plus 16 MiB, is refused. The backup and the
|
||||
content cache count the live photo as stored only with both files, the album
|
||||
folder links both, `quak get` writes both, and the content result gives the
|
||||
video as `videoPath`. A ZIP an earlier version stored is replaced on the next
|
||||
backup run, and removed from the cache when the library opens it.
|
||||
|
||||
- 2026-09-23: Settled the package metadata (issue 6). quak is not published, so
|
||||
`package.json` is marked `"private": true` and the `files` field is gone.
|
||||
`engines.node` is `>=22`, the major version `script/bootstrap` and the
|
||||
`Dockerfile` use. An `exports` map makes `.` and `./package.json` the only
|
||||
importable paths; `runMetadataBackup`, the thumbnail helpers and their types
|
||||
stay internal to the CLI.
|
||||
|
||||
- 2026-09-23: Brought the README and this file in line with the tree (issue
|
||||
111). The layout lists `src/library/` and the other source files, the backup
|
||||
layout names `failures.json` and the optional `thumbnails/`, "Opening a
|
||||
library" says what happens when the first refresh fails, the Testing section
|
||||
gives the 60-second hard cap and 20-second target for `make test` and names
|
||||
the 90-second `timeout` in the `Dockerfile` as the backstop for a hung test,
|
||||
and "Tag v1.0.0" is no longer listed as the next step.
|
||||
|
||||
- 2026-09-23: Tested `quak login` and `backup-metadata --exif` (issue 110).
|
||||
`loginCommand` takes its login function and its prompts from `CliContext`, and
|
||||
`bin/quak.ts` passes `Client.login` and the terminal prompts. Tests cover a
|
||||
login from `QUAK_EMAIL` and `QUAK_PASSWORD` with no prompt, the TOTP prompt, a
|
||||
failed login, and the saved session's modes, and show that `--exif` and
|
||||
`--all` each turn on EXIF extraction and that it is off without them.
|
||||
|
||||
- 2026-09-23: `quak backup` writes each original once and no longer fills the
|
||||
cache (issue 106). An original fetched for a backup is written by the download
|
||||
writer straight into the backup's `originals/`, and the content cache records
|
||||
it there instead of keeping its own copy; one the cache already held is still
|
||||
copied. `quak backup` opens its library with the thumbnail and originals
|
||||
precache off.
|
||||
|
||||
- 2026-09-23: `backup-metadata`, `helper list-missing-thumbnails` and
|
||||
`helper fix-missing-thumbnails` refresh before they answer (issue 100). Each
|
||||
awaits `lib.fresh()` before reading, so a file added since the cache was
|
||||
written is included, and a failed refresh prints one line and exits 1 instead
|
||||
of answering from a stale or empty cache. The README lists them among the
|
||||
commands that refresh first.
|
||||
|
||||
- 2026-09-23: `quak backup` waits for the server refresh and fails when it fails
|
||||
(issue 99). `lib.backup()` joins a refresh already running or starts one, as
|
||||
`fresh()` does, and rejects before touching any file when it fails, leaving
|
||||
`failures.json` as it was, so `quak backup` prints the error as one line and
|
||||
exits 1 instead of backing up the previous run's file list, or nothing, and
|
||||
exiting 0.
|
||||
|
||||
- 2026-09-23: CLI errors print a message instead of a stack trace (issue 102).
|
||||
An error a command throws is printed as one `quak: MESSAGE` line on stderr and
|
||||
the CLI exits 1 once output has drained. The wrapper that does this moved from
|
||||
`bin/quak.ts` to `src/cli-run.ts`, and `bin/quak.ts` now awaits
|
||||
`program.parseAsync()`.
|
||||
|
||||
- 2026-09-23: Opening a library no longer deletes another process's download in
|
||||
progress (issue 105). The download writer's temp files are named
|
||||
`.quak-<pid>-<random>.tmp`, and `removeLeftoverTempFiles`, moved from the
|
||||
backup into the download module, deletes a `.quak-*.tmp` file only when the
|
||||
process ID in its name is no longer running. The content cache calls it at
|
||||
`open()` for `originals/` and `thumbnails/`, the backup as before.
|
||||
|
||||
- 2026-09-23: Re-vendored the lint and test setup from the template (issue 96).
|
||||
Linting and testing are the `lint` and `test` phases of the `Dockerfile`;
|
||||
`script/lint` and `script/test` each build one with `--no-cache`, and the last
|
||||
stage compiles and depends on both, so `script/cibuild` is one build.
|
||||
`Dockerfile.lint`, `CHECK_EPOCH`, `LINT_EPOCH` and the tests that checked them
|
||||
are gone; `REPO_POLICIES.md` is re-copied.
|
||||
|
||||
- 2026-09-23: Stopped `helper fix-missing-thumbnails` retrying files the server
|
||||
always refuses (issue 109). Both thumbnail helpers skip a file another account
|
||||
owns without fetching it. The fixer skips a file whose recorded thumbnail size
|
||||
is 0 or unknown before downloading it, and otherwise tries smaller encodings
|
||||
(720 px quality 50 down to 160 px quality 20) until the encrypted thumbnail is
|
||||
no larger than that size, skipping the file if none fits.
|
||||
|
||||
- 2026-09-23: Tested the live-photo hash check's error paths (issue 117). Tests
|
||||
download a live photo whose ZIP names an unknown compression method, one whose
|
||||
ZIP has no image entry and one with no video entry, and check that nothing is
|
||||
stored and the error names the file ID; the unreadable one is not retried.
|
||||
|
||||
- 2026-09-23: `quak logout` ends the session on the server (issue 108). It calls
|
||||
`POST /users/logout` through the new `Client.logoutOnServer()`, then deletes
|
||||
`session.json` even when that call fails, says so and exits 1. It prints the
|
||||
account's cache directory and says it still holds decrypted data. The default
|
||||
cache path is now `defaultCacheDirectory()` in the library, shared with
|
||||
`Library.open`.
|
||||
|
||||
- 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files
|
||||
in one collection with the same title, and two collections with the same name,
|
||||
each get their ID added to the name (`IMG_0001 (12345).JPG`, `Trip (10)/`), so
|
||||
none replaces another's symlink or JSON. Each run removes symlinks into
|
||||
`originals/` for files no longer in the collection, and the folders of deleted
|
||||
or renamed collections, leaving anything else in `collections/` alone. The
|
||||
README backup layout states the naming rule.
|
||||
|
||||
- 2026-09-23: Checked downloaded originals against their recorded content hash
|
||||
(issue 68). `downloadFile`, which `quak get`, the content cache and backup all
|
||||
use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and
|
||||
stores nothing on a mismatch, failing with an error naming the file ID. A live
|
||||
photo ZIP is unpacked as it streams with `fflate` and its image and video
|
||||
hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older
|
||||
clients' `imageHash` and `videoHash` fields for live photos. A file with no
|
||||
recorded hash is stored unchecked.
|
||||
|
||||
- 2026-09-23: Kept one account's cache from mixing with another's (issue 104).
|
||||
When `metadata.json` in the cache directory was written for a different,
|
||||
non-zero user ID than the client's, `Library.open` deletes it and `mldata/`
|
||||
and starts empty, so the first refresh enumerates from 0. This only happens
|
||||
with `--cache-dir` or an explicit `cacheDirectory`; the default path already
|
||||
includes the user ID. A test opens one account's cache as another account.
|
||||
|
||||
- 2026-09-23: `backup-metadata` no longer stops on one failed ML data request
|
||||
(issue 101). Each request of up to 200 files is tried on its own; a failed one
|
||||
is logged, its files are written with the reason in `mlDataError`, and the
|
||||
command exits 1 once the dump is complete. `fetchMLData`, which only this
|
||||
command used, is gone; the command calls `fetchMLDataBatch` per batch.
|
||||
- 2026-09-23: Single-sourced the version string (issue 5). `package.json` is the
|
||||
only place it is written: `src/index.ts` imports it for `VERSION` and
|
||||
`bin/quak.ts` passes `VERSION` to commander. tsc copies `package.json` to
|
||||
`dist/package.json`, so the import resolves from the built output too, and
|
||||
`script/build` runs the built CLI with `--version` to prove it. A test checks
|
||||
that `VERSION` and `quak --version` both equal the `package.json` version.
|
||||
- 2026-09-23: Tested that `Library.close()` waits for the originals precache
|
||||
(issue 93). The test that holds a precache fetch open while `close()` runs now
|
||||
runs once with only the thumbnail fill and once with only the originals fill,
|
||||
so dropping either wait from `Precache.close()` fails a test.
|
||||
- 2026-09-23: Fixed two intermittently failing library tests (issue 90).
|
||||
`Library.close()` now returns a promise that resolves once an in-flight
|
||||
refresh (including its cache write), the ML data fetch and running precache
|
||||
sweeps have finished; the library tests await it, so `afterEach` no longer
|
||||
removes the cache directory while something is still writing into it. The
|
||||
precache test waits for both fills to report "done" instead of for its stub
|
||||
source to be called, which happened before the cache recorded the file.
|
||||
- 2026-09-23: Pinned three guards reviewers found untested (issue 89). The
|
||||
download idle deadline's timer is unref'd, so it can never keep the process
|
||||
alive, and a test checks no timer is left after a download completes or fails.
|
||||
A test covers the rejection of `#` in a request path. The EXIF scan compares
|
||||
the `Exif` header only in an APP1 segment of length 8 or more, so it never
|
||||
reads the next segment's bytes, with a test for a short one.
|
||||
- 2026-09-23: Made the download deadline an idle deadline (issue 24).
|
||||
`downloadTimeoutMs` now aborts a file or thumbnail download only after no
|
||||
bytes have arrived for that long, default 60 seconds, instead of bounding the
|
||||
whole transfer at 10 minutes, so a slow download that keeps making progress
|
||||
completes. A download that fails before reading the whole body cancels it, so
|
||||
a failed file no longer holds its connection.
|
||||
- 2026-09-23: Every `ApiClient` request URL is now built by one function next to
|
||||
the class (issue 18), so a self-hosted `apiOrigin` with a base path keeps it
|
||||
on every request, a path works with or without a leading slash, and query
|
||||
parameters are percent-encoded. A path containing `?` or `#` is rejected with
|
||||
an error instead of being silently cut.
|
||||
- 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies
|
||||
moved from `bin/quak.ts` into `src/cli-commands.ts` as functions that take
|
||||
their options and a context (output streams, session directory, cache
|
||||
directory, session loader) and return an exit code; `bin/quak.ts` only wires
|
||||
them to commander and exits with the code once stdout and stderr have drained,
|
||||
so nothing below it calls `process.exit`. `test/cli/commands.test.ts` drives
|
||||
them with a fake client: session file modes, logout, the missing and corrupt
|
||||
session paths, and the output and exit code of `whoami`, `collections`,
|
||||
`files`, `get`, `get-thumb`, `backup` and `helper list-missing-thumbnails`.
|
||||
- 2026-09-23: Hardened the backup tree's atomic copy (issue 22). `copyAtomic`
|
||||
fsyncs its temp file before the rename and the directory after it, through the
|
||||
download writer's `fsyncPath`; each backup run deletes `.quak-backup-*.tmp`
|
||||
files whose process is no longer running. The README backup layout names the
|
||||
temp files and states that the rename replaces a symlink and takes the temp
|
||||
file's permissions. Added tests for a missing and an unwritable destination
|
||||
directory for `downloadFile` and `downloadThumbnail`.
|
||||
- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue
|
||||
11). Every segment length is checked against the remaining bytes and lengths
|
||||
under 2 stop the scan, so a truncated or corrupt original can neither throw
|
||||
nor loop. A malformed or unparseable EXIF segment is recorded as
|
||||
`imageMetadata.exifError`, and a failure to read the original as
|
||||
`imageMetadataError` in the per-file JSON, instead of the field being left
|
||||
out.
|
||||
- 2026-09-22: Hardened the retry classifier (issue 80). A `POST` or `PUT` is
|
||||
replayed only when every errno in the cause chain is a connect errno, and it
|
||||
no longer follows redirects. `getRetryOptions()` returns a copy. Tests pin
|
||||
every errno the classifier names, the cause-chain depth limit, cycle
|
||||
termination, and a fresh deadline per attempt for every retrying entry point.
|
||||
The README's endpoint list is the one place that names the requests the replay
|
||||
rule covers.
|
||||
- 2026-09-22: Stopped `make test` collecting tests from checkouts nested under
|
||||
`.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a
|
||||
nested checkout ran the whole suite again; `vitest.config.ts` now adds
|
||||
`.claude/**` to vitest's default excludes, and
|
||||
`test/packaging/nested-checkout.test.ts` plants a nested checkout in a temp
|
||||
directory and fails if vitest would collect it.
|
||||
- 2026-09-22: Dropped the deprecated `@types/libsodium-wrappers-sumo` stub from
|
||||
`devDependencies` (issue 27). It shipped no declarations; the types come from
|
||||
`libsodium-wrappers-sumo` itself. `yarn.lock` regenerated by `yarn remove`.
|
||||
- 2026-09-22: Hardened the client session lifecycle (issue 10).
|
||||
`Client.fromJSON` checks every snapshot field and each key's decoded length
|
||||
and names the bad field; `toJSON` reads the token through
|
||||
`ApiClient.getAuthToken` and throws when there is none; `logout` zeroes the
|
||||
key buffers, and `collectionsSince` re-checks for logout after its request so
|
||||
it never decrypts with zeroed keys. The CLI reports a corrupt session file
|
||||
separately from a missing one (`src/cli-session.ts`).
|
||||
- 2026-09-22: Sanitized file names taken from server metadata (issue 9). A new
|
||||
`src/filename.ts` holds the one sanitizer, used by `quak get`/`get-thumb`
|
||||
without `--out`, `downloadFile`/`downloadThumbnail` without `outPath`, and the
|
||||
backup and metadata backup trees; it removes separators, control characters,
|
||||
leading dots and Windows device names, and falls back to a name built from the
|
||||
ID for an empty title. Originals-cache extensions are letters and digits only,
|
||||
else `.bin`. A user-supplied path is used as is. `decryptFile` reads a missing
|
||||
or non-string title as "" and rejects metadata that is not a JSON object.
|
||||
- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
|
||||
snippets) to match the shipped cache/API library on `next` (issue 53, issue
|
||||
13). Documented `Library.open` and its options, the default-read vs `fresh()`
|
||||
distinction (and that the CLI's read commands are fresh), the record types and
|
||||
`snapshot()`/`subscribe()`, the `albums`/`photos`/`timeline` read surface,
|
||||
`Photo` content methods, `thumbnails.ensure`, the `mldata` search surface,
|
||||
`backup()`, the three request pools (10/5/25), and the on-disk cache layout;
|
||||
noted the deferred content-hash integrity check (issue 68). Docs-only; no code
|
||||
changed.
|
||||
- 2026-09-22: Added resumable, deletion-aware enumeration to `Client` (issue 38,
|
||||
closes issue 7). `collectionsSince`/`filesSince` take a starting cursor,
|
||||
decrypt live records, surface tombstoned ids in a separate `deleted` list (a
|
||||
tombstone has nothing to decrypt, so it is a bare id, not a hollow record),
|
||||
and return the max `updationTime` seen as the cursor to resume from.
|
||||
`filesSince` refuses to loop when the diff reports `hasMore` without advancing
|
||||
the cursor (issue 7). `listCollections`/`listFiles` are now thin wrappers that
|
||||
enumerate from `sinceTime: 0` and drop deletions, so existing callers are
|
||||
unaffected.
|
||||
- 2026-09-22: Carried file size, thumbnail size, and the deletion flag through
|
||||
`decryptFile` (issue 37, foundation for the cache/API design). Live files now
|
||||
populate `file.size`/`thumbnail.size` from the server's `info` (left
|
||||
`undefined` when the server omits it), and `isDeleted` is carried from the
|
||||
diff row onto `EnteFile`. No caller change: `listFiles` still filters deleted
|
||||
rows before decrypting. Surfacing a tombstone through decryption belongs to
|
||||
the enumeration unit (issue 38).
|
||||
- 2026-08-10: Made `lint-once.test.ts` enforce what its header claims. It walked
|
||||
`make check` only, so it never read `Dockerfile` — the image CI builds through
|
||||
`script/cibuild` — and a second `prettier --check .` could be added there with
|
||||
the suite staying green. The walk now also starts at
|
||||
`.gitea/workflows/check.yml` and follows its `run:` steps, so the graph under
|
||||
test is the one CI executes rather than the one someone assumed it executes.
|
||||
The lockfile assertion was a substring check against the whole of
|
||||
`script/bootstrap`, which has two install sites and so reported the branch the
|
||||
containers never take; the two branches are now resolved separately and every
|
||||
`yarn install` in each is required to be `--frozen-lockfile`. Prettier is
|
||||
counted per occurrence instead of per line, so two invocations chained with
|
||||
`&&` no longer read as one, and edges are followed on counted lines instead of
|
||||
being skipped. Every way for the walk to reach nothing — an unknown target, an
|
||||
unknown script, a missing file, a node with no commands, an unknown node kind
|
||||
— is a thrown error rather than a quiet zero. Every assertion in the file was
|
||||
mutation-tested individually.
|
||||
- 2026-08-10: Stopped `make check` running `prettier --check .` twice. Since
|
||||
linting moved into Docker, the duplicate was one container pass and one host
|
||||
pass of the same check: `script/lint` builds `Dockerfile.lint`, which runs
|
||||
prettier as a build step, and `script/check` then called `script/fmt-check` as
|
||||
well. The host call is gone from `script/check` and from `script/precommit`;
|
||||
the container keeps checking formatting, because a successful
|
||||
`Dockerfile.lint` build is what CI treats as proof of a clean tree, and it is
|
||||
also what still fails the pre-commit hook on a badly formatted tree.
|
||||
`script/fmt-check` survives as a standalone entrypoint, whose verdict cannot
|
||||
drift from the container's. A test walks the invocation graph from each
|
||||
entrypoint — through the Makefile shims, the `script/` calls and the
|
||||
`docker build` — and asserts the prettier count, so the duplication cannot
|
||||
come back unnoticed.
|
||||
- 2026-08-10: Moved all linting into Docker. `script/lint` builds a new root
|
||||
`Dockerfile.lint`, which copies the repo into the digest-pinned node image and
|
||||
runs eslint and prettier as build steps, so a successful build is a clean
|
||||
lint; no host lint path remains and `yarn lint` is gone from `package.json`. A
|
||||
fail-closed `LINT_EPOCH` guard stops Docker serving the linter layers from
|
||||
cache, which is how a lint build returns success in under a second having
|
||||
linted nothing. The lint stage inside `Dockerfile` and its `COPY --from=lint`
|
||||
ordering hack are gone: that image now runs `make test` and `make build` only,
|
||||
because `script/check` calls `script/lint` and running it in a container would
|
||||
mean docker inside docker. `script/cibuild` builds the lint image first, then
|
||||
the test and build image.
|
||||
- 2026-08-09: Made `make docker` green and policy-conformant. Multi-stage
|
||||
Dockerfile: a lint stage runs `make fmt-check` and `make lint`, and the check
|
||||
stage takes a `COPY --from=lint` dependency on it before running `make check`
|
||||
and `make build`. `CHECK_EPOCH` and a fail-closed guard stop Docker serving
|
||||
those two layers from cache, which is what let a build report success without
|
||||
running the suite. `script/projectname` says `quak`, so the image is tagged
|
||||
`quak`; `script/bootstrap` updates apt lists before installing, so a Debian
|
||||
base works; `.dockerignore` no longer ships the compiled binary, the caches or
|
||||
agent worktrees into the build context, and keeps `.gitignore` in it for
|
||||
prettier.
|
||||
- 2026-08-09: Fixed the TypeScript build. `rootDir` is the repo root, so `bin/`
|
||||
compiles alongside `src/` instead of failing with TS6059; output is
|
||||
`dist/src/` and `dist/bin/`, which is where `main`, `types` and `bin.quak` now
|
||||
point. `script/build` verifies the declared entrypoints exist after the
|
||||
compiler runs and makes the CLI executable, the Dockerfile runs `make build`
|
||||
as well as `make check`, and a `quak` script makes the README's
|
||||
`yarn quak <command>` examples work.
|
||||
- 2026-08-09: Retry policy: no retry on 4xx (except `408` and `429`),
|
||||
exponential backoff with full jitter on 5xx, transport failures and truncated
|
||||
transfers, under per-attempt deadlines that cover the response body as well as
|
||||
the request. Downloads retry request, stream consumption and decryption as one
|
||||
unit; `postJSON` and `putJSON` are replayed only when the connection was never
|
||||
established.
|
||||
- 2026-08-09: Downloads verify the secretstream terminated on `TAG_FINAL` and
|
||||
write output atomically: a truncated body is rejected instead of landing on
|
||||
disk as a short file, and plaintext is staged in a sibling temp file and
|
||||
renamed into place, so a failed download leaves the destination untouched.
|
||||
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
|
||||
shims, README Entrypoints section
|
||||
- 2026-06-10: Decrypted collections shared by other users (sealed-box keys);
|
||||
listCollections drops deleted-collection tombstones.
|
||||
- 2026-06-10: Login hardening: dual-2FA empty-string fields handled, TOTP
|
||||
preferred when a passkey is also enrolled, interactive input via
|
||||
@inquirer/prompts.
|
||||
- 2026-06-10: Replaced sharp with pure JS (jpeg-js + exif-reader); added
|
||||
single-binary bun build and make install.
|
||||
- 2026-06-09: Added backup-metadata command (ML data always included, --exif
|
||||
opt-in); rewrote README to match the implementation; added thumbnail helper
|
||||
tests.
|
||||
- 2026-05-13: Full CLI surface: login, backup with dedup symlink layout,
|
||||
collections, files, get, get-thumb, thumbnail repair helpers.
|
||||
- 2026-05-13: Client OO API with literate usage tests; file download and
|
||||
decryption; all three metadata layers decrypted and persisted; renamed quack
|
||||
to quak.
|
||||
- 2026-05-11: SRP login flow (email OTP + TOTP) and ApiClient.
|
||||
|
||||
# Future Steps
|
||||
|
||||
- Future desktop client, separate repo:
|
||||
|
||||
@@ -1,31 +0,0 @@
|
||||
#!/bin/sh
|
||||
# Writes src/clip-keyword-vectors.ts and test/clip-keywords-sunset.json by
|
||||
# running bin/clip-keyword-vectors.py in a Python image pinned by digest, with
|
||||
# its packages pinned by hash. Run it by hand from a checkout, then `make fmt`;
|
||||
# no gate runs it. Besides the Python image, it downloads about 265 MB, most of
|
||||
# it Ente's two CLIP models.
|
||||
set -eu
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
|
||||
# python:3.13-slim (Python 3.13.17 on Debian 13.7), 2026-10-21. The wheels
|
||||
# below are the ones for this image on linux/amd64.
|
||||
IMAGE=python@sha256:57b29a7c050c4e679e4bea572e14b3965cf4b5839de2da49d55e49ffc4024f39
|
||||
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
docker run --rm -i --platform linux/amd64 \
|
||||
--user "$(id -u):$(id -g)" -e HOME=/tmp \
|
||||
-v "$ROOT:/repo" -w /repo "$IMAGE" sh -c '
|
||||
python -m venv /tmp/venv &&
|
||||
/tmp/venv/bin/pip install --quiet --disable-pip-version-check \
|
||||
--no-cache-dir --no-deps --require-hashes -r /dev/stdin &&
|
||||
/tmp/venv/bin/python bin/clip-keyword-vectors.py' <<'EOF'
|
||||
numpy==2.5.3 --hash=sha256:a5fa86b80fd24bcd1aff83ad23be44ea323de3f787be8f8b15d4a65621e25321
|
||||
onnxruntime==1.30.0 --hash=sha256:86f940afc801ea9681a4da8af84fbe95e1d9ea7d80903952cc1bfad54faad38f
|
||||
pillow==12.3.0 --hash=sha256:0847a763afefb695bc912d7c131e7e0632d4edc1d8698f58ddabec8e46b8b6d3
|
||||
tokenizers==0.23.2 --hash=sha256:41c2f84d172449b4dadb9cdc508e3e364076613c35b16e76ecfe47a60d1e3305
|
||||
EOF
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -1,158 +0,0 @@
|
||||
"""Write src/clip-keyword-vectors.ts and test/clip-keywords-sunset.json.
|
||||
|
||||
bin/clip-keyword-vectors runs this in a pinned Python image. It downloads
|
||||
Ente's Discover list, Ente's CLIP text and image models, the CLIP tokenizer
|
||||
and a public-domain photo of a sunset, checks each against its sha256, and
|
||||
writes:
|
||||
|
||||
- src/clip-keyword-vectors.ts: each Discover category of Ente's photos app,
|
||||
with the text embedding Ente's CLIP text model gives its query;
|
||||
- test/clip-keywords-sunset.json: the CLIP embedding Ente's image model gives
|
||||
the photo, prepared as Ente's apps prepare a photo.
|
||||
|
||||
Both embeddings are scaled to length 1, as Ente's apps scale them.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import urllib.request
|
||||
|
||||
import numpy as np
|
||||
import onnxruntime
|
||||
from PIL import Image, ImageOps
|
||||
from tokenizers import Tokenizer
|
||||
|
||||
ENTE_COMMIT = "f5ad70a860ed25d0c8057b0efb4bd6eaefe4b0c5"
|
||||
DISCOVER_PATH = "mobile/apps/photos/assets/discover.json"
|
||||
DISCOVER_URL = (
|
||||
f"https://raw.githubusercontent.com/ente-io/ente/{ENTE_COMMIT}/{DISCOVER_PATH}"
|
||||
)
|
||||
DISCOVER_SHA256 = "cf70ac7116b592f7921ca4cf67cdbb421a8e6b287fbe44b2be678b8d198e5c24"
|
||||
|
||||
TEXT_MODEL = "mobileclip_s2_text_opset18_quant.onnx"
|
||||
TEXT_MODEL_SHA256 = "d92f33dfcff83077fc2e0d3414250710efbb51795dfd89767bdbefb5fdc47322"
|
||||
IMAGE_MODEL = "mobileclip_s2_image_gelu_opset20.onnx"
|
||||
IMAGE_MODEL_SHA256 = "205a430af825e501c5138e5bb9abea942482a7a4fd4a680e98e47cf0830dce7e"
|
||||
|
||||
# The CLIP tokenizer Hugging Face publishes for OpenAI's CLIP. Its vocabulary
|
||||
# and merges are those of bpe_simple_vocab_16e6.txt, the vocabulary Ente's
|
||||
# apps tokenize with.
|
||||
TOKENIZER_URL = (
|
||||
"https://huggingface.co/openai/clip-vit-base-patch32/resolve/"
|
||||
"3d74acf9a28c67741b2f4f2ea7635f0aaf6f0268/tokenizer.json"
|
||||
)
|
||||
TOKENIZER_SHA256 = "b556ac8c99757ffb677208af34bc8c6721572114111a6e0aaf5fa69ff0b8d842"
|
||||
|
||||
# "Sunset off the Massachusetts coast", a NOAA photo in the public domain:
|
||||
# https://commons.wikimedia.org/wiki/File:Sunset_off_the_Massachusetts_coast_-_NOAA.jpg
|
||||
PHOTO_URL = (
|
||||
"https://upload.wikimedia.org/wikipedia/commons/d/db/"
|
||||
"Sunset_off_the_Massachusetts_coast_-_NOAA.jpg"
|
||||
)
|
||||
PHOTO_SHA256 = "be62f3289c9bf7b75af68cd6978021d54e8a531d5d4cc493f6e9fbf5e1ab74d3"
|
||||
|
||||
# Ente's text model takes 77 token ids, padded with 0; its image model takes
|
||||
# a 256x256 RGB image.
|
||||
TOKEN_COUNT = 77
|
||||
IMAGE_SIZE = 256
|
||||
|
||||
|
||||
def download(url, sha256):
|
||||
# Wikimedia refuses requests without a User-Agent.
|
||||
request = urllib.request.Request(url, headers={"User-Agent": "quak"})
|
||||
with urllib.request.urlopen(request) as response:
|
||||
data = response.read()
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
if digest != sha256:
|
||||
raise SystemExit(f"{url} has sha256 {digest}, not {sha256}")
|
||||
return data
|
||||
|
||||
|
||||
def ente_model(name, sha256):
|
||||
return onnxruntime.InferenceSession(
|
||||
download(f"https://models.ente.com/{name}", sha256),
|
||||
providers=["CPUExecutionProvider"],
|
||||
)
|
||||
|
||||
|
||||
def run(session, tensor):
|
||||
(output,) = session.run(None, {session.get_inputs()[0].name: tensor})
|
||||
embedding = output[0]
|
||||
return embedding / np.linalg.norm(embedding)
|
||||
|
||||
|
||||
def text_embedding(session, tokenizer, query):
|
||||
ids = tokenizer.encode(query).ids
|
||||
ids += [0] * (TOKEN_COUNT - len(ids))
|
||||
return run(session, np.array([ids], dtype=np.int32))
|
||||
|
||||
|
||||
def image_embedding(session, photo):
|
||||
image = ImageOps.exif_transpose(Image.open(io.BytesIO(photo))).convert("RGB")
|
||||
# Scale to cover IMAGE_SIZE x IMAGE_SIZE and crop the centre.
|
||||
square = ImageOps.fit(image, (IMAGE_SIZE, IMAGE_SIZE), Image.Resampling.BILINEAR)
|
||||
pixels = np.asarray(square, dtype=np.float32) / 255
|
||||
return run(session, pixels.transpose(2, 0, 1)[np.newaxis])
|
||||
|
||||
|
||||
def numbers(embedding):
|
||||
return ", ".join(str(value) for value in embedding)
|
||||
|
||||
|
||||
def write_categories(prompts, embeddings):
|
||||
lines = [
|
||||
"// Written by bin/clip-keyword-vectors; do not edit by hand.",
|
||||
"//",
|
||||
"// The Discover categories of Ente's photos app, in their order, from",
|
||||
f"// {DISCOVER_PATH} in https://github.com/ente-io/ente",
|
||||
f"// at commit {ENTE_COMMIT}. Each has the text",
|
||||
f"// embedding Ente's CLIP text model ({TEXT_MODEL})",
|
||||
"// gives its query, scaled to length 1. `showVideo` is false for a category",
|
||||
"// that leaves out videos.",
|
||||
"",
|
||||
"export const DISCOVER_CATEGORIES = [",
|
||||
]
|
||||
for prompt, embedding in zip(prompts, embeddings):
|
||||
lines += [
|
||||
" {",
|
||||
f" title: {json.dumps(prompt['title'])},",
|
||||
f" query: {json.dumps(prompt['query'])},",
|
||||
f" minScore: {json.dumps(prompt['minScore'])},",
|
||||
f" showVideo: {json.dumps(prompt.get('showVideo', True))},",
|
||||
f" textEmbedding: [{numbers(embedding)}],",
|
||||
" },",
|
||||
]
|
||||
lines.append("];")
|
||||
with open("src/clip-keyword-vectors.ts", "w", encoding="utf-8") as file:
|
||||
file.write("\n".join(lines) + "\n")
|
||||
|
||||
|
||||
def write_fixture(embedding):
|
||||
with open("test/clip-keywords-sunset.json", "w", encoding="utf-8") as file:
|
||||
file.write(
|
||||
"{\n"
|
||||
f' "photo": {json.dumps(PHOTO_URL)},\n'
|
||||
f' "sha256": {json.dumps(PHOTO_SHA256)},\n'
|
||||
f' "embedding": [{numbers(embedding)}]\n'
|
||||
"}\n"
|
||||
)
|
||||
|
||||
|
||||
def main():
|
||||
prompts = json.loads(download(DISCOVER_URL, DISCOVER_SHA256))["prompts"]
|
||||
tokenizer = Tokenizer.from_str(
|
||||
download(TOKENIZER_URL, TOKENIZER_SHA256).decode("utf-8")
|
||||
)
|
||||
text_model = ente_model(TEXT_MODEL, TEXT_MODEL_SHA256)
|
||||
embeddings = [
|
||||
text_embedding(text_model, tokenizer, prompt["query"]) for prompt in prompts
|
||||
]
|
||||
write_categories(prompts, embeddings)
|
||||
|
||||
image_model = ente_model(IMAGE_MODEL, IMAGE_MODEL_SHA256)
|
||||
write_fixture(image_embedding(image_model, download(PHOTO_URL, PHOTO_SHA256)))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+17
-57
@@ -2,9 +2,8 @@
|
||||
|
||||
import { stdout, stderr } from "node:process";
|
||||
import { input, password } from "@inquirer/prompts";
|
||||
import { Command, Option } from "commander";
|
||||
import { Command } from "commander";
|
||||
import envPaths from "env-paths";
|
||||
import { logLevel, logRequests, stderrLog, type Log } from "../src/cli-log.js";
|
||||
import { init } from "../src/crypto/index.js";
|
||||
import {
|
||||
type CliContext,
|
||||
@@ -30,9 +29,6 @@ const paths = envPaths("quak", { suffix: "" });
|
||||
|
||||
const program = new Command();
|
||||
|
||||
// Program options, so they are taken before or after any command's name.
|
||||
// `-v` counts its repeats, the way commander's documentation counts a repeated
|
||||
// option.
|
||||
program
|
||||
.name("quak")
|
||||
.description("CLI for the Ente end-to-end encrypted photo service")
|
||||
@@ -41,60 +37,26 @@ program
|
||||
"--cache-dir <path>",
|
||||
"Directory for the local metadata/content cache " +
|
||||
"(default: the per-user cache directory)",
|
||||
)
|
||||
.option(
|
||||
"-v, --verbose",
|
||||
"Log more on stderr: debug lines with -v, trace lines too with -vv",
|
||||
(_value: string, previous: number) => previous + 1,
|
||||
0,
|
||||
)
|
||||
.addOption(
|
||||
new Option(
|
||||
"-q, --quiet",
|
||||
"Print nothing on stderr but the error a failing command exits with",
|
||||
).conflicts("verbose"),
|
||||
)
|
||||
.configureHelp({ showGlobalOptions: true });
|
||||
|
||||
interface ProgramOptions {
|
||||
cacheDir?: string;
|
||||
verbose: number;
|
||||
quiet?: boolean;
|
||||
}
|
||||
|
||||
// The log for this run, at the level `-v` and `-q` ask for.
|
||||
const log = (): Log => {
|
||||
const { verbose, quiet } = program.opts<ProgramOptions>();
|
||||
return stderrLog(stderr, logLevel(verbose, quiet ?? false));
|
||||
};
|
||||
);
|
||||
|
||||
const context = (): CliContext => ({
|
||||
stdout,
|
||||
log: log(),
|
||||
stderr,
|
||||
sessionDir: paths.data,
|
||||
cacheDir: program.opts<ProgramOptions>().cacheDir,
|
||||
loadSession: (path) => loadSession(path, logRequests(log())),
|
||||
login: (opts) =>
|
||||
Client.login({
|
||||
...opts,
|
||||
apiOptions: { ...opts.apiOptions, ...logRequests(log()) },
|
||||
}),
|
||||
cacheDir: program.opts<{ cacheDir?: string }>().cacheDir,
|
||||
loadSession,
|
||||
login: (opts) => Client.login(opts),
|
||||
prompt: (message) => input({ message }),
|
||||
promptSecret: (message) => password({ message, mask: true }),
|
||||
});
|
||||
|
||||
const run = (command: Promise<number>): Promise<void> =>
|
||||
runCommand(command, stdout, stderr, log(), (code) => process.exit(code));
|
||||
runCommand(command, stdout, stderr, (code) => process.exit(code));
|
||||
|
||||
program
|
||||
.command("login")
|
||||
.description("Log in to an Ente account and save the session")
|
||||
.option(
|
||||
"--server <url>",
|
||||
"Address of a self-hosted Ente server's API, saved with the session " +
|
||||
"(default: QUAK_SERVER, then Ente's hosted service)",
|
||||
)
|
||||
.action((opts: { server?: string }) => run(loginCommand(context(), opts)));
|
||||
.action(() => run(loginCommand(context())));
|
||||
|
||||
program
|
||||
.command("whoami")
|
||||
@@ -116,15 +78,14 @@ program
|
||||
|
||||
program
|
||||
.command("files")
|
||||
.description("List the files in a collection, or every file in the account")
|
||||
.argument(
|
||||
"[collectionID]",
|
||||
"Collection ID (from `quak collections`); without it, every file " +
|
||||
"is listed once",
|
||||
.description("List files in a collection")
|
||||
.requiredOption(
|
||||
"--collection <id>",
|
||||
"Collection ID (from `quak collections`)",
|
||||
)
|
||||
.option("--json", "Output as JSON array")
|
||||
.action((collectionID: string | undefined, opts: { json?: boolean }) =>
|
||||
run(filesCommand(context(), collectionID, opts)),
|
||||
.action((opts: { collection: string; json?: boolean }) =>
|
||||
run(filesCommand(context(), opts)),
|
||||
);
|
||||
|
||||
program
|
||||
@@ -132,6 +93,7 @@ program
|
||||
.description("Download and decrypt a single file")
|
||||
.argument("<fileID>", "File ID (from `quak files`)")
|
||||
.option("--out <path>", "Output file path")
|
||||
.option("--collection <id>", "Accepted for compatibility; ignored")
|
||||
.action((fileID: string, opts: { out?: string }) =>
|
||||
run(getCommand(context(), fileID, opts)),
|
||||
);
|
||||
@@ -141,6 +103,7 @@ program
|
||||
.description("Download and decrypt a thumbnail")
|
||||
.argument("<fileID>", "File ID (from `quak files`)")
|
||||
.option("--out <path>", "Output file path")
|
||||
.option("--collection <id>", "Accepted for compatibility; ignored")
|
||||
.action((fileID: string, opts: { out?: string }) =>
|
||||
run(getThumbCommand(context(), fileID, opts)),
|
||||
);
|
||||
@@ -179,10 +142,7 @@ program
|
||||
{
|
||||
...context(),
|
||||
loadSession: (path) =>
|
||||
loadSession(
|
||||
path,
|
||||
logRequests(log(), UNATTENDED_RETRY_OPTIONS),
|
||||
),
|
||||
loadSession(path, { retry: UNATTENDED_RETRY_OPTIONS }),
|
||||
},
|
||||
dir,
|
||||
opts,
|
||||
|
||||
+1
-1
@@ -12,7 +12,7 @@
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"engines": {
|
||||
"node": "^22.13.0 || >=23.5.0"
|
||||
"node": ">=22"
|
||||
},
|
||||
"main": "./dist/src/index.js",
|
||||
"types": "./dist/src/index.d.ts",
|
||||
|
||||
@@ -1,85 +0,0 @@
|
||||
#!/bin/sh
|
||||
# script/build-bin: run script/build, then bundle the built dist/ into one
|
||||
# executable, bin/quak, which reports the version script/build stamped. Our own
|
||||
# extension to scripts-to-rule-them-all.
|
||||
#
|
||||
# bun compiles the binary from a copy of its own executable, so the binary
|
||||
# loads the same libraries as the bun that built it. nix's bun loads libraries
|
||||
# from the nix store: on Linux its C library, on macOS its ICU. A binary
|
||||
# compiled from it loads them too. nix-collect-garbage can delete them, and the
|
||||
# binary then no longer starts. This script therefore compiles with bun's own
|
||||
# release build, which loads only the system's libraries. It downloads that
|
||||
# build and checks it against the hash below before running it.
|
||||
set -eu
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
|
||||
# bun 1.3.13, 2026-11-11, and the SHA-256 of each release archive below. The
|
||||
# x86-64 archives are the baseline builds, which also run on CPUs without AVX2.
|
||||
# The musl archives are for Linux systems that use the musl C library, such as
|
||||
# Alpine.
|
||||
BUN_VERSION="1.3.13"
|
||||
BUN_LINUX_X64_BASELINE_SHA256="9d8a24292a7068090205daac0a5a223f5f69736f5287e37bf88d3b4031edc750"
|
||||
BUN_LINUX_X64_MUSL_BASELINE_SHA256="88ca7c7ad235b498f549eea2f770f434e9f0f5e9ba95168a2d3a1f235184c394"
|
||||
BUN_LINUX_AARCH64_SHA256="70bae41b3908b0a120e1e58c5c8af30e74afae3b8d11b0d3fdd8e787ddfb4b22"
|
||||
BUN_LINUX_AARCH64_MUSL_SHA256="5385e978107ce4934298d8d6afe9bfbb898683f6cc23e6753a0da60bc60c5b81"
|
||||
BUN_DARWIN_X64_BASELINE_SHA256="a98ba6a480f22fda9b343626b906a4e26aa53618bf85d2bc5928ecf2ba45f0ed"
|
||||
BUN_DARWIN_AARCH64_SHA256="5467e3f65dba526b9fea98f0cce04efafc0c63e169733ec27b876a3ad32da190"
|
||||
|
||||
# compile_with_downloaded_bun <archive-name> <sha256>
|
||||
compile_with_downloaded_bun() {
|
||||
tmp="$(mktemp -d)"
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
curl -fsSL -o "$tmp/$1.zip" \
|
||||
"https://github.com/oven-sh/bun/releases/download/bun-v$BUN_VERSION/$1.zip"
|
||||
# macOS always has shasum, but not every macOS has sha256sum.
|
||||
if command -v sha256sum >/dev/null 2>&1; then
|
||||
echo "$2 $tmp/$1.zip" | sha256sum -c -
|
||||
else
|
||||
echo "$2 $tmp/$1.zip" | shasum -a 256 -c -
|
||||
fi
|
||||
unzip -q "$tmp/$1.zip" -d "$tmp"
|
||||
"$tmp/$1/bun" build dist/bin/quak.js --compile --outfile bin/quak
|
||||
}
|
||||
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
"$ROOT/script/build"
|
||||
platform="$(uname -s)-$(uname -m)"
|
||||
# musl's ldd begins its version output with "musl libc"; glibc's does not.
|
||||
if ldd --version 2>&1 | grep -q '^musl libc'; then
|
||||
platform="$platform-musl"
|
||||
fi
|
||||
case "$platform" in
|
||||
Linux-x86_64)
|
||||
compile_with_downloaded_bun bun-linux-x64-baseline \
|
||||
"$BUN_LINUX_X64_BASELINE_SHA256"
|
||||
;;
|
||||
Linux-x86_64-musl)
|
||||
compile_with_downloaded_bun bun-linux-x64-musl-baseline \
|
||||
"$BUN_LINUX_X64_MUSL_BASELINE_SHA256"
|
||||
;;
|
||||
Linux-aarch64)
|
||||
compile_with_downloaded_bun bun-linux-aarch64 \
|
||||
"$BUN_LINUX_AARCH64_SHA256"
|
||||
;;
|
||||
Linux-aarch64-musl)
|
||||
compile_with_downloaded_bun bun-linux-aarch64-musl \
|
||||
"$BUN_LINUX_AARCH64_MUSL_SHA256"
|
||||
;;
|
||||
Darwin-x86_64)
|
||||
compile_with_downloaded_bun bun-darwin-x64-baseline \
|
||||
"$BUN_DARWIN_X64_BASELINE_SHA256"
|
||||
;;
|
||||
Darwin-arm64)
|
||||
compile_with_downloaded_bun bun-darwin-aarch64 \
|
||||
"$BUN_DARWIN_AARCH64_SHA256"
|
||||
;;
|
||||
*)
|
||||
echo "build-bin: bun has no release build for $(uname -s) on $(uname -m)" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
main "$@"
|
||||
+13
-65
@@ -36,15 +36,6 @@ export interface ApiClientOptions {
|
||||
retry?: RetryOptions;
|
||||
requestTimeoutMs?: number;
|
||||
downloadTimeoutMs?: number;
|
||||
// Called once for each request sent, with the status of its response or
|
||||
// the error it failed with, for a caller that logs requests. A request
|
||||
// that is retried is reported once for each attempt.
|
||||
onRequest?: (request: {
|
||||
method: string;
|
||||
url: string;
|
||||
status?: number;
|
||||
error?: unknown;
|
||||
}) => void;
|
||||
}
|
||||
|
||||
export interface StreamOptions {
|
||||
@@ -168,7 +159,6 @@ export class ApiClient {
|
||||
private readonly retry: ResolvedRetryOptions;
|
||||
private readonly requestTimeoutMs: number;
|
||||
private readonly downloadTimeoutMs: number;
|
||||
private readonly onRequest: NonNullable<ApiClientOptions["onRequest"]>;
|
||||
private token: string | undefined;
|
||||
|
||||
constructor(opts?: ApiClientOptions) {
|
||||
@@ -190,7 +180,6 @@ export class ApiClient {
|
||||
opts?.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS;
|
||||
this.downloadTimeoutMs =
|
||||
opts?.downloadTimeoutMs ?? DEFAULT_DOWNLOAD_TIMEOUT_MS;
|
||||
this.onRequest = opts?.onRequest ?? (() => {});
|
||||
this.token = opts?.authToken;
|
||||
}
|
||||
|
||||
@@ -225,33 +214,6 @@ export class ApiClient {
|
||||
return h;
|
||||
}
|
||||
|
||||
// Send one request and report it to `onRequest`, with the status of its
|
||||
// response or the error it failed with. `reportedURL` is the URL given to
|
||||
// `onRequest`, when that must not be the one sent.
|
||||
private async send(
|
||||
url: string,
|
||||
init: RequestInit & { method: string },
|
||||
reportedURL = url,
|
||||
): Promise<Response> {
|
||||
let resp: Response;
|
||||
try {
|
||||
resp = await this._fetch(url, init);
|
||||
} catch (err) {
|
||||
this.onRequest({
|
||||
method: init.method,
|
||||
url: reportedURL,
|
||||
error: err,
|
||||
});
|
||||
throw err;
|
||||
}
|
||||
this.onRequest({
|
||||
method: init.method,
|
||||
url: reportedURL,
|
||||
status: resp.status,
|
||||
});
|
||||
return resp;
|
||||
}
|
||||
|
||||
private async throwIfError(resp: Response): Promise<void> {
|
||||
if (resp.ok) return;
|
||||
const requestID = resp.headers.get("x-request-id") ?? undefined;
|
||||
@@ -295,7 +257,7 @@ export class ApiClient {
|
||||
const url = buildURL(this.apiOrigin, path, query);
|
||||
// A GET changes nothing, so it is retried under the full policy.
|
||||
return withRetry(async () => {
|
||||
const resp = await this.send(url, {
|
||||
const resp = await this._fetch(url, {
|
||||
method: "GET",
|
||||
headers: this.headers(),
|
||||
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
||||
@@ -318,7 +280,7 @@ export class ApiClient {
|
||||
// 3xx status, which is not retried.
|
||||
return withRetry(
|
||||
async () => {
|
||||
const resp = await this.send(url, {
|
||||
const resp = await this._fetch(url, {
|
||||
method: "POST",
|
||||
headers: this.headers({
|
||||
"Content-Type": "application/json",
|
||||
@@ -354,35 +316,21 @@ export class ApiClient {
|
||||
});
|
||||
}
|
||||
|
||||
// `contentMD5` is the value given to `getUploadURL`. The server signs it
|
||||
// into the URL, and storage refuses a PUT without that `Content-MD5`.
|
||||
async putFile(
|
||||
presignedURL: string,
|
||||
data: Uint8Array,
|
||||
contentMD5: string,
|
||||
): Promise<void> {
|
||||
async putFile(presignedURL: string, data: Uint8Array): Promise<void> {
|
||||
// Idempotent despite being a write: a presigned PUT stores one whole
|
||||
// object at one key in one request, so replaying it either overwrites
|
||||
// the same bytes or lands them for the first time. There is no partial
|
||||
// state to protect, hence the full policy rather than the POST rule.
|
||||
// It is reported without its query, which holds the upload's
|
||||
// signature: a log keeps what it is given.
|
||||
const { origin, pathname } = new URL(presignedURL);
|
||||
await withRetry(async () => {
|
||||
const resp = await this.send(
|
||||
presignedURL,
|
||||
{
|
||||
method: "PUT",
|
||||
headers: {
|
||||
"Content-Type": "application/octet-stream",
|
||||
"Content-Length": String(data.length),
|
||||
"Content-MD5": contentMD5,
|
||||
},
|
||||
body: data,
|
||||
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
||||
const resp = await this._fetch(presignedURL, {
|
||||
method: "PUT",
|
||||
headers: {
|
||||
"Content-Type": "application/octet-stream",
|
||||
"Content-Length": String(data.length),
|
||||
},
|
||||
origin + pathname,
|
||||
);
|
||||
body: data,
|
||||
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
||||
});
|
||||
if (!resp.ok) {
|
||||
// An ApiError, not a bare Error: without the status on the
|
||||
// error the upload path cannot be classified at all, and a
|
||||
@@ -400,7 +348,7 @@ export class ApiClient {
|
||||
// Same replay and redirect rules as `postJSON`, for the same reasons.
|
||||
return withRetry(
|
||||
async () => {
|
||||
const resp = await this.send(url, {
|
||||
const resp = await this._fetch(url, {
|
||||
method: "PUT",
|
||||
headers: this.headers({
|
||||
"Content-Type": "application/json",
|
||||
@@ -446,7 +394,7 @@ export class ApiClient {
|
||||
// headers, when no bytes have arrived either.
|
||||
const deadline = idleDeadline(this.downloadTimeoutMs);
|
||||
try {
|
||||
const resp = await this.send(url, {
|
||||
const resp = await this._fetch(url, {
|
||||
method: "GET",
|
||||
headers: this.headers(),
|
||||
signal: deadline.signal,
|
||||
|
||||
+85
-94
@@ -14,9 +14,9 @@
|
||||
// YYYY/YYYY-MM/YYYY-MM-DD/
|
||||
// YYYY-MM-DD.<fileID>.<ext> the decrypted bytes (the save path)
|
||||
// YYYY-MM-DD.<fileID>.json per-file metadata sidecar, with
|
||||
// the file's ML data, its
|
||||
// keywords, and its original's
|
||||
// EXIF, XMP and dimensions
|
||||
// the file's ML data and its
|
||||
// original's EXIF, XMP and
|
||||
// dimensions
|
||||
// collections/<name>/<title> symlink to the original
|
||||
// collections/<name>.json per-collection metadata
|
||||
// account.json the account's email and user ID
|
||||
@@ -41,9 +41,8 @@
|
||||
//
|
||||
// With `verify`, each original already at its save path is hashed as the
|
||||
// download check hashes it, and one that does not match the content hash its
|
||||
// metadata records is replaced once a matching copy is downloaded, in the same
|
||||
// run. Until then it stays as it is: a download that fails, or that does not
|
||||
// match either, leaves it in place and is recorded as failed.
|
||||
// metadata records is removed and fetched again in the same run. What is put
|
||||
// back is hashed too, and recorded as failed if it still does not match.
|
||||
//
|
||||
// Resilience (issue #8): no per-file condition aborts the run. A failed
|
||||
// download, a failed symlink, or ML data missing because the ML data fetch
|
||||
@@ -62,6 +61,7 @@
|
||||
// code.
|
||||
|
||||
import {
|
||||
createReadStream,
|
||||
lstatSync,
|
||||
mkdirSync,
|
||||
readdirSync,
|
||||
@@ -77,23 +77,25 @@ import { readFile } from "node:fs/promises";
|
||||
import { dirname, extname, join, relative, resolve } from "node:path";
|
||||
import lockfile from "proper-lockfile";
|
||||
|
||||
import { clipKeywords } from "./clip-keywords.js";
|
||||
import {
|
||||
chunkHashFinal,
|
||||
chunkHashInit,
|
||||
chunkHashUpdate,
|
||||
init,
|
||||
} from "./crypto/index.js";
|
||||
import { removeLeftoverTempFiles } from "./download/index.js";
|
||||
import { describeError } from "./errors.js";
|
||||
import {
|
||||
MAX_NAME_BYTES,
|
||||
sanitizeFileName,
|
||||
shortenKeepingExtension,
|
||||
shortenToBytes,
|
||||
withExtension,
|
||||
} from "./filename.js";
|
||||
import {
|
||||
copyAtomic,
|
||||
placeOriginal,
|
||||
replaceOriginal,
|
||||
savePath,
|
||||
storedAtSavePath,
|
||||
storedHash,
|
||||
} from "./library/content.js";
|
||||
import { representative } from "./library/records.js";
|
||||
import type { Undecryptable } from "./library/store.js";
|
||||
@@ -102,10 +104,7 @@ import type { MLData } from "./mldata-fetch.js";
|
||||
import type { Collection, EnteFile, FileMetadata } from "./model/types.js";
|
||||
import { hasTransportCode } from "./retry.js";
|
||||
|
||||
// Reports one line of the backup's progress. `level` is "error" when the line
|
||||
// reports something that failed, such as a file or an album, and is left out
|
||||
// for every other line.
|
||||
export type ProgressCallback = (message: string, level?: "error") => void;
|
||||
export type ProgressCallback = (message: string) => void;
|
||||
|
||||
// The lock in a backup directory, as `lockBackupDirectory` returns it.
|
||||
export interface BackupLock {
|
||||
@@ -153,9 +152,9 @@ export interface BackupResult {
|
||||
// Originals already at their save path and left untouched.
|
||||
skipped: number;
|
||||
// With `verify`, the originals already at their save path whose hash
|
||||
// matched, those whose hash did not (each replaced once a matching copy is
|
||||
// downloaded), and those whose metadata records no hash (left as they
|
||||
// are). All three are zero without `verify`.
|
||||
// matched, those whose hash did not (each removed and fetched again), and
|
||||
// those whose metadata records no hash (left as they are). All three are
|
||||
// zero without `verify`.
|
||||
verified: number;
|
||||
mismatched: number;
|
||||
unchecked: number;
|
||||
@@ -193,15 +192,6 @@ export interface BackupLibrary {
|
||||
fileID: number,
|
||||
destination: string,
|
||||
): Promise<{ path: string; videoPath?: string }>;
|
||||
// Fetch an original's bytes to `destination` through the content pool,
|
||||
// a live photo as its image and its video beside it, even when the cache
|
||||
// or the library's save path already holds it, and return where they
|
||||
// landed. With `verify`, an original whose hash does not match is replaced
|
||||
// with what this fetches.
|
||||
fetchOriginal(
|
||||
fileID: number,
|
||||
destination: string,
|
||||
): Promise<{ path: string; videoPath?: string }>;
|
||||
thumbnail(fileID: number): Promise<{ path: string }>;
|
||||
// How many originals the backup works on at once: the limit of the
|
||||
// library's content pool.
|
||||
@@ -213,11 +203,6 @@ export interface BackupLibrary {
|
||||
// A file's cached ML data, as `lib.mldata.forFile()` returns it, or
|
||||
// undefined when none is cached.
|
||||
mlData(fileID: number): Promise<MLData | undefined>;
|
||||
// Why the ML data the last ML data fetch that succeeded got for a file did
|
||||
// not decrypt, gunzip or parse, or undefined when it did or the fetch got
|
||||
// none. A fetch still running does not change it. A fetch that fails
|
||||
// leaves it as it was.
|
||||
mlDataError(fileID: number): unknown;
|
||||
}
|
||||
|
||||
type FailureClass = "transient" | "permanent" | "unknown";
|
||||
@@ -248,13 +233,10 @@ const isPresent = (path: string): boolean => {
|
||||
// are transient; refusals and local filesystem/decrypt errors are permanent;
|
||||
// anything else is unknown. The error code and the error as `describeError`
|
||||
// gives it are inspected, and so is each errno on the error's cause chain,
|
||||
// where Node's `fetch` puts it. An error named `TimeoutError` is transient
|
||||
// whatever its message says, as `isRetryable` retries it: each of quak's
|
||||
// deadlines aborts with one, and the download's says only that it stalled.
|
||||
// where Node's `fetch` puts it.
|
||||
const classify = (err: unknown): FailureClass => {
|
||||
if (hasTransportCode(err)) return "transient";
|
||||
const e = err as NodeJS.ErrnoException;
|
||||
if (e?.name === "TimeoutError") return "transient";
|
||||
const text = `${e?.code ?? ""} ${describeError(err)}`.toLowerCase();
|
||||
if (
|
||||
/timeout|timed out|econnreset|econnrefused|econnaborted|network|socket|eai_again|throttl|temporarily|429|500|502|503|504/.test(
|
||||
@@ -294,10 +276,9 @@ const rebuildSymlink = (linkPath: string, target: string): void => {
|
||||
// case (two names that differ only in case are one entry on a case-insensitive
|
||||
// file system), or it is too long for a file name; then it gets ` (<id>)`,
|
||||
// right before the extension when `beforeExtension` is set. To fit, the part
|
||||
// before the ID is cut short as far as it must be. When the extension leaves
|
||||
// no room for the ID and the first character of the name, the extension is
|
||||
// cut short, and the part before the ID keeps that character and any bytes
|
||||
// the cut leaves over (see `shortenKeepingExtension`). With `withJSON`, each
|
||||
// before the ID is cut short as far as it must be. When the extension alone
|
||||
// leaves no room for the ID, the extension is cut short and the part before
|
||||
// the ID keeps only the bytes that cut leaves over. With `withJSON`, each
|
||||
// entry also has the JSON file `<name>.json` beside it, as an album does: a
|
||||
// name then leaves room for `.json`, and one that is another entry's JSON name
|
||||
// gets its ID added too.
|
||||
@@ -311,9 +292,11 @@ const uniqueNames = (
|
||||
const maxBytes = MAX_NAME_BYTES - (opts.withJSON ? ".json".length : 0);
|
||||
const withID = (id: number, name: string): string => {
|
||||
const tag = ` (${id})`;
|
||||
return opts.beforeExtension
|
||||
? shortenKeepingExtension(name, maxBytes, tag)
|
||||
: shortenToBytes(name, maxBytes - Buffer.byteLength(tag)) + tag;
|
||||
const fullExt = opts.beforeExtension ? extname(name) : "";
|
||||
const stem = name.slice(0, name.length - fullExt.length);
|
||||
const ext = shortenToBytes(fullExt, maxBytes - Buffer.byteLength(tag));
|
||||
const room = maxBytes - Buffer.byteLength(tag + ext);
|
||||
return shortenToBytes(stem, room) + tag + ext;
|
||||
};
|
||||
const names = entries.map((e) => e.name);
|
||||
const suffixed = new Set<number>();
|
||||
@@ -458,6 +441,26 @@ const saveLedger = (path: string, ledger: Map<number, FailureEntry>): void => {
|
||||
);
|
||||
};
|
||||
|
||||
// The content hash of the original stored at `stored`, computed as the download
|
||||
// check computes it: over each file's bytes, read in chunks, and for a live
|
||||
// photo `<imageHash>:<videoHash>`.
|
||||
const storedHash = async (stored: {
|
||||
path: string;
|
||||
videoPath?: string;
|
||||
}): Promise<string> => {
|
||||
await init();
|
||||
const hashFile = async (path: string): Promise<string> => {
|
||||
const state = chunkHashInit();
|
||||
for await (const chunk of createReadStream(path)) {
|
||||
chunkHashUpdate(state, chunk as Buffer);
|
||||
}
|
||||
return chunkHashFinal(state);
|
||||
};
|
||||
const hash = await hashFile(stored.path);
|
||||
if (stored.videoPath === undefined) return hash;
|
||||
return `${hash}:${await hashFile(stored.videoPath)}`;
|
||||
};
|
||||
|
||||
// A file's EXIF, XMP and dimensions as its JSON holds them: what
|
||||
// `extractImageMetadata` found in its original, or why the original could not
|
||||
// be read.
|
||||
@@ -498,9 +501,8 @@ const imageMetadataFor = async (
|
||||
}
|
||||
};
|
||||
|
||||
// The file's JSON: its basic fields, its magic metadata, its ML data without
|
||||
// the CLIP embedding and the keywords that embedding gives, or the reason the
|
||||
// ML data is missing, and its image metadata.
|
||||
// The file's JSON: its basic fields, its magic metadata, its ML data or the
|
||||
// reason the ML data is missing, and its image metadata.
|
||||
const writeSidecar = (
|
||||
path: string,
|
||||
file: EnteFile,
|
||||
@@ -516,13 +518,7 @@ const writeSidecar = (
|
||||
};
|
||||
if (file.magicMetadata) meta.magicMetadata = file.magicMetadata;
|
||||
if (file.pubMagicMetadata) meta.pubMagicMetadata = file.pubMagicMetadata;
|
||||
if (ml.mlData) {
|
||||
const { clip: _clip, ...mlDataWithoutClip } = ml.mlData;
|
||||
meta.mlData = mlDataWithoutClip;
|
||||
const isVideo = file.metadata.fileType === "video";
|
||||
const keywords = clipKeywords(ml.mlData, isVideo);
|
||||
if (keywords) meta.keywords = keywords;
|
||||
}
|
||||
if (ml.mlData) meta.mlData = ml.mlData;
|
||||
if (ml.mlDataError) meta.mlDataError = ml.mlDataError;
|
||||
if (image.imageMetadata) meta.imageMetadata = image.imageMetadata;
|
||||
if (image.imageMetadataError) {
|
||||
@@ -665,7 +661,7 @@ const runLockedBackup = async (
|
||||
for (const record of lib.undecryptable()) {
|
||||
const error = `could not be decrypted: ${record.error}`;
|
||||
if (record.fileID === undefined) {
|
||||
log(`FAILED album ${record.collectionID}: ${error}`, "error");
|
||||
log(`FAILED album ${record.collectionID}: ${error}`);
|
||||
undecryptableAlbums.push(record.collectionID);
|
||||
} else if (scope.has(record.collectionID)) {
|
||||
const copies = memberships.get(record.fileID);
|
||||
@@ -686,13 +682,11 @@ const runLockedBackup = async (
|
||||
// through the content cache/pools, as `Photo.download()` does, and fetch
|
||||
// the optional thumbnails; a present file is left as is. With `verify`, a
|
||||
// present original is hashed first, and one that does not match the hash
|
||||
// its metadata records is downloaded again and replaced by
|
||||
// `replaceOriginal`, which leaves it as it was when that download fails
|
||||
// or does not match either. One that cannot be read is recorded as failed
|
||||
// and left where it is. `storeOriginal` does this for one original, and
|
||||
// records its failure.
|
||||
// its metadata records is removed and fetched like a missing one. One
|
||||
// that cannot be read is recorded as failed and left where it is.
|
||||
// `storeOriginal` does this for one original, and records its failure.
|
||||
const storeOriginal = async (file: EnteFile): Promise<void> => {
|
||||
const stored = storedAtSavePath(downloadDirectory, file);
|
||||
let stored = storedAtSavePath(downloadDirectory, file);
|
||||
let mismatch = false;
|
||||
if (stored !== undefined && verify) {
|
||||
try {
|
||||
@@ -706,11 +700,15 @@ const runLockedBackup = async (
|
||||
);
|
||||
mismatched++;
|
||||
mismatch = true;
|
||||
rmSync(stored.path);
|
||||
if (stored.videoPath !== undefined) {
|
||||
rmSync(stored.videoPath);
|
||||
}
|
||||
stored = undefined;
|
||||
}
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED verifying original ${file.metadata.title}: ${describeError(err)}`,
|
||||
"error",
|
||||
);
|
||||
recordFailure(
|
||||
file.id,
|
||||
@@ -721,30 +719,33 @@ const runLockedBackup = async (
|
||||
return;
|
||||
}
|
||||
}
|
||||
if (stored !== undefined && !mismatch) {
|
||||
if (stored !== undefined) {
|
||||
skipped++;
|
||||
return;
|
||||
}
|
||||
try {
|
||||
log(`Fetching original ${file.metadata.title} (${file.id})...`);
|
||||
if (mismatch) {
|
||||
await replaceOriginal(downloadDirectory, file, (dest) =>
|
||||
lib.fetchOriginal(file.id, dest),
|
||||
);
|
||||
} else {
|
||||
// A fetched original is written straight to its save path (a
|
||||
// live photo beside it); only one that was already cached
|
||||
// elsewhere is copied.
|
||||
await placeOriginal(downloadDirectory, file, (dest) =>
|
||||
lib.original(file.id, dest),
|
||||
// A fetched original is written straight to its save path (a
|
||||
// live photo beside it); only one that was already cached
|
||||
// elsewhere is copied.
|
||||
const placed = await placeOriginal(
|
||||
downloadDirectory,
|
||||
file,
|
||||
(dest) => lib.original(file.id, dest),
|
||||
);
|
||||
storedThisRun.add(file.id);
|
||||
// A copy from the cache is not checked as a download is and can
|
||||
// hold the same bad bytes, so what is put back after a mismatch
|
||||
// is hashed too. A bad copy stays where it is and the file fails.
|
||||
if (mismatch && (await storedHash(placed)) !== file.metadata.hash) {
|
||||
throw new Error(
|
||||
"the original put back does not match its content hash either",
|
||||
);
|
||||
}
|
||||
storedThisRun.add(file.id);
|
||||
downloaded++;
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED original ${file.metadata.title}: ${describeError(err)}`,
|
||||
"error",
|
||||
);
|
||||
recordFailure(
|
||||
file.id,
|
||||
@@ -809,18 +810,19 @@ const runLockedBackup = async (
|
||||
// Phase 2: rebuild the derived views from the model. Sidecars first, for
|
||||
// every present original (this repairs stale ones), each with the file's
|
||||
// ML data once an ML data fetch has completed, and its image metadata.
|
||||
// When the fetch got a file's ML data but could not decrypt, gunzip or
|
||||
// parse it, or when the fetch failed, a file with no cached ML data gets
|
||||
// the reason instead and is recorded as failed. The next run fetches its
|
||||
// ML data again because none is cached.
|
||||
// When the fetch fails, a file with no cached ML data gets the reason
|
||||
// instead and is recorded as failed. The next run fetches its ML data
|
||||
// again because none is cached.
|
||||
if (includeOriginals) {
|
||||
let mlDataFailure: unknown;
|
||||
let mlDataError: string | undefined;
|
||||
try {
|
||||
log("Fetching ML data...");
|
||||
await lib.fetchMLData();
|
||||
} catch (err) {
|
||||
mlDataFailure = err;
|
||||
log(`FAILED ML data: ${describeError(err)}`, "error");
|
||||
mlDataError = describeError(err);
|
||||
log(`FAILED ML data: ${mlDataError}`);
|
||||
}
|
||||
for (const file of distinct.values()) {
|
||||
if (lock.lost) throw lock.lost;
|
||||
@@ -837,19 +839,13 @@ const runLockedBackup = async (
|
||||
storedThisRun.has(file.id),
|
||||
);
|
||||
const mlData = await lib.mlData(file.id);
|
||||
const failure = lib.mlDataError(file.id) ?? mlDataFailure;
|
||||
if (mlData === undefined && failure !== undefined) {
|
||||
writeSidecar(
|
||||
path,
|
||||
file,
|
||||
{ mlDataError: describeError(failure) },
|
||||
image,
|
||||
);
|
||||
if (mlData === undefined && mlDataError !== undefined) {
|
||||
writeSidecar(path, file, { mlDataError }, image);
|
||||
recordFailure(
|
||||
file.id,
|
||||
file.metadata.title,
|
||||
collectionName.get(file.collectionID) ?? "",
|
||||
new Error("ML data", { cause: failure }),
|
||||
new Error("ML data", { cause: mlDataFailure }),
|
||||
);
|
||||
} else {
|
||||
writeSidecar(path, file, { mlData }, image);
|
||||
@@ -881,10 +877,7 @@ const runLockedBackup = async (
|
||||
downloadDirectory,
|
||||
);
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED removing old album directories: ${describeError(err)}`,
|
||||
"error",
|
||||
);
|
||||
log(`FAILED removing old album directories: ${describeError(err)}`);
|
||||
}
|
||||
|
||||
// An album whose folder or JSON cannot be written, such as one whose
|
||||
@@ -913,7 +906,6 @@ const runLockedBackup = async (
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED removing old links in ${c.name}: ${describeError(err)}`,
|
||||
"error",
|
||||
);
|
||||
}
|
||||
|
||||
@@ -932,7 +924,6 @@ const runLockedBackup = async (
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED symlink ${c.name}/${linkName}: ${describeError(err)}`,
|
||||
"error",
|
||||
);
|
||||
recordFailure(
|
||||
link.file.id,
|
||||
@@ -949,7 +940,7 @@ const runLockedBackup = async (
|
||||
metaFiles,
|
||||
);
|
||||
} catch (err) {
|
||||
log(`FAILED album ${c.name}: ${describeError(err)}`, "error");
|
||||
log(`FAILED album ${c.name}: ${describeError(err)}`);
|
||||
failedAlbums.push(c.id);
|
||||
}
|
||||
}
|
||||
|
||||
+122
-186
@@ -4,10 +4,7 @@
|
||||
// code; a thrown error is left to the caller. Nothing here calls
|
||||
// `process.exit`: `bin/quak.ts` wires these to the command line, and `run` in
|
||||
// `cli-run.ts` prints a thrown error as one line and exits once output has
|
||||
// drained. Output must stay byte-identical (see `cli-output.ts`). Each line on
|
||||
// stderr goes through `ctx.log` at its level (see `cli-log.ts`). Every command
|
||||
// that exits non-zero logs a fatal line saying why; one that exits 1 because
|
||||
// files or albums failed logs it last, saying how many and where to look.
|
||||
// drained. Output must stay byte-identical (see `cli-output.ts`).
|
||||
|
||||
import {
|
||||
copyFileSync,
|
||||
@@ -21,7 +18,6 @@ import { extname, join } from "node:path";
|
||||
import {
|
||||
type Client,
|
||||
type ClientSnapshot,
|
||||
isServerAddress,
|
||||
type LoginOptions,
|
||||
} from "./client.js";
|
||||
import { lockBackupDirectory, type BackupLock } from "./backup.js";
|
||||
@@ -37,27 +33,18 @@ import {
|
||||
originalName,
|
||||
thumbnailName,
|
||||
} from "./cli-output.js";
|
||||
import { type Log, logRefresh } from "./cli-log.js";
|
||||
import {
|
||||
cachedFilesByID,
|
||||
freshCollections,
|
||||
freshFiles,
|
||||
freshFile,
|
||||
freshFilesByID,
|
||||
} from "./cli-read.js";
|
||||
import { freshCollections, freshFiles, freshFile } from "./cli-read.js";
|
||||
import { withExtension } from "./filename.js";
|
||||
import { runMetadataBackup } from "./metadata-backup.js";
|
||||
import type { Collection, EnteFile } from "./model/types.js";
|
||||
import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js";
|
||||
|
||||
export interface CliContext {
|
||||
stdout: { write(text: string): unknown };
|
||||
// Writes a line on stderr at a log level, if the run's level prints it.
|
||||
log: Log;
|
||||
stderr: { write(text: string): unknown };
|
||||
// Directory holding `session.json`.
|
||||
sessionDir: string;
|
||||
// The `--cache-dir` global, or undefined to let the library pick its
|
||||
// per-user default keyed by the account's server and user ID.
|
||||
// per-user default keyed by the account id.
|
||||
cacheDir?: string;
|
||||
// Reads the session file into a client, or null when there is none. The
|
||||
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
|
||||
@@ -95,16 +82,16 @@ const requireSession = (ctx: CliContext): Client | undefined => {
|
||||
try {
|
||||
client = ctx.loadSession(sessionPath(ctx));
|
||||
} catch (err) {
|
||||
ctx.log("fatal", err instanceof Error ? err.message : String(err));
|
||||
ctx.log(
|
||||
"fatal",
|
||||
`Run "quak logout" and then "quak login" to replace it.`,
|
||||
ctx.stderr.write(
|
||||
`${err instanceof Error ? err.message : err}\n` +
|
||||
`Run "quak logout" and then "quak login" to replace it.\n`,
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
if (!client) {
|
||||
ctx.log("fatal", `Not logged in. Run "quak login" first.`);
|
||||
ctx.log("fatal", `Session file: ${sessionPath(ctx)}`);
|
||||
ctx.stderr.write(
|
||||
`Not logged in. Run "quak login" first.\nSession file: ${sessionPath(ctx)}\n`,
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
return client;
|
||||
@@ -120,7 +107,7 @@ const parseID = (
|
||||
): number | undefined => {
|
||||
const id = Number(text);
|
||||
if (/^[0-9]+$/.test(text) && Number.isSafeInteger(id)) return id;
|
||||
ctx.log("fatal", `Invalid ${kind} ID: ${text}`);
|
||||
ctx.stderr.write(`Invalid ${kind} ID: ${text}\n`);
|
||||
return undefined;
|
||||
};
|
||||
|
||||
@@ -130,73 +117,47 @@ const parseID = (
|
||||
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
|
||||
// originals through the on-disk cache.
|
||||
const readLibraryClient = (client: Client): LibraryClient => ({
|
||||
apiOrigin: client.apiOrigin,
|
||||
whoami: () => client.whoami(),
|
||||
collectionsSince: (args) => client.collectionsSince(args),
|
||||
filesSince: (args) => client.filesSince(args),
|
||||
contentSource: () => client.contentSource(),
|
||||
});
|
||||
|
||||
// Open a library for a single point command. The aggressive background
|
||||
// precache (issue #48) is off, because a one-shot `collections` or `get` must
|
||||
// not start downloading the whole account. No refresh starts as the library
|
||||
// opens or on a timer. Each command asks for the refresh it needs.
|
||||
// `collections` and `files` with a collection ID need none (issues #210 and
|
||||
// #217). Nor do `get` and `get-thumb` on an empty cache (issue #222). Nor does
|
||||
// `helper fix-missing-thumbnails --file` on an empty cache (issue #223). While
|
||||
// a refresh runs, each collection whose files it has fetched is logged, so a
|
||||
// large account shows progress rather than nothing.
|
||||
// Open a library for a single point command: the aggressive background precache
|
||||
// (issue #48) is off — a one-shot `collections` or `get` must not start
|
||||
// downloading the whole account — and the refresh interval is long so no second
|
||||
// refresh fires mid-command.
|
||||
const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
|
||||
Library.open({
|
||||
client: readLibraryClient(client),
|
||||
cacheDirectory: ctx.cacheDir,
|
||||
refreshOnOpen: false,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
refreshIntervalSeconds: 3600,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
onProgress: logRefresh(ctx.log),
|
||||
});
|
||||
|
||||
// Logs in to the server `--server` or `QUAK_SERVER` names, the flag first, or
|
||||
// to Ente's hosted service when neither is set. The saved session keeps the
|
||||
// server, so every later command uses it.
|
||||
export const loginCommand = async (
|
||||
ctx: CliContext,
|
||||
opts: { server?: string },
|
||||
): Promise<number> => {
|
||||
export const loginCommand = async (ctx: CliContext): Promise<number> => {
|
||||
await init();
|
||||
const server = opts.server ?? process.env.QUAK_SERVER;
|
||||
if (server !== undefined && !isServerAddress(server)) {
|
||||
ctx.log(
|
||||
"fatal",
|
||||
`Invalid server address "${server}": it must be an http or ` +
|
||||
`https URL with no query or fragment`,
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
const email = process.env.QUAK_EMAIL ?? (await ctx.prompt("Email"));
|
||||
const password =
|
||||
process.env.QUAK_PASSWORD ?? (await ctx.promptSecret("Password"));
|
||||
|
||||
ctx.log("info", "Authenticating...");
|
||||
ctx.stderr.write("Authenticating...\n");
|
||||
try {
|
||||
const client = await ctx.login({
|
||||
email,
|
||||
password,
|
||||
totp: async () => ctx.prompt("TOTP code: "),
|
||||
emailOTP: async () => ctx.prompt("Email verification code: "),
|
||||
apiOptions:
|
||||
server === undefined ? undefined : { apiOrigin: server },
|
||||
});
|
||||
|
||||
saveSession(ctx.sessionDir, client.toJSON());
|
||||
const info = client.whoami();
|
||||
ctx.log("info", `Logged in as ${info.email} (user ${info.userID})`);
|
||||
ctx.log("info", `Session saved to ${sessionPath(ctx)}`);
|
||||
ctx.stderr.write(`Logged in as ${info.email} (user ${info.userID})\n`);
|
||||
ctx.stderr.write(`Session saved to ${sessionPath(ctx)}\n`);
|
||||
} catch (err) {
|
||||
ctx.log(
|
||||
"fatal",
|
||||
`Login failed: ${err instanceof Error ? err.message : err}`,
|
||||
ctx.stderr.write(
|
||||
`Login failed: ${err instanceof Error ? err.message : err}\n`,
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
@@ -218,7 +179,7 @@ export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
|
||||
export const logoutCommand = async (ctx: CliContext): Promise<number> => {
|
||||
const path = sessionPath(ctx);
|
||||
if (!existsSync(path)) {
|
||||
ctx.log("info", "No session found.");
|
||||
ctx.stderr.write("No session found.\n");
|
||||
return 0;
|
||||
}
|
||||
await init();
|
||||
@@ -227,10 +188,7 @@ export const logoutCommand = async (ctx: CliContext): Promise<number> => {
|
||||
try {
|
||||
const client = ctx.loadSession(path);
|
||||
if (client) {
|
||||
cacheDir ??= defaultCacheDirectory(
|
||||
client.whoami().userID,
|
||||
client.apiOrigin,
|
||||
);
|
||||
cacheDir ??= defaultCacheDirectory(client.whoami().userID);
|
||||
await client.logoutOnServer();
|
||||
client.logout();
|
||||
}
|
||||
@@ -239,15 +197,16 @@ export const logoutCommand = async (ctx: CliContext): Promise<number> => {
|
||||
}
|
||||
unlinkSync(path);
|
||||
if (failure === undefined) {
|
||||
ctx.log("info", "Session ended on the server.");
|
||||
ctx.stderr.write("Session ended on the server.\n");
|
||||
} else {
|
||||
ctx.log("fatal", `Could not end the session on the server: ${failure}`);
|
||||
ctx.stderr.write(
|
||||
`Could not end the session on the server: ${failure}\n`,
|
||||
);
|
||||
}
|
||||
ctx.log("info", "Session deleted.");
|
||||
ctx.stderr.write("Session deleted.\n");
|
||||
if (cacheDir !== undefined) {
|
||||
ctx.log(
|
||||
"warning",
|
||||
`Cache directory ${cacheDir} still holds decrypted data; delete it to remove that data.`,
|
||||
ctx.stderr.write(
|
||||
`Cache directory ${cacheDir} still holds decrypted data; delete it to remove that data.\n`,
|
||||
);
|
||||
}
|
||||
return failure === undefined ? 0 : 1;
|
||||
@@ -262,10 +221,9 @@ export const collectionsCommand = async (
|
||||
if (!client) return 3;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Ask the server for the collection list alone and list it in its
|
||||
// order (issue #36 amendment, issues #52 and #210): the pre-library
|
||||
// CLI printed current state in this order, not the albums
|
||||
// projection's newest-first order.
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52): the pre-library CLI printed current state in
|
||||
// this order, not the albums projection's newest-first order.
|
||||
const collections = await freshCollections(lib);
|
||||
|
||||
if (opts.json) {
|
||||
@@ -296,32 +254,26 @@ export const collectionsCommand = async (
|
||||
}
|
||||
};
|
||||
|
||||
// Lists the files of the collection `collectionIDStr` names, or without it every
|
||||
// file in the account, each once (issue #211).
|
||||
export const filesCommand = async (
|
||||
ctx: CliContext,
|
||||
collectionIDStr: string | undefined,
|
||||
opts: { json?: boolean },
|
||||
opts: { collection: string; json?: boolean },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 3;
|
||||
const collectionID =
|
||||
collectionIDStr === undefined
|
||||
? undefined
|
||||
: parseID(ctx, "collection", collectionIDStr);
|
||||
if (collectionIDStr !== undefined && collectionID === undefined) return 1;
|
||||
const collectionID = parseID(ctx, "collection", opts.collection);
|
||||
if (collectionID === undefined) return 1;
|
||||
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Ask the server for current state (issue #36 amendment, issue #52).
|
||||
// Each file prints from its own decrypted metadata (raw title,
|
||||
// microsecond creationTime) via cli-output. The files print in the
|
||||
// order `freshFiles` returns them, not the projection's newest-first
|
||||
// order.
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52). Each file prints from its own decrypted
|
||||
// metadata (raw title, microsecond creationTime) via cli-output, and in
|
||||
// the pre-library CLI's enumeration order, not the projection's
|
||||
// newest-first order.
|
||||
const files = await freshFiles(lib, collectionID);
|
||||
if (!files) {
|
||||
ctx.log("fatal", `Collection ${collectionID} not found`);
|
||||
ctx.stderr.write(`Collection ${collectionID} not found\n`);
|
||||
return 1;
|
||||
}
|
||||
|
||||
@@ -355,42 +307,40 @@ export const getCommand = async (
|
||||
try {
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const file = await freshFile(lib, fileID);
|
||||
if (!file) {
|
||||
ctx.log("fatal", `File ${fileID} not found`);
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
ctx.stderr.write(`File ${fileID} not found\n`);
|
||||
return 1;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await lib.originalOf(file);
|
||||
const result = await photo.original();
|
||||
// Default name is the file's own title, as the pre-library CLI used
|
||||
// (not the editedName-preferring projection title) (issue #52).
|
||||
const outPath = opts.out ?? originalName(file);
|
||||
if (result.videoPath === undefined) {
|
||||
const outPath = opts.out ?? originalName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
ctx.log("info", `${result.bytes} bytes -> ${outPath}`);
|
||||
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
return 0;
|
||||
}
|
||||
// A live photo is written as its image and its video, each named after
|
||||
// the title with its own extension, as Ente's clients name them. With
|
||||
// --out, the image goes there and the video beside it.
|
||||
const imageExt = extname(result.path);
|
||||
const videoExt = extname(result.videoPath);
|
||||
const imageOut = opts.out ?? originalName(file, imageExt);
|
||||
const videoOut =
|
||||
opts.out === undefined
|
||||
? originalName(file, videoExt)
|
||||
: withExtension(opts.out, videoExt);
|
||||
const imageOut =
|
||||
opts.out ?? withExtension(outPath, extname(result.path));
|
||||
const videoOut = withExtension(outPath, extname(result.videoPath));
|
||||
if (imageOut.toLowerCase() === videoOut.toLowerCase()) {
|
||||
ctx.log(
|
||||
"fatal",
|
||||
`File ${fileID} is a live photo, and its video would also be written to ${imageOut}`,
|
||||
ctx.stderr.write(
|
||||
`File ${fileID} is a live photo, and its video would also be written to ${imageOut}\n`,
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
copyFileSync(result.path, imageOut);
|
||||
copyFileSync(result.videoPath, videoOut);
|
||||
ctx.log("info", `${result.bytes} bytes -> ${imageOut}`);
|
||||
ctx.log("info", `${statSync(videoOut).size} bytes -> ${videoOut}`);
|
||||
ctx.stderr.write(
|
||||
`${result.bytes} bytes -> ${imageOut}\n` +
|
||||
`${statSync(videoOut).size} bytes -> ${videoOut}\n`,
|
||||
);
|
||||
return 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
@@ -412,18 +362,19 @@ export const getThumbCommand = async (
|
||||
try {
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const file = await freshFile(lib, fileID);
|
||||
if (!file) {
|
||||
ctx.log("fatal", `File ${fileID} not found`);
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
ctx.stderr.write(`File ${fileID} not found\n`);
|
||||
return 1;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await lib.thumbnailOf(file);
|
||||
const result = await photo.thumbnail();
|
||||
// Default name is thumb_<file's own title>, as the pre-library CLI
|
||||
// used (not the projection title) (issue #52).
|
||||
const outPath = opts.out ?? thumbnailName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
ctx.log("info", `${result.bytes} bytes -> ${outPath}`);
|
||||
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
return 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
@@ -443,16 +394,11 @@ export const backupMetadataCommand = async (
|
||||
// Refresh first so the dump holds current account state, not what the
|
||||
// cache last held; a failed refresh throws.
|
||||
await lib.fresh();
|
||||
const result = await runMetadataBackup(lib, client, dir, {
|
||||
const { failedMLBatches } = await runMetadataBackup(lib, client, dir, {
|
||||
exif: opts.exif || opts.all,
|
||||
onProgress: (msg, level) => ctx.log(level ?? "info", msg),
|
||||
onProgress: (msg) => ctx.stderr.write(msg + "\n"),
|
||||
});
|
||||
if (result.filesWithMLDataError === 0) return 0;
|
||||
ctx.log(
|
||||
"fatal",
|
||||
`quak: the ML data of ${result.filesWithMLDataError} file(s) could not be read; see mlDataError in their JSON in ${dir}`,
|
||||
);
|
||||
return 1;
|
||||
return failedMLBatches > 0 ? 1 : 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
@@ -467,7 +413,7 @@ export const backupCommand = async (
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 3;
|
||||
|
||||
ctx.log("info", "Starting backup...");
|
||||
ctx.stderr.write("Starting backup...\n");
|
||||
// The lock is taken before the library opens and starts its refresh, so a
|
||||
// run refused while another backup of `dir` runs sends no request.
|
||||
let lock: BackupLock;
|
||||
@@ -475,7 +421,7 @@ export const backupCommand = async (
|
||||
lock = await lockBackupDirectory(dir);
|
||||
} catch (err) {
|
||||
if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;
|
||||
ctx.log("fatal", `quak: ${(err as Error).message}`);
|
||||
ctx.stderr.write(`quak: ${(err as Error).message}\n`);
|
||||
return 2;
|
||||
}
|
||||
try {
|
||||
@@ -490,48 +436,45 @@ export const backupCommand = async (
|
||||
refreshIntervalSeconds: Infinity,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
onProgress: logRefresh(ctx.log),
|
||||
});
|
||||
try {
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: dir,
|
||||
lock,
|
||||
verify: opts.verify,
|
||||
onProgress: (msg, level) => ctx.log(level ?? "info", msg),
|
||||
onProgress: (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
},
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(JSON.stringify(result, null, 2) + "\n");
|
||||
} else {
|
||||
ctx.log("info", "\n--- Backup complete ---");
|
||||
ctx.log("info", ` Total files: ${result.totalFiles}`);
|
||||
ctx.log("info", ` Downloaded: ${result.downloaded}`);
|
||||
ctx.log("info", ` Skipped: ${result.skipped}`);
|
||||
ctx.stderr.write("\n--- Backup complete ---\n");
|
||||
ctx.stderr.write(` Total files: ${result.totalFiles}\n`);
|
||||
ctx.stderr.write(` Downloaded: ${result.downloaded}\n`);
|
||||
ctx.stderr.write(` Skipped: ${result.skipped}\n`);
|
||||
if (opts.verify) {
|
||||
ctx.log("info", ` Verified: ${result.verified}`);
|
||||
ctx.log("info", ` Mismatched: ${result.mismatched}`);
|
||||
ctx.log("info", ` Unchecked: ${result.unchecked}`);
|
||||
ctx.stderr.write(` Verified: ${result.verified}\n`);
|
||||
ctx.stderr.write(` Mismatched: ${result.mismatched}\n`);
|
||||
ctx.stderr.write(` Unchecked: ${result.unchecked}\n`);
|
||||
}
|
||||
ctx.log("info", ` Failed: ${result.failed}`);
|
||||
ctx.stderr.write(` Failed: ${result.failed}\n`);
|
||||
if (result.errors.length > 0) {
|
||||
ctx.log("error", "\nFailed files:");
|
||||
ctx.stderr.write("\nFailed files:\n");
|
||||
for (const e of result.errors) {
|
||||
ctx.log(
|
||||
"error",
|
||||
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}`,
|
||||
ctx.stderr.write(
|
||||
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const albums =
|
||||
result.undecryptableAlbums.length + result.failedAlbums.length;
|
||||
if (result.failed === 0 && albums === 0) return 0;
|
||||
ctx.log(
|
||||
"fatal",
|
||||
`quak: ${result.failed} file(s) and ${albums} album(s) failed; see ${join(dir, "failures.json")} and the FAILED lines`,
|
||||
);
|
||||
return 1;
|
||||
return result.failed > 0 ||
|
||||
result.undecryptableAlbums.length > 0 ||
|
||||
result.failedAlbums.length > 0
|
||||
? 1
|
||||
: 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
@@ -552,19 +495,18 @@ export const listMissingThumbnailsCommand = async (
|
||||
// Refresh first so files added since the cache was written are
|
||||
// checked; a failed refresh throws.
|
||||
await lib.fresh();
|
||||
const missing = await listMissingThumbnails(lib, client, (msg, level) =>
|
||||
ctx.log(level ?? "info", msg),
|
||||
);
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(JSON.stringify(missing, null, 2) + "\n");
|
||||
} else {
|
||||
if (missing.length === 0) {
|
||||
ctx.log("info", "No missing thumbnails found.");
|
||||
ctx.stderr.write("No missing thumbnails found.\n");
|
||||
} else {
|
||||
ctx.log(
|
||||
"info",
|
||||
`\n${missing.length} file(s) with missing thumbnails:`,
|
||||
ctx.stderr.write(
|
||||
`\n${missing.length} file(s) with missing thumbnails:\n`,
|
||||
);
|
||||
for (const m of missing) {
|
||||
ctx.stdout.write(
|
||||
@@ -595,37 +537,31 @@ export const fixMissingThumbnailsCommand = async (
|
||||
}
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
let found: Map<number, { file: EnteFile; collection: Collection }>;
|
||||
if (fileIDs.length > 0) {
|
||||
found = await freshFilesByID(lib, fileIDs);
|
||||
} else {
|
||||
// Refresh first so files added since the cache was written are
|
||||
// checked; a failed refresh throws.
|
||||
await lib.fresh();
|
||||
ctx.log("info", "Scanning for missing thumbnails...");
|
||||
const missing = await listMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
(msg, level) => ctx.log(level ?? "info", msg),
|
||||
);
|
||||
// Refresh first so files added since the cache was written are found;
|
||||
// a failed refresh throws.
|
||||
await lib.fresh();
|
||||
if (fileIDs.length === 0) {
|
||||
ctx.stderr.write("Scanning for missing thumbnails...\n");
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
});
|
||||
fileIDs = missing.map((m) => m.fileID);
|
||||
if (fileIDs.length === 0) {
|
||||
ctx.log("info", "No missing thumbnails found.");
|
||||
ctx.stderr.write("No missing thumbnails found.\n");
|
||||
return 0;
|
||||
}
|
||||
ctx.log("info", `Found ${fileIDs.length} file(s) to fix.`);
|
||||
found = cachedFilesByID(lib, fileIDs);
|
||||
ctx.stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
|
||||
}
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
fileIDs,
|
||||
found,
|
||||
(msg, level) => ctx.log(level ?? "info", msg),
|
||||
(msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
},
|
||||
);
|
||||
|
||||
const failed = results.filter((r) => r.status === "failed").length;
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(JSON.stringify(results, null, 2) + "\n");
|
||||
} else {
|
||||
@@ -633,30 +569,30 @@ export const fixMissingThumbnailsCommand = async (
|
||||
const skipped = results.filter(
|
||||
(r) => r.status === "skipped",
|
||||
).length;
|
||||
ctx.log("info", `\n--- Done ---`);
|
||||
ctx.log("info", ` Fixed: ${fixed}`);
|
||||
ctx.log("info", ` Skipped: ${skipped}`);
|
||||
ctx.log("info", ` Failed: ${failed}`);
|
||||
const failed = results.filter((r) => r.status === "failed").length;
|
||||
ctx.stderr.write(`\n--- Done ---\n`);
|
||||
ctx.stderr.write(` Fixed: ${fixed}\n`);
|
||||
ctx.stderr.write(` Skipped: ${skipped}\n`);
|
||||
ctx.stderr.write(` Failed: ${failed}\n`);
|
||||
if (skipped > 0) {
|
||||
ctx.log("info", "\nSkipped:");
|
||||
ctx.stderr.write("\nSkipped:\n");
|
||||
for (const r of results.filter((r) => r.status === "skipped")) {
|
||||
ctx.log("info", ` ${r.fileID}\t${r.title}\t${r.reason}`);
|
||||
ctx.stderr.write(
|
||||
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
if (failed > 0) {
|
||||
ctx.log("error", "\nFailed files:");
|
||||
ctx.stderr.write("\nFailed files:\n");
|
||||
for (const r of results.filter((r) => r.status === "failed")) {
|
||||
ctx.log("error", ` ${r.fileID}\t${r.title}\t${r.reason}`);
|
||||
ctx.stderr.write(
|
||||
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (failed === 0) return 0;
|
||||
ctx.log(
|
||||
"fatal",
|
||||
`quak: ${failed} file(s) failed; see the "Failed files:" list, or the JSON with --json`,
|
||||
);
|
||||
return 1;
|
||||
return results.some((r) => r.status === "failed") ? 1 : 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
|
||||
-102
@@ -1,102 +0,0 @@
|
||||
// The CLI's log level, and the one function the commands write stderr lines
|
||||
// through.
|
||||
//
|
||||
// Every line quak logs goes to stderr, at one of six levels, from the
|
||||
// quietest: fatal, error, warning, info, debug, trace. A run prints the lines
|
||||
// at its own level and at every quieter one. Its level is info unless `-v` or
|
||||
// `-q` asks for another (see "What quak prints" in the README). A command that
|
||||
// runs and exits non-zero logs a fatal line saying why, so even `-q`, which
|
||||
// prints only fatal lines, never leaves such an exit unexplained. An error
|
||||
// line reports something that failed, such as a file or an album, and a
|
||||
// warning something that did not fail but needs the user's attention. stdout
|
||||
// carries a command's output and the prompts of `login`, and is not a log.
|
||||
// Two things on stderr are not log lines: the `error:` message for a
|
||||
// command line that quak cannot parse, and the usage printed when no command
|
||||
// is given.
|
||||
//
|
||||
// The library itself writes no lines: its layers report through callbacks, and
|
||||
// the CLI logs what they report. `logRequests` gives the API client options
|
||||
// that log each request and each retry, and `logRefresh` the library's
|
||||
// `onProgress` that logs what a refresh and an ML data fetch report.
|
||||
|
||||
import type { ApiClientOptions } from "./api/client.js";
|
||||
import { describeError } from "./errors.js";
|
||||
import type { RefreshEvent } from "./library/index.js";
|
||||
import type { RetryOptions } from "./retry.js";
|
||||
|
||||
const LEVELS = ["fatal", "error", "warning", "info", "debug", "trace"] as const;
|
||||
|
||||
export type LogLevel = (typeof LEVELS)[number];
|
||||
|
||||
// Writes `line` on stderr when the run's level prints lines at `level`.
|
||||
export type Log = (level: LogLevel, line: string) => void;
|
||||
|
||||
// The level asked for by `-v` given `verbose` times, or by `-q`: info without
|
||||
// either, debug for `-v`, trace for `-vv` and more, fatal for `-q`.
|
||||
// `bin/quak.ts` refuses `-q` and `-v` together.
|
||||
export const logLevel = (verbose: number, quiet: boolean): LogLevel => {
|
||||
if (quiet) return "fatal";
|
||||
if (verbose === 0) return "info";
|
||||
return verbose === 1 ? "debug" : "trace";
|
||||
};
|
||||
|
||||
// The log that writes each line at `level` or quieter to `stderr`, ending it
|
||||
// with a newline.
|
||||
export const stderrLog =
|
||||
(stderr: { write(text: string): unknown }, level: LogLevel): Log =>
|
||||
(lineLevel, line) => {
|
||||
if (LEVELS.indexOf(lineLevel) <= LEVELS.indexOf(level)) {
|
||||
stderr.write(line + "\n");
|
||||
}
|
||||
};
|
||||
|
||||
// API client options that log each request at trace, with the status of its
|
||||
// response or the error it failed with, and each retry at debug, with why and
|
||||
// how long it waits. `retry` is the retry policy, the default when not given.
|
||||
export const logRequests = (
|
||||
log: Log,
|
||||
retry?: RetryOptions,
|
||||
): ApiClientOptions => ({
|
||||
retry: {
|
||||
...retry,
|
||||
onRetry: (err, attempt, delayMs) =>
|
||||
log(
|
||||
"debug",
|
||||
`Attempt ${attempt} failed, retrying in ${Math.round(delayMs)} ms: ${describeError(err)}`,
|
||||
),
|
||||
},
|
||||
onRequest: ({ method, url, status, error }) =>
|
||||
log(
|
||||
"trace",
|
||||
`${method} ${url}: ${error === undefined ? `HTTP ${status}` : describeError(error)}`,
|
||||
),
|
||||
});
|
||||
|
||||
// How a log line names each operation the library reports the start and end
|
||||
// of.
|
||||
const OPERATION_NAMES: Record<RefreshEvent["operation"], string> = {
|
||||
refresh: "Refresh",
|
||||
fetchMLData: "ML data fetch",
|
||||
precacheThumbnails: "Thumbnail precache",
|
||||
precacheOriginals: "Original precache",
|
||||
};
|
||||
|
||||
// A library `onProgress` that logs, at info, each collection whose files a
|
||||
// refresh has fetched, and at debug the start and end of each refresh and ML
|
||||
// data fetch, with the error of one that failed.
|
||||
export const logRefresh =
|
||||
(log: Log) =>
|
||||
(event: RefreshEvent): void => {
|
||||
if (event.status === "progress") {
|
||||
log(
|
||||
"info",
|
||||
`Fetched the files of ${event.fetched} of ${event.total} collections`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
const error = event.error === undefined ? "" : `: ${event.error}`;
|
||||
log(
|
||||
"debug",
|
||||
`${OPERATION_NAMES[event.operation]} ${event.status}${error}`,
|
||||
);
|
||||
};
|
||||
+7
-9
@@ -9,7 +9,7 @@
|
||||
// `metadata.title`, and issue #52 requires that output stay byte-identical, so
|
||||
// the commands shape their output from the raw `EnteFile` through here.
|
||||
|
||||
import { defaultFileName } from "./filename.js";
|
||||
import { sanitizeFileName } from "./filename.js";
|
||||
import type { EnteFile, FileType, Microseconds } from "./model/types.js";
|
||||
|
||||
// One row of `quak files --json`.
|
||||
@@ -33,13 +33,11 @@ export const fileListRow = (file: EnteFile): FileListRow => ({
|
||||
export const fileListLine = (file: EnteFile): string =>
|
||||
`${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
|
||||
|
||||
// Default output path for `quak get` when `--out` is not given, made from the
|
||||
// title (see `defaultFileName`), with `ext` in place of its extension for a
|
||||
// live photo's image or video; `--out` is the user's and is used as is.
|
||||
export const originalName = (file: EnteFile, ext?: string): string =>
|
||||
defaultFileName(file.metadata.title, file.id, { ext });
|
||||
// Default output path for `quak get` when `--out` is not given. The title comes
|
||||
// from the server, so it is sanitized; `--out` is the user's and is used as is.
|
||||
export const originalName = (file: EnteFile): string =>
|
||||
sanitizeFileName(file.metadata.title, `file-${file.id}`);
|
||||
|
||||
// Default output path for `quak get-thumb` when `--out` is not given: `thumb_`
|
||||
// and the title.
|
||||
// Default output path for `quak get-thumb` when `--out` is not given.
|
||||
export const thumbnailName = (file: EnteFile): string =>
|
||||
defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
|
||||
`thumb_${originalName(file)}`;
|
||||
+39
-127
@@ -1,150 +1,62 @@
|
||||
// How the CLI's read commands obtain current data.
|
||||
//
|
||||
// `collections`, `files`, `get`, and `get-thumb` must answer for
|
||||
// `collections`, `files --collection`, `get`, and `get-thumb` must answer for
|
||||
// the account's state at the moment the command runs, not for whatever the
|
||||
// local cache last happened to hold (owner amendment, issue #36). `get`,
|
||||
// `get-thumb` and `files` without a collection ID therefore force a server
|
||||
// round-trip through `Library.fresh()` and only then read, so a change made
|
||||
// elsewhere is visible. `collections` asks the library for the collection list
|
||||
// alone through `Library.freshCollections()` (issue #210): a refresh would also
|
||||
// fetch the files of every collection, which on a large account takes long and
|
||||
// which `collections` does not print. `files` with a collection ID reads that
|
||||
// list too, then that one collection's files through `Library.freshFiles()`
|
||||
// (issue #217). On an empty cache, `get` and `get-thumb` read that list, then
|
||||
// one collection's files at a time, until they find the file (issue #222).
|
||||
// `helper fix-missing-thumbnails --file` does the same until it finds every
|
||||
// file it names (issue #223).
|
||||
// local cache last happened to hold (owner amendment, issue #36). Each helper
|
||||
// therefore forces a server round-trip through `Library.fresh()` and only then
|
||||
// reads — so a collection, file, or metadata change made elsewhere is visible.
|
||||
//
|
||||
// `collections` and `files` with a collection ID list in the server's order.
|
||||
// `files` without one lists in the library's own enumeration order
|
||||
// (`listFiles()`). Neither command uses the `albums`/`photos` projection's
|
||||
// newest-first order, which re-sorts the rows. The field values still come
|
||||
// from each record's raw metadata via `cli-output.ts`. `files` without a
|
||||
// collection ID lists every file in the account once (issue #211): the library
|
||||
// keeps a record of a file for each collection that holds it, and only the
|
||||
// first collection's record is listed.
|
||||
// `collections` and `files` also list in the library's own enumeration order —
|
||||
// `listCollections()`/`listFiles()`, the order the pre-library CLI printed —
|
||||
// rather than the `albums`/`photos` projection's newest-first order, which
|
||||
// re-sorts the rows. The field values still come from each record's raw
|
||||
// metadata via `cli-output.ts`.
|
||||
|
||||
import type { Collection, EnteFile } from "./model/types.js";
|
||||
import type { Photo, PhotosAPI } from "./library/index.js";
|
||||
|
||||
// The slice of `Library` these helpers read. `Library` satisfies it
|
||||
// structurally; a test can drive them with a stand-in that records each fresh
|
||||
// read and serves records in a known enumeration order.
|
||||
// structurally; a test can drive them with a stand-in that records the
|
||||
// `fresh()` call and serves records in a known enumeration order.
|
||||
export interface FreshReadLibrary {
|
||||
fresh(): Promise<unknown>;
|
||||
freshCollections(): Promise<Collection[]>;
|
||||
freshFiles(collection: Collection): Promise<EnteFile[]>;
|
||||
listCollections(): Collection[];
|
||||
getCollection(id: number): Collection | undefined;
|
||||
listFiles(collectionID: number): EnteFile[];
|
||||
getFileByID(fileID: number): EnteFile | undefined;
|
||||
photos: Pick<PhotosAPI, "byID">;
|
||||
}
|
||||
|
||||
// Every live collection, current as of one collections request, in the
|
||||
// server's order.
|
||||
export const freshCollections = (
|
||||
// Every live collection, current as of a forced refresh, in enumeration order.
|
||||
export const freshCollections = async (
|
||||
lib: FreshReadLibrary,
|
||||
): Promise<Collection[]> => lib.freshCollections();
|
||||
|
||||
// The files of one collection, as the server holds them now, in the server's
|
||||
// order. `undefined` (not an empty list) when the collection does not exist, so
|
||||
// the caller can tell "no such collection" from "an empty collection". Without
|
||||
// a collection ID, every file in the account as of a forced refresh, each once,
|
||||
// as held by the first collection that holds it.
|
||||
export const freshFiles = async (
|
||||
lib: FreshReadLibrary,
|
||||
collectionID?: number,
|
||||
): Promise<EnteFile[] | undefined> => {
|
||||
if (collectionID !== undefined) {
|
||||
const collections = await lib.freshCollections();
|
||||
const collection = collections.find((c) => c.id === collectionID);
|
||||
return collection ? lib.freshFiles(collection) : undefined;
|
||||
}
|
||||
): Promise<Collection[]> => {
|
||||
await lib.fresh();
|
||||
const listed = new Set<number>();
|
||||
const files: EnteFile[] = [];
|
||||
for (const collection of lib.listCollections()) {
|
||||
for (const file of lib.listFiles(collection.id)) {
|
||||
if (listed.has(file.id)) continue;
|
||||
listed.add(file.id);
|
||||
files.push(file);
|
||||
}
|
||||
}
|
||||
return files;
|
||||
return lib.listCollections();
|
||||
};
|
||||
|
||||
// One file as the server holds it now, or `undefined` when no collection holds
|
||||
// it. When the cache holds the account's collections, it refreshes and looks
|
||||
// the file up in the cache. On an empty cache, it reads the collection list,
|
||||
// then one collection's files at a time, and stops at the first collection
|
||||
// that holds the file. That search stores nothing.
|
||||
// The files of one collection, current as of a forced refresh, in enumeration
|
||||
// order. `undefined` (not an empty list) when the collection does not exist, so
|
||||
// the caller can tell "no such collection" from "an empty collection".
|
||||
export const freshFiles = async (
|
||||
lib: FreshReadLibrary,
|
||||
collectionID: number,
|
||||
): Promise<EnteFile[] | undefined> => {
|
||||
await lib.fresh();
|
||||
if (!lib.getCollection(collectionID)) return undefined;
|
||||
return lib.listFiles(collectionID);
|
||||
};
|
||||
|
||||
// One file, current as of a forced refresh, resolved to both its content
|
||||
// handle (`Photo`, for fetching bytes) and its raw record (`EnteFile`, for the
|
||||
// default output name and field values). `undefined` when the file is unknown.
|
||||
export const freshFile = async (
|
||||
lib: FreshReadLibrary,
|
||||
fileID: number,
|
||||
): Promise<EnteFile | undefined> => {
|
||||
// Only a refresh stores collections, so a cache holding any holds them all.
|
||||
if (lib.listCollections().length > 0) {
|
||||
await lib.fresh();
|
||||
return lib.getFileByID(fileID);
|
||||
}
|
||||
for (const collection of await lib.freshCollections()) {
|
||||
const files = await lib.freshFiles(collection);
|
||||
const file = files.find((f) => f.id === fileID);
|
||||
if (file) return file;
|
||||
}
|
||||
return undefined;
|
||||
};
|
||||
|
||||
// The most recently updated collection first, and the higher ID first on a
|
||||
// tie. `Library.albums.list()` lists albums in this order.
|
||||
const byNewestCollection = (a: Collection, b: Collection): number =>
|
||||
b.updationTime - a.updationTime || b.id - a.id;
|
||||
|
||||
// Each file named by `fileIDs` that the cache holds, mapped to its record and
|
||||
// the collection it was found in. Collections are searched in
|
||||
// `byNewestCollection` order, and a file is found in the first that holds it.
|
||||
// So `helper fix-missing-thumbnails` names a file after the same album as
|
||||
// `helper list-missing-thumbnails`. It reads the cache alone, so the caller
|
||||
// refreshes first.
|
||||
export const cachedFilesByID = (
|
||||
lib: FreshReadLibrary,
|
||||
fileIDs: number[],
|
||||
): Map<number, { file: EnteFile; collection: Collection }> => {
|
||||
const wanted = new Set(fileIDs);
|
||||
const found = new Map<number, { file: EnteFile; collection: Collection }>();
|
||||
const collections = [...lib.listCollections()].sort(byNewestCollection);
|
||||
for (const collection of collections) {
|
||||
for (const file of lib.listFiles(collection.id)) {
|
||||
if (wanted.has(file.id) && !found.has(file.id)) {
|
||||
found.set(file.id, { file, collection });
|
||||
}
|
||||
}
|
||||
}
|
||||
return found;
|
||||
};
|
||||
|
||||
// Each file named by `fileIDs` as the server holds it now, mapped to its record
|
||||
// and collection as `cachedFilesByID` maps it. When the cache holds the
|
||||
// account's collections, it refreshes and looks the files up in the cache. On
|
||||
// an empty cache, it reads the collection list, then one collection's files at
|
||||
// a time, and stops once it has found every file. That search stores nothing.
|
||||
export const freshFilesByID = async (
|
||||
lib: FreshReadLibrary,
|
||||
fileIDs: number[],
|
||||
): Promise<Map<number, { file: EnteFile; collection: Collection }>> => {
|
||||
// Only a refresh stores collections, so a cache holding any holds them all.
|
||||
if (lib.listCollections().length > 0) {
|
||||
await lib.fresh();
|
||||
return cachedFilesByID(lib, fileIDs);
|
||||
}
|
||||
const wanted = new Set(fileIDs);
|
||||
const found = new Map<number, { file: EnteFile; collection: Collection }>();
|
||||
const collections = await lib.freshCollections();
|
||||
for (const collection of [...collections].sort(byNewestCollection)) {
|
||||
if (found.size === wanted.size) break;
|
||||
for (const file of await lib.freshFiles(collection)) {
|
||||
if (wanted.has(file.id) && !found.has(file.id)) {
|
||||
found.set(file.id, { file, collection });
|
||||
}
|
||||
}
|
||||
}
|
||||
return found;
|
||||
): Promise<{ photo: Photo; file: EnteFile } | undefined> => {
|
||||
await lib.fresh();
|
||||
const photo = lib.photos.byID({ fileID });
|
||||
const file = lib.getFileByID(fileID);
|
||||
if (!photo || !file) return undefined;
|
||||
return { photo, file };
|
||||
};
|
||||
+9
-12
@@ -2,23 +2,21 @@
|
||||
|
||||
import type { Writable } from "node:stream";
|
||||
import { ApiError } from "./api/client.js";
|
||||
import type { Log } from "./cli-log.js";
|
||||
import { describeError } from "./errors.js";
|
||||
|
||||
// Run a command and exit with its code once stdout/stderr have drained.
|
||||
// Exiting before the drain can truncate piped output, and the library can keep
|
||||
// the event loop alive after a command returns, so a plain return could hang.
|
||||
// An error the command throws is logged as one fatal `quak: MESSAGE` line,
|
||||
// without the stack trace, and exits 1. MESSAGE is the error as
|
||||
// `describeError` gives it, so it carries the reason Node's `fetch` puts on
|
||||
// `cause`. A 401 from the server means it no longer accepts the saved session:
|
||||
// that logs one fatal line saying to log in again and exits 3, as a missing or
|
||||
// corrupt session file does.
|
||||
// An error the command throws is printed as one `quak: MESSAGE` line, without
|
||||
// the stack trace, and exits 1. MESSAGE is the error as `describeError` gives
|
||||
// it, so it carries the reason Node's `fetch` puts on `cause`. A 401 from the
|
||||
// server means it no longer accepts the saved session: that prints one line
|
||||
// saying to log in again and exits 3, as a missing or corrupt session file
|
||||
// does.
|
||||
export const run = async (
|
||||
command: Promise<number>,
|
||||
stdout: Writable,
|
||||
stderr: Writable,
|
||||
log: Log,
|
||||
exit: (code: number) => void,
|
||||
): Promise<void> => {
|
||||
let code: number;
|
||||
@@ -26,13 +24,12 @@ export const run = async (
|
||||
code = await command;
|
||||
} catch (err) {
|
||||
if (err instanceof ApiError && err.status === 401) {
|
||||
log(
|
||||
"fatal",
|
||||
`quak: the saved session is no longer valid; run "quak login"`,
|
||||
stderr.write(
|
||||
`quak: the saved session is no longer valid; run "quak login"\n`,
|
||||
);
|
||||
code = 3;
|
||||
} else {
|
||||
log("fatal", `quak: ${describeError(err)}`);
|
||||
stderr.write(`quak: ${describeError(err)}\n`);
|
||||
code = 1;
|
||||
}
|
||||
}
|
||||
|
||||
+5
-45
@@ -40,9 +40,6 @@ export interface ClientSnapshot {
|
||||
masterKey: string;
|
||||
secretKey: string;
|
||||
publicKey: string;
|
||||
// The `apiOrigin` the client was given, so a restored client talks to
|
||||
// the server that issued its token. Absent means the default server.
|
||||
apiOrigin?: string;
|
||||
}
|
||||
|
||||
// The result of a resumable enumeration. Live decrypted records and deleted
|
||||
@@ -69,19 +66,7 @@ export interface FilesPage {
|
||||
const errorMessage = (err: unknown): string =>
|
||||
err instanceof Error ? err.message : String(err);
|
||||
|
||||
// Whether `text` can be an Ente server's address: an http or https URL, which
|
||||
// may have a path, and no query or fragment, which request URLs could not
|
||||
// carry. `quak login --server` and `Client.fromJSON` both check with it.
|
||||
export const isServerAddress = (text: string): boolean =>
|
||||
URL.canParse(text) &&
|
||||
["http:", "https:"].includes(new URL(text).protocol) &&
|
||||
!/[?#]/.test(text);
|
||||
|
||||
export class Client {
|
||||
// The `apiOrigin` the client was given, or undefined when it was given
|
||||
// none, which means Ente's hosted service. The library keeps a cache per
|
||||
// server as well as per user ID.
|
||||
readonly apiOrigin: string | undefined;
|
||||
private readonly api: ApiClient;
|
||||
private readonly email: string;
|
||||
private readonly userID: number;
|
||||
@@ -97,7 +82,6 @@ export class Client {
|
||||
masterKey: Uint8Array,
|
||||
secretKey: Uint8Array,
|
||||
publicKey: Uint8Array,
|
||||
apiOrigin: string | undefined,
|
||||
) {
|
||||
this.api = api;
|
||||
this.email = email;
|
||||
@@ -105,7 +89,6 @@ export class Client {
|
||||
this.masterKey = masterKey;
|
||||
this.secretKey = secretKey;
|
||||
this.publicKey = publicKey;
|
||||
this.apiOrigin = apiOrigin;
|
||||
}
|
||||
|
||||
static async login(opts: LoginOptions): Promise<Client> {
|
||||
@@ -147,15 +130,12 @@ export class Client {
|
||||
unwrapped.masterKey,
|
||||
unwrapped.secretKey,
|
||||
unwrapped.publicKey,
|
||||
opts.apiOptions?.apiOrigin,
|
||||
);
|
||||
}
|
||||
|
||||
// Restore a client from a `toJSON()` snapshot. The snapshot usually comes
|
||||
// straight from `JSON.parse` of a file on disk, so every field is checked
|
||||
// before use; a bad one throws an error naming it. The snapshot's
|
||||
// `apiOrigin`, when it has one, is used instead of `apiOptions.apiOrigin`.
|
||||
// Needs `init()` first.
|
||||
// before use; a bad one throws an error naming it. Needs `init()` first.
|
||||
static fromJSON(snapshot: unknown, apiOptions?: ApiClientOptions): Client {
|
||||
const invalid = (field: string, problem: string): Error =>
|
||||
new Error(`Invalid session data: ${field} ${problem}`);
|
||||
@@ -172,17 +152,6 @@ export class Client {
|
||||
if (!Number.isInteger(s.userID)) {
|
||||
throw invalid("userID", "must be an integer");
|
||||
}
|
||||
if (
|
||||
s.apiOrigin !== undefined &&
|
||||
(typeof s.apiOrigin !== "string" || !isServerAddress(s.apiOrigin))
|
||||
) {
|
||||
throw invalid(
|
||||
"apiOrigin",
|
||||
"must be an http or https URL with no query or fragment",
|
||||
);
|
||||
}
|
||||
const apiOrigin =
|
||||
(s.apiOrigin as string | undefined) ?? apiOptions?.apiOrigin;
|
||||
const key = (field: string): Uint8Array => {
|
||||
const value = s[field];
|
||||
if (typeof value !== "string") {
|
||||
@@ -206,7 +175,6 @@ export class Client {
|
||||
|
||||
const api = new ApiClient({
|
||||
...apiOptions,
|
||||
apiOrigin,
|
||||
authToken: s.token as string,
|
||||
});
|
||||
return new Client(
|
||||
@@ -216,7 +184,6 @@ export class Client {
|
||||
key("masterKey"),
|
||||
key("secretKey"),
|
||||
key("publicKey"),
|
||||
apiOrigin,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -255,9 +222,6 @@ export class Client {
|
||||
masterKey: toBase64(this.masterKey),
|
||||
secretKey: toBase64(this.secretKey),
|
||||
publicKey: toBase64(this.publicKey),
|
||||
...(this.apiOrigin === undefined
|
||||
? {}
|
||||
: { apiOrigin: this.apiOrigin }),
|
||||
};
|
||||
}
|
||||
|
||||
@@ -409,17 +373,13 @@ export class Client {
|
||||
}
|
||||
|
||||
// Fetch machine-learning data (face detections + CLIP embeddings) for up
|
||||
// to a batch of files, each decrypted with its own key, and for each file
|
||||
// whose ML data did not decrypt, gunzip or parse, why not. One request;
|
||||
// the library batches at `MLDATA_BATCH_SIZE` and schedules each batch
|
||||
// through its metadata request pool.
|
||||
// to a batch of files, each decrypted with its own key. One request; the
|
||||
// library batches at `MLDATA_BATCH_SIZE` and schedules each batch through
|
||||
// its metadata request pool.
|
||||
async fetchMLData(args: {
|
||||
fileIDs: number[];
|
||||
fileKeys: Map<number, Uint8Array>;
|
||||
}): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}> {
|
||||
}): Promise<Map<number, MLData>> {
|
||||
this.assertLoggedIn();
|
||||
return fetchMLDataBatch(this.api, args.fileIDs, args.fileKeys);
|
||||
}
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -1,57 +0,0 @@
|
||||
// Keywords for a file from its CLIP embedding.
|
||||
//
|
||||
// Ente stores no keywords. A file's ML data holds a CLIP embedding, the 512
|
||||
// numbers Ente's image model gave the photo, and Ente's photos app sorts files
|
||||
// into the categories of its Discover section with it: a file falls in a
|
||||
// category when the cosine similarity of its CLIP embedding and the text
|
||||
// embedding of the category's query is at least the category's `minScore`,
|
||||
// unless the file is a video and the category leaves out videos. A file's
|
||||
// keywords are the titles of the categories it falls in. The categories and
|
||||
// their text embeddings are in `clip-keyword-vectors.ts`.
|
||||
|
||||
import { DISCOVER_CATEGORIES } from "./clip-keyword-vectors.js";
|
||||
import { clipEmbedding } from "./library/mldata.js";
|
||||
import type { MLData } from "./mldata-fetch.js";
|
||||
|
||||
// How many numbers a CLIP embedding from Ente's image model has.
|
||||
const EMBEDDING_LENGTH = 512;
|
||||
|
||||
const norm = (vector: number[]): number => Math.hypot(...vector);
|
||||
|
||||
const cosineSimilarity = (a: number[], b: number[]): number => {
|
||||
let dot = 0;
|
||||
a.forEach((value, i) => {
|
||||
dot += value * (b[i] ?? 0);
|
||||
});
|
||||
return dot / (norm(a) * norm(b));
|
||||
};
|
||||
|
||||
// The titles of the Discover categories a file falls in, highest score first,
|
||||
// or undefined when its ML data has no CLIP embedding of 512 finite numbers
|
||||
// that are not all zero. Tests pass their own `categories`.
|
||||
export const clipKeywords = (
|
||||
mlData: MLData,
|
||||
isVideo: boolean,
|
||||
categories = DISCOVER_CATEGORIES,
|
||||
): string[] | undefined => {
|
||||
const embedding = clipEmbedding(mlData);
|
||||
if (
|
||||
!embedding ||
|
||||
embedding.length !== EMBEDDING_LENGTH ||
|
||||
norm(embedding) === 0
|
||||
) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const matches: { title: string; score: number }[] = [];
|
||||
for (const category of categories) {
|
||||
if (isVideo && !category.showVideo) continue;
|
||||
const score = cosineSimilarity(embedding, category.textEmbedding);
|
||||
if (score >= category.minScore) {
|
||||
matches.push({ title: category.title, score });
|
||||
}
|
||||
}
|
||||
return matches
|
||||
.sort((a, b) => b.score - a.score)
|
||||
.map((match) => match.title);
|
||||
};
|
||||
+17
-38
@@ -16,7 +16,7 @@ import {
|
||||
streamTagFinal,
|
||||
} from "../crypto/index.js";
|
||||
import { TruncatedStreamError } from "../errors.js";
|
||||
import { defaultFileName, safeExtension, withExtension } from "../filename.js";
|
||||
import { safeExtension, sanitizeFileName, withExtension } from "../filename.js";
|
||||
import { withRetry } from "../retry.js";
|
||||
import type { ApiClient } from "../api/client.js";
|
||||
import type { EnteFile } from "../model/types.js";
|
||||
@@ -199,13 +199,11 @@ const isRunning = (pid: number): boolean => {
|
||||
};
|
||||
|
||||
// Delete the temp files a killed process left in `dir`: the writer's
|
||||
// `.quak-<pid>-<random>.tmp`, the backup copy's
|
||||
// `.quak-backup-<name>-<pid>-<random>.tmp`, and the image and video of a live
|
||||
// photo downloaded to replace a stored one, `.quak-<pid>-<random>.<ext>` (see
|
||||
// `replaceOriginal`). Only files whose process is no longer running are
|
||||
// removed, so another process writing into the same directory keeps its own. A
|
||||
// reused process ID can only keep a leftover a while longer, never remove a
|
||||
// live one.
|
||||
// `.quak-<pid>-<random>.tmp` and the backup copy's
|
||||
// `.quak-backup-<name>-<pid>-<random>.tmp`. Only files whose process is no
|
||||
// longer running are removed, so another process writing into the same
|
||||
// directory keeps its own. A reused process ID can only keep a leftover a while
|
||||
// longer, never remove a live one.
|
||||
export const removeLeftoverTempFiles = (dir: string): void => {
|
||||
let names: string[];
|
||||
try {
|
||||
@@ -214,9 +212,7 @@ export const removeLeftoverTempFiles = (dir: string): void => {
|
||||
return;
|
||||
}
|
||||
for (const name of names) {
|
||||
const match = /^\.quak-(?:.*-)?(\d+)-[0-9a-z]*\.[0-9A-Za-z]+$/.exec(
|
||||
name,
|
||||
);
|
||||
const match = /^\.quak-(?:.*-)?(\d+)-[0-9a-z]*\.tmp$/.exec(name);
|
||||
if (match && !isRunning(Number(match[1]))) {
|
||||
rmSync(join(dir, name), { force: true });
|
||||
}
|
||||
@@ -365,14 +361,10 @@ const openPart = async (
|
||||
// name starts with `image` as the image and the one whose name starts with
|
||||
// `video` as the video, and refuses a ZIP holding a second of either. It is
|
||||
// written unpacked: each part is named `destination` with the extension
|
||||
// replaced by its own entry's, and the two must differ ignoring case. When
|
||||
// `defaultName` is set, `destination` is the name `defaultFileName` made from
|
||||
// the title rather than a path the caller gave, and each part is named as
|
||||
// `defaultFileName` names it with its own extension, so that the name is cut
|
||||
// short to fit with that extension. When the file records a hash,
|
||||
// `<imageHash>:<videoHash>` must match it, each over that part's own bytes.
|
||||
// Only then are the image, then the video, renamed into place; on any failure
|
||||
// neither is stored.
|
||||
// replaced by its own entry's, and the two must differ ignoring case. When the
|
||||
// file records a hash, `<imageHash>:<videoHash>` must match it, each over that
|
||||
// part's own bytes. Only then are the image, then the video, renamed into
|
||||
// place; on any failure neither is stored.
|
||||
//
|
||||
// The ZIP is chosen by its uploader and may expand enormously, so each part is
|
||||
// written as it decompresses and never held, and the ZIP is refused once the
|
||||
@@ -392,7 +384,6 @@ const decryptLivePhoto = async (
|
||||
key: Uint8Array,
|
||||
onProgress: ProgressCallback | undefined,
|
||||
file: EnteFile,
|
||||
defaultName: boolean,
|
||||
): Promise<DownloadResult> => {
|
||||
const sliceSize = 4096;
|
||||
const dir = dirname(destination);
|
||||
@@ -491,12 +482,8 @@ const decryptLivePhoto = async (
|
||||
`live photo's image and video have the same extension, ${video.ext}`,
|
||||
);
|
||||
}
|
||||
const partPath = (ext: string): string =>
|
||||
defaultName
|
||||
? defaultFileName(file.metadata.title, file.id, { ext })
|
||||
: withExtension(destination, ext);
|
||||
const path = partPath(image.ext);
|
||||
const videoPath = partPath(video.ext);
|
||||
const path = withExtension(destination, image.ext);
|
||||
const videoPath = withExtension(destination, video.ext);
|
||||
for (const part of parts) {
|
||||
await part.handle.sync();
|
||||
await part.handle.close();
|
||||
@@ -540,9 +527,6 @@ const decryptLivePhoto = async (
|
||||
// and then removes its own temp file, and only the attempt that reaches
|
||||
// TAG_FINAL renames one into place, so a download that needed three tries still
|
||||
// performs exactly one rename over the destination.
|
||||
//
|
||||
// `defaultName` says `destination` is the name `defaultFileName` made, not a
|
||||
// path the caller gave (see `decryptLivePhoto`).
|
||||
const fetchAndDecrypt = async (
|
||||
api: ApiClient,
|
||||
openStream: () => Promise<ReadableStream<Uint8Array>>,
|
||||
@@ -551,7 +535,6 @@ const fetchAndDecrypt = async (
|
||||
destination: string,
|
||||
onProgress?: ProgressCallback,
|
||||
original?: EnteFile,
|
||||
defaultName = false,
|
||||
): Promise<DownloadResult> =>
|
||||
withRetry(async () => {
|
||||
const stream = await openStream();
|
||||
@@ -564,7 +547,6 @@ const fetchAndDecrypt = async (
|
||||
key,
|
||||
onProgress,
|
||||
original,
|
||||
defaultName,
|
||||
);
|
||||
}
|
||||
const bytesWritten = await decryptToTemp(
|
||||
@@ -596,11 +578,10 @@ export const downloadFile = async (
|
||||
outPath?: string,
|
||||
onProgress?: ProgressCallback,
|
||||
): Promise<DownloadResult> => {
|
||||
// `outPath` is the caller's and is used as is. Without it, the file is
|
||||
// written in the current directory under the name `defaultFileName` makes
|
||||
// from the title, which is the server's.
|
||||
// `outPath` is the caller's and is used as is; the title is the server's
|
||||
// and is sanitized so it can only name a file in the current directory.
|
||||
const resolvedPath =
|
||||
outPath ?? defaultFileName(file.metadata.title, file.id);
|
||||
outPath ?? sanitizeFileName(file.metadata.title, `file-${file.id}`);
|
||||
const header = fromBase64(file.file.decryptionHeader);
|
||||
return fetchAndDecrypt(
|
||||
api,
|
||||
@@ -610,7 +591,6 @@ export const downloadFile = async (
|
||||
resolvedPath,
|
||||
onProgress,
|
||||
file,
|
||||
outPath === undefined,
|
||||
);
|
||||
};
|
||||
|
||||
@@ -620,10 +600,9 @@ export const downloadThumbnail = async (
|
||||
outPath?: string,
|
||||
onProgress?: ProgressCallback,
|
||||
): Promise<DownloadResult> => {
|
||||
// As in `downloadFile`, with `thumb_` before the title.
|
||||
const resolvedPath =
|
||||
outPath ??
|
||||
defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
|
||||
`thumb_${sanitizeFileName(file.metadata.title, `file-${file.id}`)}`;
|
||||
const header = fromBase64(file.thumbnail.decryptionHeader);
|
||||
return fetchAndDecrypt(
|
||||
api,
|
||||
|
||||
+4
-56
@@ -5,9 +5,7 @@
|
||||
// used on disk goes through here, so it can only ever name one file inside the
|
||||
// directory the caller chose: never a path, never `..`, never hidden, never a
|
||||
// Windows device name. Such a name can still be too long for a file name, and
|
||||
// `shortenToBytes` or `shortenKeepingExtension` cuts one short.
|
||||
// `defaultFileName` does both for the name a file is saved under when the
|
||||
// caller gives no path.
|
||||
// `shortenToBytes` cuts one short.
|
||||
//
|
||||
// A path the user typed (`--out`, `outPath`) is not passed through here: the
|
||||
// caller is trusted, the server is not.
|
||||
@@ -32,11 +30,11 @@ export const sanitizeFileName = (name: string, fallback: string): string => {
|
||||
return RESERVED_DEVICE_NAME.test(cleaned) ? `_${cleaned}` : cleaned;
|
||||
};
|
||||
|
||||
// The extension of `title` (".jpg"), or ".bin" when it has none, it holds
|
||||
// anything but letters and digits, or it is more than 16 of them long.
|
||||
// The extension of `title` (".jpg"), or ".bin" when it has none or it holds
|
||||
// anything but letters and digits.
|
||||
export const safeExtension = (title: string): string => {
|
||||
const ext = extname(title);
|
||||
return /^\.[A-Za-z0-9]{1,16}$/.test(ext) ? ext : ".bin";
|
||||
return /^\.[A-Za-z0-9]+$/.test(ext) ? ext : ".bin";
|
||||
};
|
||||
|
||||
// `name` with its extension, if it has one, replaced by `ext` (".mov").
|
||||
@@ -59,53 +57,3 @@ export const shortenToBytes = (name: string, maxBytes: number): string => {
|
||||
}
|
||||
return name.slice(0, end);
|
||||
};
|
||||
|
||||
// `name` cut short to fit in `maxBytes` bytes with `insert` put right before
|
||||
// its extension (`IMG_0001 (12).jpg`), each cut made as `shortenToBytes`
|
||||
// makes it. The part before the extension is cut first, so the extension is
|
||||
// kept whenever it fits, but that part always keeps its first character:
|
||||
// when the extension leaves no room for `insert` and that character, the
|
||||
// extension is cut short too, and the part before it keeps the bytes that cut
|
||||
// leaves over. So a name that does not start with a dot never comes back
|
||||
// starting with one, which would hide it. `insert` is never cut. With no
|
||||
// `insert`, a name that fits comes back unchanged.
|
||||
export const shortenKeepingExtension = (
|
||||
name: string,
|
||||
maxBytes: number,
|
||||
insert = "",
|
||||
): string => {
|
||||
const fullExt = extname(name);
|
||||
const stem = name.slice(0, name.length - fullExt.length);
|
||||
const first = Array.from(stem)[0] ?? "";
|
||||
const ext = shortenToBytes(
|
||||
fullExt,
|
||||
maxBytes - Buffer.byteLength(insert + first),
|
||||
);
|
||||
const room = maxBytes - Buffer.byteLength(insert + ext);
|
||||
return shortenToBytes(stem, room) + insert + ext;
|
||||
};
|
||||
|
||||
// The name a file is saved under in the working directory when the caller
|
||||
// gives no path (`quak get` without `--out`, `downloadFile` without
|
||||
// `outPath`): `prefix` (`thumb_` for a thumbnail), then the file's title with
|
||||
// unsafe characters replaced, or `file-<id>` when the title is empty, cut
|
||||
// short so that the whole fits in a file name, keeping the title's extension
|
||||
// whenever it fits. `ext` (".mov"), when given, takes the place of the
|
||||
// title's extension before the name is cut short; a live photo's image and
|
||||
// video are each named with their own. The cut can leave a device name
|
||||
// (`CONSOLE.<long extension>` becomes `CON.<long extension>`); that name gets
|
||||
// a leading `_`, as `sanitizeFileName` gives one, and is cut a byte shorter to
|
||||
// leave room for it.
|
||||
export const defaultFileName = (
|
||||
title: string,
|
||||
fileID: number,
|
||||
opts: { prefix?: string; ext?: string } = {},
|
||||
): string => {
|
||||
const { prefix = "", ext } = opts;
|
||||
const safe = sanitizeFileName(title, `file-${fileID}`);
|
||||
const name = ext === undefined ? safe : withExtension(safe, ext);
|
||||
const room = MAX_NAME_BYTES - Buffer.byteLength(prefix);
|
||||
const cut = prefix + shortenKeepingExtension(name, room);
|
||||
if (!RESERVED_DEVICE_NAME.test(cut)) return cut;
|
||||
return `_${prefix}${shortenKeepingExtension(name, room - 1)}`;
|
||||
};
|
||||
+11
-152
@@ -13,8 +13,6 @@
|
||||
// originals, the thumbnail pool for thumbnails. The pool limits concurrency,
|
||||
// orders on-demand work ahead of background, and dedups by key so a fileID
|
||||
// requested twice while the first is still in flight downloads once.
|
||||
// `fetchOriginal` passes no key, so it is not shared with a fetch of the same
|
||||
// file already in flight.
|
||||
//
|
||||
// Integrity. The reused streaming decrypt is the enforced guarantee: every
|
||||
// chunk is authenticated and the writer renames the file into place only once
|
||||
@@ -29,10 +27,9 @@
|
||||
// It also names each original's save path under the download directory
|
||||
// (`savePath`), where `Photo.download()` and `lib.backup()` put it. A copy in
|
||||
// the cache does not count as saved there, but is copied there rather than
|
||||
// fetched again. An original stored there whose bytes do not match its content
|
||||
// hash is replaced only by a download that does (`replaceOriginal`).
|
||||
// fetched again.
|
||||
|
||||
import { createReadStream, existsSync, readFileSync, statSync } from "node:fs";
|
||||
import { existsSync, readFileSync, statSync } from "node:fs";
|
||||
import {
|
||||
chmod,
|
||||
copyFile,
|
||||
@@ -47,12 +44,6 @@ import {
|
||||
import { basename, dirname, extname, join } from "node:path";
|
||||
|
||||
import type { ApiClient } from "../api/client.js";
|
||||
import {
|
||||
chunkHashFinal,
|
||||
chunkHashInit,
|
||||
chunkHashUpdate,
|
||||
init,
|
||||
} from "../crypto/index.js";
|
||||
import {
|
||||
downloadFile,
|
||||
downloadThumbnail,
|
||||
@@ -358,26 +349,6 @@ export const storedAtSavePath = (
|
||||
return storedOriginal(dirname(path), basename(path, extname(path)), file);
|
||||
};
|
||||
|
||||
// The content hash of the original stored at `stored`, computed as the download
|
||||
// check computes it: over each file's bytes, read in chunks, and for a live
|
||||
// photo `<imageHash>:<videoHash>`.
|
||||
export const storedHash = async (stored: {
|
||||
path: string;
|
||||
videoPath?: string;
|
||||
}): Promise<string> => {
|
||||
await init();
|
||||
const hashFile = async (path: string): Promise<string> => {
|
||||
const state = chunkHashInit();
|
||||
for await (const chunk of createReadStream(path)) {
|
||||
chunkHashUpdate(state, chunk as Buffer);
|
||||
}
|
||||
return chunkHashFinal(state);
|
||||
};
|
||||
const hash = await hashFile(stored.path);
|
||||
if (stored.videoPath === undefined) return hash;
|
||||
return `${hash}:${await hashFile(stored.videoPath)}`;
|
||||
};
|
||||
|
||||
// Copy bytes into `dest` via a temp file in the same directory plus rename, so
|
||||
// `dest` appears only once it is whole ("present means complete"). As in the
|
||||
// download writer, the temp file is fsynced before the rename and the directory
|
||||
@@ -434,62 +405,6 @@ export const placeOriginal = async (
|
||||
return { path, videoPath };
|
||||
};
|
||||
|
||||
// Replace the original of `file` stored at its save path under `root` with a
|
||||
// new copy, keeping the stored one until the copy is complete and its hash,
|
||||
// computed as `storedHash` computes it, is the content hash the file's metadata
|
||||
// records. `get` is given a temporary file beside the save path and writes the
|
||||
// copy there, a live photo as its image and its video beside that file, each
|
||||
// with its own extension. Only a copy that matches is renamed over the stored
|
||||
// one, and a live photo's image and video are both hashed before either is
|
||||
// renamed. The image is renamed first: should the run stop, or the video's
|
||||
// rename fail, before the video is renamed, the new image is left beside the
|
||||
// old video, and a later backup with `verify` replaces the video if the video
|
||||
// was the part that did not match. When `get` fails or the copy does not match,
|
||||
// the temporary files are removed and the stored original stays as it was.
|
||||
export const replaceOriginal = async (
|
||||
root: string,
|
||||
file: EnteFile,
|
||||
get: (dest: string) => Promise<{ path: string; videoPath?: string }>,
|
||||
): Promise<{ path: string; videoPath?: string }> => {
|
||||
const dest = savePath(root, file);
|
||||
const dir = dirname(dest);
|
||||
// Named as the download writer names its temporary files; a live photo's
|
||||
// image and video get this name with their own extensions. The next
|
||||
// backup removes any that a killed run leaves (`removeLeftoverTempFiles`).
|
||||
const tmp = join(
|
||||
dir,
|
||||
`.quak-${process.pid}-${Math.random().toString(36).slice(2)}.tmp`,
|
||||
);
|
||||
let got: { path: string; videoPath?: string } | undefined;
|
||||
try {
|
||||
got = await get(tmp);
|
||||
if ((await storedHash(got)) !== file.metadata.hash) {
|
||||
throw new Error(
|
||||
"the downloaded original does not match its content hash either",
|
||||
);
|
||||
}
|
||||
if (got.videoPath === undefined) {
|
||||
await rename(got.path, dest);
|
||||
await fsyncPath(dir);
|
||||
return { path: dest };
|
||||
}
|
||||
const path = withExtension(dest, extname(got.path));
|
||||
const videoPath = withExtension(dest, extname(got.videoPath));
|
||||
await rename(got.path, path);
|
||||
await rename(got.videoPath, videoPath);
|
||||
const name = basename(dest, extname(dest));
|
||||
await writeLivePhotoJSON(dir, name, { path, videoPath });
|
||||
return { path, videoPath };
|
||||
} finally {
|
||||
// Whatever was not renamed into place. A failure to remove it must
|
||||
// not replace the error that explains what went wrong.
|
||||
for (const path of [tmp, got?.path, got?.videoPath]) {
|
||||
if (path === undefined) continue;
|
||||
await rm(path, { force: true }).catch(() => undefined);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
private readonly pools: RequestPools;
|
||||
private readonly source: ContentSource;
|
||||
@@ -581,34 +496,14 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
fileID: number,
|
||||
opts?: ContentOptions,
|
||||
): Promise<ContentResult> {
|
||||
const file = this.fileOrThrow(fileID);
|
||||
return this.get(file, "original", "on-demand", opts?.onProgress);
|
||||
return this.get(fileID, "original", "on-demand", opts?.onProgress);
|
||||
}
|
||||
|
||||
async thumbnail(
|
||||
fileID: number,
|
||||
opts?: ContentOptions,
|
||||
): Promise<ContentResult> {
|
||||
const file = this.fileOrThrow(fileID);
|
||||
return this.get(file, "thumbnail", "on-demand", opts?.onProgress);
|
||||
}
|
||||
|
||||
// The original of `file`, fetched and cached as `original` does. `file` is
|
||||
// used as given, so `getFile` need not know it.
|
||||
async originalOf(
|
||||
file: EnteFile,
|
||||
opts?: ContentOptions,
|
||||
): Promise<ContentResult> {
|
||||
return this.get(file, "original", "on-demand", opts?.onProgress);
|
||||
}
|
||||
|
||||
// The thumbnail of `file`, fetched and cached as `thumbnail` does. `file`
|
||||
// is used as given, so `getFile` need not know it.
|
||||
async thumbnailOf(
|
||||
file: EnteFile,
|
||||
opts?: ContentOptions,
|
||||
): Promise<ContentResult> {
|
||||
return this.get(file, "thumbnail", "on-demand", opts?.onProgress);
|
||||
return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
|
||||
}
|
||||
|
||||
// Put the original at the save path of `file` under the download directory
|
||||
@@ -635,7 +530,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
destination: string,
|
||||
): Promise<ContentResult> {
|
||||
const result = await this.acquire(
|
||||
this.fileOrThrow(fileID),
|
||||
fileID,
|
||||
"original",
|
||||
"on-demand",
|
||||
undefined,
|
||||
@@ -648,34 +543,6 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
};
|
||||
}
|
||||
|
||||
// Fetch an original through the content pool to `destination`, a live
|
||||
// photo as its image and its video beside it, and return where it landed.
|
||||
// Unlike `backupOriginal`, it fetches even when the cache or the save path
|
||||
// already holds the original, and it records nothing: what it writes is
|
||||
// the caller's. It is not shared with a fetch of the same file already
|
||||
// under way, which writes somewhere else.
|
||||
async fetchOriginal(
|
||||
fileID: number,
|
||||
destination: string,
|
||||
): Promise<{ path: string; videoPath?: string }> {
|
||||
const file = this.fileOrThrow(fileID);
|
||||
return this.pools.content.run(
|
||||
async () => {
|
||||
const fetched = await this.fetchInto(
|
||||
file,
|
||||
destination,
|
||||
"original",
|
||||
undefined,
|
||||
);
|
||||
for (const path of [fetched.path, fetched.videoPath]) {
|
||||
if (path !== undefined) await chmod(path, FILE_MODE);
|
||||
}
|
||||
return fetched;
|
||||
},
|
||||
{ priority: "on-demand" },
|
||||
);
|
||||
}
|
||||
|
||||
async ensure(args: EnsureOptions): Promise<EnsureResult[]> {
|
||||
return this.ensureThumbnails(args);
|
||||
}
|
||||
@@ -740,8 +607,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
onProgress: ((event: EnsureEvent) => void) | undefined,
|
||||
): Promise<EnsureResult> {
|
||||
try {
|
||||
const file = this.fileOrThrow(fileID);
|
||||
const result = await this.acquire(file, kind, priority, signal);
|
||||
const result = await this.acquire(fileID, kind, priority, signal);
|
||||
const status = result.cached ? "skipped" : "done";
|
||||
onProgress?.({ fileID, status, path: result.path });
|
||||
return { fileID, path: result.path };
|
||||
@@ -756,16 +622,8 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
}
|
||||
}
|
||||
|
||||
// The record `getFile` gives for `fileID`. It throws for a file `getFile`
|
||||
// does not know.
|
||||
private fileOrThrow(fileID: number): EnteFile {
|
||||
const file = this.getFile(fileID);
|
||||
if (!file) throw new Error(`content cache: unknown file ${fileID}`);
|
||||
return file;
|
||||
}
|
||||
|
||||
private async get(
|
||||
file: EnteFile,
|
||||
fileID: number,
|
||||
kind: Kind,
|
||||
priority: Priority,
|
||||
onProgress: ((event: ContentEvent) => void) | undefined,
|
||||
@@ -773,7 +631,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
const onByte: ProgressCallback | undefined = onProgress
|
||||
? (bytesDone) => onProgress({ status: "downloading", bytesDone })
|
||||
: undefined;
|
||||
const result = await this.acquire(file, kind, priority, undefined, {
|
||||
const result = await this.acquire(fileID, kind, priority, undefined, {
|
||||
onByte,
|
||||
});
|
||||
onProgress?.(
|
||||
@@ -795,7 +653,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
// photo's original is present only with its video, and is returned with
|
||||
// it.
|
||||
private async acquire(
|
||||
file: EnteFile,
|
||||
fileID: number,
|
||||
kind: Kind,
|
||||
priority: Priority,
|
||||
signal: AbortSignal | undefined,
|
||||
@@ -806,7 +664,8 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
||||
videoPath?: string;
|
||||
cached: boolean;
|
||||
}> {
|
||||
const fileID = file.id;
|
||||
const file = this.getFile(fileID);
|
||||
if (!file) throw new Error(`content cache: unknown file ${fileID}`);
|
||||
const isLivePhoto =
|
||||
kind === "original" && file.metadata.fileType === "livePhoto";
|
||||
|
||||
|
||||
+31
-222
@@ -18,17 +18,6 @@
|
||||
// the caller (the default reads stay silent and serve the last good copy). The
|
||||
// default methods and the background loop are unchanged.
|
||||
//
|
||||
// `freshCollections()` is the fresh read of the collection list alone (issue
|
||||
// #210): one collections request, no collection's files, and nothing stored,
|
||||
// for a caller such as `quak collections` that needs no more. `freshFiles()`
|
||||
// is the fresh read of one collection's files, and stores nothing either
|
||||
// (issue #217). A caller that asks for each refresh it needs can also open the
|
||||
// library with `refreshOnOpen: false`, so that `open()` starts none.
|
||||
//
|
||||
// A refresh reports through `onProgress` each collection whose files it has
|
||||
// fetched, so a large account, whose first refresh fetches the files of every
|
||||
// collection, shows progress.
|
||||
//
|
||||
// A refresh stages all of its network work first and only mutates the store
|
||||
// once every fetch has succeeded. A refresh that fails partway therefore never
|
||||
// becomes visible to reads: the last good snapshot stays in place, and the
|
||||
@@ -66,8 +55,6 @@ import {
|
||||
ContentCache,
|
||||
savePath,
|
||||
storedAtSavePath,
|
||||
type ContentOptions,
|
||||
type ContentResult,
|
||||
type ContentSource,
|
||||
type ThumbnailsAPI,
|
||||
type EnsureOptions,
|
||||
@@ -102,7 +89,6 @@ export {
|
||||
} from "./content.js";
|
||||
export { type MLDataAPI, type SimilarResult } from "./mlsearch.js";
|
||||
import type { CollectionsPage, FilesPage } from "../client.js";
|
||||
import { describeError } from "../errors.js";
|
||||
import { MLDATA_BATCH_SIZE, type MLData } from "../mldata-fetch.js";
|
||||
import type { Collection, EnteFile } from "../model/types.js";
|
||||
import { runBackup, type BackupOptions, type BackupResult } from "../backup.js";
|
||||
@@ -119,18 +105,9 @@ export {
|
||||
export const DEFAULT_REFRESH_INTERVAL_SECONDS = 3;
|
||||
|
||||
// The account's cache directory when `cacheDirectory` is not given: the
|
||||
// env-paths cache directory plus the user id, so each account has its own. An
|
||||
// account on a self-hosted server has the server's address, URL-encoded, in
|
||||
// between, because two servers can give two accounts the same user id.
|
||||
export const defaultCacheDirectory = (
|
||||
userID: number,
|
||||
apiOrigin?: string,
|
||||
): string => {
|
||||
const cache = envPaths("quak", { suffix: "" }).cache;
|
||||
return apiOrigin === undefined
|
||||
? join(cache, String(userID))
|
||||
: join(cache, encodeURIComponent(apiOrigin), String(userID));
|
||||
};
|
||||
// env-paths cache directory plus the user id, so each account has its own.
|
||||
export const defaultCacheDirectory = (userID: number): string =>
|
||||
join(envPaths("quak", { suffix: "" }).cache, String(userID));
|
||||
|
||||
// Project a metadata store into by-id records, filling each record's cache
|
||||
// paths from the content cache when one is given. Shared by the live read
|
||||
@@ -153,9 +130,6 @@ const deriveRecordsFromStore = (
|
||||
// tests drive a mock with no crypto or network; the real `Client` satisfies it
|
||||
// structurally.
|
||||
export interface LibraryClient {
|
||||
// The server the client talks to, as `Client.apiOrigin` names it. Absent
|
||||
// means Ente's hosted service.
|
||||
readonly apiOrigin?: string;
|
||||
whoami(): { email: string; userID: number };
|
||||
collectionsSince(args: { sinceTime: number }): Promise<CollectionsPage>;
|
||||
filesSince(args: {
|
||||
@@ -164,16 +138,12 @@ export interface LibraryClient {
|
||||
sinceTime: number;
|
||||
}): Promise<FilesPage>;
|
||||
// Fetch ML data (face detections + CLIP embeddings) for up to a batch of
|
||||
// files, and for each file whose ML data did not decrypt, gunzip or parse,
|
||||
// why not. Optional: a client without it simply disables ML fetching,
|
||||
// leaving the metadata refresh untouched.
|
||||
// files. Optional: a client without it simply disables ML fetching, leaving
|
||||
// the metadata refresh untouched.
|
||||
fetchMLData?(args: {
|
||||
fileIDs: number[];
|
||||
fileKeys: Map<number, Uint8Array>;
|
||||
}): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}>;
|
||||
}): Promise<Map<number, MLData>>;
|
||||
// The byte source for the on-disk content cache. Optional so a mock client
|
||||
// that only serves metadata still satisfies the interface; when absent (and
|
||||
// no explicit `contentSource` is passed to `open`) the content cache is
|
||||
@@ -184,11 +154,7 @@ export interface LibraryClient {
|
||||
// A progress event for one unit of background work. A metadata "refresh" or an
|
||||
// ML "fetchMLData" pass each fire "started" before their network work and then
|
||||
// exactly one of "done" or "failed"; "failed" carries the error message and
|
||||
// an ML "done" reports how many payloads it stored and, in `mlDataErrors`,
|
||||
// each file whose ML data did not decrypt, gunzip or parse, with the reason.
|
||||
// In between, a refresh fires "progress" after each collection whose files it
|
||||
// fetched: `fetched` is how many collections it has fetched the files of so
|
||||
// far, `total` how many it will. The precache fills
|
||||
// an ML "done" reports how many payloads it stored. The precache fills
|
||||
// ("precacheThumbnails"/"precacheOriginals", #48) fire "started"/"done" around
|
||||
// each sweep that has work, "done" reporting the count newly cached.
|
||||
export interface RefreshEvent {
|
||||
@@ -197,19 +163,17 @@ export interface RefreshEvent {
|
||||
| "fetchMLData"
|
||||
| "precacheThumbnails"
|
||||
| "precacheOriginals";
|
||||
status: "started" | "progress" | "done" | "failed";
|
||||
status: "started" | "done" | "failed";
|
||||
error?: string;
|
||||
fetched?: number;
|
||||
total?: number;
|
||||
mlDataErrors?: { fileID: number; error: string }[];
|
||||
}
|
||||
|
||||
export type RefreshProgressCallback = (event: RefreshEvent) => void;
|
||||
|
||||
export interface LibraryOptions {
|
||||
client: LibraryClient;
|
||||
// Where `metadata.json` lives. Defaults to `defaultCacheDirectory`, so
|
||||
// each account has its own cache.
|
||||
// Where `metadata.json` lives. Defaults to the env-paths cache directory
|
||||
// plus the user id, so each account has its own cache.
|
||||
cacheDirectory?: string;
|
||||
// The root of every photo's save path, where `Photo.download()` and
|
||||
// `lib.backup()` put originals. Defaults to `photos` in the working
|
||||
@@ -220,10 +184,6 @@ export interface LibraryOptions {
|
||||
// `Infinity` starts none: the library then refreshes only as it opens and
|
||||
// when `fresh()` or `backup()` asks.
|
||||
refreshIntervalSeconds?: number;
|
||||
// Whether `open()` starts a refresh. Defaults to true. With false, `open()`
|
||||
// returns at once onto what the cache holds, empty or not, and a refresh
|
||||
// starts only when `fresh()` or `backup()` asks or the timer fires.
|
||||
refreshOnOpen?: boolean;
|
||||
onProgress?: RefreshProgressCallback;
|
||||
// The bounded request pools (issue #45), shared by the ML-data fetch (the
|
||||
// metadata pool) and the content cache. Defaults to a fresh set at the
|
||||
@@ -265,11 +225,6 @@ export interface LibraryStatus {
|
||||
lastMLFetchAt?: number;
|
||||
// The most recent ML fetch pass's error, set only while it failed.
|
||||
lastMLError?: string;
|
||||
// Each file whose ML data the last ML fetch pass that succeeded got but
|
||||
// could not decrypt, gunzip or parse, with the reason. A pass still
|
||||
// running leaves it as it is. A pass that fails leaves it as it was. Such
|
||||
// ML data is not cached, so the next pass asks for it again.
|
||||
mlDataErrors: { fileID: number; error: string }[];
|
||||
// ML payloads stored on disk and CLIP embeddings in the index; undefined
|
||||
// when ML fetching is disabled.
|
||||
mlStored?: number;
|
||||
@@ -312,7 +267,6 @@ export class Library {
|
||||
// available (a metadata-only client with no explicit source).
|
||||
private readonly cache?: ContentCache;
|
||||
private readonly userID: number;
|
||||
private readonly apiOrigin: string | undefined;
|
||||
private readonly intervalMs: number;
|
||||
private readonly onProgress?: RefreshProgressCallback;
|
||||
private readonly pools: RequestPools;
|
||||
@@ -338,12 +292,6 @@ export class Library {
|
||||
private lastError?: string;
|
||||
private lastMLFetchAt?: number;
|
||||
private lastMLError?: string;
|
||||
// For each file whose ML data the last ML fetch pass that succeeded got
|
||||
// but could not decrypt, gunzip or parse, why not, by fileID. A pass
|
||||
// gathers its own and puts them here when it ends without failing, so
|
||||
// `status()` and `backup()` never see the list of a pass that is still
|
||||
// running or that failed. Kept in RAM only, never in the cache.
|
||||
private mlDataErrors = new Map<number, unknown>();
|
||||
// The plain-record projection as of the last refresh, and the GUI change
|
||||
// subscribers. A refresh that alters the projection notifies each with the
|
||||
// delta; `lastRecords` is kept current every refresh so a subscriber that
|
||||
@@ -359,7 +307,6 @@ export class Library {
|
||||
client: LibraryClient;
|
||||
store: MetadataStore;
|
||||
userID: number;
|
||||
apiOrigin: string | undefined;
|
||||
cacheDirectory: string;
|
||||
downloadDirectory: string;
|
||||
intervalMs: number;
|
||||
@@ -372,7 +319,6 @@ export class Library {
|
||||
this.client = args.client;
|
||||
this.store = args.store;
|
||||
this.userID = args.userID;
|
||||
this.apiOrigin = args.apiOrigin;
|
||||
this.cacheDirectory = args.cacheDirectory;
|
||||
this.downloadDirectory = args.downloadDirectory;
|
||||
this.intervalMs = args.intervalMs;
|
||||
@@ -417,13 +363,11 @@ export class Library {
|
||||
// then opens empty with the failure recorded in `status()`. With an
|
||||
// existing cache the first refresh runs in the background and `open()`
|
||||
// returns as soon as the cached data is ready — an unreachable server does
|
||||
// not block opening. With `refreshOnOpen: false` there is no first
|
||||
// refresh: `open()` returns onto the cache as it is.
|
||||
// not block opening.
|
||||
static async open(opts: LibraryOptions): Promise<Library> {
|
||||
const { userID } = opts.client.whoami();
|
||||
const { apiOrigin } = opts.client;
|
||||
const cacheDirectory =
|
||||
opts.cacheDirectory ?? defaultCacheDirectory(userID, apiOrigin);
|
||||
opts.cacheDirectory ?? defaultCacheDirectory(userID);
|
||||
if (opts.downloadDirectory === "") {
|
||||
throw new Error(
|
||||
"library: downloadDirectory is empty (leave it out to save " +
|
||||
@@ -435,24 +379,14 @@ export class Library {
|
||||
let store = await MetadataStore.load(metadataPath);
|
||||
// A cache directory given explicitly can hold another account's cache.
|
||||
// Its records and cursor are not this account's, so delete it and the
|
||||
// ML data beside it and start empty. A cache from another server goes
|
||||
// whole: two servers can give two files the same ID, so its originals
|
||||
// and thumbnails could be served for this account's files. A user ID
|
||||
// of 0 means the cache names no account, so nothing is stored in it.
|
||||
if (
|
||||
store.userID !== 0 &&
|
||||
(store.userID !== userID || store.apiOrigin !== apiOrigin)
|
||||
) {
|
||||
const discard = ["metadata.json", "mldata"];
|
||||
if (store.apiOrigin !== apiOrigin) {
|
||||
discard.push("originals", "thumbnails");
|
||||
}
|
||||
for (const name of discard) {
|
||||
await rm(join(cacheDirectory, name), {
|
||||
recursive: true,
|
||||
force: true,
|
||||
});
|
||||
}
|
||||
// ML data beside it and start empty. A user ID of 0 means the cache
|
||||
// was never refreshed and so holds nothing to discard.
|
||||
if (store.userID !== 0 && store.userID !== userID) {
|
||||
await rm(metadataPath, { force: true });
|
||||
await rm(join(cacheDirectory, "mldata"), {
|
||||
recursive: true,
|
||||
force: true,
|
||||
});
|
||||
store = await MetadataStore.load(metadataPath);
|
||||
}
|
||||
const intervalMs =
|
||||
@@ -516,7 +450,6 @@ export class Library {
|
||||
client: opts.client,
|
||||
store,
|
||||
userID,
|
||||
apiOrigin,
|
||||
cacheDirectory,
|
||||
downloadDirectory,
|
||||
intervalMs,
|
||||
@@ -533,14 +466,9 @@ export class Library {
|
||||
// refresh.
|
||||
precache?.start();
|
||||
|
||||
if (opts.refreshOnOpen === false) {
|
||||
// The caller asks for each refresh it needs.
|
||||
lib.scheduleNext();
|
||||
} else if (store.listCollections().length > 0) {
|
||||
if (store.loadedFromDisk) {
|
||||
// An existing copy already answers reads; refresh in the background
|
||||
// and start the interval once that first cycle settles. A cache
|
||||
// that names the account but holds no collection has nothing to
|
||||
// answer with, such as after `originalOf` on an empty cache.
|
||||
// and start the interval once that first cycle settles.
|
||||
void lib.runRefresh().then(() => lib.scheduleNext());
|
||||
} else {
|
||||
// Nothing was cached: wait for the first refresh to fill the store
|
||||
@@ -614,7 +542,6 @@ export class Library {
|
||||
lastError: this.lastError,
|
||||
lastMLFetchAt: this.lastMLFetchAt,
|
||||
lastMLError: this.lastMLError,
|
||||
mlDataErrors: this.describeMLDataErrors(),
|
||||
mlStored: ml?.stored,
|
||||
mlIndexed: ml?.indexed,
|
||||
originalsUsedBytes: originals?.usedBytes,
|
||||
@@ -642,84 +569,6 @@ export class Library {
|
||||
};
|
||||
}
|
||||
|
||||
// The fresh read of the collection list alone (issue #210): every live
|
||||
// collection that decrypts, in the server's order, from one collections
|
||||
// request. Unlike a refresh, it fetches no collection's files, which on a
|
||||
// large account take long. Nothing is stored: a refresh fetches a
|
||||
// collection's files only when the collection is new or has changed since
|
||||
// the stored copy, so a collection stored here without its files would
|
||||
// never have them fetched.
|
||||
async freshCollections(): Promise<Collection[]> {
|
||||
if (this.closed) throw new Error("the library is closed");
|
||||
const { collections } = await this.client.collectionsSince({
|
||||
sinceTime: 0,
|
||||
});
|
||||
return collections;
|
||||
}
|
||||
|
||||
// The fresh read of one collection's files (issue #217): every live file
|
||||
// in it that decrypts, in the server's order. `collection` is a record
|
||||
// `freshCollections()` returned, whose key decrypts the files. It fetches
|
||||
// no other collection's files. Like `freshCollections()`, it stores
|
||||
// nothing. It returns each file once, as the latest copy the server sent,
|
||||
// where the first copy was.
|
||||
async freshFiles(collection: Collection): Promise<EnteFile[]> {
|
||||
if (this.closed) throw new Error("the library is closed");
|
||||
const { files } = await this.client.filesSince({
|
||||
collectionID: collection.id,
|
||||
collectionKey: collection.key,
|
||||
sinceTime: 0,
|
||||
});
|
||||
const byID = new Map<number, EnteFile>();
|
||||
for (const file of files) byID.set(file.id, file);
|
||||
return [...byID.values()];
|
||||
}
|
||||
|
||||
// The original of `file`, fetched and cached as `Photo.original()` does.
|
||||
// `file` is used as given, so the store need not hold it. On an empty
|
||||
// cache, `quak get` passes a file that `freshFiles()` returned.
|
||||
async originalOf(
|
||||
file: EnteFile,
|
||||
opts?: ContentOptions,
|
||||
): Promise<ContentResult> {
|
||||
if (!this.cache) {
|
||||
throw new Error(
|
||||
"originalOf requires a library opened with a content cache",
|
||||
);
|
||||
}
|
||||
await this.saveAccountInEmptyCache();
|
||||
return this.cache.originalOf(file, opts);
|
||||
}
|
||||
|
||||
// The thumbnail of `file`, fetched and cached as `Photo.thumbnail()` does.
|
||||
// `file` is used as given, so the store need not hold it. On an empty
|
||||
// cache, `quak get-thumb` passes a file that `freshFiles()` returned.
|
||||
async thumbnailOf(
|
||||
file: EnteFile,
|
||||
opts?: ContentOptions,
|
||||
): Promise<ContentResult> {
|
||||
if (!this.cache) {
|
||||
throw new Error(
|
||||
"thumbnailOf requires a library opened with a content cache",
|
||||
);
|
||||
}
|
||||
await this.saveAccountInEmptyCache();
|
||||
return this.cache.thumbnailOf(file, opts);
|
||||
}
|
||||
|
||||
// Saves the account's user ID and server in a cache that names no account
|
||||
// yet, before `originalOf` or `thumbnailOf` stores a file there. A later
|
||||
// `open()` for an account on another server then deletes that file (see
|
||||
// `open()`), although no refresh has stored any records.
|
||||
private async saveAccountInEmptyCache(): Promise<void> {
|
||||
if (this.store.userID !== 0) return;
|
||||
this.store.userID = this.userID;
|
||||
this.store.apiOrigin = this.apiOrigin;
|
||||
this.unsaved = true;
|
||||
await this.store.save();
|
||||
this.unsaved = false;
|
||||
}
|
||||
|
||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||
// library's, each original at its save path, with a durable failure
|
||||
// ledger (issue #51). Takes the lock in that directory first, unless the
|
||||
@@ -757,13 +606,10 @@ export class Library {
|
||||
undecryptable: () => this.store.listUndecryptable(),
|
||||
original: (fileID, destination) =>
|
||||
cache!.backupOriginal(fileID, destination),
|
||||
fetchOriginal: (fileID, destination) =>
|
||||
cache!.fetchOriginal(fileID, destination),
|
||||
thumbnail: (fileID) => cache!.thumbnail(fileID),
|
||||
contentConcurrency: this.pools.content.concurrency,
|
||||
fetchMLData: () => this.fetchMLDataNow(),
|
||||
mlData: (fileID) => this.mldata.forFile({ fileID }),
|
||||
mlDataError: (fileID) => this.mlDataErrors.get(fileID),
|
||||
},
|
||||
{ ...opts, downloadDirectory },
|
||||
);
|
||||
@@ -873,40 +719,26 @@ export class Library {
|
||||
// Stage per-collection file diffs. A collection's files are
|
||||
// re-enumerated only when its updationTime has advanced past the cached
|
||||
// copy; an unchanged album's file list cannot have changed. New
|
||||
// collections enumerate from the beginning of time. Each collection
|
||||
// whose files are fetched is reported, with how many there are to
|
||||
// fetch.
|
||||
const changedCollections = page.collections.filter((collection) => {
|
||||
const known = this.store.getCollection(collection.id);
|
||||
return !known || collection.updationTime > known.updationTime;
|
||||
});
|
||||
// collections enumerate from the beginning of time.
|
||||
const filePages: { collectionID: number; page: FilesPage }[] = [];
|
||||
for (const collection of changedCollections) {
|
||||
for (const collection of page.collections) {
|
||||
const known = this.store.getCollection(collection.id);
|
||||
if (known && collection.updationTime <= known.updationTime)
|
||||
continue;
|
||||
const filePage = await this.client.filesSince({
|
||||
collectionID: collection.id,
|
||||
collectionKey: collection.key,
|
||||
sinceTime: known ? known.updationTime : 0,
|
||||
});
|
||||
filePages.push({ collectionID: collection.id, page: filePage });
|
||||
this.emit({
|
||||
operation: "refresh",
|
||||
status: "progress",
|
||||
fetched: filePages.length,
|
||||
total: changedCollections.length,
|
||||
});
|
||||
}
|
||||
|
||||
// Network work done; commit to the store and persist only if something
|
||||
// actually changed.
|
||||
let changed = false;
|
||||
|
||||
if (
|
||||
this.store.userID !== this.userID ||
|
||||
this.store.apiOrigin !== this.apiOrigin
|
||||
) {
|
||||
if (this.store.userID !== this.userID) {
|
||||
this.store.userID = this.userID;
|
||||
this.store.apiOrigin = this.apiOrigin;
|
||||
changed = true;
|
||||
}
|
||||
|
||||
@@ -1004,11 +836,8 @@ export class Library {
|
||||
|
||||
// One ML fetch pass: fetch, decrypt and store the ML data for every file
|
||||
// the store knows about that is not cached (or whose `updationTime` has
|
||||
// advanced), through the metadata pool, and update the CLIP index. For a
|
||||
// file whose ML data does not decrypt, gunzip or parse, it gathers the
|
||||
// reason instead, and puts the reasons it gathered in `mlDataErrors` when
|
||||
// it ends without failing. A failure leaves `mlDataErrors` as it was and
|
||||
// is reported through `status()` and `onProgress`, then thrown.
|
||||
// advanced), through the metadata pool, and update the CLIP index. A
|
||||
// failure is reported through `status()` and `onProgress`, then thrown.
|
||||
private async runMLFetch(): Promise<void> {
|
||||
const mldata = this.mlStore;
|
||||
// Bind so the call keeps the client as its receiver when invoked
|
||||
@@ -1018,14 +847,7 @@ export class Library {
|
||||
|
||||
const files = this.uniqueFiles();
|
||||
const needed = mldata.neededFor(files);
|
||||
// This pass's reasons, put in `this.mlDataErrors` when it ends without
|
||||
// failing. A file whose ML data did not decode in the pass before is
|
||||
// still among `needed`, so this pass gives its reason anew.
|
||||
const mlDataErrors = new Map<number, unknown>();
|
||||
if (needed.length === 0) {
|
||||
this.mlDataErrors = mlDataErrors;
|
||||
return;
|
||||
}
|
||||
if (needed.length === 0) return;
|
||||
|
||||
this.emit({ operation: "fetchMLData", status: "started" });
|
||||
try {
|
||||
@@ -1040,25 +862,20 @@ export class Library {
|
||||
for (let i = 0; i < needed.length; i += MLDATA_BATCH_SIZE) {
|
||||
if (this.closed) break;
|
||||
const batch = needed.slice(i, i + MLDATA_BATCH_SIZE);
|
||||
const { payloads, errors } = await this.pools.metadata.run(
|
||||
const payloads = await this.pools.metadata.run(
|
||||
() => fetchMLData({ fileIDs: batch, fileKeys }),
|
||||
{ priority: "background" },
|
||||
);
|
||||
for (const [id, error] of errors) {
|
||||
mlDataErrors.set(id, error);
|
||||
}
|
||||
stored += (await mldata.storeFetched(payloads, updation))
|
||||
.stored;
|
||||
}
|
||||
|
||||
this.mlDataErrors = mlDataErrors;
|
||||
this.lastMLFetchAt = Date.now();
|
||||
this.lastMLError = undefined;
|
||||
this.emit({
|
||||
operation: "fetchMLData",
|
||||
status: "done",
|
||||
fetched: stored,
|
||||
mlDataErrors: this.describeMLDataErrors(),
|
||||
});
|
||||
} catch (err) {
|
||||
const error = err instanceof Error ? err.message : String(err);
|
||||
@@ -1068,14 +885,6 @@ export class Library {
|
||||
}
|
||||
}
|
||||
|
||||
// `mlDataErrors` as `status()` and the ML "done" event give it.
|
||||
private describeMLDataErrors(): { fileID: number; error: string }[] {
|
||||
return [...this.mlDataErrors].map(([fileID, err]) => ({
|
||||
fileID,
|
||||
error: describeError(err),
|
||||
}));
|
||||
}
|
||||
|
||||
// The distinct files the store holds, one entry per fileID (a file in
|
||||
// several collections shares its ML data), each carrying the key and the
|
||||
// newest `updationTime` seen across its memberships.
|
||||
|
||||
@@ -61,7 +61,7 @@ export interface MLIndex {
|
||||
// Pull the CLIP embedding out of a payload, or undefined when it is absent or
|
||||
// misshapen. Kept strict so a bad payload is skipped rather than corrupting the
|
||||
// packed index.
|
||||
export const clipEmbedding = (payload: MLData): number[] | undefined => {
|
||||
const clipEmbedding = (payload: MLData): number[] | undefined => {
|
||||
const clip = payload.clip;
|
||||
if (typeof clip !== "object" || clip === null) return undefined;
|
||||
const embedding = (clip as { embedding?: unknown }).embedding;
|
||||
|
||||
+12
-11
@@ -1,12 +1,11 @@
|
||||
// On-disk JSON metadata store for the local cache.
|
||||
//
|
||||
// The store keeps one `metadata.json` file holding the account's server
|
||||
// state: the user id, the server's address for an account on a self-hosted
|
||||
// server, a schema version, the cursor for the incremental collections
|
||||
// listing, the decrypted collection and file records, and the collections and
|
||||
// files the refresh could not decrypt. The whole file is read into RAM on
|
||||
// load and rewritten as a whole on save; there is no partial update and no
|
||||
// lock file. A separate refresh unit populates the store from
|
||||
// state: the user id, a schema version, the cursor for the incremental
|
||||
// collections listing, the decrypted collection and file records, and the
|
||||
// collections and files the refresh could not decrypt. The whole file is read
|
||||
// into RAM on load and rewritten as a whole on save; there is no partial
|
||||
// update and no lock file. A separate refresh unit populates the store from
|
||||
// the server — this module only stores what it is given.
|
||||
//
|
||||
// The file is a cache, so it is never trusted to exist or to be intact: a
|
||||
@@ -47,7 +46,6 @@ export interface Undecryptable {
|
||||
interface StoredMetadata {
|
||||
schemaVersion: number;
|
||||
userID: number;
|
||||
apiOrigin?: string;
|
||||
collectionsSinceTime: Microseconds;
|
||||
collections: StoredCollection[];
|
||||
files: StoredFile[];
|
||||
@@ -70,10 +68,14 @@ export class MetadataStore {
|
||||
readonly path: string;
|
||||
readonly schemaVersion = METADATA_SCHEMA_VERSION;
|
||||
userID = 0;
|
||||
// The account's `apiOrigin`, or undefined for Ente's hosted service.
|
||||
apiOrigin: string | undefined;
|
||||
collectionsSinceTime: Microseconds = 0;
|
||||
|
||||
// True when `load` populated this store from a valid existing file; false
|
||||
// on a first run or a missing/corrupt/wrong-version file that loaded empty.
|
||||
// `Library.open` reads it to decide whether the first refresh may run in
|
||||
// the background (an existing copy already serves reads) or must be awaited.
|
||||
loadedFromDisk = false;
|
||||
|
||||
private readonly collections = new Map<number, Collection>();
|
||||
private readonly files = new Map<string, EnteFile>();
|
||||
private undecryptable: Undecryptable[] = [];
|
||||
@@ -100,8 +102,8 @@ export class MetadataStore {
|
||||
if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) {
|
||||
return store;
|
||||
}
|
||||
store.loadedFromDisk = true;
|
||||
store.userID = parsed.userID ?? 0;
|
||||
store.apiOrigin = parsed.apiOrigin;
|
||||
store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0;
|
||||
for (const stored of parsed.collections ?? []) {
|
||||
const collection: Collection = {
|
||||
@@ -141,7 +143,6 @@ export class MetadataStore {
|
||||
const model: StoredMetadata = {
|
||||
schemaVersion: METADATA_SCHEMA_VERSION,
|
||||
userID: this.userID,
|
||||
apiOrigin: this.apiOrigin,
|
||||
collectionsSinceTime: this.collectionsSinceTime,
|
||||
collections: [...this.collections.values()].map((c) => ({
|
||||
...c,
|
||||
|
||||
+13
-48
@@ -1,16 +1,10 @@
|
||||
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import * as jpeg from "jpeg-js";
|
||||
import { clipKeywords } from "./clip-keywords.js";
|
||||
import type { Client } from "./client.js";
|
||||
import { describeError } from "./errors.js";
|
||||
import { readExifTags } from "./exif.js";
|
||||
import type { Library, Photo } from "./library/index.js";
|
||||
import {
|
||||
MAX_NAME_BYTES,
|
||||
sanitizeFileName,
|
||||
shortenToBytes,
|
||||
} from "./filename.js";
|
||||
import { sanitizeFileName } from "./filename.js";
|
||||
import {
|
||||
fetchMLDataBatch,
|
||||
MLDATA_BATCH_SIZE,
|
||||
@@ -18,10 +12,7 @@ import {
|
||||
} from "./mldata-fetch.js";
|
||||
import type { EnteFile } from "./model/types.js";
|
||||
|
||||
// Reports one line of the dump's progress. `level` is "error" when the line
|
||||
// reports ML data that could not be read, and is left out for every other
|
||||
// line.
|
||||
export type ProgressCallback = (message: string, level?: "error") => void;
|
||||
export type ProgressCallback = (message: string) => void;
|
||||
|
||||
export interface MetadataBackupOptions {
|
||||
exif?: boolean;
|
||||
@@ -98,18 +89,16 @@ const extractExif = async (
|
||||
|
||||
// Dump every decrypted metadata layer the account holds into a directory tree
|
||||
// of plain JSON: account, per-collection, and per-file records including the
|
||||
// private and public magic metadata and the ML data, with the keywords of its
|
||||
// CLIP embedding in place of the embedding (see clip-keywords.ts). Collections
|
||||
// private and public magic metadata and (by default) the ML data. Collections
|
||||
// and files are enumerated from the library's cache, which the caller refreshes
|
||||
// first. Returns how many files have no ML data because their ML data request
|
||||
// failed or their ML data did not decrypt, gunzip or parse; they are still
|
||||
// first. Returns how many ML data requests failed; their files are still
|
||||
// written, with `mlDataError` in place of `mlData`.
|
||||
export const runMetadataBackup = async (
|
||||
lib: Library,
|
||||
client: Client,
|
||||
outDir: string,
|
||||
opts?: MetadataBackupOptions,
|
||||
): Promise<{ filesWithMLDataError: number }> => {
|
||||
): Promise<{ failedMLBatches: number }> => {
|
||||
const log = opts?.onProgress ?? (() => {});
|
||||
const wantExif = opts?.exif ?? false;
|
||||
|
||||
@@ -136,15 +125,7 @@ export const runMetadataBackup = async (
|
||||
const col = lib.getCollection(album.collectionID);
|
||||
if (!col) continue;
|
||||
|
||||
// The album's ID keeps the folders apart, so the name after it can be
|
||||
// cut short to fit in a file name.
|
||||
const prefix = `${col.id}-`;
|
||||
const dirName =
|
||||
prefix +
|
||||
shortenToBytes(
|
||||
sanitizeFileName(col.name, "unnamed"),
|
||||
MAX_NAME_BYTES - prefix.length,
|
||||
);
|
||||
const dirName = `${col.id}-${sanitizeFileName(col.name, "unnamed")}`;
|
||||
const colDir = join(outDir, "collections", dirName);
|
||||
mkdirSync(colDir, { recursive: true });
|
||||
|
||||
@@ -183,34 +164,26 @@ export const runMetadataBackup = async (
|
||||
}
|
||||
|
||||
// One failed request (retries exhausted) must not end the dump: its files
|
||||
// get the reason in `mlDataError` and the other batches go on. So does a
|
||||
// file whose ML data does not decrypt, gunzip or parse.
|
||||
// get the reason in `mlDataError` and the other batches go on.
|
||||
log("Fetching ML data (face detections, CLIP embeddings)...");
|
||||
const mlDataMap = new Map<number, MLData>();
|
||||
const mlDataErrors = new Map<number, string>();
|
||||
let failedMLBatches = 0;
|
||||
const fileIDs = [...fileKeys.keys()];
|
||||
for (let i = 0; i < fileIDs.length; i += MLDATA_BATCH_SIZE) {
|
||||
const batch = fileIDs.slice(i, i + MLDATA_BATCH_SIZE);
|
||||
try {
|
||||
const { payloads, errors } = await fetchMLDataBatch(
|
||||
const result = await fetchMLDataBatch(
|
||||
client.getApiClient(),
|
||||
batch,
|
||||
fileKeys,
|
||||
);
|
||||
for (const [id, payload] of payloads) mlDataMap.set(id, payload);
|
||||
for (const [id, err] of errors) {
|
||||
const reason = describeError(err);
|
||||
log(
|
||||
`ML data of file ${id} could not be read: ${reason}`,
|
||||
"error",
|
||||
);
|
||||
mlDataErrors.set(id, reason);
|
||||
}
|
||||
for (const [id, payload] of result) mlDataMap.set(id, payload);
|
||||
} catch (err) {
|
||||
const reason = err instanceof Error ? err.message : String(err);
|
||||
failedMLBatches++;
|
||||
log(
|
||||
`ML data request for ${batch.length} file(s) failed: ${reason}`,
|
||||
"error",
|
||||
);
|
||||
for (const id of batch) mlDataErrors.set(id, reason);
|
||||
}
|
||||
@@ -232,16 +205,8 @@ export const runMetadataBackup = async (
|
||||
if (file.pubMagicMetadata)
|
||||
fileMeta.pubMagicMetadata = file.pubMagicMetadata;
|
||||
|
||||
// The ML data without the CLIP embedding, and the keywords that
|
||||
// embedding gives.
|
||||
const ml = mlDataMap.get(file.id);
|
||||
if (ml) {
|
||||
const { clip: _clip, ...mlDataWithoutClip } = ml;
|
||||
fileMeta.mlData = mlDataWithoutClip;
|
||||
const isVideo = file.metadata.fileType === "video";
|
||||
const keywords = clipKeywords(ml, isVideo);
|
||||
if (keywords) fileMeta.keywords = keywords;
|
||||
}
|
||||
if (ml) fileMeta.mlData = ml;
|
||||
const mlError = mlDataErrors.get(file.id);
|
||||
if (mlError) fileMeta.mlDataError = mlError;
|
||||
|
||||
@@ -264,5 +229,5 @@ export const runMetadataBackup = async (
|
||||
}
|
||||
|
||||
log("Metadata backup complete.");
|
||||
return { filesWithMLDataError: mlDataErrors.size };
|
||||
return { failedMLBatches };
|
||||
};
|
||||
+24
-33
@@ -28,52 +28,43 @@ interface RawRemoteFileData {
|
||||
updatedAt?: number;
|
||||
}
|
||||
|
||||
// Decrypt one entry with its file key and gunzip the JSON payload. Throws why
|
||||
// when the entry does not decrypt, gunzip or parse, or parses to something
|
||||
// other than a JSON object, such as `null`, a number or an array.
|
||||
const decodeEntry = (entry: RawRemoteFileData, key: Uint8Array): MLData => {
|
||||
const decrypted = decryptBlob(
|
||||
fromBase64(entry.encryptedData),
|
||||
fromBase64(entry.decryptionHeader),
|
||||
key,
|
||||
);
|
||||
const json = gunzipSync(Buffer.from(decrypted)).toString("utf-8");
|
||||
const payload: unknown = JSON.parse(json);
|
||||
if (
|
||||
typeof payload !== "object" ||
|
||||
payload === null ||
|
||||
Array.isArray(payload)
|
||||
) {
|
||||
throw new Error("not a JSON object");
|
||||
// Decrypt one entry with its file key and gunzip the JSON payload. Returns
|
||||
// undefined when the key is unknown or the entry does not decrypt/parse, so one
|
||||
// corrupt file never fails a whole batch.
|
||||
const decodeEntry = (
|
||||
entry: RawRemoteFileData,
|
||||
key: Uint8Array | undefined,
|
||||
): MLData | undefined => {
|
||||
if (!key) return undefined;
|
||||
try {
|
||||
const decrypted = decryptBlob(
|
||||
fromBase64(entry.encryptedData),
|
||||
fromBase64(entry.decryptionHeader),
|
||||
key,
|
||||
);
|
||||
const json = gunzipSync(Buffer.from(decrypted)).toString("utf-8");
|
||||
return JSON.parse(json) as MLData;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
return payload as MLData;
|
||||
};
|
||||
|
||||
// Fetch ML data for up to `MLDATA_BATCH_SIZE` ids in a single request. This is
|
||||
// the unit the request pools schedule; callers with more ids split them into
|
||||
// batches and submit each batch to the pool. Returns the payload of each file
|
||||
// whose entry decodes and, in `errors`, why not for each file whose entry does
|
||||
// not, so one corrupt file never fails a whole batch. An entry for a file
|
||||
// whose key is not known, which quak did not ask for, is skipped.
|
||||
// batches and submit each batch to the pool.
|
||||
export const fetchMLDataBatch = async (
|
||||
api: ApiClient,
|
||||
fileIDs: number[],
|
||||
fileKeys: Map<number, Uint8Array>,
|
||||
): Promise<{ payloads: Map<number, MLData>; errors: Map<number, unknown> }> => {
|
||||
): Promise<Map<number, MLData>> => {
|
||||
const { data } = await api.postJSON<{ data: RawRemoteFileData[] }>(
|
||||
"/files/data/fetch",
|
||||
{ type: "mldata", fileIDs },
|
||||
);
|
||||
const payloads = new Map<number, MLData>();
|
||||
const errors = new Map<number, unknown>();
|
||||
const result = new Map<number, MLData>();
|
||||
for (const entry of data ?? []) {
|
||||
const key = fileKeys.get(entry.fileID);
|
||||
if (!key) continue;
|
||||
try {
|
||||
payloads.set(entry.fileID, decodeEntry(entry, key));
|
||||
} catch (err) {
|
||||
errors.set(entry.fileID, err);
|
||||
}
|
||||
const payload = decodeEntry(entry, fileKeys.get(entry.fileID));
|
||||
if (payload) result.set(entry.fileID, payload);
|
||||
}
|
||||
return { payloads, errors };
|
||||
return result;
|
||||
};
|
||||
+8
-35
@@ -24,9 +24,6 @@ export interface RetryOptions {
|
||||
sleep?: (ms: number) => Promise<void>;
|
||||
// Injected so the jitter is reproducible under test.
|
||||
random?: () => number;
|
||||
// Called before each wait with the error that failed attempt number
|
||||
// `attempt` and how long the wait is, for a caller that logs retries.
|
||||
onRetry?: (err: unknown, attempt: number, delayMs: number) => void;
|
||||
}
|
||||
|
||||
export type ResolvedRetryOptions = Required<RetryOptions>;
|
||||
@@ -38,7 +35,6 @@ export const DEFAULT_RETRY_OPTIONS: ResolvedRetryOptions = {
|
||||
sleep: (ms: number): Promise<void> =>
|
||||
new Promise((resolve) => setTimeout(resolve, ms)),
|
||||
random: Math.random,
|
||||
onRetry: () => {},
|
||||
};
|
||||
|
||||
// For a run nobody is watching, such as `quak backup` from cron. A request that
|
||||
@@ -59,18 +55,9 @@ export const resolveRetryOptions = (
|
||||
maxDelayMs: opts?.maxDelayMs ?? DEFAULT_RETRY_OPTIONS.maxDelayMs,
|
||||
sleep: opts?.sleep ?? DEFAULT_RETRY_OPTIONS.sleep,
|
||||
random: opts?.random ?? DEFAULT_RETRY_OPTIONS.random,
|
||||
onRetry: opts?.onRetry ?? DEFAULT_RETRY_OPTIONS.onRetry,
|
||||
});
|
||||
|
||||
// Transport failures: the request did not complete, for reasons below HTTP.
|
||||
//
|
||||
// The errnos are Node's. The binary `make install` builds runs under bun,
|
||||
// whose `fetch` rejects with a plain `Error` that has no `cause` and a `code`
|
||||
// of bun's own. Bun gives `ConnectionRefused` for a refused connection and for
|
||||
// a TLS handshake that failed, and `ConnectionRefused` or `FailedToOpenSocket`
|
||||
// for a name that does not resolve. For a connection reset or closed before
|
||||
// the response ended, it gives Node's `ECONNRESET`. These were found by
|
||||
// running that binary, built with bun 1.3.13, against listeners on 127.0.0.1.
|
||||
const TRANSPORT_CODES = new Set([
|
||||
"ECONNRESET",
|
||||
"ECONNABORTED",
|
||||
@@ -83,9 +70,6 @@ const TRANSPORT_CODES = new Set([
|
||||
"ENETUNREACH",
|
||||
"ENETRESET",
|
||||
"ENETDOWN",
|
||||
// Bun's.
|
||||
"ConnectionRefused",
|
||||
"FailedToOpenSocket",
|
||||
]);
|
||||
|
||||
// The subset of the above that can only be reported before a TCP connection
|
||||
@@ -103,12 +87,6 @@ const TRANSPORT_CODES = new Set([
|
||||
// ambiguity this set exists to exclude. They stay in `TRANSPORT_CODES`, so
|
||||
// they remain retryable for idempotent calls; only replay eligibility is
|
||||
// narrowed. See `isSafeToReplay`.
|
||||
//
|
||||
// Bun's `ConnectionRefused` and `FailedToOpenSocket` are absent too, so a
|
||||
// `POST` or `PUT` is not replayed on them. When a connection bun kept open from
|
||||
// an earlier request closes before any answer, bun sends the request again on
|
||||
// a new connection, and if that one fails it gives one of these codes although
|
||||
// the server may already have the request.
|
||||
const CONNECT_CODES = new Set(["ENOTFOUND", "EAI_AGAIN", "ECONNREFUSED"]);
|
||||
|
||||
// Collect every `code` in an error's cause chain. undici does not put the
|
||||
@@ -127,7 +105,7 @@ const causeCodes = (err: unknown): { codes: string[]; complete: boolean } => {
|
||||
return { codes, complete };
|
||||
};
|
||||
|
||||
// Does the error's cause chain hold the code of a transport failure, one of
|
||||
// Does the error's cause chain hold the errno of a transport failure, one of
|
||||
// `TRANSPORT_CODES`? `isRetryable` retries such an error, and `classify` in
|
||||
// `src/backup.ts` calls it transient.
|
||||
export const hasTransportCode = (err: unknown): boolean =>
|
||||
@@ -174,8 +152,7 @@ export const isRetryable = (err: unknown): boolean => {
|
||||
// deliberately literal. Demanding a recognised `cause` instead would
|
||||
// classify real network failures as permanent and fail backups that should
|
||||
// have succeeded; the cost of the imprecision is bounded by the attempt
|
||||
// count. Bun's fetch rejects with a plain `Error` instead, which is
|
||||
// recognised by its code (see `TRANSPORT_CODES`).
|
||||
// count.
|
||||
if (err instanceof TypeError) return true;
|
||||
return hasTransportCode(err);
|
||||
};
|
||||
@@ -193,13 +170,11 @@ export const isRetryable = (err: unknown): boolean => {
|
||||
//
|
||||
// Everything else is ambiguous. A 5xx proves the server did process the
|
||||
// request. A reset or a broken pipe can arrive after it was fully sent and
|
||||
// acted on. A routing errno can be delivered on an established socket. Bun's
|
||||
// `ConnectionRefused` and `FailedToOpenSocket` can come from a new connection
|
||||
// bun opened to resend a request that already went out on a connection it kept
|
||||
// open. A deadline says nothing at all about the server's state. So every code in the
|
||||
// cause chain must be one of `CONNECT_CODES`: one other code anywhere in the
|
||||
// chain is doubt, and doubt is not replayed. A chain longer than the walk is
|
||||
// doubt too: the links below the limit were never read.
|
||||
// acted on. A routing errno can be delivered on an established socket. A
|
||||
// deadline says nothing at all about the server's state. So every errno in the
|
||||
// cause chain must be a connect errno: one other errno anywhere in the chain
|
||||
// is doubt, and doubt is not replayed. A chain longer than the walk is doubt
|
||||
// too: the links below the limit were never read.
|
||||
export const isSafeToReplay = (err: unknown): boolean => {
|
||||
const { codes, complete } = causeCodes(err);
|
||||
return (
|
||||
@@ -239,9 +214,7 @@ export const withRetry = async <T>(
|
||||
// and its status intact.
|
||||
throw err;
|
||||
}
|
||||
const delayMs = backoffMs(attempt, policy);
|
||||
policy.onRetry(err, attempt, delayMs);
|
||||
await policy.sleep(delayMs);
|
||||
await policy.sleep(backoffMs(attempt, policy));
|
||||
}
|
||||
}
|
||||
};
|
||||
+38
-18
@@ -5,7 +5,7 @@ import type { Client } from "./client.js";
|
||||
import type { Library } from "./library/index.js";
|
||||
import { ApiError } from "./api/client.js";
|
||||
import { encryptBlob, toBase64 } from "./crypto/index.js";
|
||||
import type { Collection, EnteFile } from "./model/types.js";
|
||||
import type { EnteFile } from "./model/types.js";
|
||||
|
||||
// The server refuses a thumbnail larger than the one it already records for the
|
||||
// file (`thumbnail.size`, the encrypted size), so these encodings are tried
|
||||
@@ -49,10 +49,7 @@ export interface ThumbnailFixResult {
|
||||
reason?: string;
|
||||
}
|
||||
|
||||
// Reports one line of progress. `level` is "error" when the line reports a
|
||||
// file that could not be checked or fixed, and is left out for every other
|
||||
// line.
|
||||
export type ProgressCallback = (message: string, level?: "error") => void;
|
||||
export type ProgressCallback = (message: string) => void;
|
||||
|
||||
// Enumerate every file the library knows about, newest album first, each file
|
||||
// once, and report those whose server-side thumbnail is missing. "Missing" is
|
||||
@@ -114,7 +111,6 @@ export const listMissingThumbnails = async (
|
||||
} else {
|
||||
log(
|
||||
`[${album.name}] Could not check ${photo.title}: ${err instanceof Error ? err.message : String(err)} (not reported as missing)`,
|
||||
"error",
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -226,18 +222,16 @@ const encryptThumbnailWithin = (
|
||||
return undefined;
|
||||
};
|
||||
|
||||
// Regenerate and upload a thumbnail for each requested file. `found` holds the
|
||||
// record of each requested file that a collection holds, and the collection it
|
||||
// was found in. Each original is read through the library's content cache, by
|
||||
// `lib.originalOf(file)`. The new thumbnail is a JPEG, encrypted under the
|
||||
// file's own key. It is uploaded, then registered with the server. Each file is
|
||||
// Regenerate and upload a thumbnail for each requested file. Originals are read
|
||||
// through the library's content cache (`photo.original()`); the generated
|
||||
// thumbnail is JPEG-encoded, encrypted under the file's own key, and registered
|
||||
// with the server — the encrypt-and-upload path is unchanged. Each file is
|
||||
// resolved to one outcome (fixed / skipped / failed) and a failure on one file
|
||||
// never stops the others.
|
||||
export const fixMissingThumbnails = async (
|
||||
lib: Library,
|
||||
client: Client,
|
||||
fileIDs: number[],
|
||||
found: Map<number, { file: EnteFile; collection: Collection }>,
|
||||
onProgress?: ProgressCallback,
|
||||
): Promise<ThumbnailFixResult[]> => {
|
||||
const log = onProgress ?? (() => {});
|
||||
@@ -245,8 +239,31 @@ export const fixMissingThumbnails = async (
|
||||
const api = client.getApiClient();
|
||||
const { userID } = client.whoami();
|
||||
|
||||
// Resolve each requested fileID to its file record and owning album by
|
||||
// enumerating the library, each file taken from the first album that holds
|
||||
// it. The raw `EnteFile` carries the per-file key the thumbnail is
|
||||
// encrypted under, which the projected records deliberately do not.
|
||||
const wanted = new Set(fileIDs);
|
||||
const fileMap = new Map<
|
||||
number,
|
||||
{ file: EnteFile; collectionName: string }
|
||||
>();
|
||||
for (const album of lib.albums.list()) {
|
||||
for (const photo of album.photos.list()) {
|
||||
if (!wanted.has(photo.fileID) || fileMap.has(photo.fileID))
|
||||
continue;
|
||||
const file = lib.getFile(album.collectionID, photo.fileID);
|
||||
if (file) {
|
||||
fileMap.set(photo.fileID, {
|
||||
file,
|
||||
collectionName: album.name,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (const fileID of fileIDs) {
|
||||
const entry = found.get(fileID);
|
||||
const entry = fileMap.get(fileID);
|
||||
if (!entry) {
|
||||
results.push({
|
||||
fileID,
|
||||
@@ -258,8 +275,7 @@ export const fixMissingThumbnails = async (
|
||||
continue;
|
||||
}
|
||||
|
||||
const { file } = entry;
|
||||
const collectionName = entry.collection.name;
|
||||
const { file, collectionName } = entry;
|
||||
const title = file.metadata.title;
|
||||
|
||||
const skipReason = reasonToSkip(file, userID);
|
||||
@@ -277,10 +293,15 @@ export const fixMissingThumbnails = async (
|
||||
const maxSize = file.thumbnail.size!;
|
||||
|
||||
try {
|
||||
const photo = lib.photos.byID({ fileID });
|
||||
if (!photo) {
|
||||
throw new Error("file not present in the library cache");
|
||||
}
|
||||
|
||||
log(
|
||||
`[${collectionName}] Downloading ${title} for thumbnail generation...`,
|
||||
);
|
||||
const { path } = await lib.originalOf(file);
|
||||
const { path } = await photo.original();
|
||||
const fileBytes = new Uint8Array(readFileSync(path));
|
||||
|
||||
if (!isJpeg(fileBytes)) {
|
||||
@@ -325,7 +346,7 @@ export const fixMissingThumbnails = async (
|
||||
ciphertext.length,
|
||||
md5,
|
||||
);
|
||||
await api.putFile(url, ciphertext, md5);
|
||||
await api.putFile(url, ciphertext);
|
||||
await api.updateThumbnail(file.id, objectKey, toBase64(header));
|
||||
|
||||
log(`[${collectionName}] Thumbnail uploaded for ${title}`);
|
||||
@@ -338,7 +359,6 @@ export const fixMissingThumbnails = async (
|
||||
} catch (err) {
|
||||
log(
|
||||
`[${collectionName}] FAILED ${title}: ${err instanceof Error ? err.message : err}`,
|
||||
"error",
|
||||
);
|
||||
results.push({
|
||||
fileID,
|
||||
|
||||
+4
-72
@@ -780,7 +780,7 @@ describe("ApiClient timeouts", () => {
|
||||
|
||||
await client.getJSON("/a");
|
||||
await client.postJSON("/b", {});
|
||||
await client.putFile("https://s3.example/x", new Uint8Array([1]), "m");
|
||||
await client.putFile("https://s3.example/x", new Uint8Array([1]));
|
||||
await client.putJSON("/c", {});
|
||||
|
||||
for (const call of calls) {
|
||||
@@ -825,8 +825,7 @@ describe("ApiClient timeouts", () => {
|
||||
[
|
||||
"putFile",
|
||||
() => new Response(null, { status: 200 }),
|
||||
(c) =>
|
||||
c.putFile("https://s3.example/x", new Uint8Array([1]), "m"),
|
||||
(c) => c.putFile("https://s3.example/x", new Uint8Array([1])),
|
||||
],
|
||||
[
|
||||
"getFileStream",
|
||||
@@ -1050,7 +1049,7 @@ describe("ApiClient error typing", () => {
|
||||
const client = new ApiClient({ fetch, retry: noWait });
|
||||
|
||||
const err: unknown = await client
|
||||
.putFile("https://s3.example/obj", new Uint8Array([1, 2]), "m")
|
||||
.putFile("https://s3.example/obj", new Uint8Array([1, 2]))
|
||||
.catch((e: unknown) => e);
|
||||
|
||||
expect(err).toBeInstanceOf(ApiError);
|
||||
@@ -1068,11 +1067,7 @@ describe("ApiClient error typing", () => {
|
||||
);
|
||||
const client = new ApiClient({ fetch, retry: noWait });
|
||||
|
||||
await client.putFile(
|
||||
"https://s3.example/obj",
|
||||
new Uint8Array([1, 2]),
|
||||
"m",
|
||||
);
|
||||
await client.putFile("https://s3.example/obj", new Uint8Array([1, 2]));
|
||||
expect(calls).toHaveLength(2);
|
||||
});
|
||||
|
||||
@@ -1227,66 +1222,3 @@ describe("ApiClient non-idempotent requests", () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// `onRequest` hears of each request the client sends, as its response arrives
|
||||
// or as it fails. The CLI logs these with `-vv`. A retried request is reported
|
||||
// once for each attempt.
|
||||
describe("ApiClient onRequest", () => {
|
||||
it("reports each request with its status, or with the error it failed with", async () => {
|
||||
const refused = new TypeError("fetch failed");
|
||||
const { fetch } = scriptedFetch(
|
||||
refused,
|
||||
textResponse("unavailable", 503),
|
||||
jsonResponse({ collections: [] }),
|
||||
jsonResponse({}),
|
||||
streamResponse(new Uint8Array([1, 2, 3])),
|
||||
);
|
||||
const requests: unknown[] = [];
|
||||
const client = new ApiClient({
|
||||
fetch,
|
||||
retry: noWait,
|
||||
onRequest: (request) => requests.push(request),
|
||||
});
|
||||
|
||||
await client.getJSON("/collections/v2", { sinceTime: 0 });
|
||||
await client.postJSON("/users/logout", {});
|
||||
await readAll(await client.getFileStream(7));
|
||||
|
||||
const collections = "https://api.ente.io/collections/v2?sinceTime=0";
|
||||
expect(requests).toEqual([
|
||||
{ method: "GET", url: collections, error: refused },
|
||||
{ method: "GET", url: collections, status: 503 },
|
||||
{ method: "GET", url: collections, status: 200 },
|
||||
{
|
||||
method: "POST",
|
||||
url: "https://api.ente.io/users/logout",
|
||||
status: 200,
|
||||
},
|
||||
{
|
||||
method: "GET",
|
||||
url: "https://files.ente.io/?fileID=7",
|
||||
status: 200,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("reports a presigned upload without its query, which holds the upload's signature", async () => {
|
||||
const { fetch } = scriptedFetch(new Response(null, { status: 200 }));
|
||||
const requests: unknown[] = [];
|
||||
const client = new ApiClient({
|
||||
fetch,
|
||||
retry: noWait,
|
||||
onRequest: (request) => requests.push(request),
|
||||
});
|
||||
|
||||
await client.putFile(
|
||||
"https://s3.example/obj?X-Amz-Signature=abc",
|
||||
new Uint8Array([1, 2]),
|
||||
"m",
|
||||
);
|
||||
|
||||
expect(requests).toEqual([
|
||||
{ method: "PUT", url: "https://s3.example/obj", status: 200 },
|
||||
]);
|
||||
});
|
||||
});
|
||||
+2
-13
@@ -80,20 +80,13 @@ describe("ApiClient.putFile", () => {
|
||||
const client = new ApiClient({ fetch, authToken: "secret-tok" });
|
||||
const data = new Uint8Array([1, 2, 3, 4, 5]);
|
||||
|
||||
await client.putFile(
|
||||
"https://s3.example.com/presigned?sig=abc",
|
||||
data,
|
||||
"md5base64==",
|
||||
);
|
||||
await client.putFile("https://s3.example.com/presigned?sig=abc", data);
|
||||
|
||||
expect(calls[0]!.url).toBe("https://s3.example.com/presigned?sig=abc");
|
||||
expect(calls[0]!.init?.method).toBe("PUT");
|
||||
const headers = new Headers(calls[0]!.init?.headers as HeadersInit);
|
||||
expect(headers.get("Content-Type")).toBe("application/octet-stream");
|
||||
expect(headers.get("Content-Length")).toBe("5");
|
||||
// The server signs the MD5 into the URL; storage refuses the PUT
|
||||
// without it.
|
||||
expect(headers.get("Content-MD5")).toBe("md5base64==");
|
||||
// Must NOT leak auth headers to S3
|
||||
expect(headers.has("X-Auth-Token")).toBe(false);
|
||||
expect(headers.has("X-Client-Package")).toBe(false);
|
||||
@@ -106,11 +99,7 @@ describe("ApiClient.putFile", () => {
|
||||
const client = new ApiClient({ fetch });
|
||||
|
||||
await expect(
|
||||
client.putFile(
|
||||
"https://s3.example.com/bad",
|
||||
new Uint8Array(10),
|
||||
"md5base64==",
|
||||
),
|
||||
client.putFile("https://s3.example.com/bad", new Uint8Array(10)),
|
||||
).rejects.toThrow(/403/);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
* data this file encrypts with libsodium, as Ente's clients encrypt it. So
|
||||
* quak's own code fetches each original, decrypts it, checks it against the
|
||||
* hash its uploader recorded and unpacks a live photo, and quak's own retries
|
||||
* repeat a failed request. Three tests run `lib.backup()` instead, as a program
|
||||
* repeat a failed request. Two tests run `lib.backup()` instead, as a program
|
||||
* using quak as a library does.
|
||||
*
|
||||
* The account, user 42, has three albums:
|
||||
@@ -54,7 +54,6 @@ import {
|
||||
saveSession,
|
||||
type CliContext,
|
||||
} from "../../src/cli-commands.js";
|
||||
import { stderrLog } from "../../src/cli-log.js";
|
||||
import { run } from "../../src/cli-run.js";
|
||||
import { loadSession } from "../../src/cli-session.js";
|
||||
import { Client } from "../../src/client.js";
|
||||
@@ -86,17 +85,10 @@ const DAY = "2024/2024-03/2024-03-01";
|
||||
const TOAST = new TextEncoder().encode("a few seconds of the wedding toast");
|
||||
|
||||
// Ente's ML data for beach.jpg: a face it found, and the photo's CLIP
|
||||
// embedding, here that of a photo of a sunset (see clip-keywords.test.ts).
|
||||
// embedding.
|
||||
const ML_DATA = {
|
||||
face: { faces: [{ faceID: "100_1", score: 0.98 }] },
|
||||
clip: {
|
||||
embedding: JSON.parse(
|
||||
readFileSync(
|
||||
new URL("../clip-keywords-sunset.json", import.meta.url),
|
||||
"utf-8",
|
||||
),
|
||||
).embedding,
|
||||
},
|
||||
clip: { embedding: [0.5, 0.6, 0.7] },
|
||||
};
|
||||
|
||||
// Each file's ID, its owner, its original as uploaded, and the metadata its
|
||||
@@ -283,15 +275,13 @@ afterAll(() => {
|
||||
// Run `quak backup <dir>` as `bin/quak.ts` runs it, sending its requests
|
||||
// through `fetch`, and return its exit code and everything it printed on
|
||||
// stderr and on stdout. Its requests are retried as those of `quak backup`
|
||||
// are, except that a retry does not wait. A download is given up after
|
||||
// `downloadTimeoutMs` with no bytes received, 60 seconds unless given. Its
|
||||
// cache is beside `dir`, so each run of one directory finds the cache the run
|
||||
// before it left, as cron's runs find the user's.
|
||||
// are, except that a retry does not wait. Its cache is beside `dir`, so each
|
||||
// run of one directory finds the cache the run before it left, as cron's runs
|
||||
// find the user's.
|
||||
const backup = async (
|
||||
dir: string,
|
||||
opts: { verify?: boolean; json?: boolean } = {},
|
||||
fetch = server,
|
||||
downloadTimeoutMs?: number,
|
||||
): Promise<{ code: number; stderr: string; stdout: string }> => {
|
||||
let printed = "";
|
||||
const stderr = new Writable({
|
||||
@@ -311,24 +301,22 @@ const backup = async (
|
||||
const prompted = async (): Promise<never> => {
|
||||
throw new Error("quak backup prompted");
|
||||
};
|
||||
const log = stderrLog(stderr, "info");
|
||||
const ctx: CliContext = {
|
||||
stdout,
|
||||
log,
|
||||
stderr,
|
||||
sessionDir,
|
||||
cacheDir: `${dir}-cache`,
|
||||
loadSession: (path) =>
|
||||
loadSession(path, {
|
||||
fetch,
|
||||
retry: { ...UNATTENDED_RETRY_OPTIONS, sleep: async () => {} },
|
||||
downloadTimeoutMs,
|
||||
}),
|
||||
login: prompted,
|
||||
prompt: prompted,
|
||||
promptSecret: prompted,
|
||||
};
|
||||
const code = await new Promise<number>((resolve) => {
|
||||
void run(backupCommand(ctx, dir, opts), stdout, stderr, log, resolve);
|
||||
void run(backupCommand(ctx, dir, opts), stdout, stderr, resolve);
|
||||
});
|
||||
return { code, stderr: printed, stdout: output };
|
||||
};
|
||||
@@ -418,7 +406,7 @@ describe("the first backup of the account", () => {
|
||||
expect(downloads.sort()).toEqual([100, 200, 300]);
|
||||
});
|
||||
|
||||
// The tree under "The backup directory" in the README. Each original is at
|
||||
// The tree "Backup layout" in the README describes. Each original is at
|
||||
// its save path, `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, with
|
||||
// the extension of its title; the live photo is its image and its video,
|
||||
// each with the extension it had inside the ZIP, and a `.livephoto.json`
|
||||
@@ -493,10 +481,7 @@ describe("the first backup of the account", () => {
|
||||
},
|
||||
updationTime: TAKEN,
|
||||
});
|
||||
// The JSON holds the ML data without its CLIP embedding, and the
|
||||
// keywords the embedding gives in its place.
|
||||
expect(beach.mlData).toEqual({ face: ML_DATA.face });
|
||||
expect(beach.keywords).toEqual(["Sunset"]);
|
||||
expect(beach.mlData).toEqual(ML_DATA);
|
||||
expect(beach.imageMetadata.exif.Make.description).toBe("Canon");
|
||||
expect(beach.imageMetadata.exif.Model.description).toBe("EOS R5");
|
||||
|
||||
@@ -507,7 +492,6 @@ describe("the first backup of the account", () => {
|
||||
metadata: { title: "toast.mp4", fileType: "video" },
|
||||
});
|
||||
expect(toast).not.toHaveProperty("mlData");
|
||||
expect(toast).not.toHaveProperty("keywords");
|
||||
expect(toast).not.toHaveProperty("imageMetadata");
|
||||
expect(toast).not.toHaveProperty("imageMetadataError");
|
||||
|
||||
@@ -776,81 +760,6 @@ describe("quak backup", () => {
|
||||
},
|
||||
);
|
||||
|
||||
// The download of toast.mp4 fails as bun's `fetch` reports a failure, in
|
||||
// the binary `make install` builds: a plain `Error` with bun's own code
|
||||
// and message, and no cause. These are what bun gave for a refused
|
||||
// connection, a name that did not resolve and a reset connection. The
|
||||
// file is tried as often as under Node, and goes into `failures.json`
|
||||
// with that message, as a transient failure.
|
||||
it.each([
|
||||
[
|
||||
"ConnectionRefused",
|
||||
"Unable to connect. Is the computer able to access the url?",
|
||||
],
|
||||
["FailedToOpenSocket", "Was there a typo in the url or port?"],
|
||||
[
|
||||
"ECONNRESET",
|
||||
"The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()",
|
||||
],
|
||||
])(
|
||||
"records a download that failed with bun's %s in failures.json, as transient",
|
||||
async (bunCode, message) => {
|
||||
let attempts = 0;
|
||||
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||
if (downloadOf(input) === 200) {
|
||||
attempts++;
|
||||
throw Object.assign(new Error(message), {
|
||||
code: bunCode,
|
||||
errno: 0,
|
||||
});
|
||||
}
|
||||
return server(input, init);
|
||||
};
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
|
||||
const { code } = await backup(dir, {}, fetch);
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(attempts).toBe(UNATTENDED_RETRY_OPTIONS.attempts);
|
||||
const failures = readJSON(join(dir, "failures.json"));
|
||||
expect(failures.files["200"]).toMatchObject({
|
||||
classification: "transient",
|
||||
error: message,
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
// The download of toast.mp4 stalls: the server sends the headers and then
|
||||
// never a byte of the body. Each attempt is given up after 50 ms, quak's
|
||||
// own deadline for this run, with a `TimeoutError` whose message says
|
||||
// only that the download stalled. The file goes into `failures.json` as a
|
||||
// transient failure, as a request that ran out of time is retried.
|
||||
it("records a stalled download in failures.json, as transient", async () => {
|
||||
let attempts = 0;
|
||||
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||
if (downloadOf(input) === 200) {
|
||||
attempts++;
|
||||
return new Response(
|
||||
new ReadableStream<Uint8Array>({
|
||||
pull: () => new Promise<void>(() => {}),
|
||||
}),
|
||||
);
|
||||
}
|
||||
return server(input, init);
|
||||
};
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
|
||||
const { code } = await backup(dir, {}, fetch, 50);
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(attempts).toBe(UNATTENDED_RETRY_OPTIONS.attempts);
|
||||
const failures = readJSON(join(dir, "failures.json"));
|
||||
expect(failures.files["200"]).toMatchObject({
|
||||
classification: "transient",
|
||||
error: "download stalled: no bytes received for 50 ms",
|
||||
});
|
||||
});
|
||||
|
||||
// The name of Ente's API server does not resolve when quak fetches the ML
|
||||
// data, after every original has been backed up. No file has ML data
|
||||
// cached yet, so each goes into `failures.json` with the reason, as a
|
||||
@@ -890,133 +799,6 @@ describe("quak backup", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
// The server sends beach.jpg's ML data encrypted under a key that is not
|
||||
// the file's, so quak cannot decrypt it. The file's JSON has the reason
|
||||
// in place of the ML data, and the file goes into `failures.json` with
|
||||
// it, so the exit code is 1. The other two files have no ML data, which
|
||||
// is no failure.
|
||||
it("records a file whose ML data does not decrypt in its JSON and in failures.json, and exits 1", async () => {
|
||||
const otherKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const mlData = encryptStreamBlob(
|
||||
gzipSync(JSON.stringify(ML_DATA)),
|
||||
otherKey,
|
||||
);
|
||||
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||
if (new URL(String(input)).pathname !== "/files/data/fetch") {
|
||||
return server(input, init);
|
||||
}
|
||||
const data = [
|
||||
{
|
||||
fileID: BEACH.id,
|
||||
encryptedData: toBase64(mlData.ciphertext),
|
||||
decryptionHeader: toBase64(mlData.header),
|
||||
},
|
||||
];
|
||||
return new Response(JSON.stringify({ data }), {
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
};
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
|
||||
const { code, stderr } = await backup(dir, {}, fetch);
|
||||
|
||||
expect(code).toBe(1);
|
||||
const reason = "secretstream chunk authentication failed";
|
||||
const beachJSON = readJSON(join(dir, DAY, "2024-03-01.100.json"));
|
||||
expect(beachJSON).not.toHaveProperty("mlData");
|
||||
expect(beachJSON.mlDataError).toBe(reason);
|
||||
const failures = readJSON(join(dir, "failures.json"));
|
||||
expect(Object.keys(failures.files)).toEqual(["100"]);
|
||||
expect(failures.files["100"]).toMatchObject({
|
||||
fileID: 100,
|
||||
title: "beach.jpg",
|
||||
error: `ML data: ${reason}`,
|
||||
});
|
||||
expect(stderr).toContain(`beach.jpg (id 100): ML data: ${reason}\n`);
|
||||
for (const fileID of [200, 300]) {
|
||||
const json = readJSON(join(dir, DAY, `2024-03-01.${fileID}.json`));
|
||||
expect(json).not.toHaveProperty("mlData");
|
||||
expect(json).not.toHaveProperty("mlDataError");
|
||||
}
|
||||
});
|
||||
|
||||
// A program using quak as a library backs the account up, and another ML
|
||||
// data fetch, as one the library's refresh starts, fails while the backup
|
||||
// writes the JSON files. The backup's own fetch got beach.jpg's ML data
|
||||
// under a key that is not the file's; the fetch that fails leaves that
|
||||
// reason in place, so beach.jpg's JSON and `failures.json` still give it.
|
||||
it("keeps the reason in beach.jpg's JSON and in failures.json when an ML data fetch fails while lib.backup() writes the JSON files", async () => {
|
||||
const otherKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const mlData = encryptStreamBlob(
|
||||
gzipSync(JSON.stringify(ML_DATA)),
|
||||
otherKey,
|
||||
);
|
||||
let mlDataFetchFails = false;
|
||||
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||
if (new URL(String(input)).pathname !== "/files/data/fetch") {
|
||||
return server(input, init);
|
||||
}
|
||||
if (mlDataFetchFails) throw new Error("the connection was reset");
|
||||
const data = [
|
||||
{
|
||||
fileID: BEACH.id,
|
||||
encryptedData: toBase64(mlData.ciphertext),
|
||||
decryptionHeader: toBase64(mlData.header),
|
||||
},
|
||||
];
|
||||
return new Response(JSON.stringify({ data }), {
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
};
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
const lib = await Library.open({
|
||||
client: loadSession(join(sessionDir, "session.json"), {
|
||||
fetch,
|
||||
retry: { ...UNATTENDED_RETRY_OPTIONS, sleep: async () => {} },
|
||||
})!,
|
||||
cacheDirectory: `${dir}-cache`,
|
||||
downloadDirectory: dir,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
try {
|
||||
// The backup reads a file's cached ML data just before it writes
|
||||
// that file's JSON. The first time, a refresh starts an ML data
|
||||
// fetch whose request fails, and the backup goes on once that
|
||||
// fetch has failed.
|
||||
const forFile = lib.mldata.forFile.bind(lib.mldata);
|
||||
vi.spyOn(lib.mldata, "forFile").mockImplementationOnce(
|
||||
async (args) => {
|
||||
mlDataFetchFails = true;
|
||||
await lib.fresh();
|
||||
await vi.waitFor(
|
||||
() => expect(lib.status().lastMLError).toBeDefined(),
|
||||
{ timeout: 2000, interval: 5 },
|
||||
);
|
||||
return forFile(args);
|
||||
},
|
||||
);
|
||||
|
||||
const result = await lib.backup();
|
||||
|
||||
expect(result.failed).toBe(1);
|
||||
const reason = "secretstream chunk authentication failed";
|
||||
const beachJSON = readJSON(join(dir, DAY, "2024-03-01.100.json"));
|
||||
expect(beachJSON).not.toHaveProperty("mlData");
|
||||
expect(beachJSON.mlDataError).toBe(reason);
|
||||
const failures = readJSON(join(dir, "failures.json"));
|
||||
expect(Object.keys(failures.files)).toEqual(["100"]);
|
||||
expect(failures.files["100"]).toMatchObject({
|
||||
fileID: 100,
|
||||
title: "beach.jpg",
|
||||
error: `ML data: ${reason}`,
|
||||
});
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// Something changed beach.jpg on disk after it was backed up. A run with
|
||||
// --verify hashes every stored original, finds that one no longer matches
|
||||
// the hash Ente recorded, says so, and downloads it again; it put back the
|
||||
@@ -1117,9 +899,6 @@ describe("a backup whose lock is lost", () => {
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toBe(
|
||||
"Starting backup...\n" +
|
||||
"Fetched the files of 1 of 3 collections\n" +
|
||||
"Fetched the files of 2 of 3 collections\n" +
|
||||
"Fetched the files of 3 of 3 collections\n" +
|
||||
"Refreshing library...\n" +
|
||||
"Fetching original beach.jpg (100)...\n" +
|
||||
"Fetching original IMG_0002.HEIC (300)...\n" +
|
||||
|
||||
@@ -42,7 +42,6 @@ import {
|
||||
saveSession,
|
||||
type CliContext,
|
||||
} from "../../src/cli-commands.js";
|
||||
import { stderrLog } from "../../src/cli-log.js";
|
||||
import { loadSession } from "../../src/cli-session.js";
|
||||
import { Client, type ClientSnapshot } from "../../src/client.js";
|
||||
import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
|
||||
@@ -305,7 +304,7 @@ const context = (ente: FakeEnte): CliContext => {
|
||||
saveSession(join(root, "session"), ente.snapshot());
|
||||
return {
|
||||
stdout,
|
||||
log: stderrLog(stderr, "info"),
|
||||
stderr,
|
||||
sessionDir: join(root, "session"),
|
||||
cacheDir: join(root, "cache"),
|
||||
loadSession: (path) =>
|
||||
@@ -371,24 +370,6 @@ describe("quak backup with records it cannot decrypt", () => {
|
||||
expect(stderr.text).toContain(" Failed: 2\n");
|
||||
});
|
||||
|
||||
// The album is an error line. The last line is fatal and counts the files
|
||||
// and the album, so `-q`, which prints fatal lines only, still says why
|
||||
// the backup exits 1.
|
||||
it("logs the album at error, and ends with one fatal line counting the files and the album", async () => {
|
||||
const ctx = {
|
||||
...context(brokenAccount()),
|
||||
log: stderrLog(stderr, "error"),
|
||||
};
|
||||
expect(await backupCommand(ctx, backupDir(), {})).toBe(1);
|
||||
|
||||
expect(stderr.text).toMatch(
|
||||
/^FAILED album 2: could not be decrypted: /,
|
||||
);
|
||||
expect(stderr.text.split("\n").at(-2)).toBe(
|
||||
`quak: 2 file(s) and 1 album(s) failed; see ${join(backupDir(), "failures.json")} and the FAILED lines`,
|
||||
);
|
||||
});
|
||||
|
||||
it("names each album it cannot decrypt in the --json result", async () => {
|
||||
expect(await backup(brokenAccount(), { json: true })).toBe(1);
|
||||
|
||||
@@ -567,7 +548,7 @@ describe("the read commands with records quak cannot decrypt", () => {
|
||||
expect(await collectionsCommand(context(ente), {})).toBe(0);
|
||||
expect(stdout.text).toBe("1\talbum\tTrip\n");
|
||||
|
||||
expect(await filesCommand(context(ente), "1", {})).toBe(0);
|
||||
expect(await filesCommand(context(ente), { collection: "1" })).toBe(0);
|
||||
expect(stdout.text).toBe("10\timage\tphoto-10.jpg\n");
|
||||
});
|
||||
});
|
||||
+44
-202
@@ -52,7 +52,6 @@ import { tmpdir } from "node:os";
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
|
||||
import { runBackup, type BackupLibrary } from "../../src/backup.js";
|
||||
import { DISCOVER_CATEGORIES } from "../../src/clip-keyword-vectors.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import type { ContentSource } from "../../src/library/content.js";
|
||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||
@@ -192,12 +191,9 @@ class HostileClient extends MockClient {
|
||||
}
|
||||
|
||||
// A content source that writes byte buffers of the expected length and can be
|
||||
// told to fail one fileID's original, as a server error or as a 404, or to
|
||||
// write other bytes for it, to exercise per-file resilience.
|
||||
// told to fail one fileID's original, to exercise per-file resilience.
|
||||
interface StubSource extends ContentSource {
|
||||
failID?: number;
|
||||
notFoundID?: number;
|
||||
otherBytesID?: number;
|
||||
failThumbID?: number;
|
||||
originalCalls: number;
|
||||
}
|
||||
@@ -208,10 +204,8 @@ const stubSource = (): StubSource => {
|
||||
original: async ({ file: f, destination }) => {
|
||||
s.originalCalls++;
|
||||
if (s.failID === f.id) throw new Error("HTTP 500 from server");
|
||||
if (s.notFoundID === f.id) throw new Error("HTTP 404 from server");
|
||||
const size = SIZE_BY_ID[f.id] ?? 10;
|
||||
const fill = s.otherBytesID === f.id ? 2 : 0;
|
||||
writeFileSync(destination, Buffer.alloc(size, fill));
|
||||
writeFileSync(destination, Buffer.alloc(size));
|
||||
return { bytesWritten: size };
|
||||
},
|
||||
thumbnail: async ({ file: f, destination }) => {
|
||||
@@ -702,26 +696,17 @@ describe("lib.backup", () => {
|
||||
// A child that has already exited: its process ID is not running.
|
||||
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
|
||||
const leftover = `.quak-backup-2026-03-01.100.jpg-${exitedPID}-abc123.tmp`;
|
||||
// A live photo's image and video downloaded to replace a stored one,
|
||||
// before they were renamed into place.
|
||||
const replacement = [
|
||||
`.quak-${exitedPID}-abc789.heic`,
|
||||
`.quak-${exitedPID}-abc789.mov`,
|
||||
];
|
||||
// This test's own process stands in for a backup running at the same
|
||||
// time.
|
||||
const inProgress = `.quak-backup-2026-03-01.101.jpg-${process.pid}-def456.tmp`;
|
||||
for (const name of [leftover, ...replacement, inProgress]) {
|
||||
writeFileSync(join(day, name), "partial");
|
||||
}
|
||||
writeFileSync(join(day, leftover), "partial");
|
||||
writeFileSync(join(day, inProgress), "partial");
|
||||
const lib = await openLibrary(stubSource());
|
||||
|
||||
await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
const names = readdirSync(day);
|
||||
expect(names).not.toContain(leftover);
|
||||
expect(names).not.toContain(replacement[0]);
|
||||
expect(names).not.toContain(replacement[1]);
|
||||
expect(names).toContain(inProgress);
|
||||
lib.close();
|
||||
});
|
||||
@@ -947,30 +932,19 @@ describe("the backup lock", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// Also serves ML data: file 100 has `ML_PAYLOAD`, whose CLIP embedding is that
|
||||
// of a photo of a sunset (see clip-keywords.test.ts), and the other two have
|
||||
// none. While `mlError` is set, every ML data request fails with it.
|
||||
const SUNSET_EMBEDDING: number[] = JSON.parse(
|
||||
readFileSync(
|
||||
new URL("../clip-keywords-sunset.json", import.meta.url),
|
||||
"utf-8",
|
||||
),
|
||||
).embedding;
|
||||
const ML_PAYLOAD = {
|
||||
face: { faces: [] },
|
||||
clip: { embedding: SUNSET_EMBEDDING },
|
||||
};
|
||||
// Also serves ML data: file 100 has `ML_PAYLOAD`, the other two have none.
|
||||
// While `mlError` is set, every ML data request fails with it.
|
||||
const ML_PAYLOAD = { face: { faces: [] }, clip: { embedding: [0.5, 0.25] } };
|
||||
|
||||
class MLClient extends MockClient {
|
||||
mlError?: string;
|
||||
async fetchMLData(args: { fileIDs: number[] }): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}> {
|
||||
async fetchMLData(args: {
|
||||
fileIDs: number[];
|
||||
}): Promise<Map<number, MLData>> {
|
||||
if (this.mlError) throw new Error(this.mlError);
|
||||
const payloads = new Map<number, MLData>();
|
||||
if (args.fileIDs.includes(100)) payloads.set(100, ML_PAYLOAD);
|
||||
return { payloads, errors: new Map() };
|
||||
const result = new Map<number, MLData>();
|
||||
if (args.fileIDs.includes(100)) result.set(100, ML_PAYLOAD);
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -979,99 +953,16 @@ const fileJSON = (outDir: string, fileID: number): Record<string, unknown> =>
|
||||
JSON.parse(readFileSync(saved(outDir, `${fileID}.json`), "utf-8"));
|
||||
|
||||
describe("ML data in each file's JSON", () => {
|
||||
// The JSON holds the ML data without its CLIP embedding, and in its place
|
||||
// the keywords the embedding gives.
|
||||
it("writes a file's ML data and keywords, and no ML field for a file that has none", async () => {
|
||||
it("writes a file's ML data, and no ML field for a file that has none", async () => {
|
||||
const lib = await openLibrary(stubSource(), new MLClient());
|
||||
const outDir = join(root, "backup");
|
||||
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } });
|
||||
expect(fileJSON(outDir, 100).keywords).toEqual(["Sunset"]);
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
|
||||
expect(fileJSON(outDir, 101)).not.toHaveProperty("mlData");
|
||||
expect(fileJSON(outDir, 101)).not.toHaveProperty("mlDataError");
|
||||
expect(fileJSON(outDir, 101)).not.toHaveProperty("keywords");
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
// A file's ML data can hold faces and no CLIP embedding. Its JSON then has
|
||||
// the ML data and no keywords.
|
||||
it("writes no keywords for ML data without a CLIP embedding", async () => {
|
||||
const faces = { face: { faces: [{ faceID: "101_1", score: 0.9 }] } };
|
||||
|
||||
// Gives file 101 ML data holding only faces.
|
||||
class FacesClient extends MockClient {
|
||||
async fetchMLData(args: { fileIDs: number[] }): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}> {
|
||||
const payloads = new Map<number, MLData>();
|
||||
if (args.fileIDs.includes(101)) payloads.set(101, faces);
|
||||
return { payloads, errors: new Map() };
|
||||
}
|
||||
}
|
||||
const lib = await openLibrary(stubSource(), new FacesClient());
|
||||
const outDir = join(root, "backup");
|
||||
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(fileJSON(outDir, 101).mlData).toEqual(faces);
|
||||
expect(fileJSON(outDir, 101)).not.toHaveProperty("keywords");
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
// Ente's photos app keeps videos out of some Discover categories, Identity
|
||||
// among them. Here a photo and a video share one CLIP embedding, Identity's
|
||||
// own text embedding, so the photo gets the keyword Identity and the video
|
||||
// does not.
|
||||
it("keeps a video out of a category that leaves out videos", async () => {
|
||||
const identity = DISCOVER_CATEGORIES.find(
|
||||
(c) => c.title === "Identity",
|
||||
)!;
|
||||
expect(identity.showVideo).toBe(false);
|
||||
const toast = file(102, 1, "toast.mp4");
|
||||
const video = {
|
||||
...toast,
|
||||
metadata: { ...toast.metadata, fileType: "video" as const },
|
||||
};
|
||||
|
||||
// Serves beach.jpg and the video in the Vacation album, and gives
|
||||
// each ML data holding only that embedding.
|
||||
class VideoClient extends MockClient {
|
||||
override async filesSince(args: {
|
||||
collectionID: number;
|
||||
}): Promise<FilesPage> {
|
||||
const files =
|
||||
args.collectionID === 1
|
||||
? [file(100, 1, "beach.jpg"), video]
|
||||
: [];
|
||||
return { files, deleted: [], cursor: 1 };
|
||||
}
|
||||
async fetchMLData(args: { fileIDs: number[] }): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}> {
|
||||
const payloads = new Map<number, MLData>();
|
||||
for (const id of args.fileIDs) {
|
||||
payloads.set(id, {
|
||||
clip: { embedding: identity.textEmbedding },
|
||||
});
|
||||
}
|
||||
return { payloads, errors: new Map() };
|
||||
}
|
||||
}
|
||||
const lib = await openLibrary(stubSource(), new VideoClient());
|
||||
const outDir = join(root, "backup");
|
||||
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(fileJSON(outDir, 100).keywords).toContain("Identity");
|
||||
expect(fileJSON(outDir, 102)).toHaveProperty("keywords");
|
||||
expect(fileJSON(outDir, 102).keywords).not.toContain("Identity");
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
@@ -1087,7 +978,7 @@ describe("ML data in each file's JSON", () => {
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
// File 100's ML data was cached by the first run and is kept.
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } });
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
|
||||
expect(fileJSON(outDir, 100)).not.toHaveProperty("mlDataError");
|
||||
for (const fileID of [101, 200]) {
|
||||
expect(fileJSON(outDir, fileID).mlDataError).toBe(
|
||||
@@ -1120,7 +1011,7 @@ describe("ML data in each file's JSON", () => {
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(result.skipped).toBe(3);
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } });
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
|
||||
for (const fileID of [100, 101, 200]) {
|
||||
expect(fileJSON(outDir, fileID)).not.toHaveProperty("mlDataError");
|
||||
}
|
||||
@@ -1403,8 +1294,7 @@ describe("image metadata in each file's JSON", () => {
|
||||
});
|
||||
|
||||
// With `verify`, each original already stored is hashed, and one that does not
|
||||
// match the content hash its metadata records is downloaded again, and replaced
|
||||
// only by a download that matches.
|
||||
// match the content hash its metadata records is downloaded again.
|
||||
describe("backup with verify", () => {
|
||||
// MockClient's files, each recording the hash of what `stubSource` writes
|
||||
// for it, except diagram.png (200), which records none.
|
||||
@@ -1447,12 +1337,6 @@ describe("backup with verify", () => {
|
||||
const good = Buffer.alloc(SIZE_BY_ID[100]!);
|
||||
const corrupt = Buffer.alloc(SIZE_BY_ID[100]!, 1);
|
||||
|
||||
// The temporary files left in the date folder of the backup in `outDir`.
|
||||
const tempFiles = (outDir: string): string[] =>
|
||||
readdirSync(join(outDir, DAY)).filter((name) =>
|
||||
name.startsWith(".quak-"),
|
||||
);
|
||||
|
||||
it("leaves an original that matches its hash, and one with no hash, as they are", async () => {
|
||||
const { lib, source, outDir } = await backedUp();
|
||||
const calls = source.originalCalls;
|
||||
@@ -1497,16 +1381,13 @@ describe("backup with verify", () => {
|
||||
expect(log.filter((msg) => msg.startsWith("MISMATCH"))).toEqual([
|
||||
"MISMATCH original beach.jpg (100): its bytes do not match its content hash",
|
||||
]);
|
||||
expect(tempFiles(outDir)).toEqual([]);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
// A copy whose hash does not match is still mostly the photo, so the
|
||||
// backup keeps it until it holds a better one.
|
||||
it("keeps an original that does not match its hash when its download answers 404", async () => {
|
||||
it("records a failed download in failures.json, with the original removed", async () => {
|
||||
const { lib, source, outDir } = await backedUp();
|
||||
writeFileSync(saved(outDir, "100.jpg"), corrupt);
|
||||
source.notFoundID = 100;
|
||||
source.failID = 100;
|
||||
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: outDir,
|
||||
@@ -1519,18 +1400,18 @@ describe("backup with verify", () => {
|
||||
failed: 1,
|
||||
});
|
||||
expect(Object.keys(readLedger(outDir).files)).toEqual(["100"]);
|
||||
expect(readFileSync(saved(outDir, "100.jpg"))).toEqual(corrupt);
|
||||
expect(
|
||||
readlinkSync(join(outDir, "collections", "Vacation", "beach.jpg")),
|
||||
).toBe(linkTo("100.jpg"));
|
||||
expect(tempFiles(outDir)).toEqual([]);
|
||||
expect(existsSync(saved(outDir, "100.jpg"))).toBe(false);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it("keeps an original that does not match its hash when its download does not match either", async () => {
|
||||
const { lib, source, outDir } = await backedUp();
|
||||
writeFileSync(saved(outDir, "100.jpg"), corrupt);
|
||||
source.otherBytesID = 100;
|
||||
it("records in failures.json an original the cache puts back with the same bad bytes", async () => {
|
||||
const lib = await openLibrary(stubSource(), new HashedClient());
|
||||
const outDir = join(root, "backup");
|
||||
// The cache holds a bad copy, and a backup copies an original the
|
||||
// cache holds to its save path.
|
||||
const cached = await lib.photos.byID({ fileID: 100 })!.original();
|
||||
writeFileSync(cached.path, corrupt);
|
||||
await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: outDir,
|
||||
@@ -1543,39 +1424,10 @@ describe("backup with verify", () => {
|
||||
failed: 1,
|
||||
});
|
||||
expect(result.errors.map((e) => e.error)).toEqual([
|
||||
"the downloaded original does not match its content hash either",
|
||||
"the original put back does not match its content hash either",
|
||||
]);
|
||||
expect(Object.keys(readLedger(outDir).files)).toEqual(["100"]);
|
||||
expect(readFileSync(saved(outDir, "100.jpg"))).toEqual(corrupt);
|
||||
expect(tempFiles(outDir)).toEqual([]);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
// The cache can hold the same bad bytes, as a copy from it is not checked
|
||||
// as a download is, so the replacement is always downloaded.
|
||||
it("downloads the replacement even when the cache holds the original", async () => {
|
||||
const source = stubSource();
|
||||
const lib = await openLibrary(source, new HashedClient());
|
||||
const outDir = join(root, "backup");
|
||||
// The cache holds a bad copy, and a backup copies an original the
|
||||
// cache holds to its save path.
|
||||
const cached = await lib.photos.byID({ fileID: 100 })!.original();
|
||||
writeFileSync(cached.path, corrupt);
|
||||
await lib.backup({ downloadDirectory: outDir });
|
||||
const calls = source.originalCalls;
|
||||
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: outDir,
|
||||
verify: true,
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({
|
||||
downloaded: 1,
|
||||
mismatched: 1,
|
||||
failed: 0,
|
||||
});
|
||||
expect(source.originalCalls).toBe(calls + 1);
|
||||
expect(readFileSync(saved(outDir, "100.jpg"))).toEqual(good);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
@@ -1659,16 +1511,12 @@ describe("backup album folders", () => {
|
||||
writeFileSync(path, `original ${fileID}`);
|
||||
return { path };
|
||||
},
|
||||
fetchOriginal: async () => {
|
||||
throw new Error("no verify in this stand-in");
|
||||
},
|
||||
thumbnail: async () => {
|
||||
throw new Error("no thumbnails in this stand-in");
|
||||
},
|
||||
contentConcurrency: 5,
|
||||
fetchMLData: async () => {},
|
||||
mlData: async () => undefined,
|
||||
mlDataError: () => undefined,
|
||||
whoami: () => ({ email: "backup@example.com", userID: USER_ID }),
|
||||
});
|
||||
|
||||
@@ -1889,11 +1737,12 @@ describe("backup album folders", () => {
|
||||
expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 1 });
|
||||
});
|
||||
|
||||
// The extensions of these titles leave no room for the ID and the "a"
|
||||
// before them, so each is cut short to 250 bytes, which leaves one, for
|
||||
// the "a". File 1's extension is 303 bytes and file 2's 271. Neither
|
||||
// extension is only letters and digits, so both originals are saved with
|
||||
// `.bin`, a name that fits.
|
||||
// The extensions of these titles leave no room for the ID, so each is cut
|
||||
// short and the name before the ID keeps only the bytes that cut leaves
|
||||
// over. File 1's extension is 303 bytes and is cut to 251, which leaves
|
||||
// none. File 2's is 271 bytes and is cut to 250, since "€" takes three
|
||||
// bytes, which leaves one, for the "a". Neither extension is only letters
|
||||
// and digits, so both originals are saved with `.bin`, a name that fits.
|
||||
it("cuts the extension of a title short when it leaves no room for the ID", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf([
|
||||
@@ -1908,7 +1757,7 @@ describe("backup album folders", () => {
|
||||
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
const link = `a (1).${"L".repeat(249)}`;
|
||||
const link = ` (1).${"L".repeat(250)}`;
|
||||
const link2 = `a (2).${"€".repeat(83)}`;
|
||||
expect(Buffer.byteLength(link)).toBe(255);
|
||||
expect(Buffer.byteLength(link2)).toBe(255);
|
||||
@@ -1923,9 +1772,9 @@ describe("backup album folders", () => {
|
||||
|
||||
// Files 1 and 3 share a title, so each gets its ID before the
|
||||
// extension. File 2's title is too long, and its extension leaves no
|
||||
// room for the ID and the "x" before it. Were the ID put at the end of
|
||||
// file 2's title cut short, its link would have file 1's name, and the
|
||||
// link made second would replace the first.
|
||||
// room for the ID. Were the ID put at the end of file 2's title cut
|
||||
// short, its link would have file 1's name, and the link made second
|
||||
// would replace the first.
|
||||
it("keeps names unique when a title's extension leaves no room for the ID", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const shared = `x.${"Q".repeat(245)} (2)`;
|
||||
@@ -1945,8 +1794,8 @@ describe("backup album folders", () => {
|
||||
expect(result.failed).toBe(0);
|
||||
expect(tree(outDir)).toEqual([
|
||||
"Trip/",
|
||||
`Trip/ (2).${"Q".repeat(250)} -> ${linkTo("2.bin")}`,
|
||||
`Trip/x (1).${"Q".repeat(245)} (2) -> ${linkTo("1.bin")}`,
|
||||
`Trip/x (2).${"Q".repeat(249)} -> ${linkTo("2.bin")}`,
|
||||
`Trip/x (3).${"Q".repeat(245)} (2) -> ${linkTo("3.bin")}`,
|
||||
"Trip.json",
|
||||
]);
|
||||
@@ -2395,12 +2244,11 @@ describe("backup of live photos", () => {
|
||||
Buffer.from(IMAGE),
|
||||
);
|
||||
expect(readFileSync(video)).toEqual(Buffer.from(VIDEO));
|
||||
expect(readdirSync(join(outDir, DAY)).sort()).toEqual(stored);
|
||||
expect(tree(outDir)).toEqual(linked);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it("keeps both files of a live photo that does not match its hash when downloading it again fails", async () => {
|
||||
it("removes both files of a live photo that does not match its hash when downloading it again fails", async () => {
|
||||
const { file: live, body } = await asLivePhoto(
|
||||
file(500, 10, "IMG_0500.HEIC"),
|
||||
);
|
||||
@@ -2418,14 +2266,8 @@ describe("backup of live photos", () => {
|
||||
|
||||
expect(result).toMatchObject({ mismatched: 1, failed: 1 });
|
||||
expect(Object.keys(readLedger(outDir).files)).toEqual(["500"]);
|
||||
expect(readFileSync(saved(outDir, "500.heic"))).toEqual(
|
||||
Buffer.from(IMAGE),
|
||||
);
|
||||
expect(readFileSync(saved(outDir, "500.mov"), "utf-8")).toBe(
|
||||
"another few seconds of video",
|
||||
);
|
||||
expect(readdirSync(join(outDir, DAY)).sort()).toEqual(stored);
|
||||
expect(tree(outDir)).toEqual(linked);
|
||||
expect(existsSync(saved(outDir, "500.heic"))).toBe(false);
|
||||
expect(existsSync(saved(outDir, "500.mov"))).toBe(false);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
|
||||
+9
-89
@@ -1,20 +1,10 @@
|
||||
/**
|
||||
* Tests for the retry options `bin/quak.ts` loads the saved session with
|
||||
* (`quak backup` gets the unattended ones, every other command the default),
|
||||
* each with the callbacks that log its requests and retries, for `quak files`
|
||||
* taking its collection ID as an optional argument, for `quak get` and
|
||||
* `quak get-thumb` refusing `--collection`, and for `quak login` taking
|
||||
* `--server` and sending its first request to the server it names.
|
||||
* Tests for the retry options `bin/quak.ts` loads the saved session with:
|
||||
* `quak backup` gets the unattended ones, every other command the default.
|
||||
*
|
||||
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
|
||||
* replaced by one that records the options it is given and reports no session,
|
||||
* so the command stops before it makes any request. `quak login` reads no
|
||||
* session, so its test stubs `fetch` instead and records each request.
|
||||
*
|
||||
* The first load of the sources `bin/quak.ts` imports compiles them, and the
|
||||
* first `init()` starts libsodium. On a loaded host that takes longer than a
|
||||
* test's timeout, so both happen here, before the tests run. Each test's
|
||||
* `vi.resetModules()` then only runs the compiled sources again.
|
||||
* so the command stops before it makes any request.
|
||||
*/
|
||||
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
@@ -22,13 +12,8 @@ import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { afterAll, afterEach, describe, expect, it, vi } from "vitest";
|
||||
import type { ApiClientOptions } from "../../src/api/client.js";
|
||||
import "../../src/cli-commands.js";
|
||||
import { init } from "../../src/crypto/index.js";
|
||||
import "../../src/index.js";
|
||||
import { UNATTENDED_RETRY_OPTIONS } from "../../src/retry.js";
|
||||
|
||||
await init();
|
||||
|
||||
const loaded = vi.hoisted(() => [] as (ApiClientOptions | undefined)[]);
|
||||
|
||||
vi.mock("../../src/cli-session.js", () => ({
|
||||
@@ -41,27 +26,20 @@ vi.mock("../../src/cli-session.js", () => ({
|
||||
const argv = process.argv;
|
||||
const dir = mkdtempSync(join(tmpdir(), "quak-bin-test-"));
|
||||
|
||||
// Run `quak <args>` to completion and return what it wrote on stderr, which is
|
||||
// kept off the terminal. The exit is swallowed.
|
||||
const quak = async (...args: string[]): Promise<string> => {
|
||||
let written = "";
|
||||
vi.spyOn(process.stderr, "write").mockImplementation((text) => {
|
||||
written += String(text);
|
||||
return true;
|
||||
});
|
||||
// Run `quak <args>` to completion. The "Not logged in" message and the exit
|
||||
// are swallowed.
|
||||
const quak = async (...args: string[]): Promise<void> => {
|
||||
vi.spyOn(process.stderr, "write").mockImplementation(() => true);
|
||||
vi.spyOn(process, "exit").mockImplementation(() => undefined as never);
|
||||
process.argv = ["node", "quak", ...args];
|
||||
vi.resetModules();
|
||||
await import("../../bin/quak.js");
|
||||
return written;
|
||||
};
|
||||
|
||||
afterEach(() => {
|
||||
process.argv = argv;
|
||||
loaded.length = 0;
|
||||
vi.restoreAllMocks();
|
||||
vi.unstubAllGlobals();
|
||||
vi.unstubAllEnvs();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
@@ -81,71 +59,13 @@ describe("bin/quak.ts session loading", () => {
|
||||
attempts,
|
||||
baseDelayMs,
|
||||
maxDelayMs,
|
||||
onRetry: expect.any(Function),
|
||||
}),
|
||||
onRequest: expect.any(Function),
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("loads the session for another command with the default retry options", async () => {
|
||||
it("loads the session for another command with the default options", async () => {
|
||||
await quak("collections");
|
||||
expect(loaded).toEqual([
|
||||
{
|
||||
retry: { onRetry: expect.any(Function) },
|
||||
onRequest: expect.any(Function),
|
||||
},
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
// The command line takes `files` with or without a collection ID, and the
|
||||
// command runs as far as its session check either way (issue #211).
|
||||
describe("bin/quak.ts files", () => {
|
||||
it("runs without a collection ID", async () => {
|
||||
expect(await quak("files")).toMatch(/^Not logged in/);
|
||||
});
|
||||
|
||||
it("runs with a collection ID", async () => {
|
||||
expect(await quak("files", "7")).toMatch(/^Not logged in/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("bin/quak.ts get and get-thumb", () => {
|
||||
it.each(["get", "get-thumb"])(
|
||||
"%s refuses --collection as an unknown option",
|
||||
async (command) => {
|
||||
expect(await quak(command, "1", "--collection", "2")).toMatch(
|
||||
/^error: unknown option '--collection'/,
|
||||
);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
describe("bin/quak.ts login", () => {
|
||||
// The command refuses an address that is not a URL before it asks for
|
||||
// anything or sends any request.
|
||||
it("hands --server to the command", async () => {
|
||||
expect(await quak("login", "--server", "ente.example.com")).toMatch(
|
||||
/^Invalid server address "ente\.example\.com"/,
|
||||
);
|
||||
});
|
||||
|
||||
// `fetch` answers 404, so the login fails at its first request. A login
|
||||
// that succeeded would save its session in this user's real data
|
||||
// directory, since env-paths is not mocked here.
|
||||
it("sends its first request to the server --server names", async () => {
|
||||
const requests: string[] = [];
|
||||
vi.stubGlobal("fetch", async (input: string) => {
|
||||
const { origin, pathname } = new URL(input);
|
||||
requests.push(origin + pathname);
|
||||
return new Response("not found", { status: 404 });
|
||||
});
|
||||
vi.stubEnv("QUAK_EMAIL", "login@example.com");
|
||||
vi.stubEnv("QUAK_PASSWORD", "password");
|
||||
await quak("login", "--server", "https://ente.example.com");
|
||||
expect(requests).toEqual([
|
||||
"https://ente.example.com/users/srp/attributes",
|
||||
]);
|
||||
expect(loaded).toEqual([undefined]);
|
||||
});
|
||||
});
|
||||
+47
-970
File diff suppressed because it is too large.
Load diff
@@ -1,359 +0,0 @@
|
||||
/**
|
||||
* Tests for the CLI's log levels, set with `-v`/`--verbose` and `-q`/`--quiet`
|
||||
* (`src/cli-log.ts`, issue #212).
|
||||
*
|
||||
* Every line quak prints on stderr is a log line at one of six levels, from
|
||||
* the quietest: fatal, error, warning, info, debug, trace. A run prints the
|
||||
* lines at its own level and at every quieter one. The level is info unless
|
||||
* asked otherwise: `-v` raises it to debug, `-vv` to trace, and more repeats
|
||||
* stay at trace. `-q` lowers it to fatal, so a run prints only the line a
|
||||
* command prints as it fails with a non-zero exit code. stdout holds the
|
||||
* command's output at every level.
|
||||
*
|
||||
* Most tests run `bin/quak.ts`, as the smoke test does, against an account of
|
||||
* one album holding one photo: the saved session is in a temporary directory,
|
||||
* and `fetch` is a fake of the Ente server that serves the album and the photo
|
||||
* encrypted as Ente's clients encrypt them. Each run has a cache directory of
|
||||
* its own, so its refresh fetches the album's files, as on a first run.
|
||||
*/
|
||||
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import sodium from "libsodium-wrappers-sumo";
|
||||
import {
|
||||
afterAll,
|
||||
afterEach,
|
||||
beforeAll,
|
||||
describe,
|
||||
expect,
|
||||
it,
|
||||
vi,
|
||||
} from "vitest";
|
||||
|
||||
import { ApiClient } from "../../src/api/client.js";
|
||||
import { saveSession } from "../../src/cli-commands.js";
|
||||
import { logRefresh, logRequests, type Log } from "../../src/cli-log.js";
|
||||
import { init, toBase64 } from "../../src/crypto/index.js";
|
||||
import { encryptSecretbox, encryptStreamBlob } from "../ente-server.js";
|
||||
|
||||
// `bin/quak.ts` reads the saved session from the data directory env-paths
|
||||
// gives it, which here is a temporary directory made before the first run.
|
||||
const paths = vi.hoisted(() => ({ data: "" }));
|
||||
vi.mock("env-paths", () => ({ default: () => paths }));
|
||||
|
||||
const USER_ID = 42;
|
||||
const root = mkdtempSync(join(tmpdir(), "quak-log-test-"));
|
||||
const argv = process.argv;
|
||||
|
||||
class ExitCalled extends Error {}
|
||||
|
||||
// Run `quak <args>` to its exit, with a cache of its own, and return its exit
|
||||
// code and what it wrote on stdout and on stderr, which are kept off the
|
||||
// terminal. `process.exit` is stubbed to throw, so nothing runs after it, as
|
||||
// nothing does after a real exit.
|
||||
const quak = async (
|
||||
...args: string[]
|
||||
): Promise<{ code: number; stdout: string; stderr: string }> => {
|
||||
let stdout = "";
|
||||
let stderr = "";
|
||||
let code = NaN;
|
||||
vi.spyOn(process.stdout, "write").mockImplementation((text) => {
|
||||
stdout += String(text);
|
||||
return true;
|
||||
});
|
||||
vi.spyOn(process.stderr, "write").mockImplementation((text) => {
|
||||
stderr += String(text);
|
||||
return true;
|
||||
});
|
||||
vi.spyOn(process, "exit").mockImplementation((exitCode) => {
|
||||
code = Number(exitCode ?? 0);
|
||||
throw new ExitCalled();
|
||||
});
|
||||
const cache = mkdtempSync(join(root, "cache-"));
|
||||
process.argv = ["node", "quak", "--cache-dir", cache, ...args];
|
||||
vi.resetModules();
|
||||
await expect(import("../../bin/quak.js")).rejects.toBeInstanceOf(
|
||||
ExitCalled,
|
||||
);
|
||||
return { code, stdout, stderr };
|
||||
};
|
||||
|
||||
// Album 1, "Trips", of user 42, holding photo 100, "beach.jpg". The album's
|
||||
// key is encrypted with the account's master key, the photo's key with the
|
||||
// album's, and the photo's metadata with its own key. Only the session and
|
||||
// these two records are needed: the tests never log in.
|
||||
beforeAll(async () => {
|
||||
await init();
|
||||
await sodium.ready;
|
||||
const masterKey = sodium.crypto_secretbox_keygen();
|
||||
const keyPair = sodium.crypto_box_keypair();
|
||||
paths.data = join(root, "session");
|
||||
saveSession(paths.data, {
|
||||
email: "log@example.com",
|
||||
userID: USER_ID,
|
||||
token: "token",
|
||||
masterKey: toBase64(masterKey),
|
||||
secretKey: toBase64(keyPair.privateKey),
|
||||
publicKey: toBase64(keyPair.publicKey),
|
||||
});
|
||||
|
||||
const albumKey = sodium.crypto_secretbox_keygen();
|
||||
const albumKeyBox = encryptSecretbox(albumKey, masterKey);
|
||||
const name = encryptSecretbox(new TextEncoder().encode("Trips"), albumKey);
|
||||
const album = {
|
||||
id: 1,
|
||||
owner: { id: USER_ID },
|
||||
encryptedKey: toBase64(albumKeyBox.ciphertext),
|
||||
keyDecryptionNonce: toBase64(albumKeyBox.nonce),
|
||||
encryptedName: toBase64(name.ciphertext),
|
||||
nameDecryptionNonce: toBase64(name.nonce),
|
||||
type: "album",
|
||||
updationTime: 1,
|
||||
};
|
||||
const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const photoKeyBox = encryptSecretbox(photoKey, albumKey);
|
||||
const metadata = encryptStreamBlob(
|
||||
new TextEncoder().encode(
|
||||
JSON.stringify({
|
||||
title: "beach.jpg",
|
||||
fileType: 0,
|
||||
creationTime: 1000,
|
||||
modificationTime: 1000,
|
||||
}),
|
||||
),
|
||||
photoKey,
|
||||
);
|
||||
const photo = {
|
||||
id: 100,
|
||||
collectionID: 1,
|
||||
ownerID: USER_ID,
|
||||
encryptedKey: toBase64(photoKeyBox.ciphertext),
|
||||
keyDecryptionNonce: toBase64(photoKeyBox.nonce),
|
||||
metadata: {
|
||||
encryptedData: toBase64(metadata.ciphertext),
|
||||
decryptionHeader: toBase64(metadata.header),
|
||||
},
|
||||
// These tests fetch no original and no thumbnail.
|
||||
file: { decryptionHeader: "" },
|
||||
thumbnail: { decryptionHeader: "" },
|
||||
updationTime: 1,
|
||||
};
|
||||
|
||||
const json = (body: unknown): Response =>
|
||||
new Response(JSON.stringify(body), {
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
vi.stubGlobal("fetch", async (input: string) => {
|
||||
const { pathname } = new URL(input);
|
||||
if (pathname === "/collections/v2") {
|
||||
return json({ collections: [album] });
|
||||
}
|
||||
if (pathname === "/collections/v2/diff") {
|
||||
return json({ diff: [photo], hasMore: false });
|
||||
}
|
||||
return new Response("not found", { status: 404 });
|
||||
});
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
process.argv = argv;
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
vi.unstubAllGlobals();
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// What `quak files` writes on stdout at every level.
|
||||
const FILES = "100\timage\tbeach.jpg\n";
|
||||
|
||||
// What it writes on stderr at each level. Info is the refresh's progress, one
|
||||
// line for the one album whose files it fetched. Debug adds the start and end
|
||||
// of the refresh, and trace each request with the status of its answer.
|
||||
const INFO = "Fetched the files of 1 of 1 collections\n";
|
||||
const DEBUG = "Refresh started\n" + INFO + "Refresh done\n";
|
||||
const TRACE =
|
||||
"Refresh started\n" +
|
||||
"GET https://api.ente.io/collections/v2?sinceTime=0: HTTP 200\n" +
|
||||
"GET https://api.ente.io/collections/v2/diff?collectionID=1&sinceTime=0: HTTP 200\n" +
|
||||
INFO +
|
||||
"Refresh done\n";
|
||||
|
||||
describe("the log level", () => {
|
||||
it("is info by default", async () => {
|
||||
expect(await quak("files")).toEqual({
|
||||
code: 0,
|
||||
stdout: FILES,
|
||||
stderr: INFO,
|
||||
});
|
||||
});
|
||||
|
||||
it("is debug with -v", async () => {
|
||||
expect(await quak("-v", "files")).toEqual({
|
||||
code: 0,
|
||||
stdout: FILES,
|
||||
stderr: DEBUG,
|
||||
});
|
||||
});
|
||||
|
||||
it("is trace with -vv", async () => {
|
||||
expect(await quak("-vv", "files")).toEqual({
|
||||
code: 0,
|
||||
stdout: FILES,
|
||||
stderr: TRACE,
|
||||
});
|
||||
});
|
||||
|
||||
it("stays at trace with -vvv", async () => {
|
||||
expect((await quak("-vvv", "files")).stderr).toBe(TRACE);
|
||||
});
|
||||
|
||||
it("counts --verbose --verbose as -vv", async () => {
|
||||
expect((await quak("--verbose", "--verbose", "files")).stderr).toBe(
|
||||
TRACE,
|
||||
);
|
||||
});
|
||||
|
||||
// Both are options of the program, taken before or after the command's
|
||||
// name, as `--cache-dir` is.
|
||||
it("is set by an option after the command's name too", async () => {
|
||||
expect((await quak("files", "-v")).stderr).toBe(DEBUG);
|
||||
expect((await quak("files", "--json", "-vv")).stderr).toBe(TRACE);
|
||||
expect((await quak("files", "-q")).stderr).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
describe("-q", () => {
|
||||
it("prints nothing on stderr when the command works", async () => {
|
||||
expect(await quak("-q", "files")).toEqual({
|
||||
code: 0,
|
||||
stdout: FILES,
|
||||
stderr: "",
|
||||
});
|
||||
});
|
||||
|
||||
// No album holds file 999, so fixing it fails. The summary from
|
||||
// `--- Done ---` to the end of the `Failed files:` list is left out. The
|
||||
// line saying why the command failed is the fatal one, and it is printed.
|
||||
it("prints only the line a failing command exits with", async () => {
|
||||
const fix = ["helper", "fix-missing-thumbnails", "--file", "999"];
|
||||
expect(await quak("-q", ...fix)).toEqual({
|
||||
code: 1,
|
||||
stdout: "",
|
||||
stderr: `quak: 1 file(s) failed; see the "Failed files:" list, or the JSON with --json\n`,
|
||||
});
|
||||
});
|
||||
|
||||
it("is refused with -v, in one line, before the command runs", async () => {
|
||||
for (const args of [
|
||||
["-q", "-v", "files"],
|
||||
["files", "--verbose", "--quiet"],
|
||||
]) {
|
||||
const { code, stdout, stderr } = await quak(...args);
|
||||
expect(code).toBe(1);
|
||||
expect(stdout).toBe("");
|
||||
expect(stderr).toBe(
|
||||
"error: option '-q, --quiet' cannot be used with option '-v, --verbose'\n",
|
||||
);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// `--help` lists both options, and each command's help lists them too, since
|
||||
// each command takes them.
|
||||
describe("--help", () => {
|
||||
it("lists -v and -q for the program and for each command", async () => {
|
||||
for (const args of [["--help"], ["files", "--help"]]) {
|
||||
const { stdout } = await quak(...args);
|
||||
expect(stdout).toContain("-v, --verbose");
|
||||
expect(stdout).toContain("-q, --quiet");
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// The library writes no lines itself. Its layers report through callbacks,
|
||||
// and the CLI logs what they report.
|
||||
describe("what the library reports", () => {
|
||||
const collect = (): { log: Log; lines: string[] } => {
|
||||
const lines: string[] = [];
|
||||
return { log: (level, line) => lines.push(`${level}: ${line}`), lines };
|
||||
};
|
||||
|
||||
// A request answered 503 and retried: each attempt is a request, logged
|
||||
// at trace with its status, and the retry is logged at debug with how
|
||||
// long it waits and why.
|
||||
it("logs each request at trace and each retry at debug", async () => {
|
||||
const { log, lines } = collect();
|
||||
const answers = [
|
||||
new Response("unavailable", { status: 503 }),
|
||||
new Response("{}", {
|
||||
headers: { "content-type": "application/json" },
|
||||
}),
|
||||
];
|
||||
const client = new ApiClient({
|
||||
...logRequests(log, {
|
||||
sleep: async () => {},
|
||||
random: () => 0.5,
|
||||
}),
|
||||
fetch: async () => answers.shift()!,
|
||||
});
|
||||
|
||||
await client.getJSON("/collections/v2", { sinceTime: 0 });
|
||||
|
||||
const url = "https://api.ente.io/collections/v2?sinceTime=0";
|
||||
expect(lines).toEqual([
|
||||
`trace: GET ${url}: HTTP 503`,
|
||||
"debug: Attempt 1 failed, retrying in 250 ms: HTTP 503",
|
||||
`trace: GET ${url}: HTTP 200`,
|
||||
]);
|
||||
});
|
||||
|
||||
it("logs a request that failed with its error", async () => {
|
||||
const { log, lines } = collect();
|
||||
const client = new ApiClient({
|
||||
...logRequests(log, { attempts: 1 }),
|
||||
fetch: async () => {
|
||||
throw new TypeError("fetch failed", {
|
||||
cause: new Error("connect ECONNREFUSED 1.2.3.4:443"),
|
||||
});
|
||||
},
|
||||
});
|
||||
|
||||
await expect(client.getJSON("/collections/v2")).rejects.toThrow();
|
||||
|
||||
expect(lines).toEqual([
|
||||
"trace: GET https://api.ente.io/collections/v2: fetch failed: connect ECONNREFUSED 1.2.3.4:443",
|
||||
]);
|
||||
});
|
||||
|
||||
// Each operation is named in plain words.
|
||||
it("logs a refresh's progress at info, and its start and end at debug", () => {
|
||||
const { log, lines } = collect();
|
||||
const onProgress = logRefresh(log);
|
||||
|
||||
onProgress({ operation: "refresh", status: "started" });
|
||||
onProgress({
|
||||
operation: "refresh",
|
||||
status: "progress",
|
||||
fetched: 1,
|
||||
total: 2,
|
||||
});
|
||||
onProgress({
|
||||
operation: "refresh",
|
||||
status: "failed",
|
||||
error: "HTTP 503",
|
||||
});
|
||||
onProgress({ operation: "fetchMLData", status: "started" });
|
||||
onProgress({ operation: "fetchMLData", status: "done", fetched: 3 });
|
||||
|
||||
expect(lines).toEqual([
|
||||
"debug: Refresh started",
|
||||
"info: Fetched the files of 1 of 2 collections",
|
||||
"debug: Refresh failed: HTTP 503",
|
||||
"debug: ML data fetch started",
|
||||
"debug: ML data fetch done",
|
||||
]);
|
||||
});
|
||||
});
|
||||
@@ -41,14 +41,12 @@ import { beforeAll, afterAll, describe, expect, it, vi } from "vitest";
|
||||
import { init, toBase64, encryptBlob } from "../../src/crypto/index.js";
|
||||
import * as jpegJs from "jpeg-js";
|
||||
import { Client } from "../../src/client.js";
|
||||
import { DISCOVER_CATEGORIES } from "../../src/clip-keyword-vectors.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import {
|
||||
runMetadataBackup,
|
||||
type MetadataBackupOptions,
|
||||
} from "../../src/metadata-backup.js";
|
||||
import { backupMetadataCommand } from "../../src/cli-commands.js";
|
||||
import { stderrLog, type LogLevel } from "../../src/cli-log.js";
|
||||
import {
|
||||
buildAccount,
|
||||
buildMetaFetch,
|
||||
@@ -57,8 +55,8 @@ import {
|
||||
type MetaMockState,
|
||||
} from "../ente-server.js";
|
||||
|
||||
// One file per ML data request, so each file of the mock account is fetched in
|
||||
// a request of its own and one of them can fail on its own.
|
||||
// One file per ML data request, so the two files of the mock account are
|
||||
// fetched in two requests and one of them can fail on its own.
|
||||
vi.mock("../../src/mldata-fetch.js", async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import("../../src/mldata-fetch.js")>()),
|
||||
MLDATA_BATCH_SIZE: 1,
|
||||
@@ -67,21 +65,6 @@ vi.mock("../../src/mldata-fetch.js", async (importOriginal) => ({
|
||||
const TEST_EMAIL = "metabackup@example.com";
|
||||
const TEST_PASSWORD = "metapass";
|
||||
|
||||
// The CLIP embedding of a photo of a sunset (see clip-keywords.test.ts), which
|
||||
// the ML data of beach.jpg holds.
|
||||
const SUNSET_EMBEDDING: number[] = JSON.parse(
|
||||
readFileSync(
|
||||
new URL("../clip-keywords-sunset.json", import.meta.url),
|
||||
"utf-8",
|
||||
),
|
||||
).embedding;
|
||||
|
||||
// Identity, a Discover category of Ente's photos app that leaves out videos.
|
||||
const IDENTITY = DISCOVER_CATEGORIES.find((c) => c.title === "Identity")!;
|
||||
|
||||
// ML data holding faces and no CLIP embedding, which file 302 has.
|
||||
const FACES_ONLY = { face: { faces: [{ faceID: "face-def", score: 0.9 }] } };
|
||||
|
||||
let mock: MetaMockState;
|
||||
let testDir: string;
|
||||
|
||||
@@ -220,65 +203,6 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
|
||||
updationTime: 1710000000000000,
|
||||
};
|
||||
|
||||
// A file in coll 10 with the ML data given. `fileType` is Ente's number: 0
|
||||
// an image, 1 a video.
|
||||
const vacationFile = (
|
||||
id: number,
|
||||
title: string,
|
||||
fileType: number,
|
||||
mlData: object,
|
||||
) => {
|
||||
const fileKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const { ciphertext: encKey, nonce } = encryptSecretbox(fileKey, ck1);
|
||||
const meta = encryptStreamBlob(
|
||||
new TextEncoder().encode(
|
||||
JSON.stringify({
|
||||
title,
|
||||
fileType,
|
||||
creationTime: 1700000000000000,
|
||||
modificationTime: 1700000000000000,
|
||||
}),
|
||||
),
|
||||
fileKey,
|
||||
);
|
||||
const ml = encryptBlob(
|
||||
new Uint8Array(gzipSync(Buffer.from(JSON.stringify(mlData)))),
|
||||
fileKey,
|
||||
);
|
||||
return {
|
||||
raw: {
|
||||
id,
|
||||
collectionID: 10,
|
||||
ownerID: 42,
|
||||
encryptedKey: toBase64(encKey),
|
||||
keyDecryptionNonce: toBase64(nonce),
|
||||
metadata: {
|
||||
encryptedData: toBase64(meta.ciphertext),
|
||||
decryptionHeader: toBase64(meta.header),
|
||||
},
|
||||
file: {
|
||||
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
|
||||
},
|
||||
thumbnail: {
|
||||
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
|
||||
},
|
||||
updationTime: 1700000000000000,
|
||||
},
|
||||
mlData: {
|
||||
encryptedData: toBase64(ml.ciphertext),
|
||||
decryptionHeader: toBase64(ml.header),
|
||||
},
|
||||
};
|
||||
};
|
||||
// Files 300 and 301 in coll 10: a photo and a video whose ML data holds
|
||||
// only a CLIP embedding, Identity's own text embedding.
|
||||
const identityML = { clip: { embedding: IDENTITY.textEmbedding } };
|
||||
const identityPhoto = vacationFile(300, "passport.jpg", 0, identityML);
|
||||
const identityVideo = vacationFile(301, "passport.mp4", 1, identityML);
|
||||
// File 302 in coll 10: a photo whose ML data holds faces and no CLIP
|
||||
// embedding.
|
||||
const facesPhoto = vacationFile(302, "party.jpg", 0, FACES_ONLY);
|
||||
|
||||
// Encrypt ML data for file 100 (gzipped JSON, encrypted with file key)
|
||||
const mlPayload = JSON.stringify({
|
||||
face: {
|
||||
@@ -305,7 +229,7 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
|
||||
clip: {
|
||||
version: 1,
|
||||
client: "test",
|
||||
embedding: SUNSET_EMBEDDING,
|
||||
embedding: [0.5, 0.6, 0.7],
|
||||
},
|
||||
});
|
||||
const gzipped = gzipSync(Buffer.from(mlPayload));
|
||||
@@ -345,23 +269,12 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
|
||||
keyAttributes: account.keyAttributes,
|
||||
encryptedToken: account.encryptedToken,
|
||||
collections: [rawColl1, rawColl2],
|
||||
filesByCollection: {
|
||||
10: [
|
||||
rawFile1,
|
||||
identityPhoto.raw,
|
||||
identityVideo.raw,
|
||||
facesPhoto.raw,
|
||||
],
|
||||
20: [rawFile2],
|
||||
},
|
||||
filesByCollection: { 10: [rawFile1], 20: [rawFile2] },
|
||||
encryptedMLData: {
|
||||
100: {
|
||||
encryptedData: toBase64(mlCiphertext),
|
||||
decryptionHeader: toBase64(mlHeader),
|
||||
},
|
||||
300: identityPhoto.mlData,
|
||||
301: identityVideo.mlData,
|
||||
302: facesPhoto.mlData,
|
||||
},
|
||||
fileCiphertexts: { 100: encFileBody1 },
|
||||
};
|
||||
@@ -518,9 +431,7 @@ describe("quak backup-metadata", () => {
|
||||
expect(account.email).toBe(TEST_EMAIL);
|
||||
});
|
||||
|
||||
// The CLIP embedding is left out of the ML data, and the keywords it gives
|
||||
// are written in its place.
|
||||
it("fetches and decrypts ML data by default, with keywords for the CLIP embedding", async () => {
|
||||
it("fetches and decrypts ML data by default", async () => {
|
||||
const outDir = join(testDir, "ml-data");
|
||||
await runBackup(outDir);
|
||||
|
||||
@@ -540,41 +451,8 @@ describe("quak backup-metadata", () => {
|
||||
expect(fileMeta.mlData.face.faces[0].faceID).toBe("face-abc");
|
||||
expect(fileMeta.mlData.face.faces[0].score).toBeCloseTo(0.98);
|
||||
expect(fileMeta.mlData.face.faces[0].detection.box.x).toBeCloseTo(0.1);
|
||||
expect(fileMeta.mlData).not.toHaveProperty("clip");
|
||||
expect(fileMeta.keywords).toEqual(["Sunset"]);
|
||||
});
|
||||
|
||||
// Ente's photos app keeps videos out of some Discover categories, Identity
|
||||
// among them. Files 300 and 301, a photo and a video, share one CLIP
|
||||
// embedding, Identity's own text embedding, so the photo gets the keyword
|
||||
// Identity and the video does not.
|
||||
it("keeps a video out of a category that leaves out videos", async () => {
|
||||
expect(IDENTITY.showVideo).toBe(false);
|
||||
const outDir = join(testDir, "video-keywords");
|
||||
await runBackup(outDir);
|
||||
|
||||
const vacation = join(outDir, "collections", "10-Vacation");
|
||||
const fileJSON = (fileID: number) =>
|
||||
JSON.parse(readFileSync(join(vacation, `${fileID}.json`), "utf-8"));
|
||||
expect(fileJSON(300).keywords).toContain("Identity");
|
||||
expect(fileJSON(301)).toHaveProperty("keywords");
|
||||
expect(fileJSON(301).keywords).not.toContain("Identity");
|
||||
});
|
||||
|
||||
// A file's ML data can hold faces and no CLIP embedding. File 302's JSON
|
||||
// then has the ML data and no keywords.
|
||||
it("writes no keywords for ML data without a CLIP embedding", async () => {
|
||||
const outDir = join(testDir, "no-clip");
|
||||
await runBackup(outDir);
|
||||
|
||||
const fileMeta = JSON.parse(
|
||||
readFileSync(
|
||||
join(outDir, "collections", "10-Vacation", "302.json"),
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
expect(fileMeta.mlData).toEqual(FACES_ONLY);
|
||||
expect(fileMeta).not.toHaveProperty("keywords");
|
||||
expect(fileMeta.mlData.clip).toBeDefined();
|
||||
expect(fileMeta.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]);
|
||||
});
|
||||
|
||||
it("extracts EXIF from downloaded files when --exif is set", async () => {
|
||||
@@ -611,18 +489,15 @@ describe("quak backup-metadata", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
// Run the CLI command against `fetch`, the mock unless given, logging at
|
||||
// `level`, and return its exit code, stderr and output directory.
|
||||
const runCommand = async (
|
||||
fetch = buildMetaFetch(mock),
|
||||
level: LogLevel = "info",
|
||||
) => {
|
||||
describe("quak backup-metadata when an ML data request fails", () => {
|
||||
// Run the CLI command against the mock and return its exit code, stderr
|
||||
// and output directory.
|
||||
const runCommand = async (failMLDataFor?: number) => {
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: {
|
||||
fetch,
|
||||
fetch: buildMetaFetch(mock, failMLDataFor),
|
||||
retry: { sleep: async () => {} },
|
||||
},
|
||||
});
|
||||
@@ -631,10 +506,7 @@ describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
const code = await backupMetadataCommand(
|
||||
{
|
||||
stdout: { write: () => true },
|
||||
log: stderrLog(
|
||||
{ write: (text: string) => (stderr += text) },
|
||||
level,
|
||||
),
|
||||
stderr: { write: (text: string) => (stderr += text) },
|
||||
sessionDir: testDir,
|
||||
cacheDir: mkdtempSync(join(testDir, "cache-")),
|
||||
loadSession: () => client,
|
||||
@@ -645,16 +517,11 @@ describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
return { code, stderr, outDir };
|
||||
};
|
||||
|
||||
// The failed request is logged at error, so it is the first line printed
|
||||
// at that level.
|
||||
it("writes every file, marks the failed batch's files, and exits 1", async () => {
|
||||
const { code, stderr, outDir } = await runCommand(
|
||||
buildMetaFetch(mock, 200),
|
||||
"error",
|
||||
);
|
||||
const { code, stderr, outDir } = await runCommand(200);
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toMatch(/^ML data request for 1 file\(s\) failed: /);
|
||||
expect(stderr).toContain("ML data request for 1 file(s) failed");
|
||||
|
||||
const ok = JSON.parse(
|
||||
readFileSync(
|
||||
@@ -662,7 +529,7 @@ describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
expect(ok.keywords).toEqual(["Sunset"]);
|
||||
expect(ok.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]);
|
||||
expect(ok.mlDataError).toBeUndefined();
|
||||
|
||||
const failed = JSON.parse(
|
||||
@@ -676,53 +543,6 @@ describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
expect(failed.mlDataError).toContain("500");
|
||||
});
|
||||
|
||||
// The server sends ML data for diagram.png encrypted under a key that is
|
||||
// not the file's, so quak cannot decrypt it. That is logged at error, and
|
||||
// the last line is fatal, so `-q`, which prints fatal lines only, still
|
||||
// says why the command exits 1.
|
||||
it("writes the reason for ML data that does not decrypt, and exits 1", async () => {
|
||||
const otherKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const { header, ciphertext } = encryptBlob(
|
||||
new Uint8Array(gzipSync(Buffer.from("{}"))),
|
||||
otherKey,
|
||||
);
|
||||
const { code, stderr, outDir } = await runCommand(
|
||||
buildMetaFetch({
|
||||
...mock,
|
||||
encryptedMLData: {
|
||||
...mock.encryptedMLData,
|
||||
200: {
|
||||
encryptedData: toBase64(ciphertext),
|
||||
decryptionHeader: toBase64(header),
|
||||
},
|
||||
},
|
||||
}),
|
||||
"error",
|
||||
);
|
||||
|
||||
expect(code).toBe(1);
|
||||
const reason = "secretstream chunk authentication failed";
|
||||
expect(stderr).toBe(
|
||||
`ML data of file 200 could not be read: ${reason}\n` +
|
||||
`quak: the ML data of 1 file(s) could not be read; see mlDataError in their JSON in ${outDir}\n`,
|
||||
);
|
||||
const failed = JSON.parse(
|
||||
readFileSync(
|
||||
join(outDir, "collections", "20-__Work", "200.json"),
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
expect(failed.mlData).toBeUndefined();
|
||||
expect(failed.mlDataError).toBe(reason);
|
||||
const ok = JSON.parse(
|
||||
readFileSync(
|
||||
join(outDir, "collections", "10-Vacation", "100.json"),
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
expect(ok.keywords).toEqual(["Sunset"]);
|
||||
});
|
||||
|
||||
it("exits 0 when every ML data request succeeds", async () => {
|
||||
const { code } = await runCommand();
|
||||
expect(code).toBe(0);
|
||||
|
||||
+47
-178
@@ -3,25 +3,17 @@
|
||||
* issue #36, issue #52).
|
||||
*
|
||||
* The `collections`, `files`, `get`, and `get-thumb` commands must answer for
|
||||
* current server state, not the local cache. `get`, `get-thumb` and `files`
|
||||
* without a collection ID force a `Library.fresh()` round-trip before they
|
||||
* read. The stand-in library below serves no cached record until `fresh()` has
|
||||
* been awaited, so a helper that read without refreshing would come back empty
|
||||
* and fail here. `collections` asks the library for the collection list alone
|
||||
* (`freshCollections()`, issue #210). `files` with a collection ID asks for
|
||||
* that list, then for that one collection's files (`freshFiles()`, issue
|
||||
* #217). Neither may wait for a refresh, which would fetch the files of every
|
||||
* collection. On an empty cache, `get` and `get-thumb` ask for that list, then
|
||||
* for one collection's files at a time, until they find the file (issue #222).
|
||||
* `helper fix-missing-thumbnails --file` does the same until it finds every
|
||||
* file it names, reading the most recently updated collection first (issue
|
||||
* #223).
|
||||
* current server state, not the local cache, so each helper forces a
|
||||
* `Library.fresh()` round-trip before it reads. The stand-in library below
|
||||
* serves nothing until `fresh()` has been awaited, so a helper that read
|
||||
* without refreshing would come back empty and fail here.
|
||||
*
|
||||
* `collections` and `files` also list in the order the library gives them, not
|
||||
* `collections` and `files` also list in the library's enumeration order
|
||||
* (`listCollections`/`listFiles`) — the order the pre-library CLI printed — not
|
||||
* the albums/photos projection's newest-first order. The fixtures are seeded in
|
||||
* an order that a newest-first sort would rearrange, so a regression to the
|
||||
* projection order would fail here too. Field values still come from the raw
|
||||
* metadata via `cli-output.ts`.
|
||||
* an enumeration order that a newest-first sort would rearrange, so a
|
||||
* regression to the projection order would fail here too. Field values still
|
||||
* come from the raw metadata via `cli-output.ts`.
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
@@ -30,10 +22,10 @@ import {
|
||||
freshCollections,
|
||||
freshFiles,
|
||||
freshFile,
|
||||
freshFilesByID,
|
||||
type FreshReadLibrary,
|
||||
} from "../../src/cli-read.js";
|
||||
import { fileListRow } from "../../src/cli-output.js";
|
||||
import type { Photo } from "../../src/library/index.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
|
||||
const collection = (id: number, updationTime: number): Collection => ({
|
||||
@@ -67,15 +59,11 @@ const file = (
|
||||
updationTime: creationTime,
|
||||
});
|
||||
|
||||
// A library that reveals its cached records only after `fresh()` has been
|
||||
// awaited, and serves them in the enumeration order it was given.
|
||||
// `freshCollections()` and `freshFiles()` answer in that order without a
|
||||
// refresh.
|
||||
// A library that reveals its records only after `fresh()` has been awaited, and
|
||||
// serves them in the enumeration order it was given. `photos.byID` returns a
|
||||
// stand-in `Photo` carrying just the fileID the helper passes through.
|
||||
class FakeLibrary implements FreshReadLibrary {
|
||||
freshCalls = 0;
|
||||
freshCollectionsCalls = 0;
|
||||
// The ID of each collection `freshFiles()` was asked for.
|
||||
freshFilesCalls: number[] = [];
|
||||
private refreshed = false;
|
||||
|
||||
constructor(
|
||||
@@ -89,20 +77,14 @@ class FakeLibrary implements FreshReadLibrary {
|
||||
return {};
|
||||
}
|
||||
|
||||
async freshCollections(): Promise<Collection[]> {
|
||||
this.freshCollectionsCalls++;
|
||||
return this.collections;
|
||||
}
|
||||
|
||||
async freshFiles(collection: Collection): Promise<EnteFile[]> {
|
||||
this.freshFilesCalls.push(collection.id);
|
||||
return this.files.filter((f) => f.collectionID === collection.id);
|
||||
}
|
||||
|
||||
listCollections(): Collection[] {
|
||||
return this.refreshed ? this.collections : [];
|
||||
}
|
||||
|
||||
getCollection(id: number): Collection | undefined {
|
||||
return this.listCollections().find((c) => c.id === id);
|
||||
}
|
||||
|
||||
listFiles(collectionID: number): EnteFile[] {
|
||||
return this.refreshed
|
||||
? this.files.filter((f) => f.collectionID === collectionID)
|
||||
@@ -113,10 +95,18 @@ class FakeLibrary implements FreshReadLibrary {
|
||||
if (!this.refreshed) return undefined;
|
||||
return this.files.find((f) => f.id === fileID);
|
||||
}
|
||||
|
||||
photos = {
|
||||
byID: ({ fileID }: { fileID: number }): Photo | undefined => {
|
||||
if (!this.refreshed) return undefined;
|
||||
if (!this.files.some((f) => f.id === fileID)) return undefined;
|
||||
return { fileID } as unknown as Photo;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
||||
it("freshCollections reads the collection list alone, in enumeration order", async () => {
|
||||
it("freshCollections refreshes first, then lists in enumeration order", async () => {
|
||||
// Enumeration order 2, 1, 3; a newest-first sort would be 3, 2, 1.
|
||||
const lib = new FakeLibrary(
|
||||
[collection(2, 200), collection(1, 300), collection(3, 100)],
|
||||
@@ -124,8 +114,7 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
||||
);
|
||||
const rows = await freshCollections(lib);
|
||||
|
||||
expect(lib.freshCollectionsCalls).toBe(1);
|
||||
expect(lib.freshCalls).toBe(0);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(rows.map((c) => c.id)).toEqual([2, 1, 3]);
|
||||
// The projection's newest-first order is a different sequence, so this
|
||||
// is not accidentally that order.
|
||||
@@ -136,24 +125,18 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
||||
expect(rows.map((c) => c.id)).not.toEqual(newestFirst);
|
||||
});
|
||||
|
||||
it("freshFiles with a collection ID reads that collection's files alone and keeps their order and raw fields", async () => {
|
||||
// Given in the order 10, 11, 12; creationTimes ascending, so a
|
||||
it("freshFiles refreshes first, lists in enumeration order, keeps raw fields", async () => {
|
||||
// Enumeration order by id 10, 11, 12; creationTimes ascending, so a
|
||||
// newest-first sort would reverse them.
|
||||
const files = [
|
||||
file(10, 1, 1_700_000_000_000_000),
|
||||
file(11, 1, 1_700_000_000_000_001),
|
||||
file(12, 1, 1_700_000_000_000_002),
|
||||
file(20, 2, 1_700_000_000_000_003),
|
||||
];
|
||||
const lib = new FakeLibrary(
|
||||
[collection(1, 100), collection(2, 200)],
|
||||
files,
|
||||
);
|
||||
const lib = new FakeLibrary([collection(1, 100)], files);
|
||||
|
||||
const rows = await freshFiles(lib, 1);
|
||||
expect(lib.freshCalls).toBe(0);
|
||||
expect(lib.freshCollectionsCalls).toBe(1);
|
||||
expect(lib.freshFilesCalls).toEqual([1]);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(rows?.map((f) => f.id)).toEqual([10, 11, 12]);
|
||||
|
||||
// Field values come from raw metadata: microsecond creationTime and the
|
||||
@@ -183,144 +166,30 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
// Without a collection ID, `files` lists every file in the account once
|
||||
// (issue #211). File 11 is in collections 1 and 2; collection 2 is
|
||||
// enumerated first, so 11 is listed once, with collection 2's record.
|
||||
it("freshFiles without a collection ID lists every file once, from the first collection holding it", async () => {
|
||||
const files = [
|
||||
file(10, 1, 1_700_000_000_000_000),
|
||||
file(11, 1, 1_700_000_000_000_001),
|
||||
file(20, 2, 1_700_000_000_000_002),
|
||||
file(11, 2, 1_700_000_000_000_001),
|
||||
];
|
||||
const lib = new FakeLibrary(
|
||||
[collection(2, 100), collection(1, 200)],
|
||||
files,
|
||||
);
|
||||
|
||||
const rows = await freshFiles(lib);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(rows?.map((f) => [f.id, f.collectionID])).toEqual([
|
||||
[20, 2],
|
||||
[11, 2],
|
||||
[10, 1],
|
||||
]);
|
||||
});
|
||||
|
||||
it("freshFiles returns undefined for an unknown collection", async () => {
|
||||
const lib = new FakeLibrary([collection(1, 100)], []);
|
||||
const rows = await freshFiles(lib, 999);
|
||||
expect(lib.freshCalls).toBe(0);
|
||||
expect(lib.freshFilesCalls).toEqual([]);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(rows).toBeUndefined();
|
||||
});
|
||||
|
||||
// Three collections, each holding one file.
|
||||
const threeCollections = (): FakeLibrary =>
|
||||
new FakeLibrary(
|
||||
[collection(1, 100), collection(2, 200), collection(3, 300)],
|
||||
[
|
||||
file(10, 1, 1_700_000_000_000_000),
|
||||
file(20, 2, 1_700_000_000_000_001),
|
||||
file(30, 3, 1_700_000_000_000_002),
|
||||
],
|
||||
it("freshFile refreshes first, then resolves the photo and its raw record", async () => {
|
||||
const f = file(10, 1, 1_700_000_000_000_000);
|
||||
const lib = new FakeLibrary([collection(1, 100)], [f]);
|
||||
|
||||
const resolved = await freshFile(lib, 10);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(resolved?.photo.fileID).toBe(10);
|
||||
expect(resolved?.file.metadata.title).toBe("file-10.jpg");
|
||||
expect(resolved?.file.metadata.creationTime).toBe(
|
||||
1_700_000_000_000_000,
|
||||
);
|
||||
|
||||
it("freshFile on an empty cache reads each collection's files in the server's order until one holds the file", async () => {
|
||||
const lib = threeCollections();
|
||||
|
||||
const found = await freshFile(lib, 20);
|
||||
expect(lib.freshCalls).toBe(0);
|
||||
expect(lib.freshCollectionsCalls).toBe(1);
|
||||
expect(lib.freshFilesCalls).toEqual([1, 2]);
|
||||
expect(found?.metadata.title).toBe("file-20.jpg");
|
||||
expect(found?.metadata.creationTime).toBe(1_700_000_000_000_001);
|
||||
});
|
||||
|
||||
// The stand-in's cache holds what the server holds once `fresh()` has run.
|
||||
it("freshFile on a cache that holds the collections refreshes, then looks the file up in the cache", async () => {
|
||||
const lib = threeCollections();
|
||||
await lib.fresh();
|
||||
|
||||
const found = await freshFile(lib, 30);
|
||||
expect(lib.freshCalls).toBe(2);
|
||||
expect(lib.freshCollectionsCalls).toBe(0);
|
||||
expect(lib.freshFilesCalls).toEqual([]);
|
||||
expect(found?.collectionID).toBe(3);
|
||||
});
|
||||
|
||||
it("freshFile on an empty cache returns undefined for an unknown file, after reading every collection's files", async () => {
|
||||
const lib = threeCollections();
|
||||
|
||||
const found = await freshFile(lib, 404);
|
||||
expect(lib.freshCalls).toBe(0);
|
||||
expect(lib.freshFilesCalls).toEqual([1, 2, 3]);
|
||||
expect(found).toBeUndefined();
|
||||
});
|
||||
|
||||
// The server lists collections 1, 2 and 3, and collection 3 was updated
|
||||
// most recently. File 11 is in collections 1 and 2.
|
||||
const fileInTwoCollections = (): FakeLibrary =>
|
||||
new FakeLibrary(
|
||||
[collection(1, 100), collection(2, 200), collection(3, 300)],
|
||||
[
|
||||
file(10, 1, 1_700_000_000_000_000),
|
||||
file(11, 1, 1_700_000_000_000_001),
|
||||
file(11, 2, 1_700_000_000_000_001),
|
||||
file(20, 2, 1_700_000_000_000_002),
|
||||
file(30, 3, 1_700_000_000_000_003),
|
||||
],
|
||||
);
|
||||
|
||||
it("freshFilesByID on an empty cache reads the most recently updated collection's files first, until it has found every file", async () => {
|
||||
const lib = fileInTwoCollections();
|
||||
|
||||
const found = await freshFilesByID(lib, [11, 20]);
|
||||
expect(lib.freshCalls).toBe(0);
|
||||
expect(lib.freshCollectionsCalls).toBe(1);
|
||||
expect(lib.freshFilesCalls).toEqual([3, 2]);
|
||||
expect(found.get(11)?.collection.id).toBe(2);
|
||||
expect(found.get(11)?.file.collectionID).toBe(2);
|
||||
expect(found.get(20)?.collection.id).toBe(2);
|
||||
});
|
||||
|
||||
// Collection 1 was updated last, though collections 2 and 3 have higher
|
||||
// IDs. File 11 is in collections 1 and 2.
|
||||
it("freshFilesByID on an empty cache reads the most recently updated collection first, though its ID is the lowest", async () => {
|
||||
const lib = new FakeLibrary(
|
||||
[collection(1, 400), collection(2, 200), collection(3, 300)],
|
||||
[
|
||||
file(11, 1, 1_700_000_000_000_001),
|
||||
file(11, 2, 1_700_000_000_000_001),
|
||||
],
|
||||
);
|
||||
|
||||
const found = await freshFilesByID(lib, [11]);
|
||||
expect(lib.freshFilesCalls).toEqual([1]);
|
||||
expect(found.get(11)?.collection.id).toBe(1);
|
||||
expect(found.get(11)?.file.collectionID).toBe(1);
|
||||
});
|
||||
|
||||
// The stand-in's cache holds what the server holds once `fresh()` has run.
|
||||
it("freshFilesByID on a cache that holds the collections refreshes, then finds each file in the most recently updated collection that holds it", async () => {
|
||||
const lib = fileInTwoCollections();
|
||||
await lib.fresh();
|
||||
|
||||
const found = await freshFilesByID(lib, [11, 404]);
|
||||
expect(lib.freshCalls).toBe(2);
|
||||
expect(lib.freshCollectionsCalls).toBe(0);
|
||||
expect(lib.freshFilesCalls).toEqual([]);
|
||||
expect(found.get(11)?.collection.id).toBe(2);
|
||||
expect(found.get(11)?.file.collectionID).toBe(2);
|
||||
expect(found.has(404)).toBe(false);
|
||||
});
|
||||
|
||||
it("freshFilesByID on an empty cache reads every collection's files when a file is in none of them", async () => {
|
||||
const lib = fileInTwoCollections();
|
||||
|
||||
const found = await freshFilesByID(lib, [10, 404]);
|
||||
expect(lib.freshFilesCalls).toEqual([3, 2, 1]);
|
||||
expect(found.get(10)?.collection.id).toBe(1);
|
||||
expect(found.has(404)).toBe(false);
|
||||
it("freshFile returns undefined for an unknown file", async () => {
|
||||
const lib = new FakeLibrary([collection(1, 100)], []);
|
||||
const resolved = await freshFile(lib, 404);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(resolved).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -6,7 +6,6 @@ import { PassThrough } from "node:stream";
|
||||
import { describe, it, expect } from "vitest";
|
||||
|
||||
import { ApiError } from "../../src/api/client.js";
|
||||
import { stderrLog } from "../../src/cli-log.js";
|
||||
import { run } from "../../src/cli-run.js";
|
||||
|
||||
// A stream whose written text is kept in `text`; writes finish at once, so
|
||||
@@ -24,13 +23,7 @@ const runToExit = async (
|
||||
const stdout = collector();
|
||||
const stderr = collector();
|
||||
const code = await new Promise<number>((resolve) => {
|
||||
void run(
|
||||
command,
|
||||
stdout.stream,
|
||||
stderr.stream,
|
||||
stderrLog(stderr.stream, "info"),
|
||||
resolve,
|
||||
);
|
||||
void run(command, stdout.stream, stderr.stream, resolve);
|
||||
});
|
||||
return { code, stdout: stdout.text(), stderr: stderr.text() };
|
||||
};
|
||||
|
||||
@@ -47,35 +47,6 @@ describe("Client.toJSON", () => {
|
||||
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
|
||||
});
|
||||
|
||||
it("round-trips a snapshot that names a server", () => {
|
||||
const snapshot = {
|
||||
...validSnapshot(),
|
||||
apiOrigin: "https://ente.example.com",
|
||||
};
|
||||
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
|
||||
});
|
||||
|
||||
it("names the server apiOptions gave fromJSON", async () => {
|
||||
const apiOrigin = "https://ente.example.com";
|
||||
const urls: string[] = [];
|
||||
const client = Client.fromJSON(validSnapshot(), {
|
||||
apiOrigin,
|
||||
fetch: async (input) => {
|
||||
urls.push(String(input));
|
||||
return new Response(JSON.stringify({ collections: [] }), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
},
|
||||
});
|
||||
expect(client.toJSON().apiOrigin).toBe(apiOrigin);
|
||||
|
||||
await client.listCollections();
|
||||
expect(urls).toEqual([
|
||||
"https://ente.example.com/collections/v2?sinceTime=0",
|
||||
]);
|
||||
});
|
||||
|
||||
it("throws instead of emitting a snapshot without a token", () => {
|
||||
const client = Client.fromJSON(validSnapshot());
|
||||
client.getApiClient().clearAuthToken();
|
||||
@@ -106,10 +77,6 @@ describe("Client.fromJSON", () => {
|
||||
["publicKey", undefined],
|
||||
["publicKey", "not base64!"],
|
||||
["publicKey", shortKey],
|
||||
["apiOrigin", 7],
|
||||
["apiOrigin", ""],
|
||||
["apiOrigin", "ente.example.com"],
|
||||
["apiOrigin", "ftp://ente.example.com"],
|
||||
])("rejects %s = %j, naming the field", (field, value) => {
|
||||
const snapshot: Record<string, unknown> = { ...validSnapshot() };
|
||||
snapshot[field] = value;
|
||||
@@ -118,30 +85,6 @@ describe("Client.fromJSON", () => {
|
||||
);
|
||||
});
|
||||
|
||||
// The token belongs to the snapshot's server, so that server wins over
|
||||
// the one in `apiOptions`.
|
||||
it("sends requests to the snapshot's server", async () => {
|
||||
const urls: string[] = [];
|
||||
const client = Client.fromJSON(
|
||||
{ ...validSnapshot(), apiOrigin: "https://ente.example.com/" },
|
||||
{
|
||||
apiOrigin: "https://other.example.com",
|
||||
fetch: async (input) => {
|
||||
urls.push(String(input));
|
||||
return new Response(JSON.stringify({ collections: [] }), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
await client.listCollections();
|
||||
expect(urls).toEqual([
|
||||
"https://ente.example.com/collections/v2?sinceTime=0",
|
||||
]);
|
||||
});
|
||||
|
||||
it.each([null, "a string", 42])("rejects a non-object %j", (value) => {
|
||||
expect(() => Client.fromJSON(value)).toThrow(
|
||||
/^Invalid session data: not a JSON object/,
|
||||
|
||||
@@ -1,109 +0,0 @@
|
||||
{
|
||||
"photo": "https://upload.wikimedia.org/wikipedia/commons/d/db/Sunset_off_the_Massachusetts_coast_-_NOAA.jpg",
|
||||
"sha256": "be62f3289c9bf7b75af68cd6978021d54e8a531d5d4cc493f6e9fbf5e1ab74d3",
|
||||
"embedding": [
|
||||
-0.0021123015, -0.04437615, -0.062354308, 0.006249992, 0.022141185,
|
||||
0.011159019, 0.01227392, -0.00030532264, 0.03186437, 0.09196922,
|
||||
0.039111234, -0.037084445, -0.032962628, -0.019820206, -0.013141825,
|
||||
0.0047183544, 0.017137334, 0.046581306, 0.0047622235, 0.04982521,
|
||||
0.023128154, 0.032080866, 0.037709303, 0.019241406, 0.0032100384,
|
||||
-0.014275843, 0.049337693, 0.025953151, 0.014263995, -0.0062328726,
|
||||
-0.040504098, -0.015083303, -0.008123875, -0.006233639, -0.05352683,
|
||||
-0.0048083956, 0.0763486, -0.013155067, 0.025114346, 0.022016674,
|
||||
0.0001588023, -0.010341572, -0.0037518279, -0.019181008, 0.04549337,
|
||||
-0.0010725886, 0.004636866, 0.017177915, -0.0117317885, 0.0026097358,
|
||||
0.030738998, 0.013158572, -0.08856403, -0.0121391015, 0.011477229,
|
||||
-0.0033104012, 0.035524756, -0.018607879, -0.0071739843, -0.031906743,
|
||||
0.0056713386, -0.0044195866, 0.00292471, 0.00061686034, -0.043075223,
|
||||
0.09040427, 0.001522537, -0.011445652, 0.009609886, -0.0053201183,
|
||||
-0.0057452377, 0.00972853, -0.0024397348, -0.021255083, 0.015893811,
|
||||
-0.006020548, -0.09428023, 0.014277583, -0.017975299, -0.011878315,
|
||||
0.011825521, 0.00014203975, -0.0020437292, -0.05305343, -0.024840152,
|
||||
-0.011336872, -0.002796541, -0.00036171873, 0.004530499, 0.044486605,
|
||||
0.021602718, -0.01292857, 0.024530916, 0.019559948, -0.039928272,
|
||||
-0.0681174, 0.049101, 0.02461532, 0.0365531, -0.04070358, -0.011011939,
|
||||
-0.0027188526, -0.020370876, 0.012904856, 0.043157287, -0.051658224,
|
||||
-0.3726012, 0.019549074, -0.0499252, -0.04800559, -0.011493859,
|
||||
0.02893256, 0.060341954, 0.0013929688, -0.051283866, -0.024286803,
|
||||
0.044124287, -0.005995806, -0.062817656, 0.007369614, -0.021470634,
|
||||
-0.028147545, 0.00411659, 0.04234773, 0.03787647, 0.008579682,
|
||||
-0.038568925, -0.027498025, -0.028416378, 0.0022012168, 0.010906191,
|
||||
-0.013745787, 0.018267838, 0.013531097, -0.061559845, 0.004682517,
|
||||
-0.005993303, 0.0028113858, -0.0028286583, 0.0026029332, -0.06693403,
|
||||
-0.0072381413, -0.02419584, 0.0037009937, -0.0011296495, -0.030310456,
|
||||
0.0005999609, 0.009945354, 0.024915686, 0.038672045, 0.023009432,
|
||||
-0.05093051, -0.09101409, -0.06686167, -0.06211481, -0.022600474,
|
||||
0.0024902367, -0.0067171026, -0.016769778, -0.0032803628, 0.008623083,
|
||||
-0.0016110836, 0.014925073, 0.033811048, 0.012829449, -0.052575592,
|
||||
0.03728281, -0.0051201633, 0.021238064, -0.031086605, -0.0043402985,
|
||||
0.25054494, -0.028396713, -0.010725523, -0.023445267, 0.021751683,
|
||||
0.003130971, 0.012270548, -0.014836475, 0.014466885, 0.0021478906,
|
||||
-0.088153616, 0.015356315, 0.039639276, -0.040283255, 0.014923924,
|
||||
-0.029303031, -0.004475201, 0.047299225, -0.06726505, -0.012491676,
|
||||
-0.002080582, 0.03506341, 0.04146718, 0.0074114986, -0.009582633,
|
||||
0.006623749, -0.05454898, -0.0034387247, -0.014780821, 0.03698328,
|
||||
-0.034441486, 0.22032379, -0.021792699, 0.008001907, 0.0341019,
|
||||
0.0025784625, -0.0071689896, -0.014151225, -0.050547987, -0.030372301,
|
||||
-0.03288388, -0.042987447, -0.0033656165, 0.021502906, -0.0030374376,
|
||||
-0.024413, -0.03506603, 0.030891743, 0.0069994032, 0.004887366,
|
||||
0.030707076, 0.0065348223, 0.0008240555, 0.016809713, -0.024151647,
|
||||
-0.0040458064, 0.044567548, 0.03908212, -0.061782435, 0.0026002831,
|
||||
0.02003242, -0.0115451, -0.045135964, -0.0663542, 0.005240975,
|
||||
0.0010777109, -0.009276002, -0.011919081, 0.027654119, -0.013634551,
|
||||
0.0129154315, -0.013195512, 0.040571272, -0.099290505, -0.04656693,
|
||||
-0.20979854, 0.04181004, -0.06769166, -0.04121946, 0.018260444,
|
||||
0.005617506, -0.15694222, 0.032472793, 0.014189598, 0.008345103,
|
||||
-0.041118417, 0.0074527003, -0.011494104, 0.004906474, -0.0027100458,
|
||||
0.0009994594, -0.041061398, -0.045274835, 0.015187793, 0.034275036,
|
||||
-0.02813289, 0.02098019, -0.015043198, 0.03716667, 0.013542642,
|
||||
-0.008104835, 0.014965486, -0.027978558, 0.021163663, -0.006942737,
|
||||
-0.034969814, -0.038961593, 0.016802894, -0.012318175, -0.047821593,
|
||||
-0.01933216, 0.0064432374, 0.0322358, -0.020437784, 0.02659447,
|
||||
-0.03022022, -0.019443465, -0.094178624, -0.017009107, 0.03860671,
|
||||
-0.00023695534, -0.033089854, -0.014677462, 0.012356399, 0.018040499,
|
||||
-0.19360113, 0.011710817, -0.019871691, 0.020124774, 0.024948105,
|
||||
-0.024573864, -0.028349072, -0.041731454, 0.032646254, 0.015098165,
|
||||
0.0018447012, -0.01129152, -0.02481687, 0.036048, 0.20797811,
|
||||
0.04317757, -0.017239755, 0.033431605, -0.011384422, 0.0099174455,
|
||||
-0.19739223, -0.008901062, -0.056426458, 0.014591864, 0.0044474266,
|
||||
0.008209326, -0.012241819, 0.042553645, -0.021552453, 0.008611734,
|
||||
0.048944127, -0.013872049, 0.019179989, 0.008183264, -0.053223096,
|
||||
-0.027327595, -0.0033542586, -0.012353858, -0.010941209, 0.010212546,
|
||||
0.029714134, -0.011385593, -0.0013529479, 0.014505385, 0.01148663,
|
||||
0.012613636, -0.0074430504, -0.014008101, -0.028146038, -0.042778917,
|
||||
0.02143165, 0.016812252, -0.06286215, 0.02085532, -0.035128277,
|
||||
0.0241282, 0.01397908, -0.034992814, 0.013153766, 0.0068236147,
|
||||
0.013594184, -0.06859393, 0.051849406, 0.029675119, -0.0037699796,
|
||||
-0.00013216575, 0.02916647, 0.031859048, -0.015006051, 0.0001439966,
|
||||
0.0026584554, -0.0046948595, 0.039754897, 0.034873154, 0.0012870439,
|
||||
-0.011516082, -0.074089095, 0.019241294, 0.00076195505, 0.0027334464,
|
||||
-0.036206458, -0.0011109262, -0.048838835, -0.00204685, 0.01179991,
|
||||
0.0068136845, 0.009852495, 0.017308988, 0.03159545, 0.058803353,
|
||||
0.038116634, 0.008836143, -0.010460637, -0.010994099, -0.0012926942,
|
||||
-0.048370842, 0.037696566, -0.08093239, -0.0041242186, -0.005866373,
|
||||
0.02048607, -0.12549639, 0.011705739, -0.0021425558, 0.018078592,
|
||||
-0.038228087, 0.004245971, -0.11560507, 0.01381203, -0.02734893,
|
||||
0.029595908, 0.101082794, -0.042842254, -0.08397301, -0.1546237,
|
||||
-0.037624564, 0.0069606556, -0.008020479, 0.0043268185, 0.016504597,
|
||||
0.044554908, -0.05240209, 0.020519335, -0.050994243, -0.019861093,
|
||||
0.022066997, -0.0188911, 0.012919696, -0.022872692, -0.012961374,
|
||||
-0.020071112, 0.0074553066, 0.06402014, 0.04065211, 0.014139608,
|
||||
0.06221546, -0.02574805, 0.010114545, 0.020619567, 0.032264773,
|
||||
-0.009148828, 0.022849955, -0.016593235, -0.032456633, -0.006737303,
|
||||
-0.0037493357, -0.065912865, -0.0024181998, -0.0037048983, -0.019304488,
|
||||
0.009085037, 0.057529982, -0.0378338, 0.01792042, 0.01885012,
|
||||
-0.0043600523, 0.017382368, -0.027792772, 0.006749118, -0.0053297766,
|
||||
-0.011628587, 0.015736576, 0.024801968, 0.011543913, -0.01159873,
|
||||
-0.000759976, 0.042183932, 0.05764386, -0.06422416, 0.010577374,
|
||||
-0.02261249, 0.02282793, -0.05141075, -0.11287078, -0.020670144,
|
||||
-0.020953743, 0.030885734, -0.012597732, -0.121472664, 0.028447151,
|
||||
-0.0415664, -0.010061708, 0.06441338, 0.0271207, -0.0016693748,
|
||||
-0.0045694243, 0.020342957, 0.030528478, 0.007129517, -0.0029600926,
|
||||
-0.007061397, 0.06036486, -0.004736018, -0.0090786135, -0.07601116,
|
||||
-0.039556272, -0.051269546, -0.019473685, 0.029858151, 0.028464556,
|
||||
0.012424476, 0.013742254, 0.008742927, 0.006065751, -0.009892663,
|
||||
0.07699999, -0.003443197, 0.0027863425, 0.029197995, 0.0211013,
|
||||
0.01098379, 0.016684106, -0.028917195, 0.008122037, 0.0265241,
|
||||
-0.04461223
|
||||
]
|
||||
}
|
||||
@@ -1,129 +0,0 @@
|
||||
// `clipKeywords`: the keywords quak writes for a file in place of its CLIP
|
||||
// embedding.
|
||||
//
|
||||
// Ente's ML data for a file holds a CLIP embedding, 512 numbers its image
|
||||
// model gave the photo. Ente's photos app puts a file in a category of its
|
||||
// Discover section when the cosine similarity of that embedding and the text
|
||||
// embedding of the category's query is at least the category's minimum score,
|
||||
// and keeps videos out of some categories. `clipKeywords` applies that rule
|
||||
// and returns the titles of the categories a file falls in, highest score
|
||||
// first; `quak backup` and `quak backup-metadata` write them as `keywords`.
|
||||
|
||||
import { readFileSync } from "node:fs";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { DISCOVER_CATEGORIES } from "../src/clip-keyword-vectors.js";
|
||||
import { clipKeywords } from "../src/clip-keywords.js";
|
||||
|
||||
// 512 numbers: the ones given, then zeros.
|
||||
const vector = (...start: number[]): number[] => [
|
||||
...start,
|
||||
...new Array<number>(512 - start.length).fill(0),
|
||||
];
|
||||
|
||||
const withEmbedding = (embedding: unknown) => ({
|
||||
face: { faces: [] },
|
||||
clip: { embedding },
|
||||
});
|
||||
|
||||
// Two made-up categories, each with a text embedding along one axis. The
|
||||
// embedding [3, 4] has a cosine similarity of 3/5 with "Across" and 4/5 with
|
||||
// "Up". Videos are kept out of "Across".
|
||||
const ACROSS = {
|
||||
title: "Across",
|
||||
query: "across",
|
||||
minScore: 0.6,
|
||||
showVideo: false,
|
||||
textEmbedding: vector(1),
|
||||
};
|
||||
const UP = {
|
||||
title: "Up",
|
||||
query: "up",
|
||||
minScore: 0.5,
|
||||
showVideo: true,
|
||||
textEmbedding: vector(0, 1),
|
||||
};
|
||||
|
||||
describe("clipKeywords", () => {
|
||||
it("gives each category whose minimum score the similarity reaches, highest score first", () => {
|
||||
expect(
|
||||
clipKeywords(withEmbedding(vector(3, 4)), false, [ACROSS, UP]),
|
||||
).toEqual(["Up", "Across"]);
|
||||
});
|
||||
|
||||
it("leaves out a category when the similarity is just below its minimum score", () => {
|
||||
const stricter = { ...ACROSS, minScore: 0.6 + 1e-9 };
|
||||
|
||||
expect(
|
||||
clipKeywords(withEmbedding(vector(3, 4)), false, [stricter, UP]),
|
||||
).toEqual(["Up"]);
|
||||
});
|
||||
|
||||
it("keeps a video out of a category that leaves out videos", () => {
|
||||
expect(
|
||||
clipKeywords(withEmbedding(vector(3, 4)), true, [ACROSS, UP]),
|
||||
).toEqual(["Up"]);
|
||||
});
|
||||
|
||||
it("gives an empty list for an embedding that falls in no category", () => {
|
||||
expect(
|
||||
clipKeywords(withEmbedding(vector(0, 0, 1)), false, [ACROSS, UP]),
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
it.each([
|
||||
["no CLIP embedding", { face: { faces: [] } }],
|
||||
["a CLIP part that is not an object", { clip: null }],
|
||||
["a CLIP part without an embedding", { clip: {} }],
|
||||
["an embedding of 511 numbers", withEmbedding(vector(3, 4).slice(1))],
|
||||
["an embedding with NaN in it", withEmbedding(vector(3, NaN))],
|
||||
["an embedding with Infinity in it", withEmbedding(vector(Infinity))],
|
||||
["an embedding of strings", withEmbedding(vector(3, 4).map(String))],
|
||||
["an embedding of zeros", withEmbedding(vector())],
|
||||
])("gives no keywords for ML data with %s", (_, mlData) => {
|
||||
expect(clipKeywords(mlData, false, [ACROSS, UP])).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
// The CLIP embedding Ente's image model gives a public-domain NOAA photo of a
|
||||
// sunset, made by bin/clip-keyword-vectors from the photo at `photo`.
|
||||
const SUNSET: { photo: string; sha256: string; embedding: number[] } =
|
||||
JSON.parse(
|
||||
readFileSync(
|
||||
new URL("clip-keywords-sunset.json", import.meta.url),
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
|
||||
describe("Ente's Discover categories", () => {
|
||||
it("are the 16 of Ente's photos app, each with a text embedding of norm 1", () => {
|
||||
expect(DISCOVER_CATEGORIES.map((c) => c.title)).toEqual([
|
||||
"Identity",
|
||||
"Screenshots",
|
||||
"QR Codes",
|
||||
"Receipts",
|
||||
"Notes",
|
||||
"Memes",
|
||||
"Visiting Cards",
|
||||
"Babies",
|
||||
"Pets",
|
||||
"Selfies",
|
||||
"Wallpapers",
|
||||
"Food",
|
||||
"Celebrations",
|
||||
"Sunset",
|
||||
"Hills",
|
||||
"Greenery",
|
||||
]);
|
||||
for (const category of DISCOVER_CATEGORIES) {
|
||||
expect(category.textEmbedding).toHaveLength(512);
|
||||
expect(Math.hypot(...category.textEmbedding)).toBeCloseTo(1, 5);
|
||||
}
|
||||
});
|
||||
|
||||
it("give a photo of a sunset the keyword Sunset and no other", () => {
|
||||
expect(
|
||||
clipKeywords({ clip: { embedding: SUNSET.embedding } }, false),
|
||||
).toEqual(["Sunset"]);
|
||||
});
|
||||
});
|
||||
@@ -658,26 +658,6 @@ describe("downloadFile", () => {
|
||||
expect(readdirSync(parent)).toEqual(["cwd"]);
|
||||
});
|
||||
|
||||
// A file system refuses a file name over 255 bytes.
|
||||
it("cuts a title too long for a file name short, keeping its extension", async () => {
|
||||
const plaintext = new Uint8Array([1, 2, 3]);
|
||||
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const { header, ciphertext } = encryptFileBody(plaintext, key);
|
||||
const thumbPush =
|
||||
sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
|
||||
const file = buildMockEnteFile(key, header, thumbPush.header);
|
||||
file.metadata.title = `${"t".repeat(296)}.jpg`;
|
||||
const dir = mkdtempSync(join(testDir, "long-title-"));
|
||||
|
||||
const api = new ApiClient({ fetch: mockFetchForBody(ciphertext) });
|
||||
const result = await inDirectory(dir, () => downloadFile(api, file));
|
||||
|
||||
const name = `${"t".repeat(251)}.jpg`;
|
||||
expect(Buffer.byteLength(name)).toBe(255);
|
||||
expect(result.path).toBe(name);
|
||||
expect(readdirSync(dir)).toEqual([name]);
|
||||
});
|
||||
|
||||
it("uses an explicit outPath verbatim, even one with ..", async () => {
|
||||
// The caller is trusted: its path is not sanitized.
|
||||
const { api, file } = fixtureFor(
|
||||
@@ -771,28 +751,6 @@ describe("downloadThumbnail", () => {
|
||||
expect(result.path).toBe("thumb__etc_passwd");
|
||||
expect(readdirSync(dir)).toEqual(["thumb__etc_passwd"]);
|
||||
});
|
||||
|
||||
it("leaves room for thumb_ when it cuts a title too long for a file name short", async () => {
|
||||
const plaintext = new Uint8Array([0xff, 0xd8, 0xff, 0xe0]);
|
||||
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const filePush =
|
||||
sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
|
||||
const { header: thumbHeader, ciphertext: thumbCipher } =
|
||||
encryptFileBody(plaintext, key);
|
||||
const file = buildMockEnteFile(key, filePush.header, thumbHeader);
|
||||
file.metadata.title = `${"t".repeat(296)}.jpg`;
|
||||
const dir = mkdtempSync(join(testDir, "long-thumb-title-"));
|
||||
|
||||
const api = new ApiClient({ fetch: mockFetchForBody(thumbCipher) });
|
||||
const result = await inDirectory(dir, () =>
|
||||
downloadThumbnail(api, file),
|
||||
);
|
||||
|
||||
const name = `thumb_${"t".repeat(245)}.jpg`;
|
||||
expect(Buffer.byteLength(name)).toBe(255);
|
||||
expect(result.path).toBe(name);
|
||||
expect(readdirSync(dir)).toEqual([name]);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -1923,53 +1881,6 @@ describe("downloadFile live photos", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
// A file system refuses a file name over 255 bytes. Each name is cut
|
||||
// short once its own extension is in it.
|
||||
const imageName = `${"t".repeat(250)}.heic`;
|
||||
const videoName = `${"t".repeat(251)}.mov`;
|
||||
|
||||
// The title fits, but ".heic" is one byte longer than ".jpg".
|
||||
it("cuts the image's name short for a 255-byte title ending in .jpg when no outPath is given", async () => {
|
||||
const title = `${"t".repeat(251)}.jpg`;
|
||||
const t = setup(livePhotoZip(), { ...livePhoto, title });
|
||||
|
||||
const result = await inDirectory(t.dir, () =>
|
||||
downloadFile(t.api, t.file),
|
||||
);
|
||||
|
||||
expect(Buffer.byteLength(title)).toBe(255);
|
||||
expect(Buffer.byteLength(imageName)).toBe(255);
|
||||
expect(Buffer.byteLength(videoName)).toBe(255);
|
||||
expect(result.path).toBe(imageName);
|
||||
expect(result.videoPath).toBe(videoName);
|
||||
expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]);
|
||||
});
|
||||
|
||||
it("cuts the names of the image and the video short for a 300-byte title with no extension when no outPath is given", async () => {
|
||||
const t = setup(livePhotoZip(), {
|
||||
...livePhoto,
|
||||
title: "t".repeat(300),
|
||||
});
|
||||
|
||||
const result = await inDirectory(t.dir, () =>
|
||||
downloadFile(t.api, t.file),
|
||||
);
|
||||
|
||||
expect(result.path).toBe(imageName);
|
||||
expect(result.videoPath).toBe(videoName);
|
||||
expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]);
|
||||
});
|
||||
|
||||
// The caller's path is used as is; only the extension is each part's own.
|
||||
it("keeps an outPath too long for a file name as given", async () => {
|
||||
const t = setup(livePhotoZip(), livePhoto);
|
||||
const outPath = join(t.dir, `${"t".repeat(251)}.jpg`);
|
||||
|
||||
await expect(downloadFile(t.api, t.file, outPath)).rejects.toThrow(
|
||||
/ENAMETOOLONG/,
|
||||
);
|
||||
});
|
||||
|
||||
it("gives each part its own extension from the ZIP, letters and digits only", async () => {
|
||||
const zip = livePhotoZip({ "image.JPG": IMAGE, "video.m-4v": VIDEO });
|
||||
const t = setup(zip, livePhoto);
|
||||
|
||||
+2
-5
@@ -19,11 +19,8 @@ import { SRP, SrpServer } from "fast-srp-hap";
|
||||
import { toBase64, deriveKEK, deriveLoginSubkey } from "../src/crypto/index.js";
|
||||
import type { KeyAttributes } from "../src/auth/types.js";
|
||||
|
||||
// The account's password is stretched at libsodium's lowest Argon2id cost.
|
||||
// Every login stretches it again, and at a realistic cost that takes seconds on
|
||||
// a loaded host.
|
||||
const TEST_OPS = 1;
|
||||
const TEST_MEM = 8192;
|
||||
const TEST_OPS = 2;
|
||||
const TEST_MEM = 64 * 1024 * 1024;
|
||||
|
||||
export interface MetaMockState {
|
||||
verifier: Buffer;
|
||||
|
||||
@@ -15,10 +15,8 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
defaultFileName,
|
||||
safeExtension,
|
||||
sanitizeFileName,
|
||||
shortenKeepingExtension,
|
||||
shortenToBytes,
|
||||
} from "../../src/filename.js";
|
||||
|
||||
@@ -96,12 +94,6 @@ describe("safeExtension", () => {
|
||||
expect(safeExtension("x.jp g")).toBe(".bin");
|
||||
expect(safeExtension("x.jpg\0")).toBe(".bin");
|
||||
});
|
||||
|
||||
it("keeps an extension of 16 letters and digits, and uses .bin for a longer one", () => {
|
||||
expect(safeExtension(`x.${"e".repeat(16)}`)).toBe(`.${"e".repeat(16)}`);
|
||||
expect(safeExtension(`x.${"e".repeat(17)}`)).toBe(".bin");
|
||||
expect(safeExtension(`b.${"e".repeat(300)}`)).toBe(".bin");
|
||||
});
|
||||
});
|
||||
|
||||
describe("shortenToBytes", () => {
|
||||
@@ -126,103 +118,3 @@ describe("shortenToBytes", () => {
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("shortenKeepingExtension", () => {
|
||||
it("returns a name that fits unchanged", () => {
|
||||
expect(shortenKeepingExtension("beach.jpg", 9)).toBe("beach.jpg");
|
||||
expect(shortenKeepingExtension("beach", 5)).toBe("beach");
|
||||
});
|
||||
|
||||
it("cuts the part before the extension short and keeps the extension", () => {
|
||||
expect(shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255)).toBe(
|
||||
`${"t".repeat(251)}.jpg`,
|
||||
);
|
||||
expect(shortenKeepingExtension("é".repeat(150), 255)).toBe(
|
||||
"é".repeat(127),
|
||||
);
|
||||
});
|
||||
|
||||
// A name cut down to its extension would start with a dot, and so be
|
||||
// hidden.
|
||||
it("keeps the first character before the extension, cutting the extension short to leave room for it", () => {
|
||||
expect(shortenKeepingExtension(`b.${"e".repeat(300)}`, 255)).toBe(
|
||||
`b.${"e".repeat(253)}`,
|
||||
);
|
||||
expect(shortenKeepingExtension(`ab.${"e".repeat(300)}`, 255)).toBe(
|
||||
`a.${"e".repeat(253)}`,
|
||||
);
|
||||
expect(shortenKeepingExtension(`é.${"e".repeat(300)}`, 255)).toBe(
|
||||
`é.${"e".repeat(252)}`,
|
||||
);
|
||||
// "€" takes three bytes, so the extension is cut to 253 and the
|
||||
// part before it keeps the two bytes left over, for the "a".
|
||||
expect(shortenKeepingExtension(`a.${"€".repeat(90)}`, 255)).toBe(
|
||||
`a.${"€".repeat(84)}`,
|
||||
);
|
||||
});
|
||||
|
||||
it("puts `insert` right before the extension and leaves room for it", () => {
|
||||
expect(shortenKeepingExtension("beach.jpg", 255, " (7)")).toBe(
|
||||
"beach (7).jpg",
|
||||
);
|
||||
expect(
|
||||
shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255, " (7)"),
|
||||
).toBe(`${"t".repeat(247)} (7).jpg`);
|
||||
expect(
|
||||
shortenKeepingExtension(`b.${"e".repeat(300)}`, 255, " (7)"),
|
||||
).toBe(`b (7).${"e".repeat(249)}`);
|
||||
});
|
||||
});
|
||||
|
||||
describe("defaultFileName", () => {
|
||||
it("is the title when it is safe and fits", () => {
|
||||
expect(defaultFileName("IMG_0001.HEIC", 42)).toBe("IMG_0001.HEIC");
|
||||
});
|
||||
|
||||
it("replaces unsafe characters, and is file-<id> for an empty title", () => {
|
||||
expect(defaultFileName("../x.jpg", 42)).toBe("__x.jpg");
|
||||
expect(defaultFileName("", 42)).toBe("file-42");
|
||||
});
|
||||
|
||||
it("cuts a title over 255 bytes short, keeping its extension", () => {
|
||||
expect(defaultFileName(`${"t".repeat(296)}.jpg`, 42)).toBe(
|
||||
`${"t".repeat(251)}.jpg`,
|
||||
);
|
||||
expect(defaultFileName(`b.${"e".repeat(300)}`, 42)).toBe(
|
||||
`b.${"e".repeat(253)}`,
|
||||
);
|
||||
});
|
||||
|
||||
it("puts `prefix` before the title and leaves room for it", () => {
|
||||
expect(defaultFileName("beach.jpg", 42, { prefix: "thumb_" })).toBe(
|
||||
"thumb_beach.jpg",
|
||||
);
|
||||
expect(
|
||||
defaultFileName(`${"t".repeat(296)}.jpg`, 42, { prefix: "thumb_" }),
|
||||
).toBe(`thumb_${"t".repeat(245)}.jpg`);
|
||||
});
|
||||
|
||||
// A live photo's image and video are each named with their own
|
||||
// extension, and the name is cut short once that extension is in it.
|
||||
it("puts `ext` in place of the title's extension before cutting the name short", () => {
|
||||
expect(defaultFileName("IMG_0001.HEIC", 42, { ext: ".mov" })).toBe(
|
||||
"IMG_0001.mov",
|
||||
);
|
||||
expect(defaultFileName("", 42, { ext: ".mov" })).toBe("file-42.mov");
|
||||
expect(
|
||||
defaultFileName(`${"t".repeat(251)}.jpg`, 42, { ext: ".heic" }),
|
||||
).toBe(`${"t".repeat(250)}.heic`);
|
||||
expect(defaultFileName("t".repeat(300), 42, { ext: ".heic" })).toBe(
|
||||
`${"t".repeat(250)}.heic`,
|
||||
);
|
||||
});
|
||||
|
||||
// Cutting `CONSOLE` or `NULL` short before a long extension leaves `CON`
|
||||
// or `NUL`, so the name gets the `_` `sanitizeFileName` gives a device
|
||||
// name, and the part before the extension gives up a byte for it.
|
||||
it("never returns a Windows device name, even once cut short", () => {
|
||||
const ext = `.${"e".repeat(251)}`;
|
||||
expect(defaultFileName(`CONSOLE${ext}`, 42)).toBe(`_CO${ext}`);
|
||||
expect(defaultFileName(`NULL${ext}`, 42)).toBe(`_NU${ext}`);
|
||||
});
|
||||
});
|
||||
@@ -253,12 +253,6 @@ describe("Library content wiring", () => {
|
||||
await expect(
|
||||
lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
|
||||
).rejects.toThrow(/content cache/i);
|
||||
await expect(lib.originalOf(file(1, 1))).rejects.toThrow(
|
||||
/content cache/i,
|
||||
);
|
||||
await expect(lib.thumbnailOf(file(1, 1))).rejects.toThrow(
|
||||
/content cache/i,
|
||||
);
|
||||
await lib.close();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -375,25 +375,6 @@ describe("ContentCache.original / thumbnail", () => {
|
||||
await cache.open();
|
||||
await expect(cache.original(999)).rejects.toThrow(/unknown file/i);
|
||||
});
|
||||
|
||||
// `getFile` knows no file here, so the cache can fetch only the file it is
|
||||
// given.
|
||||
it("originalOf and thumbnailOf fetch and cache the file they are given", async () => {
|
||||
const { cache, source } = buildCache({ files: [] });
|
||||
await cache.open();
|
||||
|
||||
const original = await cache.originalOf(file(5));
|
||||
const thumbnail = await cache.thumbnailOf(file(5));
|
||||
|
||||
expect(source.originalCalls).toEqual([5]);
|
||||
expect(source.thumbnailCalls).toEqual([5]);
|
||||
expect(original.path).toBe(join(cacheDir, "originals", "5.jpg"));
|
||||
expect(thumbnail.path).toBe(join(cacheDir, "thumbnails", "5.jpg"));
|
||||
expect(cache.pathsFor(5)).toEqual({
|
||||
originalPath: original.path,
|
||||
thumbnailPath: thumbnail.path,
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
describe("ContentCache.ensureThumbnails", () => {
|
||||
|
||||
@@ -14,9 +14,6 @@
|
||||
* three.
|
||||
* 3. A refresh that fails rejects the fresh read (currency was unavailable),
|
||||
* while the default reads stay silent and keep serving the last good copy.
|
||||
* 4. `freshCollections()` reads the collection list alone, with one request
|
||||
* and no collection's files, and stores nothing, so the next refresh still
|
||||
* fetches the files of every collection (issue #210).
|
||||
*
|
||||
* The client is the same metadata-only mock the background-refresh tests use:
|
||||
* no crypto, no network, scripted pages, and a record of each call. A long
|
||||
@@ -273,70 +270,3 @@ describe("Library.fresh", () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("Library.freshCollections", () => {
|
||||
let dir: string;
|
||||
let cacheDirectory: string;
|
||||
|
||||
beforeEach(() => {
|
||||
dir = mkdtempSync(join(tmpdir(), "quak-fresh-collections-"));
|
||||
cacheDirectory = join(dir, "cache");
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// A refresh fetches a collection's files only when the collection is new
|
||||
// or its updationTime has advanced past the stored copy's. Had the
|
||||
// collection list been stored, the next refresh would take every
|
||||
// collection for one it already holds the files of, and fetch none.
|
||||
it("asks for the collection list alone and stores nothing", async () => {
|
||||
const client = new MockClient();
|
||||
const page = {
|
||||
collections: [collection(2, 100), collection(1, 110)],
|
||||
deleted: [],
|
||||
cursor: 110,
|
||||
};
|
||||
client.collectionsQueue.push(page, page);
|
||||
client.filesFor(1, {
|
||||
files: [file(1001, 1, 90)],
|
||||
deleted: [],
|
||||
cursor: 90,
|
||||
});
|
||||
client.filesFor(2, {
|
||||
files: [file(2001, 2, 95)],
|
||||
deleted: [],
|
||||
cursor: 95,
|
||||
});
|
||||
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshOnOpen: false,
|
||||
refreshIntervalSeconds: SLOW_INTERVAL,
|
||||
});
|
||||
try {
|
||||
// One request, the whole list in the server's order, and no
|
||||
// collection's files.
|
||||
const collections = await lib.freshCollections();
|
||||
expect(collections.map((c) => c.id)).toEqual([2, 1]);
|
||||
expect(collections.map((c) => c.name)).toEqual([
|
||||
"album-2",
|
||||
"album-1",
|
||||
]);
|
||||
expect(client.collectionsSinceTimes).toEqual([0]);
|
||||
expect(client.filesCalls).toEqual([]);
|
||||
expect(lib.listCollections()).toEqual([]);
|
||||
|
||||
// The next refresh still asks from the beginning and fetches the
|
||||
// files of both collections.
|
||||
const reads = await lib.fresh();
|
||||
expect(client.collectionsSinceTimes).toEqual([0, 0]);
|
||||
expect(reads.photos.byID({ fileID: 1001 })?.fileID).toBe(1001);
|
||||
expect(reads.photos.byID({ fileID: 2001 })?.fileID).toBe(2001);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -19,22 +19,13 @@
|
||||
* refresh fails (offline start from cache).
|
||||
* 5. `close()` stops the timer and is idempotent, and its promise resolves
|
||||
* only once an in-flight refresh has written the cache file.
|
||||
* 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id,
|
||||
* with a self-hosted server's address before the user id.
|
||||
* 7. `open()` branches on the cache: a cache that holds no collection awaits
|
||||
* the first refresh (it has nothing to serve yet); an existing cache serves
|
||||
* its copy at once and refreshes in the background, so a slow or dead
|
||||
* server never stalls opening.
|
||||
* 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id.
|
||||
* 7. `open()` branches on the cache: an empty cache awaits the first refresh
|
||||
* (it has nothing to serve yet); an existing cache serves its copy at once
|
||||
* and refreshes in the background, so a slow or dead server never stalls
|
||||
* opening.
|
||||
* 8. A save failure that leaves RAM ahead of disk keeps `status().lastError`
|
||||
* set and keeps retrying the write; a later empty refresh does not clear it.
|
||||
* 9. With `refreshOnOpen: false`, `open()` starts no refresh at all, and a
|
||||
* refresh reports through `onProgress` each collection whose files it
|
||||
* fetched.
|
||||
* 10. A cache another account filled starts empty. One filled from another
|
||||
* server also loses its originals and thumbnails, since two servers can
|
||||
* give two files the same ID. `thumbnailOf` on an empty cache saves the
|
||||
* account's user ID and server. A refresh saves the account's server, so a
|
||||
* self-hosted account keeps its own cache from one `open()` to the next.
|
||||
*
|
||||
* The client is a mock: no crypto, no network. It serves scripted pages and
|
||||
* records the `sinceTime` each call carried so cursor threading is provable.
|
||||
@@ -55,13 +46,7 @@
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
import {
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import envPaths from "env-paths";
|
||||
@@ -119,7 +104,6 @@ const file = (
|
||||
*/
|
||||
class MockClient {
|
||||
userID = USER_ID;
|
||||
apiOrigin?: string;
|
||||
failCollections = false;
|
||||
collectionsQueue: CollectionsPage[] = [];
|
||||
filesByCollection = new Map<number, FilesPage[]>();
|
||||
@@ -597,145 +581,6 @@ describe("Library.open and background refresh", () => {
|
||||
}
|
||||
});
|
||||
|
||||
// `originalOf` on an empty cache leaves such a cache behind: it names the
|
||||
// account before it stores the original.
|
||||
it("awaits the first refresh on a cache that names the account but holds no collection", async () => {
|
||||
const seed = await MetadataStore.load(
|
||||
join(cacheDirectory, "metadata.json"),
|
||||
);
|
||||
seed.userID = USER_ID;
|
||||
await seed.save();
|
||||
let releaseFirstFetch: (page: CollectionsPage) => void = () => {};
|
||||
const gate = new Promise<CollectionsPage>((resolve) => {
|
||||
releaseFirstFetch = resolve;
|
||||
});
|
||||
const client = new MockClient();
|
||||
client.collectionsSince = async () => gate;
|
||||
|
||||
let opened = false;
|
||||
const openPromise = Library.open({ client, cacheDirectory }).then(
|
||||
(l) => {
|
||||
opened = true;
|
||||
return l;
|
||||
},
|
||||
);
|
||||
await new Promise((r) => setTimeout(r, 20));
|
||||
expect(opened).toBe(false);
|
||||
|
||||
releaseFirstFetch({
|
||||
collections: [collection(1, 100)],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
const lib = await openPromise;
|
||||
try {
|
||||
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// A caller that asks for each refresh it needs, as the CLI's read commands
|
||||
// do, opens without one: on an empty cache the refresh `open()` would
|
||||
// await fetches the files of every collection, which `quak collections`
|
||||
// never needs (issue #210).
|
||||
it("starts no refresh as it opens when refreshOnOpen is false", async () => {
|
||||
const client = new MockClient();
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 100)],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshOnOpen: false,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
});
|
||||
try {
|
||||
expect(client.collectionsSinceTimes).toEqual([]);
|
||||
expect(lib.listCollections()).toEqual([]);
|
||||
expect(lib.status().lastRefreshAt).toBeUndefined();
|
||||
|
||||
// The refresh comes when the caller asks for it.
|
||||
await lib.fresh();
|
||||
expect(client.collectionsSinceTimes).toEqual([0]);
|
||||
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// A refresh fetches the files of each collection that changed, one
|
||||
// collection after another, and stores nothing until it has them all. On
|
||||
// a large account that takes a while, so it reports each collection whose
|
||||
// files it has fetched, and how many it will fetch in all (issue #210).
|
||||
it("reports each collection whose files a refresh fetched", async () => {
|
||||
const events: RefreshEvent[] = [];
|
||||
const client = new MockClient();
|
||||
client.collectionsQueue.push({
|
||||
collections: [
|
||||
collection(1, 100),
|
||||
collection(2, 100),
|
||||
collection(3, 100),
|
||||
],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
onProgress: (e) => events.push(e),
|
||||
});
|
||||
try {
|
||||
expect(events).toEqual([
|
||||
{ operation: "refresh", status: "started" },
|
||||
{
|
||||
operation: "refresh",
|
||||
status: "progress",
|
||||
fetched: 1,
|
||||
total: 3,
|
||||
},
|
||||
{
|
||||
operation: "refresh",
|
||||
status: "progress",
|
||||
fetched: 2,
|
||||
total: 3,
|
||||
},
|
||||
{
|
||||
operation: "refresh",
|
||||
status: "progress",
|
||||
fetched: 3,
|
||||
total: 3,
|
||||
},
|
||||
{ operation: "refresh", status: "done" },
|
||||
]);
|
||||
|
||||
// Only collection 2 changed since, so the next refresh fetches
|
||||
// its files alone and counts one.
|
||||
events.length = 0;
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(2, 200)],
|
||||
deleted: [],
|
||||
cursor: 200,
|
||||
});
|
||||
await lib.fresh();
|
||||
expect(events.filter((e) => e.status === "progress")).toEqual([
|
||||
{
|
||||
operation: "refresh",
|
||||
status: "progress",
|
||||
fetched: 1,
|
||||
total: 1,
|
||||
},
|
||||
]);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps a save failure visible until a save actually succeeds", async () => {
|
||||
const saveSpy = vi
|
||||
.spyOn(MetadataStore.prototype, "save")
|
||||
@@ -893,165 +738,4 @@ describe("Library.open and background refresh", () => {
|
||||
else process.env.XDG_CACHE_HOME = prev;
|
||||
}
|
||||
});
|
||||
|
||||
it("puts a self-hosted server's address, URL-encoded, before the user id in the default cacheDirectory", async () => {
|
||||
const xdg = join(dir, "xdg-cache");
|
||||
const prev = process.env.XDG_CACHE_HOME;
|
||||
process.env.XDG_CACHE_HOME = xdg;
|
||||
try {
|
||||
const client = new MockClient();
|
||||
client.apiOrigin = "https://ente.example.com:8443/api";
|
||||
const lib = await Library.open({ client });
|
||||
try {
|
||||
expect(lib.cacheDirectory).toBe(
|
||||
join(
|
||||
envPaths("quak", { suffix: "" }).cache,
|
||||
"https%3A%2F%2Fente.example.com%3A8443%2Fapi",
|
||||
String(USER_ID),
|
||||
),
|
||||
);
|
||||
expect(lib.cacheDirectory.startsWith(xdg)).toBe(true);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
} finally {
|
||||
if (prev === undefined) delete process.env.XDG_CACHE_HOME;
|
||||
else process.env.XDG_CACHE_HOME = prev;
|
||||
}
|
||||
});
|
||||
|
||||
// A cache that account USER_ID on `apiOrigin` filled: its records, and an
|
||||
// original and a thumbnail of file 1001.
|
||||
const fillCache = async (apiOrigin?: string): Promise<void> => {
|
||||
const seed = await MetadataStore.load(
|
||||
join(cacheDirectory, "metadata.json"),
|
||||
);
|
||||
seed.userID = USER_ID;
|
||||
seed.apiOrigin = apiOrigin;
|
||||
seed.collectionsSinceTime = 500;
|
||||
seed.putCollection(collection(1, 400));
|
||||
seed.putFile(file(1001, 1, 400));
|
||||
await seed.save();
|
||||
for (const name of ["originals", "thumbnails"]) {
|
||||
mkdirSync(join(cacheDirectory, name));
|
||||
writeFileSync(join(cacheDirectory, name, "1001.jpg"), "1001");
|
||||
}
|
||||
};
|
||||
|
||||
it("deletes the whole cache an account on another server filled", async () => {
|
||||
await fillCache("https://ente.example.com");
|
||||
const client = new MockClient();
|
||||
client.apiOrigin = "https://photos.example.org";
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: 3600,
|
||||
});
|
||||
try {
|
||||
expect(client.collectionsSinceTimes).toEqual([0]);
|
||||
expect(lib.listCollections()).toEqual([]);
|
||||
for (const name of ["originals", "thumbnails"]) {
|
||||
expect(existsSync(join(cacheDirectory, name, "1001.jpg"))).toBe(
|
||||
false,
|
||||
);
|
||||
}
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// A server gives each file its own ID, so another account's cached file
|
||||
// is never served for this account's.
|
||||
it("keeps the originals and thumbnails another account on the same server cached", async () => {
|
||||
await fillCache();
|
||||
const client = new MockClient();
|
||||
client.userID = USER_ID + 1;
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: 3600,
|
||||
});
|
||||
try {
|
||||
expect(client.collectionsSinceTimes).toEqual([0]);
|
||||
expect(lib.listCollections()).toEqual([]);
|
||||
for (const name of ["originals", "thumbnails"]) {
|
||||
expect(existsSync(join(cacheDirectory, name, "1001.jpg"))).toBe(
|
||||
true,
|
||||
);
|
||||
}
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// `quak get-thumb` on an empty cache stores a thumbnail before any refresh
|
||||
// runs. `open()` for an account on another server deletes that thumbnail
|
||||
// only if the cache names the account that stored it.
|
||||
it("thumbnailOf on an empty cache saves the account's user ID and server in metadata.json", async () => {
|
||||
const client = new MockClient();
|
||||
client.apiOrigin = "https://ente.example.com";
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshOnOpen: false,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
contentSource: {
|
||||
original: async () => {
|
||||
throw new Error("original not expected");
|
||||
},
|
||||
thumbnail: async ({ destination }) => {
|
||||
writeFileSync(destination, "thumb");
|
||||
return { bytesWritten: 5 };
|
||||
},
|
||||
},
|
||||
});
|
||||
try {
|
||||
await lib.thumbnailOf(file(1001, 1, 400));
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
|
||||
const saved = await MetadataStore.load(
|
||||
join(cacheDirectory, "metadata.json"),
|
||||
);
|
||||
expect(saved.userID).toBe(USER_ID);
|
||||
expect(saved.apiOrigin).toBe("https://ente.example.com");
|
||||
});
|
||||
|
||||
// A refresh saves the account's server in `metadata.json`. Without it,
|
||||
// `open()` would take a self-hosted account's own cache for another
|
||||
// server's and delete it.
|
||||
it("keeps a self-hosted account's own cache from one open to the next", async () => {
|
||||
const client = new MockClient();
|
||||
client.apiOrigin = "https://ente.example.com";
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 100)],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
const first = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
});
|
||||
await first.close();
|
||||
mkdirSync(join(cacheDirectory, "originals"));
|
||||
writeFileSync(join(cacheDirectory, "originals", "1001.jpg"), "1001");
|
||||
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
});
|
||||
try {
|
||||
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
|
||||
expect(
|
||||
existsSync(join(cacheDirectory, "originals", "1001.jpg")),
|
||||
).toBe(true);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
// The second open's refresh started from the saved cursor.
|
||||
expect(client.collectionsSinceTimes).toEqual([0, 100]);
|
||||
});
|
||||
});
|
||||
+9
-150
@@ -12,11 +12,9 @@
|
||||
*
|
||||
* 2. `Library` wiring: after each refresh the library fetches ML data through
|
||||
* the metadata pool for every known file not yet cached, is incremental on
|
||||
* later refreshes, and refetches a file whose `updationTime` advanced. A
|
||||
* file whose ML data did not decode is reported with the reason, which
|
||||
* stays listed until the next pass ends, and asked for again. Opening a
|
||||
* cache directory written by another account starts empty, its ML data
|
||||
* included (issue #104).
|
||||
* later refreshes, and refetches a file whose `updationTime` advanced.
|
||||
* Opening a cache directory written by another account starts empty,
|
||||
* its ML data included (issue #104).
|
||||
*
|
||||
* Embedding values are chosen to be exactly representable as float32 so the
|
||||
* round-trip through `clip.f32` compares equal.
|
||||
@@ -28,7 +26,7 @@ import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { MLDataStore } from "../../src/library/mldata.js";
|
||||
import { Library, type RefreshEvent } from "../../src/library/index.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||
import type { MLData } from "../../src/mldata-fetch.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
@@ -258,14 +256,12 @@ const file = (
|
||||
|
||||
// A mock client that serves scripted collection/file pages and per-file ML
|
||||
// payloads, recording every ML fetch request so incremental behaviour is
|
||||
// provable. A file in `mlErrorByFile` has ML data that did not decode, for
|
||||
// the reason given there.
|
||||
// provable.
|
||||
class MLMockClient {
|
||||
userID = USER_ID;
|
||||
collectionsQueue: CollectionsPage[] = [];
|
||||
filesByCollection = new Map<number, FilesPage[]>();
|
||||
mlByFile = new Map<number, MLData>();
|
||||
mlErrorByFile = new Map<number, Error>();
|
||||
mlFetchCalls: number[][] = [];
|
||||
|
||||
whoami(): { email: string; userID: number } {
|
||||
@@ -302,20 +298,14 @@ class MLMockClient {
|
||||
async fetchMLData(args: {
|
||||
fileIDs: number[];
|
||||
fileKeys: Map<number, Uint8Array>;
|
||||
}): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}> {
|
||||
}): Promise<Map<number, MLData>> {
|
||||
this.mlFetchCalls.push([...args.fileIDs]);
|
||||
const payloads = new Map<number, MLData>();
|
||||
const errors = new Map<number, unknown>();
|
||||
const result = new Map<number, MLData>();
|
||||
for (const id of args.fileIDs) {
|
||||
const p = this.mlByFile.get(id);
|
||||
if (p) payloads.set(id, p);
|
||||
const error = this.mlErrorByFile.get(id);
|
||||
if (error) errors.set(id, error);
|
||||
if (p) result.set(id, p);
|
||||
}
|
||||
return { payloads, errors };
|
||||
return result;
|
||||
}
|
||||
|
||||
filesFor(collectionID: number, ...pages: FilesPage[]): void {
|
||||
@@ -459,137 +449,6 @@ describe("Library ML-data fetch on refresh", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("reports a file whose ML data did not decode through status() and onProgress, and asks for it again", async () => {
|
||||
const client = new MLMockClient();
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 100)],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
client.filesFor(1, {
|
||||
files: [file(1001, 1, 90), file(1002, 1, 95)],
|
||||
deleted: [],
|
||||
cursor: 95,
|
||||
});
|
||||
client.mlByFile.set(1001, payload([0.5, 0.25, 0.75]));
|
||||
client.mlErrorByFile.set(
|
||||
1002,
|
||||
new Error("secretstream chunk authentication failed"),
|
||||
);
|
||||
const events: RefreshEvent[] = [];
|
||||
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
onProgress: (event) => events.push(event),
|
||||
});
|
||||
try {
|
||||
await vi.waitFor(
|
||||
() => expect(lib.status().lastMLFetchAt).toBeGreaterThan(0),
|
||||
{ timeout: 2000, interval: 5 },
|
||||
);
|
||||
const reasons = [
|
||||
{
|
||||
fileID: 1002,
|
||||
error: "secretstream chunk authentication failed",
|
||||
},
|
||||
];
|
||||
expect(lib.status().mlDataErrors).toEqual(reasons);
|
||||
expect(lib.status().mlStored).toBe(1);
|
||||
expect(events.filter((e) => e.operation === "fetchMLData")).toEqual(
|
||||
[
|
||||
{ operation: "fetchMLData", status: "started" },
|
||||
{
|
||||
operation: "fetchMLData",
|
||||
status: "done",
|
||||
fetched: 1,
|
||||
mlDataErrors: reasons,
|
||||
},
|
||||
],
|
||||
);
|
||||
|
||||
// Nothing was cached for 1002, so the pass a refresh starts asks
|
||||
// for it again, and once its ML data decodes the reason is gone.
|
||||
client.mlErrorByFile.clear();
|
||||
client.mlByFile.set(1002, payload([1, -2, 0.5]));
|
||||
await lib.fresh();
|
||||
await vi.waitFor(
|
||||
() => {
|
||||
expect(lib.status().mlStored).toBe(2);
|
||||
expect(lib.status().mlDataErrors).toEqual([]);
|
||||
},
|
||||
{ timeout: 2000, interval: 5 },
|
||||
);
|
||||
expect(client.mlFetchCalls.at(-1)).toEqual([1002]);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("still lists the last pass's reasons while the next pass runs", async () => {
|
||||
const client = new MLMockClient();
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 100)],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
client.filesFor(1, {
|
||||
files: [file(1002, 1, 95)],
|
||||
deleted: [],
|
||||
cursor: 95,
|
||||
});
|
||||
client.mlErrorByFile.set(
|
||||
1002,
|
||||
new Error("secretstream chunk authentication failed"),
|
||||
);
|
||||
const reasons = [
|
||||
{ fileID: 1002, error: "secretstream chunk authentication failed" },
|
||||
];
|
||||
let release!: () => void;
|
||||
const held = new Promise<void>((r) => (release = r));
|
||||
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
});
|
||||
try {
|
||||
await vi.waitFor(
|
||||
() => expect(lib.status().lastMLFetchAt).toBeGreaterThan(0),
|
||||
{ timeout: 2000, interval: 5 },
|
||||
);
|
||||
expect(lib.status().mlDataErrors).toEqual(reasons);
|
||||
|
||||
// Hold the next ML data fetch open until the test releases it.
|
||||
let fetchStarted!: () => void;
|
||||
const started = new Promise<void>((r) => (fetchStarted = r));
|
||||
const realFetch = client.fetchMLData.bind(client);
|
||||
client.fetchMLData = async (args) => {
|
||||
fetchStarted();
|
||||
await held;
|
||||
return realFetch(args);
|
||||
};
|
||||
client.mlErrorByFile.clear();
|
||||
client.mlByFile.set(1002, payload([1, -2, 0.5]));
|
||||
|
||||
// The pass a refresh starts asks for 1002 again. While it is
|
||||
// held, the reason from the pass before is still listed; once it
|
||||
// ends, its own empty list takes the place of that one.
|
||||
await lib.fresh();
|
||||
await started;
|
||||
expect(lib.status().mlDataErrors).toEqual(reasons);
|
||||
release();
|
||||
await vi.waitFor(
|
||||
() => expect(lib.status().mlDataErrors).toEqual([]),
|
||||
{ timeout: 2000, interval: 5 },
|
||||
);
|
||||
} finally {
|
||||
release();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("close() resolves only after a running ML data fetch has stored its payloads", async () => {
|
||||
const client = new MLMockClient();
|
||||
client.collectionsQueue.push({
|
||||
|
||||
@@ -1,54 +0,0 @@
|
||||
// `fetchMLDataBatch`, against a server that sends one file's ML data.
|
||||
//
|
||||
// ML data that decrypts and gunzips but is not a JSON object is reported with
|
||||
// the reason, as ML data that does not parse is, and is never returned as a
|
||||
// payload: the library would fail to store it.
|
||||
|
||||
import { gzipSync } from "node:zlib";
|
||||
import sodium from "libsodium-wrappers-sumo";
|
||||
import { beforeAll, describe, expect, it } from "vitest";
|
||||
|
||||
import { ApiClient } from "../../src/api/client.js";
|
||||
import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
|
||||
import { fetchMLDataBatch } from "../../src/mldata-fetch.js";
|
||||
|
||||
beforeAll(async () => {
|
||||
await init();
|
||||
await sodium.ready;
|
||||
});
|
||||
|
||||
// Fetch the ML data of file 1 from a server that sends `json`, gzipped and
|
||||
// encrypted with the file's key.
|
||||
const fetchServing = async (json: string) => {
|
||||
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const { header, ciphertext } = encryptBlob(
|
||||
new Uint8Array(gzipSync(json)),
|
||||
key,
|
||||
);
|
||||
const data = [
|
||||
{
|
||||
fileID: 1,
|
||||
encryptedData: toBase64(ciphertext),
|
||||
decryptionHeader: toBase64(header),
|
||||
},
|
||||
];
|
||||
const api = new ApiClient({
|
||||
fetch: async () =>
|
||||
new Response(JSON.stringify({ data }), {
|
||||
headers: { "content-type": "application/json" },
|
||||
}),
|
||||
});
|
||||
return fetchMLDataBatch(api, [1], new Map([[1, key]]));
|
||||
};
|
||||
|
||||
describe("fetchMLDataBatch", () => {
|
||||
it.each(["null", "5", "[]"])(
|
||||
"gives a reason for ML data that is %s",
|
||||
async (json) => {
|
||||
const { payloads, errors } = await fetchServing(json);
|
||||
|
||||
expect(payloads.size).toBe(0);
|
||||
expect(errors.get(1)).toEqual(new Error("not a JSON object"));
|
||||
},
|
||||
);
|
||||
});
|
||||
@@ -1,241 +0,0 @@
|
||||
// `script/build-bin` compiles `bin/quak`, the single binary that the README's
|
||||
// cron setup runs. bun compiles it from a copy of its own executable, so the
|
||||
// binary loads the same libraries as the bun that built it. nix's bun loads
|
||||
// libraries from the nix store, which garbage collection can delete: on Linux
|
||||
// its C library, on macOS its ICU. The script therefore downloads bun's own
|
||||
// release build, which loads only the system's libraries.
|
||||
//
|
||||
// What the script promises, and why each test matters:
|
||||
//
|
||||
// - Each system, machine and C library that bun publishes a build for gets its
|
||||
// own archive: Linux on x86-64 or arm64, with glibc or with musl, as on
|
||||
// Alpine, and macOS on Intel or Apple silicon. A bun built for another
|
||||
// system, machine or C library does not start, and `make install` fails.
|
||||
// - The archive's SHA-256 must match the one in the script before anything in
|
||||
// the archive runs. Without that check, a tampered download would build the
|
||||
// binary that cron runs.
|
||||
// - Any other system or machine stops with an error, and never falls back to
|
||||
// nix's bun. Building with nix's bun would bring back the dependency on the
|
||||
// nix store.
|
||||
//
|
||||
// Each test copies the script into a fresh directory, which the script then
|
||||
// treats as the checkout, and runs it there. Stubs first on PATH stand in for
|
||||
// `uname`, `ldd`, `curl`, `unzip`, `nix-shell` and `shasum`, so no test
|
||||
// downloads anything or runs a real bun.
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import { execFileSync, spawnSync } from "node:child_process";
|
||||
import {
|
||||
chmodSync,
|
||||
copyFileSync,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
readFileSync,
|
||||
rmSync,
|
||||
symlinkSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
|
||||
|
||||
// The stubs, keyed by the command each replaces. `uname` and `ldd` describe the
|
||||
// system that runBuildBin passes in the environment. `curl`, `unzip`,
|
||||
// `nix-shell` and `shasum` write the arguments they were given, one per line,
|
||||
// to a file named after the command in $STUB_LOG_DIR, which also shows that
|
||||
// they ran.
|
||||
const stubs: Record<string, string> = {
|
||||
uname: `
|
||||
case "$1" in
|
||||
-s) echo "$STUB_UNAME_S" ;;
|
||||
-m) echo "$STUB_UNAME_M" ;;
|
||||
esac`,
|
||||
// The script looks only at the start of the line: "musl libc" or not.
|
||||
ldd: `
|
||||
if [ "$STUB_LIBC" = musl ]; then
|
||||
echo "musl libc (x86_64)"
|
||||
else
|
||||
echo "ldd (GNU libc) 2.36"
|
||||
fi`,
|
||||
// Writes a file that is not the archive to the path after -o.
|
||||
curl: `
|
||||
printf '%s\\n' "$@" > "$STUB_LOG_DIR/curl"
|
||||
while [ $# -gt 1 ]; do
|
||||
if [ "$1" = -o ]; then echo "not the archive" > "$2"; fi
|
||||
shift
|
||||
done`,
|
||||
// Unpacks a stub bun where the real archive puts bun: in a directory named
|
||||
// after the archive, inside the directory after -d. That bun writes
|
||||
// bin/quak, as a real compile would.
|
||||
unzip: `
|
||||
printf '%s\\n' "$@" > "$STUB_LOG_DIR/unzip"
|
||||
for arg in "$@"; do
|
||||
case "$arg" in *.zip) archive="$arg" ;; esac
|
||||
if [ "$previous" = -d ]; then directory="$arg"; fi
|
||||
previous="$arg"
|
||||
done
|
||||
bun="$directory/$(basename "$archive" .zip)/bun"
|
||||
mkdir -p "$(dirname "$bun")"
|
||||
echo '#!/bin/sh' > "$bun"
|
||||
echo 'mkdir -p bin && echo binary > bin/quak' >> "$bun"
|
||||
chmod +x "$bun"`,
|
||||
"nix-shell": `
|
||||
printf '%s\\n' "$@" > "$STUB_LOG_DIR/nix-shell"`,
|
||||
// The script checks the hash with `shasum` where there is no `sha256sum`.
|
||||
// The stub fails, as `shasum` does when the hash does not match.
|
||||
shasum: `
|
||||
printf '%s\\n' "$@" > "$STUB_LOG_DIR/shasum"
|
||||
exit 1`,
|
||||
};
|
||||
|
||||
let checkout = "";
|
||||
|
||||
const writeScript = (path: string, body: string): void => {
|
||||
writeFileSync(path, `#!/bin/sh\n${body}\n`);
|
||||
chmodSync(path, 0o755);
|
||||
};
|
||||
|
||||
// A checkout holding the script, a stub script/build that does nothing, the
|
||||
// stubs, and the directory they log to.
|
||||
beforeEach(() => {
|
||||
checkout = mkdtempSync(join(tmpdir(), "quak-build-bin-"));
|
||||
mkdirSync(join(checkout, "script"));
|
||||
copyFileSync(
|
||||
join(repoRoot, "script/build-bin"),
|
||||
join(checkout, "script/build-bin"),
|
||||
);
|
||||
chmodSync(join(checkout, "script/build-bin"), 0o755);
|
||||
writeScript(join(checkout, "script/build"), "exit 0");
|
||||
mkdirSync(join(checkout, "stubs"));
|
||||
for (const [command, body] of Object.entries(stubs)) {
|
||||
writeScript(join(checkout, "stubs", command), body);
|
||||
}
|
||||
mkdirSync(join(checkout, "log"));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(checkout, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// Runs the script with nothing in its environment but the stubs' settings and a
|
||||
// PATH. Unless the caller passes one, the PATH finds the stubs first and then
|
||||
// the host's commands.
|
||||
const runBuildBin = (
|
||||
system: string,
|
||||
machine: string,
|
||||
{
|
||||
libc = "glibc",
|
||||
path = `${join(checkout, "stubs")}:${process.env.PATH}`,
|
||||
} = {},
|
||||
) =>
|
||||
spawnSync(join(checkout, "script/build-bin"), {
|
||||
cwd: checkout,
|
||||
encoding: "utf-8",
|
||||
env: {
|
||||
PATH: path,
|
||||
STUB_UNAME_S: system,
|
||||
STUB_UNAME_M: machine,
|
||||
STUB_LIBC: libc,
|
||||
STUB_LOG_DIR: join(checkout, "log"),
|
||||
},
|
||||
});
|
||||
|
||||
// The real commands that the script and the stubs run.
|
||||
const hostCommands = [
|
||||
"basename",
|
||||
"chmod",
|
||||
"dirname",
|
||||
"grep",
|
||||
"mkdir",
|
||||
"mktemp",
|
||||
"rm",
|
||||
];
|
||||
|
||||
// A Mac has no `ldd`, and not every Mac has `sha256sum`. This makes a PATH that
|
||||
// finds neither: the stubs other than `ldd`, then links to hostCommands.
|
||||
const makeMacPath = (): string => {
|
||||
rmSync(join(checkout, "stubs", "ldd"));
|
||||
mkdirSync(join(checkout, "host-commands"));
|
||||
for (const command of hostCommands) {
|
||||
const target = execFileSync("sh", ["-c", `command -v ${command}`], {
|
||||
encoding: "utf-8",
|
||||
}).trim();
|
||||
symlinkSync(target, join(checkout, "host-commands", command));
|
||||
}
|
||||
return `${join(checkout, "stubs")}:${join(checkout, "host-commands")}`;
|
||||
};
|
||||
|
||||
const ran = (command: string): boolean =>
|
||||
existsSync(join(checkout, "log", command));
|
||||
|
||||
const argumentsOf = (command: string): string[] =>
|
||||
readFileSync(join(checkout, "log", command), "utf-8")
|
||||
.trimEnd()
|
||||
.split("\n");
|
||||
|
||||
describe("script/build-bin", () => {
|
||||
// The stub curl writes a file that is not the archive, so the hash check
|
||||
// must stop the script before unzip runs. A script without the check would
|
||||
// run unzip, and the stub bun it unpacks would write bin/quak. On Linux the
|
||||
// check must use `sha256sum`, since a stock Alpine has no `shasum`.
|
||||
it.each([
|
||||
["Linux", "x86_64", "glibc", "bun-linux-x64-baseline.zip"],
|
||||
["Linux", "x86_64", "musl", "bun-linux-x64-musl-baseline.zip"],
|
||||
["Linux", "aarch64", "glibc", "bun-linux-aarch64.zip"],
|
||||
["Linux", "aarch64", "musl", "bun-linux-aarch64-musl.zip"],
|
||||
])(
|
||||
"on %s %s with %s, downloads %s and stops when its hash does not match",
|
||||
(system, machine, libc, archive) => {
|
||||
const result = runBuildBin(system, machine, { libc });
|
||||
expect(argumentsOf("curl")).toContain(
|
||||
"https://github.com/oven-sh/bun/releases/download/bun-v1.3.13/" +
|
||||
archive,
|
||||
);
|
||||
expect(ran("shasum")).toBe(false);
|
||||
expect(result.status).not.toBe(0);
|
||||
expect(ran("unzip")).toBe(false);
|
||||
expect(ran("nix-shell")).toBe(false);
|
||||
expect(existsSync(join(checkout, "bin/quak"))).toBe(false);
|
||||
},
|
||||
);
|
||||
|
||||
// On the PATH from makeMacPath the script finds no `sha256sum` and checks
|
||||
// the hash with the stub `shasum`. That stub fails, so the script must stop
|
||||
// before unzip runs.
|
||||
// On Apple silicon `uname -m` says arm64, where bun's archive says aarch64.
|
||||
it.each([
|
||||
["Darwin", "x86_64", "bun-darwin-x64-baseline.zip"],
|
||||
["Darwin", "arm64", "bun-darwin-aarch64.zip"],
|
||||
])(
|
||||
"on %s %s, downloads %s and stops when shasum rejects its hash",
|
||||
(system, machine, archive) => {
|
||||
const result = runBuildBin(system, machine, {
|
||||
path: makeMacPath(),
|
||||
});
|
||||
expect(argumentsOf("curl")).toContain(
|
||||
"https://github.com/oven-sh/bun/releases/download/bun-v1.3.13/" +
|
||||
archive,
|
||||
);
|
||||
expect(argumentsOf("shasum")).toEqual(["-a", "256", "-c", "-"]);
|
||||
expect(result.status).not.toBe(0);
|
||||
expect(ran("unzip")).toBe(false);
|
||||
expect(ran("nix-shell")).toBe(false);
|
||||
expect(existsSync(join(checkout, "bin/quak"))).toBe(false);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
["Linux", "riscv64"],
|
||||
["FreeBSD", "amd64"],
|
||||
])(
|
||||
"stops on %s %s, which bun publishes no build for",
|
||||
(system, machine) => {
|
||||
const result = runBuildBin(system, machine);
|
||||
expect(result.status).not.toBe(0);
|
||||
expect(ran("curl")).toBe(false);
|
||||
expect(ran("nix-shell")).toBe(false);
|
||||
},
|
||||
);
|
||||
});
|
||||
@@ -77,14 +77,6 @@ const recordingSleep = (): {
|
||||
const errnoError = (code: string, message = code): Error =>
|
||||
Object.assign(new Error(message), { code });
|
||||
|
||||
/**
|
||||
* An error shaped like what bun's `fetch` rejects with for a failure below
|
||||
* HTTP: a plain `Error` with bun's own message, a `code`, an `errno` of 0, and
|
||||
* no `cause`.
|
||||
*/
|
||||
const bunError = (code: string, message: string): Error =>
|
||||
Object.assign(new Error(message), { code, errno: 0 });
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Classification
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -252,82 +244,6 @@ describe("isRetryable: transport failures", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("isRetryable and isSafeToReplay: bun's fetch errors", () => {
|
||||
// The binary `make install` builds runs quak under bun, whose `fetch`
|
||||
// does not reject as Node's does. Each shape here was found by running
|
||||
// that binary, built with bun 1.3.13, against listeners on 127.0.0.1 and
|
||||
// a name under `.invalid`.
|
||||
|
||||
it("retries a refused connection, and does not replay it", () => {
|
||||
// Bun gives the same error when the TLS handshake fails, and when a
|
||||
// new connection it opened to resend a request is refused. That
|
||||
// request may already have reached the server on a connection bun
|
||||
// kept open from an earlier request.
|
||||
const refused = bunError(
|
||||
"ConnectionRefused",
|
||||
"Unable to connect. Is the computer able to access the url?",
|
||||
);
|
||||
expect(isRetryable(refused)).toBe(true);
|
||||
expect(isSafeToReplay(refused)).toBe(false);
|
||||
});
|
||||
|
||||
it("retries a name that does not resolve, with either code bun gives, and does not replay it", () => {
|
||||
// For the same name bun gave `ConnectionRefused` on some requests and
|
||||
// `FailedToOpenSocket` on others. Either can also come from a new
|
||||
// connection bun opened to resend a request, as above.
|
||||
for (const unresolved of [
|
||||
bunError(
|
||||
"ConnectionRefused",
|
||||
"Unable to connect. Is the computer able to access the url?",
|
||||
),
|
||||
bunError(
|
||||
"FailedToOpenSocket",
|
||||
"Was there a typo in the url or port?",
|
||||
),
|
||||
]) {
|
||||
expect(isRetryable(unresolved)).toBe(true);
|
||||
expect(isSafeToReplay(unresolved)).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("retries a connection closed before the response ended, and does not replay it", () => {
|
||||
// Bun gives Node's errno, `ECONNRESET`, for a reset and for a close,
|
||||
// before the response or part-way through the body. The request may
|
||||
// already have reached the server.
|
||||
const reset = bunError(
|
||||
"ECONNRESET",
|
||||
"The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()",
|
||||
);
|
||||
expect(isRetryable(reset)).toBe(true);
|
||||
expect(isSafeToReplay(reset)).toBe(false);
|
||||
});
|
||||
|
||||
it("retries a request that ran out of time, and does not replay it", () => {
|
||||
// What bun's fetch rejects with when `AbortSignal.timeout()`, which
|
||||
// bounds each JSON request, fires.
|
||||
const timedOut = new DOMException(
|
||||
"The operation timed out.",
|
||||
"TimeoutError",
|
||||
);
|
||||
expect(isRetryable(timedOut)).toBe(true);
|
||||
expect(isSafeToReplay(timedOut)).toBe(false);
|
||||
});
|
||||
|
||||
it("does not retry a certificate bun rejects", () => {
|
||||
// A certificate that fails verification fails the same way however
|
||||
// often it is asked. Under Node it is retried only because every
|
||||
// `TypeError` is.
|
||||
expect(
|
||||
isRetryable(
|
||||
bunError(
|
||||
"DEPTH_ZERO_SELF_SIGNED_CERT",
|
||||
"self signed certificate",
|
||||
),
|
||||
),
|
||||
).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("isRetryable: stream truncation versus corruption", () => {
|
||||
it("retries a truncated stream", () => {
|
||||
// Truncation is a transfer that stopped early. The bytes that did
|
||||
@@ -555,36 +471,6 @@ describe("withRetry", () => {
|
||||
expect(delays).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("tells `onRetry` of each attempt it retries, before the wait", async () => {
|
||||
// A caller that logs retries, as the CLI does with `-v`, learns why
|
||||
// each attempt failed and how long the next one is put off. The last
|
||||
// attempt is not retried, so it is not reported.
|
||||
const { sleep, delays } = recordingSleep();
|
||||
const retries: [string, number, number][] = [];
|
||||
let calls = 0;
|
||||
const failure = withRetry(
|
||||
() => {
|
||||
calls++;
|
||||
return Promise.reject(new ApiError(`attempt ${calls}`, 503));
|
||||
},
|
||||
{
|
||||
attempts: 3,
|
||||
sleep,
|
||||
random: () => 0.5,
|
||||
onRetry: (err, attempt, delayMs) => {
|
||||
retries.push([(err as Error).message, attempt, delayMs]);
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
await expect(failure).rejects.toThrow("attempt 3");
|
||||
expect(retries).toEqual([
|
||||
["attempt 1", 1, 250],
|
||||
["attempt 2", 2, 500],
|
||||
]);
|
||||
expect(delays).toEqual([250, 500]);
|
||||
});
|
||||
|
||||
it("does not retry at all when attempts is 1", async () => {
|
||||
const { sleep, delays } = recordingSleep();
|
||||
let calls = 0;
|
||||
@@ -738,7 +624,6 @@ describe("retry defaults", () => {
|
||||
expect(resolved.maxDelayMs).toBe(DEFAULT_RETRY_OPTIONS.maxDelayMs);
|
||||
expect(typeof resolved.sleep).toBe("function");
|
||||
expect(typeof resolved.random).toBe("function");
|
||||
expect(typeof resolved.onRetry).toBe("function");
|
||||
});
|
||||
|
||||
it("resolves to the defaults when given nothing", () => {
|
||||
|
||||
@@ -6,12 +6,10 @@
|
||||
* working thumbnails, others return 404 or empty bodies. The tests
|
||||
* verify that the detection and repair logic handles each case correctly.
|
||||
*
|
||||
* `listMissingThumbnails` takes an open `Library` to enumerate the files. Both
|
||||
* helpers take the `Client` for the API operations: the thumbnail existence
|
||||
* check, and the encrypt-and-upload path. `fixMissingThumbnails` takes the
|
||||
* record of each file, which these tests look up with `cachedFilesByID`. It
|
||||
* reads each original through the library's content cache, by
|
||||
* `lib.originalOf(file)`.
|
||||
* As of issue #52 both helpers take an open `Library` for enumeration and the
|
||||
* `Client` for the API operations that stay unchanged (the thumbnail existence
|
||||
* check, and the encrypt-and-upload path). `fixMissingThumbnails` reads each
|
||||
* original through the library's content cache (`photo.original()`).
|
||||
*
|
||||
* `fixMissingThumbnails` is the most complex function in quak: it
|
||||
* downloads the original file, generates a JPEG thumbnail with jpeg-js,
|
||||
@@ -43,7 +41,6 @@ import {
|
||||
import { SRP, SrpServer } from "fast-srp-hap";
|
||||
import { Client } from "../../src/client.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import { cachedFilesByID } from "../../src/cli-read.js";
|
||||
import {
|
||||
listMissingThumbnails,
|
||||
fixMissingThumbnails,
|
||||
@@ -57,11 +54,8 @@ import type { RetryOptions } from "../../src/retry.js";
|
||||
|
||||
const TEST_EMAIL = "thumb@example.com";
|
||||
const TEST_PASSWORD = "thumbpass";
|
||||
// The password is stretched at libsodium's lowest Argon2id cost. Each test
|
||||
// builds an account and logs in, and at a realistic cost that takes seconds on
|
||||
// a loaded host.
|
||||
const TEST_OPS = 1;
|
||||
const TEST_MEM = 8192;
|
||||
const TEST_OPS = 2;
|
||||
const TEST_MEM = 64 * 1024 * 1024;
|
||||
const TEST_TIME = 1700000000000000;
|
||||
|
||||
interface ThumbMockState {
|
||||
@@ -273,7 +267,6 @@ const buildThumbMock = async (opts?: {
|
||||
|
||||
const buildThumbFetch = (m: ThumbMockState) => {
|
||||
let srpServer: SrpServer;
|
||||
let signedMD5: string | undefined;
|
||||
|
||||
return (async (
|
||||
input: RequestInfo | URL,
|
||||
@@ -366,9 +359,8 @@ const buildThumbFetch = (m: ThumbMockState) => {
|
||||
return new Response("not found", { status: 404 });
|
||||
}
|
||||
|
||||
// Upload URL minting. The real server signs the MD5 into the URL.
|
||||
// Upload URL minting
|
||||
if (path === "/files/upload-url") {
|
||||
signedMD5 = JSON.parse(init?.body as string).contentMD5;
|
||||
return json({
|
||||
objectKey: `42/thumb-${Date.now()}`,
|
||||
url: "https://s3.mock.test/presigned-put",
|
||||
@@ -377,11 +369,6 @@ const buildThumbFetch = (m: ThumbMockState) => {
|
||||
|
||||
// Presigned PUT (S3 upload)
|
||||
if (url.startsWith("https://s3.mock.test/")) {
|
||||
// Storage refuses a PUT that lacks the signed `Content-MD5`.
|
||||
const headers = new Headers(init?.headers);
|
||||
if (headers.get("Content-MD5") !== signedMD5) {
|
||||
return new Response("AccessDenied", { status: 400 });
|
||||
}
|
||||
const body = init?.body;
|
||||
// Store the uploaded bytes for later inspection
|
||||
if (body instanceof Uint8Array) {
|
||||
@@ -557,22 +544,11 @@ describe("listMissingThumbnails", () => {
|
||||
const client = await login(counted.fetch, { ...noWait });
|
||||
const lib = await openLib(client);
|
||||
|
||||
const errors: string[] = [];
|
||||
const missing = await listMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
(msg, level) => {
|
||||
if (level === "error") errors.push(msg);
|
||||
},
|
||||
);
|
||||
const missing = await listMissingThumbnails(lib, client);
|
||||
lib.close();
|
||||
|
||||
// Only the genuinely empty thumbnail is reported. The file that
|
||||
// could not be checked is reported as an error.
|
||||
// Only the genuinely empty thumbnail is reported.
|
||||
expect(missing.map((m) => m.fileID)).toEqual([101]);
|
||||
expect(errors).toEqual([
|
||||
expect.stringMatching(/^\[.*\] Could not check file-102\.jpg: /),
|
||||
]);
|
||||
// And the 500 was retried rather than accepted as an answer: four
|
||||
// attempts is the library default.
|
||||
expect(counted.matched()).toBe(4);
|
||||
@@ -669,12 +645,7 @@ describe("fixMissingThumbnails", () => {
|
||||
const client = await login(buildThumbFetch(fixMock));
|
||||
const lib = await openLib(client);
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
[101],
|
||||
cachedFilesByID(lib, [101]),
|
||||
);
|
||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
||||
lib.close();
|
||||
|
||||
expect(results.length).toBe(1);
|
||||
@@ -711,12 +682,7 @@ describe("fixMissingThumbnails", () => {
|
||||
const client = await login(buildThumbFetch(fixMock));
|
||||
const lib = await openLib(client);
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
[999],
|
||||
cachedFilesByID(lib, [999]),
|
||||
);
|
||||
const results = await fixMissingThumbnails(lib, client, [999]);
|
||||
lib.close();
|
||||
|
||||
expect(results.length).toBe(1);
|
||||
@@ -733,16 +699,7 @@ describe("fixMissingThumbnails", () => {
|
||||
const client = await login(buildThumbFetch(fixMock));
|
||||
const lib = await openLib(client);
|
||||
|
||||
const errors: string[] = [];
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
[101, 102],
|
||||
cachedFilesByID(lib, [101, 102]),
|
||||
(msg, level) => {
|
||||
if (level === "error") errors.push(msg);
|
||||
},
|
||||
);
|
||||
const results = await fixMissingThumbnails(lib, client, [101, 102]);
|
||||
lib.close();
|
||||
|
||||
expect(results.length).toBe(2);
|
||||
@@ -750,10 +707,6 @@ describe("fixMissingThumbnails", () => {
|
||||
const failure = results.find((r) => r.fileID === 102)!;
|
||||
expect(success.status).toBe("fixed");
|
||||
expect(failure.status).toBe("failed");
|
||||
// The failed file, and only it, is reported as an error.
|
||||
expect(errors).toEqual([
|
||||
expect.stringMatching(/^\[.*\] FAILED file-102\.jpg: /),
|
||||
]);
|
||||
});
|
||||
|
||||
it("skips a non-JPEG image and a video as unsupported, not failed (issue #17)", async () => {
|
||||
@@ -769,7 +722,6 @@ describe("fixMissingThumbnails", () => {
|
||||
lib,
|
||||
client,
|
||||
[101, 103, 104],
|
||||
cachedFilesByID(lib, [101, 103, 104]),
|
||||
);
|
||||
lib.close();
|
||||
|
||||
@@ -804,12 +756,7 @@ describe("fixMissingThumbnails", () => {
|
||||
const client = await login(counted.fetch);
|
||||
const lib = await openLib(client);
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
[101],
|
||||
cachedFilesByID(lib, [101]),
|
||||
);
|
||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
||||
lib.close();
|
||||
|
||||
expect(results[0]!.status).toBe("skipped");
|
||||
@@ -830,12 +777,7 @@ describe("fixMissingThumbnails", () => {
|
||||
const client = await login(counted.fetch);
|
||||
const lib = await openLib(client);
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
[101],
|
||||
cachedFilesByID(lib, [101]),
|
||||
);
|
||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
||||
lib.close();
|
||||
|
||||
expect(results[0]!.status).toBe("skipped");
|
||||
@@ -876,12 +818,7 @@ describe("fixMissingThumbnails", () => {
|
||||
const client = await login(buildThumbFetch(fixMock));
|
||||
const lib = await openLib(client);
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
[101],
|
||||
cachedFilesByID(lib, [101]),
|
||||
);
|
||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
||||
lib.close();
|
||||
|
||||
expect(results[0]!.status).toBe("fixed");
|
||||
|
||||
@@ -3,15 +3,8 @@ import { configDefaults, defineConfig } from "vitest/config";
|
||||
// vitest does not read .gitignore when looking for tests. A checkout nested
|
||||
// under .claude/ has its own test/ tree, and without this exclude the suite
|
||||
// runs once per nested checkout and still reports success.
|
||||
//
|
||||
// vitest starts one worker per CPU core by default, and a shared host has
|
||||
// dozens. Four keep the suite well inside the Dockerfile's time limit. vitest
|
||||
// 2 rejects a maxWorkers below its default minWorkers, so minWorkers is set
|
||||
// too.
|
||||
export default defineConfig({
|
||||
test: {
|
||||
exclude: [...configDefaults.exclude, ".claude/**"],
|
||||
minWorkers: 1,
|
||||
maxWorkers: 4,
|
||||
},
|
||||
});
|
||||
Reference in new issue
Block a user