Compare commits
9
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b95f986a36 | ||
|
|
30fc349b37 | ||
|
|
1f409cbbf2 | ||
|
|
aeb275e137 | ||
|
|
67794bf5f8 | ||
|
|
b5fcbf4400 | ||
|
|
a1ea47f9a8 | ||
|
|
f65c237e2c | ||
|
|
af1491966c |
No files matched your search
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
- branch from `next`
|
- branch from `next`
|
||||||
- do the work in Next Step
|
- do the work in Next Step
|
||||||
- move Next Step to the top of Completed Steps
|
- delete Next Step from `TODO.md`
|
||||||
- move the top item of Future Steps into Next Step
|
- move the top item of Future Steps into Next Step
|
||||||
- commit (`TODO.md` changes in the same commit as the work)
|
- commit (`TODO.md` changes in the same commit as the work)
|
||||||
- push
|
- push
|
||||||
@@ -24,723 +24,6 @@ only sneak has, so it waits on him.
|
|||||||
Tagging and releases are decided by sneak alone, and happen only when he
|
Tagging and releases are decided by sneak alone, and happen only when he
|
||||||
declares one.
|
declares one.
|
||||||
|
|
||||||
# Completed Steps
|
|
||||||
|
|
||||||
- 2026-10-25: `quak files COLLECTION` fetches the files of that collection alone
|
|
||||||
(https://git.eeqj.de/sneak/quak/issues/217). It used to wait for a refresh,
|
|
||||||
which on an empty cache fetched the files of every collection first. It now
|
|
||||||
reads the collection list through `lib.freshCollections()`, then brings that
|
|
||||||
one collection up to date through the new `lib.refreshCollection()`. That
|
|
||||||
stores and saves its files as a refresh would, and leaves the refresh cursor
|
|
||||||
where it was. When the stored copy is current, it asks for none of the files.
|
|
||||||
It never runs at the same time as a refresh. Once a refresh has received the
|
|
||||||
account's collections, it stores no collection the cache does not hold, and
|
|
||||||
`quak files COLLECTION` waits for a refresh instead. A refresh from a cursor
|
|
||||||
of 0 replaces all the cache holds, so collections keep the server's order. A
|
|
||||||
library opened on a cache that only `lib.refreshCollection()` wrote awaits its
|
|
||||||
first refresh. `quak files` with no argument still waits for a refresh. The
|
|
||||||
store lists what did not decrypt by collection ID and file ID.
|
|
||||||
|
|
||||||
- 2026-10-22: The JSON `quak backup` and `quak backup-metadata` write for a file
|
|
||||||
holds `keywords` in place of the CLIP embedding in its ML data
|
|
||||||
(https://git.eeqj.de/sneak/quak/issues/218). The keywords are the titles of
|
|
||||||
the Discover categories of Ente's photos app that the file falls in, by the
|
|
||||||
app's rule: the cosine similarity of the file's CLIP embedding and the text
|
|
||||||
embedding of a category's query is at least the category's minimum score, and
|
|
||||||
videos stay out of the categories that leave them out. `src/clip-keywords.ts`
|
|
||||||
applies the rule. `src/clip-keyword-vectors.ts` holds the 16 categories with
|
|
||||||
their text embeddings. `bin/clip-keyword-vectors` writes it with Ente's CLIP
|
|
||||||
text model, and writes the CLIP embedding of a public-domain photo of a sunset
|
|
||||||
that the tests use.
|
|
||||||
|
|
||||||
- 2026-10-18: `-v`/`--verbose` raises the CLI's log level once for each repeat,
|
|
||||||
and `-q`/`--quiet` prints only fatal errors
|
|
||||||
(https://git.eeqj.de/sneak/quak/issues/212). The levels are fatal, error,
|
|
||||||
warning, info, debug and trace; the default is info, everything quak printed
|
|
||||||
on stderr before, `-v` gives debug and `-vv` trace. `src/cli-log.ts` holds the
|
|
||||||
level and the log the commands write stderr lines through. Debug adds each
|
|
||||||
retry, which `withRetry` reports through the new `onRetry`, and the start and
|
|
||||||
end of each refresh and ML data fetch; trace adds each request, which
|
|
||||||
`ApiClient` reports through the new `onRequest`. `--json` no longer keeps the
|
|
||||||
progress lines off stderr, so `backup` and `--json` runs log a refresh's
|
|
||||||
progress too. Every line that reports a failed file or album is at error,
|
|
||||||
which the progress callbacks of `backup`, `backup-metadata` and the thumbnail
|
|
||||||
helpers now say, and a command that exits 1 because files or albums failed
|
|
||||||
ends with one fatal line saying how many and where to look.
|
|
||||||
|
|
||||||
- 2026-10-18: `quak files` takes the collection ID as an optional argument
|
|
||||||
instead of the `--collection` option, which is gone
|
|
||||||
(https://git.eeqj.de/sneak/quak/issues/211). With no argument it lists every
|
|
||||||
file in the account once, collection by collection; a file in several
|
|
||||||
collections is listed with the first of them, which is its `collectionID` in
|
|
||||||
`--json`. The fresh read in `src/cli-read.ts` does this.
|
|
||||||
|
|
||||||
- 2026-10-18: Names quak builds from an album name or a title fit in a file name
|
|
||||||
(https://git.eeqj.de/sneak/quak/issues/205). `shortenKeepingExtension` in
|
|
||||||
`src/filename.ts` cuts a name short to a number of bytes, keeping its
|
|
||||||
extension when it fits and always its first character, and can put text right
|
|
||||||
before the extension; the backup's `withID` uses it for an ID.
|
|
||||||
`defaultFileName` there gives the name a file is saved under when the caller
|
|
||||||
gives no path: a prefix such as `thumb_` and the title with unsafe characters
|
|
||||||
replaced, cut short with it to 255 bytes, and, for a live photo's image or
|
|
||||||
video, with that part's own extension before the cut. `quak get`,
|
|
||||||
`quak get-thumb`, `downloadFile` and `downloadThumbnail` use it without
|
|
||||||
`--out` or `outPath`. `backup-metadata` cuts an album's name short so that its
|
|
||||||
folder, `<id>-<name>`, fits in 255 bytes. `safeExtension` gives `.bin` for an
|
|
||||||
extension of more than 16 letters and digits, so a save path's name always
|
|
||||||
fits.
|
|
||||||
|
|
||||||
- 2026-10-18: `quak collections` no longer waits for the files of every
|
|
||||||
collection (issue 210). On an empty cache it opened its library with a refresh
|
|
||||||
that fetched and decrypted the files of each collection in turn, printing
|
|
||||||
nothing and storing nothing until all were done, so on a large account it
|
|
||||||
seemed to hang. It now reads the collection list alone through the new
|
|
||||||
`lib.freshCollections()`, one request that stores nothing. The read commands
|
|
||||||
open their library with the new `refreshOnOpen: false`, so each makes only the
|
|
||||||
refresh it asks for. A refresh sends a `"progress"` event after each
|
|
||||||
collection whose files it fetched, and the commands that wait on one print it
|
|
||||||
on stderr, except with `--json`.
|
|
||||||
|
|
||||||
- 2026-10-16: ML data the server sends that does not decrypt, gunzip or parse is
|
|
||||||
no longer taken for no ML data (issue 198). `fetchMLDataBatch` returns why for
|
|
||||||
each such file beside the payloads; ML data that parses to something other
|
|
||||||
than a JSON object, such as `null`, is one of them. The library keeps the
|
|
||||||
reasons from its last ML data fetch that succeeded in memory, not in the
|
|
||||||
cache, and gives them in `status().mlDataErrors` and in the `fetchMLData`
|
|
||||||
event with status `"done"`; a fetch still running or one that fails does not
|
|
||||||
change them. `quak backup` writes the reason as `mlDataError` in the file's
|
|
||||||
JSON when no ML data is cached for it, and records the file in `failures.json`
|
|
||||||
with an error starting `ML data: `; `backup-metadata` writes `mlDataError`
|
|
||||||
too. Both exit 1. Tests serve ML data encrypted under a key that is not the
|
|
||||||
file's.
|
|
||||||
|
|
||||||
- 2026-10-16: With `--verify`, an original whose hash does not match is no
|
|
||||||
longer removed before its replacement is downloaded (issue 199).
|
|
||||||
`replaceOriginal` in `src/library/content.ts` downloads the replacement to a
|
|
||||||
temporary file beside it, a live photo's image and video each to its own,
|
|
||||||
hashes it and renames it over the original only when it matches. When the
|
|
||||||
download fails or does not match either, the original keeps its bytes and
|
|
||||||
links, and the file goes into `failures.json`. The replacement is always
|
|
||||||
downloaded, through the new `fetchOriginal` of the content cache, never copied
|
|
||||||
from the cache.
|
|
||||||
|
|
||||||
- 2026-10-16: The `~/bin/quak` that `make install` builds retries a refused
|
|
||||||
connection and a name that does not resolve, and the backup records either as
|
|
||||||
transient (issue 206). Bun's `fetch` reports them with codes of its own,
|
|
||||||
`ConnectionRefused` and `FailedToOpenSocket`, which `src/retry.ts` now counts
|
|
||||||
as transport failures. A `POST` or `PUT` is not replayed on them, because bun
|
|
||||||
also gives them when it resends a request on a new connection after a
|
|
||||||
connection it kept open closed, and the server may already have the request.
|
|
||||||
Bun already gave `ECONNRESET` for a reset and a `TimeoutError` for a deadline.
|
|
||||||
`classify` in `src/backup.ts` now records any `TimeoutError` as transient, so
|
|
||||||
a stalled download, whose error says only that it stalled, is no longer
|
|
||||||
`unknown`. The shapes were found by running the binary against local
|
|
||||||
listeners.
|
|
||||||
|
|
||||||
- 2026-10-15: An album whose folder or JSON `quak backup` cannot write no longer
|
|
||||||
ends the run before the other albums and `failures.json` are written (issue
|
|
||||||
197). The run prints `FAILED album <name>: <reason>`, lists the album's ID in
|
|
||||||
`failedAlbums` in the result and in `--json`, goes on, and exits 1. An album
|
|
||||||
whose folder would have the name of another album's JSON, as `x.json` beside
|
|
||||||
`x` would, gets its ID added. A folder or link name over 255 bytes is cut
|
|
||||||
short between two characters, by `shortenToBytes` in `src/filename.ts`, and
|
|
||||||
gets the ID.
|
|
||||||
|
|
||||||
- 2026-10-15: A network failure is printed and recorded with its reason, not as
|
|
||||||
just `fetch failed` (issue 196). `describeError` in `src/errors.ts` gives an
|
|
||||||
error's message followed by the message of each error on its `cause` chain,
|
|
||||||
such as `fetch failed: connect ECONNREFUSED 1.2.3.4:443`, and each address's
|
|
||||||
reason when a host has several and none could be reached. `run` prints that,
|
|
||||||
and the backup logs it and records it in `failures.json`, where `classify` now
|
|
||||||
calls the failure transient when the chain holds an errno that
|
|
||||||
`hasTransportCode` in `src/retry.ts` recognises, for a failed ML data fetch
|
|
||||||
too. The walk along the chain moved from `src/retry.ts` to `src/errors.ts`,
|
|
||||||
which both use. Tests refuse every connection of a backup, then only one
|
|
||||||
file's download, and fail the ML data fetch.
|
|
||||||
|
|
||||||
- 2026-10-15: The README says that `make install` needs nix (issue 200), in
|
|
||||||
Getting Started and in "Running the backup from cron": `make build-bin` runs
|
|
||||||
bun through `nix-shell -p bun`. The `Makefile` is unchanged.
|
|
||||||
|
|
||||||
- 2026-10-14: `lib.backup()`, and so `quak backup`, works on up to the content
|
|
||||||
pool's limit of originals at once, 5 by default, instead of one at a time
|
|
||||||
(issue 182). That many workers take files from one shared list, each doing all
|
|
||||||
the work for one original, its `--verify` hashing included, so no more
|
|
||||||
originals than that are open at once. `BackupLibrary` says the limit as
|
|
||||||
`contentConcurrency`. A test holds each download at the fake Ente server to
|
|
||||||
count those in flight. Once a worker has thrown, as one does when `onProgress`
|
|
||||||
throws, or the lock is lost, no worker takes another file; the backup fails
|
|
||||||
once the originals under way are in place, and a lost lock stops it before it
|
|
||||||
fetches ML data.
|
|
||||||
|
|
||||||
- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's
|
|
||||||
review, though that issue closed on 2026-10-03 (issue 194). It now names the
|
|
||||||
issues open on the tracker.
|
|
||||||
|
|
||||||
- 2026-10-14: A backup whose lock is lost no longer dies with a stack trace
|
|
||||||
(issue 184). `lockBackupDirectory` gives proper-lockfile an `onCompromised`
|
|
||||||
that records why the lock was lost instead of throwing from its timer, and
|
|
||||||
returns the lock, which `quak backup` passes to `lib.backup()` as `lock`. The
|
|
||||||
backup then stops before it starts another file with
|
|
||||||
`lost the lock on DIR: REASON`, which `quak backup` prints as one line before
|
|
||||||
it exits 1. Releasing a lost lock leaves `backup.lock` alone. Tests in
|
|
||||||
`test/cli/backup-end-to-end.test.ts` remove the lock, or put another backup's
|
|
||||||
in its place, while a download is held open.
|
|
||||||
|
|
||||||
- 2026-10-14: An album or file quak cannot decrypt, or whose metadata is not
|
|
||||||
JSON, no longer fails the refresh, so one bad record no longer stops
|
|
||||||
`quak backup` from storing anything (issue 181). The refresh leaves it out, an
|
|
||||||
album with its files, and the library keeps it with the reason in
|
|
||||||
`metadata.json` until the server sends a copy that decrypts or deletes it.
|
|
||||||
`quak backup` stores every other file, records each such file in
|
|
||||||
`failures.json` with an error starting `could not be decrypted:`, names each
|
|
||||||
such album's ID in a printed line, or with `--json` in `undecryptableAlbums`,
|
|
||||||
and exits 1 while any remain. A file whose copy in another album decrypts is
|
|
||||||
not recorded: its original is stored from that copy, and the album where it
|
|
||||||
does not decrypt neither links nor lists it. The read commands answer from
|
|
||||||
what decrypted.
|
|
||||||
|
|
||||||
- 2026-10-14: `quak backup` no longer asks the server for changes every 3
|
|
||||||
seconds while it runs (issue 183). It opens its library with
|
|
||||||
`refreshIntervalSeconds: Infinity`, which the library now takes as no
|
|
||||||
background timer, so the backup refreshes only as it starts and no timed
|
|
||||||
refresh starts another ML data pass during the run. A test holds a backup's
|
|
||||||
download open while two hours pass and counts its `/collections/v2` requests.
|
|
||||||
|
|
||||||
- 2026-10-14: `test/cli/backup-end-to-end.test.ts` runs `quak backup` as cron
|
|
||||||
does (issue 186): `backupCommand` and `run`, the real `loadSession` reading a
|
|
||||||
session saved after logging in, and every request answered by a fake of the
|
|
||||||
Ente HTTP API serving data encrypted with libsodium. It covers the backup tree
|
|
||||||
of an account with two albums of its own and one shared with it, a second run
|
|
||||||
that downloads nothing, a download that succeeds after two 503s, a 404 that
|
|
||||||
lands in `failures.json` with exit 1, a refresh answered 401 with exit 3, and
|
|
||||||
`--verify` putting back an original changed on disk. The fake server and the
|
|
||||||
account it logs in moved from the backup-metadata test into
|
|
||||||
`test/ente-server.ts`, unchanged.
|
|
||||||
|
|
||||||
- 2026-10-14: `files --collection`, `get`, `get-thumb` and
|
|
||||||
`helper fix-missing-thumbnails --file` take an ID only when it is written in
|
|
||||||
decimal digits and is a safe integer (issue 185). Before, `Number()` read
|
|
||||||
`0x10` as 16, `1e3` as 1000 and an empty string as 0, and `--file` dropped an
|
|
||||||
ID it could not read without a word. Anything else now prints
|
|
||||||
`Invalid file ID: <arg>` or `Invalid collection ID: <arg>` and exits 1 before
|
|
||||||
any request; one bad `--file` ID fails the whole command. `parseID` in
|
|
||||||
`src/cli-commands.ts` does the check for all four.
|
|
||||||
|
|
||||||
- 2026-10-06: The README says how to run `quak backup` from cron (issue 170):
|
|
||||||
log in once as the job's user, a crontab with a backup every night and
|
|
||||||
`--verify` on Sundays appending to a log file, and the `HOME` and
|
|
||||||
`XDG_DATA_HOME` the job needs to find the saved session. A table gives each
|
|
||||||
exit code: 0, the backup is complete; 1, files are in `failures.json` or
|
|
||||||
another error stopped the run; 2, another backup of the directory is running;
|
|
||||||
3, there is no usable session. The introduction lists everything the backup
|
|
||||||
keeps for each file.
|
|
||||||
|
|
||||||
- 2026-10-06: `quak backup --verify` and `lib.backup({ verify: true })` hash
|
|
||||||
each original already at its save path as the download check does, streamed, a
|
|
||||||
live photo as `<imageHash>:<videoHash>` (issue 168). One that does not match
|
|
||||||
the content hash its metadata records is logged, removed (both files of a live
|
|
||||||
photo) and put back in the same run, downloaded or copied from the cache, and
|
|
||||||
what is put back is hashed too. One that still does not match, or whose
|
|
||||||
download fails, goes into `failures.json`. One with no recorded hash is left
|
|
||||||
alone. The result, `--json` and the summary gain `verified`, `mismatched` and
|
|
||||||
`unchecked`. Without `--verify` nothing is hashed.
|
|
||||||
|
|
||||||
- 2026-10-06: Two backups of the same directory never run at once (issue 169).
|
|
||||||
`lib.backup()` takes a lock, `backup.lock` in its download directory, made
|
|
||||||
with `proper-lockfile`, before its refresh, and removes it when it ends,
|
|
||||||
whether it succeeds or fails. A second backup of the directory, from another
|
|
||||||
process or the same one, fails at once with an error naming the directory.
|
|
||||||
`quak backup` takes the lock before it opens its library, so a refused run
|
|
||||||
sends no request; it prints the error as one line and exits 2. A lock that has
|
|
||||||
gone 10 seconds untouched, left by a run that could not remove it, is taken
|
|
||||||
over by the next run.
|
|
||||||
|
|
||||||
- 2026-10-06: `quak backup` writes each original's EXIF, XMP and dimensions into
|
|
||||||
the file's JSON as `imageMetadata`, what `backup-metadata --exif` records
|
|
||||||
(issue 167): for a live photo from its image, for a video nothing, and `{}`
|
|
||||||
for an original with none of them. A failed read puts the reason in
|
|
||||||
`imageMetadataError` and fails neither the file nor the run. An original is
|
|
||||||
read when the run stores it or when its JSON has neither field; otherwise the
|
|
||||||
field is taken from that JSON. The hand-built JPEGs moved to
|
|
||||||
`test/exif-jpeg.ts`, beside the HEIC.
|
|
||||||
|
|
||||||
- 2026-10-06: When the server answers HTTP 401 and that ends a command that
|
|
||||||
loads the saved session, because the server no longer accepts its token, the
|
|
||||||
command prints one line,
|
|
||||||
`quak: the saved session is no longer valid; run "quak login"`, and exits 3
|
|
||||||
(issue 164). `quak backup` meets that 401 on the refresh it starts with,
|
|
||||||
before it touches any file. For the same commands, a missing or corrupt
|
|
||||||
session file keeps its message and also exits 3, so a cron job can tell that
|
|
||||||
the user must log in again. `quak logout` is unchanged. `run` in
|
|
||||||
`src/cli-run.ts` recognises the 401, which reaches it unchanged.
|
|
||||||
|
|
||||||
- 2026-10-06: `quak backup` retries a failed request for longer than the other
|
|
||||||
commands do (issue 165). `src/retry.ts` exports `UNATTENDED_RETRY_OPTIONS`
|
|
||||||
beside the unchanged default: 10 attempts, a 1 s base delay and a 60 s cap, so
|
|
||||||
a request that keeps failing waits at most 243 s before it gives up.
|
|
||||||
`bin/quak.ts` loads the backup's session with them, so its refresh, ML data
|
|
||||||
and downloads all use them. What is retried and the backoff formula are
|
|
||||||
unchanged.
|
|
||||||
|
|
||||||
- 2026-10-06: The files this repository copies from `sneak/prompts` are copied
|
|
||||||
again from its commit `dd4027b` (issue 171). `.gitignore` and `.dockerignore`
|
|
||||||
keep out more secret files, and this repository's build artifacts follow the
|
|
||||||
copied content. `script/check` runs `script/fmt-check` again, `script/cibuild`
|
|
||||||
runs `script/bootstrap` and `script/check` before the image build,
|
|
||||||
`script/fmt` and `script/fmt-check` find the pinned yarn under nvm, and the
|
|
||||||
image's last stage marks `/app` safe for git.
|
|
||||||
|
|
||||||
- 2026-10-06: `quak backup` writes the account and album records
|
|
||||||
`backup-metadata` writes (issue 166): `account.json` with the account's
|
|
||||||
`email` and `userID`, and in each album's JSON its `ownerID`, `isShared`,
|
|
||||||
`updationTime` and, when present, its three layers of magic metadata. Each
|
|
||||||
file's JSON gains its `updationTime`.
|
|
||||||
|
|
||||||
- 2026-10-05: `quak backup` writes each file's ML data (its faces and CLIP
|
|
||||||
embedding) into the file's JSON as `mlData`, the payload
|
|
||||||
`lib.mldata.forFile()` returns (issue 163). `lib.backup()` waits for an ML
|
|
||||||
data fetch before it writes the JSON files. When that fetch fails, each file
|
|
||||||
whose ML data is not cached gets the reason in `mlDataError` and counts as
|
|
||||||
failed, and the next run fetches it again.
|
|
||||||
|
|
||||||
- 2026-10-03: The download-albums example test no longer fails on a disk with
|
|
||||||
under 50 GiB free (issue 160). It opens its libraries with
|
|
||||||
`freeBelowBytes: 0`, so the free space of the disk it runs on cannot shrink
|
|
||||||
the cache, evict the original it cached and add a fetch to the count. No other
|
|
||||||
test's result depends on that free space. The 50 GiB default is unchanged.
|
|
||||||
|
|
||||||
- 2026-10-02: `docker build .` stamps the commit's tag or short commit, not
|
|
||||||
`dev` (issue 154). `script/build` writes the version `script/version` prints
|
|
||||||
into `dist/package.json`: the `VERSION` environment variable or build arg when
|
|
||||||
one is given, otherwise `git describe --tags --always`, otherwise the version
|
|
||||||
`package.json` declares. `.dockerignore` sends `.git`, and a checkout with
|
|
||||||
`.git` whose version comes out empty, `dev` or `unknown` fails the build.
|
|
||||||
`make build-bin` bundles the built `dist/`, so the single binary reports the
|
|
||||||
same version.
|
|
||||||
|
|
||||||
- 2026-10-02: `photo.exif()` returns every EXIF tag in the file as `ExifTags`,
|
|
||||||
keyed by tag name, each as exifreader decodes it, not only the thirteen common
|
|
||||||
fields (issue 156). The embedded thumbnail's tags are under `Thumbnail`,
|
|
||||||
without the thumbnail image. The thirteen typed methods stay, each picking its
|
|
||||||
field from the tags `exif()` returns, typed as in `PhotoExif`. The example
|
|
||||||
script's JSON files now carry every tag.
|
|
||||||
|
|
||||||
- 2026-10-01: The content cache no longer looks for a live photo that an earlier
|
|
||||||
version cached as one ZIP (issue 151). When the cache opens, a live photo's
|
|
||||||
file that no JSON file names is now always left alone.
|
|
||||||
|
|
||||||
- 2026-10-01: `examples/download-albums.ts` logs in, opens the library, and for
|
|
||||||
every album downloads each photo to its save path, writes the photo's record
|
|
||||||
and EXIF fields to a JSON file beside it, and writes the album's photos to
|
|
||||||
`albums/<collectionID>.json` (issue 144). The build compiles it to
|
|
||||||
`dist/examples/`; the README's "Examples" section says how to run it.
|
|
||||||
|
|
||||||
- 2026-10-01: A `Photo` has one async method for each field of `exif()`, named
|
|
||||||
and typed as in `PhotoExif`: `make()`, `model()`, `lensModel()`,
|
|
||||||
`dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`,
|
|
||||||
`iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()`
|
|
||||||
and `gpsAltitude()` (issue 148). Each calls `exif()` and returns its one
|
|
||||||
field, or undefined when the file lacks it. `Photo` implements a type with one
|
|
||||||
method per `PhotoExif` field, so the build's type check fails when a field has
|
|
||||||
no method. A test checks, on the JPEG and the HEIC, that each method gives the
|
|
||||||
same value as `exif()`.
|
|
||||||
|
|
||||||
- 2026-10-01: Each original's save path is
|
|
||||||
`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
|
|
||||||
download directory, which defaults to `photos` in the working directory (issue
|
|
||||||
143). The date is the photo's `takenAt` in the machine's time zone.
|
|
||||||
`photo.savePath` is always a string, with or without a content cache, and
|
|
||||||
`isLocal` is true only when the original is at its save path.
|
|
||||||
`photo.download()` puts the original there, copied from the cache when the
|
|
||||||
cache holds it and fetched otherwise. `lib.backup()` does the same for each
|
|
||||||
file, writes each file's JSON beside its original, and links `collections/` to
|
|
||||||
the save paths.
|
|
||||||
|
|
||||||
- 2026-10-01: `photo.exif()` and `backup-metadata --exif` read EXIF from
|
|
||||||
HEIC/HEIF originals, a live photo's HEIC image included, as well as JPEG and
|
|
||||||
the other image formats `exifreader` reads (issue 145). `exifreader` replaces
|
|
||||||
`exif-reader` and the JPEG segment scan; `PhotoExif` is unchanged. The `exif`
|
|
||||||
field of `backup-metadata --exif` is now exifreader's tag output, and
|
|
||||||
`exifRaw` holds the whole EXIF block it could not read. The tests use a real
|
|
||||||
HEIC, `test/exif.heic`.
|
|
||||||
|
|
||||||
- 2026-10-01: A `Photo` has `savePath`, `isLocal`, `content()`, `exif()`,
|
|
||||||
`modifiedAt`, `hash` and `year` (issue 141). `savePath` is where
|
|
||||||
`lib.backup()` writes the original under the library's download directory; for
|
|
||||||
a live photo not yet stored, it carries the title's extension, and the backup
|
|
||||||
may store the image under a different one. `isLocal` says whether the whole
|
|
||||||
original is there. Both look only at the disk. `content()` returns the
|
|
||||||
original's bytes and `exif()` the common EXIF fields of a JPEG; both may
|
|
||||||
download the original, and `exif()` downloads no video. `PhotoRecord` gains
|
|
||||||
`modifiedAt` and `hash`, and the JPEG EXIF scan moved to `src/exif.ts`.
|
|
||||||
|
|
||||||
- 2026-09-29: The "Workflow" list at the top of this file now says to branch
|
|
||||||
from `next` and open a pull request that targets `next`, that the repository
|
|
||||||
manager squash-merges a reviewed pull request into `next`, and that only sneak
|
|
||||||
merges `next` into `main`, as the README does (issue 137).
|
|
||||||
|
|
||||||
- 2026-09-29: The README's "Development workflow" and "For LLMs" sections now
|
|
||||||
say work branches from `next`, every pull request targets `next`, the
|
|
||||||
repository manager squash-merges reviewed pull requests into `next`, and only
|
|
||||||
sneak merges `next` into `main` (issue 135).
|
|
||||||
|
|
||||||
- 2026-09-29: Brought the README and this file in line with `next` after the
|
|
||||||
milestone merge (issue 132). The Next Step says no implementation work is open
|
|
||||||
and the cache design (issue 36) waits on sneak's review. README corrections:
|
|
||||||
the Getting Started comments say when the library refreshes; most, not all,
|
|
||||||
Makefile targets call a script; `script/lint` and `script/test` have no host
|
|
||||||
path, though `yarn test` does; the SRP handshake uses `fast-srp-hap`, outside
|
|
||||||
`crypto/`, and a thumbnail upload's MD5 uses `node:crypto`; the SRP password
|
|
||||||
is the first 16 bytes of a 32-byte subkey; the key attributes and token come
|
|
||||||
after SRP, after the TOTP code SRP may ask for, or after the email OTP that
|
|
||||||
replaces SRP when the account has email MFA on, and quak cannot answer a
|
|
||||||
passkey; the auth token is sent as URL-safe base64 with padding; every
|
|
||||||
`TypeError` is retried; each download attempt writes its own temporary files,
|
|
||||||
two for a live photo, and only the attempt that completes renames them into
|
|
||||||
place; `runMetadataBackup` records a failed download in the file's JSON; a
|
|
||||||
second `client.logout()` does not throw; `quak logout` with no session exits
|
|
||||||
0, and names the cache directory only when it knows it; `login`, `whoami` and
|
|
||||||
`logout` open no library; `--exif` records XMP and, for a JPEG, EXIF, and no
|
|
||||||
IPTC; which commands take `--json`; a failed ML data request may not have been
|
|
||||||
retried; the thumbnail fixer is not limited to baseline JPEG; `quak backup`
|
|
||||||
still fetches ML data; a backup's JSON holds the basic metadata fields quak
|
|
||||||
keeps and the private and public magic metadata, not every decrypted field,
|
|
||||||
and the README no longer lists what the magic metadata holds; the default
|
|
||||||
cache directory is the per-user one, not an XDG path on macOS; pinned
|
|
||||||
originals can exceed `cacheOriginalsMaxBytes`; which tests cover which
|
|
||||||
operations; a default read is only as current as the last refresh whose
|
|
||||||
requests all succeeded; `fresh()` and `lib.backup()` join a refresh already
|
|
||||||
running; a `Photo` has no `thumbnailPath` or `originalPath`; and `408` and
|
|
||||||
`429` are retried.
|
|
||||||
|
|
||||||
- 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks
|
|
||||||
that the image's and the video's temp files are fsynced before either is
|
|
||||||
renamed into place, and the directory after both renames, as the `writeAtomic`
|
|
||||||
test does for one file.
|
|
||||||
|
|
||||||
- 2026-09-23: Live photos are stored as their image and their video (issue 107).
|
|
||||||
A live photo, which Ente stores as one ZIP, is unpacked as it downloads into
|
|
||||||
`<fileID>.<ext>` for the image and for the video, each with its extension from
|
|
||||||
the ZIP, beside `<fileID>.livephoto.json`, which names the two. Both are
|
|
||||||
checked against the recorded hash and renamed into place only when both are
|
|
||||||
complete. A ZIP holding a second image or video, or whose two files come to
|
|
||||||
more than 20 times its size plus 16 MiB, is refused. The backup and the
|
|
||||||
content cache count the live photo as stored only with both files, the album
|
|
||||||
folder links both, `quak get` writes both, and the content result gives the
|
|
||||||
video as `videoPath`. A ZIP an earlier version stored is replaced on the next
|
|
||||||
backup run, and removed from the cache when the library opens it.
|
|
||||||
|
|
||||||
- 2026-09-23: Settled the package metadata (issue 6). quak is not published, so
|
|
||||||
`package.json` is marked `"private": true` and the `files` field is gone.
|
|
||||||
`engines.node` is `>=22`, the major version `script/bootstrap` and the
|
|
||||||
`Dockerfile` use. An `exports` map makes `.` and `./package.json` the only
|
|
||||||
importable paths; `runMetadataBackup`, the thumbnail helpers and their types
|
|
||||||
stay internal to the CLI.
|
|
||||||
|
|
||||||
- 2026-09-23: Brought the README and this file in line with the tree (issue
|
|
||||||
111). The layout lists `src/library/` and the other source files, the backup
|
|
||||||
layout names `failures.json` and the optional `thumbnails/`, "Opening a
|
|
||||||
library" says what happens when the first refresh fails, the Testing section
|
|
||||||
gives the 60-second hard cap and 20-second target for `make test` and names
|
|
||||||
the 90-second `timeout` in the `Dockerfile` as the backstop for a hung test,
|
|
||||||
and "Tag v1.0.0" is no longer listed as the next step.
|
|
||||||
|
|
||||||
- 2026-09-23: Tested `quak login` and `backup-metadata --exif` (issue 110).
|
|
||||||
`loginCommand` takes its login function and its prompts from `CliContext`, and
|
|
||||||
`bin/quak.ts` passes `Client.login` and the terminal prompts. Tests cover a
|
|
||||||
login from `QUAK_EMAIL` and `QUAK_PASSWORD` with no prompt, the TOTP prompt, a
|
|
||||||
failed login, and the saved session's modes, and show that `--exif` and
|
|
||||||
`--all` each turn on EXIF extraction and that it is off without them.
|
|
||||||
|
|
||||||
- 2026-09-23: `quak backup` writes each original once and no longer fills the
|
|
||||||
cache (issue 106). An original fetched for a backup is written by the download
|
|
||||||
writer straight into the backup's `originals/`, and the content cache records
|
|
||||||
it there instead of keeping its own copy; one the cache already held is still
|
|
||||||
copied. `quak backup` opens its library with the thumbnail and originals
|
|
||||||
precache off.
|
|
||||||
|
|
||||||
- 2026-09-23: `backup-metadata`, `helper list-missing-thumbnails` and
|
|
||||||
`helper fix-missing-thumbnails` refresh before they answer (issue 100). Each
|
|
||||||
awaits `lib.fresh()` before reading, so a file added since the cache was
|
|
||||||
written is included, and a failed refresh prints one line and exits 1 instead
|
|
||||||
of answering from a stale or empty cache. The README lists them among the
|
|
||||||
commands that refresh first.
|
|
||||||
|
|
||||||
- 2026-09-23: `quak backup` waits for the server refresh and fails when it fails
|
|
||||||
(issue 99). `lib.backup()` joins a refresh already running or starts one, as
|
|
||||||
`fresh()` does, and rejects before touching any file when it fails, leaving
|
|
||||||
`failures.json` as it was, so `quak backup` prints the error as one line and
|
|
||||||
exits 1 instead of backing up the previous run's file list, or nothing, and
|
|
||||||
exiting 0.
|
|
||||||
|
|
||||||
- 2026-09-23: CLI errors print a message instead of a stack trace (issue 102).
|
|
||||||
An error a command throws is printed as one `quak: MESSAGE` line on stderr and
|
|
||||||
the CLI exits 1 once output has drained. The wrapper that does this moved from
|
|
||||||
`bin/quak.ts` to `src/cli-run.ts`, and `bin/quak.ts` now awaits
|
|
||||||
`program.parseAsync()`.
|
|
||||||
|
|
||||||
- 2026-09-23: Opening a library no longer deletes another process's download in
|
|
||||||
progress (issue 105). The download writer's temp files are named
|
|
||||||
`.quak-<pid>-<random>.tmp`, and `removeLeftoverTempFiles`, moved from the
|
|
||||||
backup into the download module, deletes a `.quak-*.tmp` file only when the
|
|
||||||
process ID in its name is no longer running. The content cache calls it at
|
|
||||||
`open()` for `originals/` and `thumbnails/`, the backup as before.
|
|
||||||
|
|
||||||
- 2026-09-23: Re-vendored the lint and test setup from the template (issue 96).
|
|
||||||
Linting and testing are the `lint` and `test` phases of the `Dockerfile`;
|
|
||||||
`script/lint` and `script/test` each build one with `--no-cache`, and the last
|
|
||||||
stage compiles and depends on both, so `script/cibuild` is one build.
|
|
||||||
`Dockerfile.lint`, `CHECK_EPOCH`, `LINT_EPOCH` and the tests that checked them
|
|
||||||
are gone; `REPO_POLICIES.md` is re-copied.
|
|
||||||
|
|
||||||
- 2026-09-23: Stopped `helper fix-missing-thumbnails` retrying files the server
|
|
||||||
always refuses (issue 109). Both thumbnail helpers skip a file another account
|
|
||||||
owns without fetching it. The fixer skips a file whose recorded thumbnail size
|
|
||||||
is 0 or unknown before downloading it, and otherwise tries smaller encodings
|
|
||||||
(720 px quality 50 down to 160 px quality 20) until the encrypted thumbnail is
|
|
||||||
no larger than that size, skipping the file if none fits.
|
|
||||||
|
|
||||||
- 2026-09-23: Tested the live-photo hash check's error paths (issue 117). Tests
|
|
||||||
download a live photo whose ZIP names an unknown compression method, one whose
|
|
||||||
ZIP has no image entry and one with no video entry, and check that nothing is
|
|
||||||
stored and the error names the file ID; the unreadable one is not retried.
|
|
||||||
|
|
||||||
- 2026-09-23: `quak logout` ends the session on the server (issue 108). It calls
|
|
||||||
`POST /users/logout` through the new `Client.logoutOnServer()`, then deletes
|
|
||||||
`session.json` even when that call fails, says so and exits 1. It prints the
|
|
||||||
account's cache directory and says it still holds decrypted data. The default
|
|
||||||
cache path is now `defaultCacheDirectory()` in the library, shared with
|
|
||||||
`Library.open`.
|
|
||||||
|
|
||||||
- 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files
|
|
||||||
in one collection with the same title, and two collections with the same name,
|
|
||||||
each get their ID added to the name (`IMG_0001 (12345).JPG`, `Trip (10)/`), so
|
|
||||||
none replaces another's symlink or JSON. Each run removes symlinks into
|
|
||||||
`originals/` for files no longer in the collection, and the folders of deleted
|
|
||||||
or renamed collections, leaving anything else in `collections/` alone. The
|
|
||||||
README backup layout states the naming rule.
|
|
||||||
|
|
||||||
- 2026-09-23: Checked downloaded originals against their recorded content hash
|
|
||||||
(issue 68). `downloadFile`, which `quak get`, the content cache and backup all
|
|
||||||
use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and
|
|
||||||
stores nothing on a mismatch, failing with an error naming the file ID. A live
|
|
||||||
photo ZIP is unpacked as it streams with `fflate` and its image and video
|
|
||||||
hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older
|
|
||||||
clients' `imageHash` and `videoHash` fields for live photos. A file with no
|
|
||||||
recorded hash is stored unchecked.
|
|
||||||
|
|
||||||
- 2026-09-23: Kept one account's cache from mixing with another's (issue 104).
|
|
||||||
When `metadata.json` in the cache directory was written for a different,
|
|
||||||
non-zero user ID than the client's, `Library.open` deletes it and `mldata/`
|
|
||||||
and starts empty, so the first refresh enumerates from 0. This only happens
|
|
||||||
with `--cache-dir` or an explicit `cacheDirectory`; the default path already
|
|
||||||
includes the user ID. A test opens one account's cache as another account.
|
|
||||||
|
|
||||||
- 2026-09-23: `backup-metadata` no longer stops on one failed ML data request
|
|
||||||
(issue 101). Each request of up to 200 files is tried on its own; a failed one
|
|
||||||
is logged, its files are written with the reason in `mlDataError`, and the
|
|
||||||
command exits 1 once the dump is complete. `fetchMLData`, which only this
|
|
||||||
command used, is gone; the command calls `fetchMLDataBatch` per batch.
|
|
||||||
- 2026-09-23: Single-sourced the version string (issue 5). `package.json` is the
|
|
||||||
only place it is written: `src/index.ts` imports it for `VERSION` and
|
|
||||||
`bin/quak.ts` passes `VERSION` to commander. tsc copies `package.json` to
|
|
||||||
`dist/package.json`, so the import resolves from the built output too, and
|
|
||||||
`script/build` runs the built CLI with `--version` to prove it. A test checks
|
|
||||||
that `VERSION` and `quak --version` both equal the `package.json` version.
|
|
||||||
- 2026-09-23: Tested that `Library.close()` waits for the originals precache
|
|
||||||
(issue 93). The test that holds a precache fetch open while `close()` runs now
|
|
||||||
runs once with only the thumbnail fill and once with only the originals fill,
|
|
||||||
so dropping either wait from `Precache.close()` fails a test.
|
|
||||||
- 2026-09-23: Fixed two intermittently failing library tests (issue 90).
|
|
||||||
`Library.close()` now returns a promise that resolves once an in-flight
|
|
||||||
refresh (including its cache write), the ML data fetch and running precache
|
|
||||||
sweeps have finished; the library tests await it, so `afterEach` no longer
|
|
||||||
removes the cache directory while something is still writing into it. The
|
|
||||||
precache test waits for both fills to report "done" instead of for its stub
|
|
||||||
source to be called, which happened before the cache recorded the file.
|
|
||||||
- 2026-09-23: Pinned three guards reviewers found untested (issue 89). The
|
|
||||||
download idle deadline's timer is unref'd, so it can never keep the process
|
|
||||||
alive, and a test checks no timer is left after a download completes or fails.
|
|
||||||
A test covers the rejection of `#` in a request path. The EXIF scan compares
|
|
||||||
the `Exif` header only in an APP1 segment of length 8 or more, so it never
|
|
||||||
reads the next segment's bytes, with a test for a short one.
|
|
||||||
- 2026-09-23: Made the download deadline an idle deadline (issue 24).
|
|
||||||
`downloadTimeoutMs` now aborts a file or thumbnail download only after no
|
|
||||||
bytes have arrived for that long, default 60 seconds, instead of bounding the
|
|
||||||
whole transfer at 10 minutes, so a slow download that keeps making progress
|
|
||||||
completes. A download that fails before reading the whole body cancels it, so
|
|
||||||
a failed file no longer holds its connection.
|
|
||||||
- 2026-09-23: Every `ApiClient` request URL is now built by one function next to
|
|
||||||
the class (issue 18), so a self-hosted `apiOrigin` with a base path keeps it
|
|
||||||
on every request, a path works with or without a leading slash, and query
|
|
||||||
parameters are percent-encoded. A path containing `?` or `#` is rejected with
|
|
||||||
an error instead of being silently cut.
|
|
||||||
- 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies
|
|
||||||
moved from `bin/quak.ts` into `src/cli-commands.ts` as functions that take
|
|
||||||
their options and a context (output streams, session directory, cache
|
|
||||||
directory, session loader) and return an exit code; `bin/quak.ts` only wires
|
|
||||||
them to commander and exits with the code once stdout and stderr have drained,
|
|
||||||
so nothing below it calls `process.exit`. `test/cli/commands.test.ts` drives
|
|
||||||
them with a fake client: session file modes, logout, the missing and corrupt
|
|
||||||
session paths, and the output and exit code of `whoami`, `collections`,
|
|
||||||
`files`, `get`, `get-thumb`, `backup` and `helper list-missing-thumbnails`.
|
|
||||||
- 2026-09-23: Hardened the backup tree's atomic copy (issue 22). `copyAtomic`
|
|
||||||
fsyncs its temp file before the rename and the directory after it, through the
|
|
||||||
download writer's `fsyncPath`; each backup run deletes `.quak-backup-*.tmp`
|
|
||||||
files whose process is no longer running. The README backup layout names the
|
|
||||||
temp files and states that the rename replaces a symlink and takes the temp
|
|
||||||
file's permissions. Added tests for a missing and an unwritable destination
|
|
||||||
directory for `downloadFile` and `downloadThumbnail`.
|
|
||||||
- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue
|
|
||||||
11). Every segment length is checked against the remaining bytes and lengths
|
|
||||||
under 2 stop the scan, so a truncated or corrupt original can neither throw
|
|
||||||
nor loop. A malformed or unparseable EXIF segment is recorded as
|
|
||||||
`imageMetadata.exifError`, and a failure to read the original as
|
|
||||||
`imageMetadataError` in the per-file JSON, instead of the field being left
|
|
||||||
out.
|
|
||||||
- 2026-09-22: Hardened the retry classifier (issue 80). A `POST` or `PUT` is
|
|
||||||
replayed only when every errno in the cause chain is a connect errno, and it
|
|
||||||
no longer follows redirects. `getRetryOptions()` returns a copy. Tests pin
|
|
||||||
every errno the classifier names, the cause-chain depth limit, cycle
|
|
||||||
termination, and a fresh deadline per attempt for every retrying entry point.
|
|
||||||
The README's endpoint list is the one place that names the requests the replay
|
|
||||||
rule covers.
|
|
||||||
- 2026-09-22: Stopped `make test` collecting tests from checkouts nested under
|
|
||||||
`.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a
|
|
||||||
nested checkout ran the whole suite again; `vitest.config.ts` now adds
|
|
||||||
`.claude/**` to vitest's default excludes, and
|
|
||||||
`test/packaging/nested-checkout.test.ts` plants a nested checkout in a temp
|
|
||||||
directory and fails if vitest would collect it.
|
|
||||||
- 2026-09-22: Dropped the deprecated `@types/libsodium-wrappers-sumo` stub from
|
|
||||||
`devDependencies` (issue 27). It shipped no declarations; the types come from
|
|
||||||
`libsodium-wrappers-sumo` itself. `yarn.lock` regenerated by `yarn remove`.
|
|
||||||
- 2026-09-22: Hardened the client session lifecycle (issue 10).
|
|
||||||
`Client.fromJSON` checks every snapshot field and each key's decoded length
|
|
||||||
and names the bad field; `toJSON` reads the token through
|
|
||||||
`ApiClient.getAuthToken` and throws when there is none; `logout` zeroes the
|
|
||||||
key buffers, and `collectionsSince` re-checks for logout after its request so
|
|
||||||
it never decrypts with zeroed keys. The CLI reports a corrupt session file
|
|
||||||
separately from a missing one (`src/cli-session.ts`).
|
|
||||||
- 2026-09-22: Sanitized file names taken from server metadata (issue 9). A new
|
|
||||||
`src/filename.ts` holds the one sanitizer, used by `quak get`/`get-thumb`
|
|
||||||
without `--out`, `downloadFile`/`downloadThumbnail` without `outPath`, and the
|
|
||||||
backup and metadata backup trees; it removes separators, control characters,
|
|
||||||
leading dots and Windows device names, and falls back to a name built from the
|
|
||||||
ID for an empty title. Originals-cache extensions are letters and digits only,
|
|
||||||
else `.bin`. A user-supplied path is used as is. `decryptFile` reads a missing
|
|
||||||
or non-string title as "" and rejects metadata that is not a JSON object.
|
|
||||||
- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
|
|
||||||
snippets) to match the shipped cache/API library on `next` (issue 53, issue
|
|
||||||
13). Documented `Library.open` and its options, the default-read vs `fresh()`
|
|
||||||
distinction (and that the CLI's read commands are fresh), the record types and
|
|
||||||
`snapshot()`/`subscribe()`, the `albums`/`photos`/`timeline` read surface,
|
|
||||||
`Photo` content methods, `thumbnails.ensure`, the `mldata` search surface,
|
|
||||||
`backup()`, the three request pools (10/5/25), and the on-disk cache layout;
|
|
||||||
noted the deferred content-hash integrity check (issue 68). Docs-only; no code
|
|
||||||
changed.
|
|
||||||
- 2026-09-22: Added resumable, deletion-aware enumeration to `Client` (issue 38,
|
|
||||||
closes issue 7). `collectionsSince`/`filesSince` take a starting cursor,
|
|
||||||
decrypt live records, surface tombstoned ids in a separate `deleted` list (a
|
|
||||||
tombstone has nothing to decrypt, so it is a bare id, not a hollow record),
|
|
||||||
and return the max `updationTime` seen as the cursor to resume from.
|
|
||||||
`filesSince` refuses to loop when the diff reports `hasMore` without advancing
|
|
||||||
the cursor (issue 7). `listCollections`/`listFiles` are now thin wrappers that
|
|
||||||
enumerate from `sinceTime: 0` and drop deletions, so existing callers are
|
|
||||||
unaffected.
|
|
||||||
- 2026-09-22: Carried file size, thumbnail size, and the deletion flag through
|
|
||||||
`decryptFile` (issue 37, foundation for the cache/API design). Live files now
|
|
||||||
populate `file.size`/`thumbnail.size` from the server's `info` (left
|
|
||||||
`undefined` when the server omits it), and `isDeleted` is carried from the
|
|
||||||
diff row onto `EnteFile`. No caller change: `listFiles` still filters deleted
|
|
||||||
rows before decrypting. Surfacing a tombstone through decryption belongs to
|
|
||||||
the enumeration unit (issue 38).
|
|
||||||
- 2026-08-10: Made `lint-once.test.ts` enforce what its header claims. It walked
|
|
||||||
`make check` only, so it never read `Dockerfile` — the image CI builds through
|
|
||||||
`script/cibuild` — and a second `prettier --check .` could be added there with
|
|
||||||
the suite staying green. The walk now also starts at
|
|
||||||
`.gitea/workflows/check.yml` and follows its `run:` steps, so the graph under
|
|
||||||
test is the one CI executes rather than the one someone assumed it executes.
|
|
||||||
The lockfile assertion was a substring check against the whole of
|
|
||||||
`script/bootstrap`, which has two install sites and so reported the branch the
|
|
||||||
containers never take; the two branches are now resolved separately and every
|
|
||||||
`yarn install` in each is required to be `--frozen-lockfile`. Prettier is
|
|
||||||
counted per occurrence instead of per line, so two invocations chained with
|
|
||||||
`&&` no longer read as one, and edges are followed on counted lines instead of
|
|
||||||
being skipped. Every way for the walk to reach nothing — an unknown target, an
|
|
||||||
unknown script, a missing file, a node with no commands, an unknown node kind
|
|
||||||
— is a thrown error rather than a quiet zero. Every assertion in the file was
|
|
||||||
mutation-tested individually.
|
|
||||||
- 2026-08-10: Stopped `make check` running `prettier --check .` twice. Since
|
|
||||||
linting moved into Docker, the duplicate was one container pass and one host
|
|
||||||
pass of the same check: `script/lint` builds `Dockerfile.lint`, which runs
|
|
||||||
prettier as a build step, and `script/check` then called `script/fmt-check` as
|
|
||||||
well. The host call is gone from `script/check` and from `script/precommit`;
|
|
||||||
the container keeps checking formatting, because a successful
|
|
||||||
`Dockerfile.lint` build is what CI treats as proof of a clean tree, and it is
|
|
||||||
also what still fails the pre-commit hook on a badly formatted tree.
|
|
||||||
`script/fmt-check` survives as a standalone entrypoint, whose verdict cannot
|
|
||||||
drift from the container's. A test walks the invocation graph from each
|
|
||||||
entrypoint — through the Makefile shims, the `script/` calls and the
|
|
||||||
`docker build` — and asserts the prettier count, so the duplication cannot
|
|
||||||
come back unnoticed.
|
|
||||||
- 2026-08-10: Moved all linting into Docker. `script/lint` builds a new root
|
|
||||||
`Dockerfile.lint`, which copies the repo into the digest-pinned node image and
|
|
||||||
runs eslint and prettier as build steps, so a successful build is a clean
|
|
||||||
lint; no host lint path remains and `yarn lint` is gone from `package.json`. A
|
|
||||||
fail-closed `LINT_EPOCH` guard stops Docker serving the linter layers from
|
|
||||||
cache, which is how a lint build returns success in under a second having
|
|
||||||
linted nothing. The lint stage inside `Dockerfile` and its `COPY --from=lint`
|
|
||||||
ordering hack are gone: that image now runs `make test` and `make build` only,
|
|
||||||
because `script/check` calls `script/lint` and running it in a container would
|
|
||||||
mean docker inside docker. `script/cibuild` builds the lint image first, then
|
|
||||||
the test and build image.
|
|
||||||
- 2026-08-09: Made `make docker` green and policy-conformant. Multi-stage
|
|
||||||
Dockerfile: a lint stage runs `make fmt-check` and `make lint`, and the check
|
|
||||||
stage takes a `COPY --from=lint` dependency on it before running `make check`
|
|
||||||
and `make build`. `CHECK_EPOCH` and a fail-closed guard stop Docker serving
|
|
||||||
those two layers from cache, which is what let a build report success without
|
|
||||||
running the suite. `script/projectname` says `quak`, so the image is tagged
|
|
||||||
`quak`; `script/bootstrap` updates apt lists before installing, so a Debian
|
|
||||||
base works; `.dockerignore` no longer ships the compiled binary, the caches or
|
|
||||||
agent worktrees into the build context, and keeps `.gitignore` in it for
|
|
||||||
prettier.
|
|
||||||
- 2026-08-09: Fixed the TypeScript build. `rootDir` is the repo root, so `bin/`
|
|
||||||
compiles alongside `src/` instead of failing with TS6059; output is
|
|
||||||
`dist/src/` and `dist/bin/`, which is where `main`, `types` and `bin.quak` now
|
|
||||||
point. `script/build` verifies the declared entrypoints exist after the
|
|
||||||
compiler runs and makes the CLI executable, the Dockerfile runs `make build`
|
|
||||||
as well as `make check`, and a `quak` script makes the README's
|
|
||||||
`yarn quak <command>` examples work.
|
|
||||||
- 2026-08-09: Retry policy: no retry on 4xx (except `408` and `429`),
|
|
||||||
exponential backoff with full jitter on 5xx, transport failures and truncated
|
|
||||||
transfers, under per-attempt deadlines that cover the response body as well as
|
|
||||||
the request. Downloads retry request, stream consumption and decryption as one
|
|
||||||
unit; `postJSON` and `putJSON` are replayed only when the connection was never
|
|
||||||
established.
|
|
||||||
- 2026-08-09: Downloads verify the secretstream terminated on `TAG_FINAL` and
|
|
||||||
write output atomically: a truncated body is rejected instead of landing on
|
|
||||||
disk as a short file, and plaintext is staged in a sibling temp file and
|
|
||||||
renamed into place, so a failed download leaves the destination untouched.
|
|
||||||
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
|
|
||||||
shims, README Entrypoints section
|
|
||||||
- 2026-06-10: Decrypted collections shared by other users (sealed-box keys);
|
|
||||||
listCollections drops deleted-collection tombstones.
|
|
||||||
- 2026-06-10: Login hardening: dual-2FA empty-string fields handled, TOTP
|
|
||||||
preferred when a passkey is also enrolled, interactive input via
|
|
||||||
@inquirer/prompts.
|
|
||||||
- 2026-06-10: Replaced sharp with pure JS (jpeg-js + exif-reader); added
|
|
||||||
single-binary bun build and make install.
|
|
||||||
- 2026-06-09: Added backup-metadata command (ML data always included, --exif
|
|
||||||
opt-in); rewrote README to match the implementation; added thumbnail helper
|
|
||||||
tests.
|
|
||||||
- 2026-05-13: Full CLI surface: login, backup with dedup symlink layout,
|
|
||||||
collections, files, get, get-thumb, thumbnail repair helpers.
|
|
||||||
- 2026-05-13: Client OO API with literate usage tests; file download and
|
|
||||||
decryption; all three metadata layers decrypted and persisted; renamed quack
|
|
||||||
to quak.
|
|
||||||
- 2026-05-11: SRP login flow (email OTP + TOTP) and ApiClient.
|
|
||||||
|
|
||||||
# Future Steps
|
# Future Steps
|
||||||
|
|
||||||
- Future desktop client, separate repo:
|
- Future desktop client, separate repo:
|
||||||
|
|||||||
+1
-1
@@ -12,7 +12,7 @@
|
|||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"engines": {
|
"engines": {
|
||||||
"node": ">=22"
|
"node": "^22.13.0 || >=23.5.0"
|
||||||
},
|
},
|
||||||
"main": "./dist/src/index.js",
|
"main": "./dist/src/index.js",
|
||||||
"types": "./dist/src/index.d.ts",
|
"types": "./dist/src/index.d.ts",
|
||||||
|
|||||||
+35
-24
@@ -37,9 +37,16 @@ import {
|
|||||||
thumbnailName,
|
thumbnailName,
|
||||||
} from "./cli-output.js";
|
} from "./cli-output.js";
|
||||||
import { type Log, logRefresh } from "./cli-log.js";
|
import { type Log, logRefresh } from "./cli-log.js";
|
||||||
import { freshCollections, freshFiles, freshFile } from "./cli-read.js";
|
import {
|
||||||
|
cachedFilesByID,
|
||||||
|
freshCollections,
|
||||||
|
freshFiles,
|
||||||
|
freshFile,
|
||||||
|
freshFilesByID,
|
||||||
|
} from "./cli-read.js";
|
||||||
import { withExtension } from "./filename.js";
|
import { withExtension } from "./filename.js";
|
||||||
import { runMetadataBackup } from "./metadata-backup.js";
|
import { runMetadataBackup } from "./metadata-backup.js";
|
||||||
|
import type { Collection, EnteFile } from "./model/types.js";
|
||||||
import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js";
|
import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js";
|
||||||
|
|
||||||
export interface CliContext {
|
export interface CliContext {
|
||||||
@@ -130,12 +137,13 @@ const readLibraryClient = (client: Client): LibraryClient => ({
|
|||||||
|
|
||||||
// Open a library for a single point command. The aggressive background
|
// Open a library for a single point command. The aggressive background
|
||||||
// precache (issue #48) is off, because a one-shot `collections` or `get` must
|
// precache (issue #48) is off, because a one-shot `collections` or `get` must
|
||||||
// not start downloading the whole account. No refresh starts as it opens or on
|
// not start downloading the whole account. No refresh starts as the library
|
||||||
// a timer. Each command asks for the refresh it needs. `collections` needs none
|
// opens or on a timer. Each command asks for the refresh it needs.
|
||||||
// (issue #210). `files COLLECTION` needs one only when `refreshCollection()`
|
// `collections` and `files` with a collection ID need none (issues #210 and
|
||||||
// resolves to `false` (issue #217). While a refresh runs,
|
// #217). Nor do `get` and `get-thumb` on an empty cache (issue #222). Nor does
|
||||||
// each collection whose files it has fetched is logged, so a large account
|
// `helper fix-missing-thumbnails --file` on an empty cache (issue #223). While
|
||||||
// shows progress rather than nothing.
|
// a refresh runs, each collection whose files it has fetched is logged, so a
|
||||||
|
// large account shows progress rather than nothing.
|
||||||
const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
|
const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
|
||||||
Library.open({
|
Library.open({
|
||||||
client: readLibraryClient(client),
|
client: readLibraryClient(client),
|
||||||
@@ -284,11 +292,11 @@ export const filesCommand = async (
|
|||||||
|
|
||||||
const lib = await openReadLibrary(ctx, client);
|
const lib = await openReadLibrary(ctx, client);
|
||||||
try {
|
try {
|
||||||
// Force a server round-trip and list in enumeration order (issue #36
|
// Ask the server for current state (issue #36 amendment, issue #52).
|
||||||
// amendment, issue #52). Each file prints from its own decrypted
|
// Each file prints from its own decrypted metadata (raw title,
|
||||||
// metadata (raw title, microsecond creationTime) via cli-output, and in
|
// microsecond creationTime) via cli-output. The files print in the
|
||||||
// the pre-library CLI's enumeration order, not the projection's
|
// order `freshFiles` returns them, not the projection's newest-first
|
||||||
// newest-first order.
|
// order.
|
||||||
const files = await freshFiles(lib, collectionID);
|
const files = await freshFiles(lib, collectionID);
|
||||||
if (!files) {
|
if (!files) {
|
||||||
ctx.log("fatal", `Collection ${collectionID} not found`);
|
ctx.log("fatal", `Collection ${collectionID} not found`);
|
||||||
@@ -325,14 +333,13 @@ export const getCommand = async (
|
|||||||
try {
|
try {
|
||||||
// Force a server round-trip so the file resolves against current state
|
// Force a server round-trip so the file resolves against current state
|
||||||
// (issue #36 amendment, issue #52).
|
// (issue #36 amendment, issue #52).
|
||||||
const resolved = await freshFile(lib, fileID);
|
const file = await freshFile(lib, fileID);
|
||||||
if (!resolved) {
|
if (!file) {
|
||||||
ctx.log("fatal", `File ${fileID} not found`);
|
ctx.log("fatal", `File ${fileID} not found`);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
const { photo, file } = resolved;
|
|
||||||
|
|
||||||
const result = await photo.original();
|
const result = await lib.originalOf(file);
|
||||||
// Default name is the file's own title, as the pre-library CLI used
|
// Default name is the file's own title, as the pre-library CLI used
|
||||||
// (not the editedName-preferring projection title) (issue #52).
|
// (not the editedName-preferring projection title) (issue #52).
|
||||||
if (result.videoPath === undefined) {
|
if (result.videoPath === undefined) {
|
||||||
@@ -383,14 +390,13 @@ export const getThumbCommand = async (
|
|||||||
try {
|
try {
|
||||||
// Force a server round-trip so the file resolves against current state
|
// Force a server round-trip so the file resolves against current state
|
||||||
// (issue #36 amendment, issue #52).
|
// (issue #36 amendment, issue #52).
|
||||||
const resolved = await freshFile(lib, fileID);
|
const file = await freshFile(lib, fileID);
|
||||||
if (!resolved) {
|
if (!file) {
|
||||||
ctx.log("fatal", `File ${fileID} not found`);
|
ctx.log("fatal", `File ${fileID} not found`);
|
||||||
return 1;
|
return 1;
|
||||||
}
|
}
|
||||||
const { photo, file } = resolved;
|
|
||||||
|
|
||||||
const result = await photo.thumbnail();
|
const result = await lib.thumbnailOf(file);
|
||||||
// Default name is thumb_<file's own title>, as the pre-library CLI
|
// Default name is thumb_<file's own title>, as the pre-library CLI
|
||||||
// used (not the projection title) (issue #52).
|
// used (not the projection title) (issue #52).
|
||||||
const outPath = opts.out ?? thumbnailName(file);
|
const outPath = opts.out ?? thumbnailName(file);
|
||||||
@@ -567,10 +573,13 @@ export const fixMissingThumbnailsCommand = async (
|
|||||||
}
|
}
|
||||||
const lib = await openReadLibrary(ctx, client);
|
const lib = await openReadLibrary(ctx, client);
|
||||||
try {
|
try {
|
||||||
// Refresh first so files added since the cache was written are found;
|
let found: Map<number, { file: EnteFile; collection: Collection }>;
|
||||||
// a failed refresh throws.
|
if (fileIDs.length > 0) {
|
||||||
await lib.fresh();
|
found = await freshFilesByID(lib, fileIDs);
|
||||||
if (fileIDs.length === 0) {
|
} else {
|
||||||
|
// Refresh first so files added since the cache was written are
|
||||||
|
// checked; a failed refresh throws.
|
||||||
|
await lib.fresh();
|
||||||
ctx.log("info", "Scanning for missing thumbnails...");
|
ctx.log("info", "Scanning for missing thumbnails...");
|
||||||
const missing = await listMissingThumbnails(
|
const missing = await listMissingThumbnails(
|
||||||
lib,
|
lib,
|
||||||
@@ -583,12 +592,14 @@ export const fixMissingThumbnailsCommand = async (
|
|||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
ctx.log("info", `Found ${fileIDs.length} file(s) to fix.`);
|
ctx.log("info", `Found ${fileIDs.length} file(s) to fix.`);
|
||||||
|
found = cachedFilesByID(lib, fileIDs);
|
||||||
}
|
}
|
||||||
|
|
||||||
const results = await fixMissingThumbnails(
|
const results = await fixMissingThumbnails(
|
||||||
lib,
|
lib,
|
||||||
client,
|
client,
|
||||||
fileIDs,
|
fileIDs,
|
||||||
|
found,
|
||||||
(msg, level) => ctx.log(level ?? "info", msg),
|
(msg, level) => ctx.log(level ?? "info", msg),
|
||||||
);
|
);
|
||||||
|
|
||||||
|
|||||||
+10
-7
@@ -1,15 +1,18 @@
|
|||||||
// The CLI's log level, and the one function the commands write stderr lines
|
// The CLI's log level, and the one function the commands write stderr lines
|
||||||
// through.
|
// through.
|
||||||
//
|
//
|
||||||
// Every line quak prints on stderr is a log line at one of six levels, from
|
// Every line quak logs goes to stderr, at one of six levels, from the
|
||||||
// the quietest: fatal, error, warning, info, debug, trace. A run prints the
|
// quietest: fatal, error, warning, info, debug, trace. A run prints the lines
|
||||||
// lines at its own level and at every quieter one. Its level is info unless
|
// at its own level and at every quieter one. Its level is info unless `-v` or
|
||||||
// `-v` or `-q` asks for another (see "CLI surface" in the README). Every
|
// `-q` asks for another (see "What quak prints" in the README). A command that
|
||||||
// command that exits non-zero logs a fatal line saying why, so even `-q`, which
|
// runs and exits non-zero logs a fatal line saying why, so even `-q`, which
|
||||||
// prints only fatal lines, never leaves a non-zero exit unexplained. An error
|
// prints only fatal lines, never leaves such an exit unexplained. An error
|
||||||
// line reports something that failed, such as a file or an album, and a
|
// line reports something that failed, such as a file or an album, and a
|
||||||
// warning something that did not fail but needs the user's attention. stdout
|
// warning something that did not fail but needs the user's attention. stdout
|
||||||
// carries only a command's output and is not a log.
|
// carries a command's output and the prompts of `login`, and is not a log.
|
||||||
|
// Two things on stderr are not log lines: the `error:` message for a
|
||||||
|
// command line that quak cannot parse, and the usage printed when no command
|
||||||
|
// is given.
|
||||||
//
|
//
|
||||||
// The library itself writes no lines: its layers report through callbacks, and
|
// The library itself writes no lines: its layers report through callbacks, and
|
||||||
// the CLI logs what they report. `logRequests` gives the API client options
|
// the CLI logs what they report. `logRequests` gives the API client options
|
||||||
|
|||||||
+102
-53
@@ -2,47 +2,40 @@
|
|||||||
//
|
//
|
||||||
// `collections`, `files`, `get`, and `get-thumb` must answer for
|
// `collections`, `files`, `get`, and `get-thumb` must answer for
|
||||||
// the account's state at the moment the command runs, not for whatever the
|
// the account's state at the moment the command runs, not for whatever the
|
||||||
// local cache last happened to hold (owner amendment, issue #36). `files`
|
// local cache last happened to hold (owner amendment, issue #36). `get`,
|
||||||
// without a collection ID, `get` and `get-thumb` therefore force a server
|
// `get-thumb` and `files` without a collection ID therefore force a server
|
||||||
// round-trip through `Library.fresh()`, and only then read. A collection, file,
|
// round-trip through `Library.fresh()` and only then read, so a change made
|
||||||
// or metadata change made elsewhere is then visible. `collections` asks the
|
// elsewhere is visible. `collections` asks the library for the collection list
|
||||||
// library for the collection list alone through `Library.freshCollections()`
|
// alone through `Library.freshCollections()` (issue #210): a refresh would also
|
||||||
// (issue #210). A refresh would also fetch the files of every collection, which
|
// fetch the files of every collection, which on a large account takes long and
|
||||||
// `collections` does not print, and on a large account that takes long.
|
// which `collections` does not print. `files` with a collection ID reads that
|
||||||
// `files COLLECTION` reads the collection list the same way, for the same
|
// list too, then that one collection's files through `Library.freshFiles()`
|
||||||
// reason. It then brings that one collection up to date through
|
// (issue #217). On an empty cache, `get` and `get-thumb` read that list, then
|
||||||
// `Library.refreshCollection()` (issue #217), and fetches no other collection's
|
// one collection's files at a time, until they find the file (issue #222).
|
||||||
// files. Once a refresh has received the account's collections,
|
// `helper fix-missing-thumbnails --file` does the same until it finds every
|
||||||
// `refreshCollection()` stores no collection the library does not hold.
|
// file it names (issue #223).
|
||||||
// `files COLLECTION` then forces a refresh instead.
|
|
||||||
//
|
//
|
||||||
// `collections` lists in the server's order. `files` lists in the library's
|
// `collections` and `files` with a collection ID list in the server's order.
|
||||||
// own enumeration order: `listCollections()`, then `listFiles()` for each.
|
// `files` without one lists in the library's own enumeration order
|
||||||
// Those are the orders the pre-library CLI printed. The `albums`/`photos`
|
// (`listFiles()`). Neither command uses the `albums`/`photos` projection's
|
||||||
// projection would re-sort the rows newest first. After a refresh, the
|
// newest-first order, which re-sorts the rows. The field values still come
|
||||||
// library holds collections in the order the server first sent them. The
|
// from each record's raw metadata via `cli-output.ts`. `files` without a
|
||||||
// field values still come from each record's raw metadata via
|
// collection ID lists every file in the account once (issue #211): the library
|
||||||
// `cli-output.ts`. `files` without a collection ID lists every file in the
|
// keeps a record of a file for each collection that holds it, and only the
|
||||||
// account once (issue #211).
|
// first collection's record is listed.
|
||||||
// The library keeps a record of a file for each collection that holds it, and
|
|
||||||
// only the first collection's record is listed.
|
|
||||||
|
|
||||||
import type { Collection, EnteFile } from "./model/types.js";
|
import type { Collection, EnteFile } from "./model/types.js";
|
||||||
import type { Photo, PhotosAPI } from "./library/index.js";
|
|
||||||
|
|
||||||
// The slice of `Library` these helpers read. `Library` satisfies it
|
// The slice of `Library` these helpers read. `Library` satisfies it
|
||||||
// structurally. A test can drive them with a stand-in that records the
|
// structurally; a test can drive them with a stand-in that records each fresh
|
||||||
// `fresh()`, `freshCollections()` and `refreshCollection()` calls. The
|
// read and serves records in a known enumeration order.
|
||||||
// stand-in serves records in a known enumeration order.
|
|
||||||
export interface FreshReadLibrary {
|
export interface FreshReadLibrary {
|
||||||
fresh(): Promise<unknown>;
|
fresh(): Promise<unknown>;
|
||||||
freshCollections(): Promise<Collection[]>;
|
freshCollections(): Promise<Collection[]>;
|
||||||
refreshCollection(collection: Collection): Promise<boolean>;
|
freshFiles(collection: Collection): Promise<EnteFile[]>;
|
||||||
getCollection(id: number): Collection | undefined;
|
|
||||||
listCollections(): Collection[];
|
listCollections(): Collection[];
|
||||||
listFiles(collectionID: number): EnteFile[];
|
listFiles(collectionID: number): EnteFile[];
|
||||||
getFileByID(fileID: number): EnteFile | undefined;
|
getFileByID(fileID: number): EnteFile | undefined;
|
||||||
photos: Pick<PhotosAPI, "byID">;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Every live collection, current as of one collections request, in the
|
// Every live collection, current as of one collections request, in the
|
||||||
@@ -51,14 +44,11 @@ export const freshCollections = (
|
|||||||
lib: FreshReadLibrary,
|
lib: FreshReadLibrary,
|
||||||
): Promise<Collection[]> => lib.freshCollections();
|
): Promise<Collection[]> => lib.freshCollections();
|
||||||
|
|
||||||
// The files of one collection, in enumeration order. `undefined` (not an empty
|
// The files of one collection, as the server holds them now, in the server's
|
||||||
// list) when the collection does not exist, so the caller can tell "no such
|
// order. `undefined` (not an empty list) when the collection does not exist, so
|
||||||
// collection" from "an empty collection". They are current as of one
|
// the caller can tell "no such collection" from "an empty collection". Without
|
||||||
// collections request and the requests that bring that collection alone up to
|
// a collection ID, every file in the account as of a forced refresh, each once,
|
||||||
// date. When `refreshCollection()` resolves to `false`, they are current as of
|
// as held by the first collection that holds it.
|
||||||
// a forced refresh instead. Without a collection ID, every file in the account,
|
|
||||||
// current as of a forced refresh. Each file is listed once, as held by the
|
|
||||||
// first collection that holds it.
|
|
||||||
export const freshFiles = async (
|
export const freshFiles = async (
|
||||||
lib: FreshReadLibrary,
|
lib: FreshReadLibrary,
|
||||||
collectionID?: number,
|
collectionID?: number,
|
||||||
@@ -66,12 +56,7 @@ export const freshFiles = async (
|
|||||||
if (collectionID !== undefined) {
|
if (collectionID !== undefined) {
|
||||||
const collections = await lib.freshCollections();
|
const collections = await lib.freshCollections();
|
||||||
const collection = collections.find((c) => c.id === collectionID);
|
const collection = collections.find((c) => c.id === collectionID);
|
||||||
if (!collection) return undefined;
|
return collection ? lib.freshFiles(collection) : undefined;
|
||||||
if (!(await lib.refreshCollection(collection))) {
|
|
||||||
await lib.fresh();
|
|
||||||
if (!lib.getCollection(collectionID)) return undefined;
|
|
||||||
}
|
|
||||||
return lib.listFiles(collectionID);
|
|
||||||
}
|
}
|
||||||
await lib.fresh();
|
await lib.fresh();
|
||||||
const listed = new Set<number>();
|
const listed = new Set<number>();
|
||||||
@@ -86,16 +71,80 @@ export const freshFiles = async (
|
|||||||
return files;
|
return files;
|
||||||
};
|
};
|
||||||
|
|
||||||
// One file, current as of a forced refresh, resolved to both its content
|
// One file as the server holds it now, or `undefined` when no collection holds
|
||||||
// handle (`Photo`, for fetching bytes) and its raw record (`EnteFile`, for the
|
// it. When the cache holds the account's collections, it refreshes and looks
|
||||||
// default output name and field values). `undefined` when the file is unknown.
|
// the file up in the cache. On an empty cache, it reads the collection list,
|
||||||
|
// then one collection's files at a time, and stops at the first collection
|
||||||
|
// that holds the file. That search stores nothing.
|
||||||
export const freshFile = async (
|
export const freshFile = async (
|
||||||
lib: FreshReadLibrary,
|
lib: FreshReadLibrary,
|
||||||
fileID: number,
|
fileID: number,
|
||||||
): Promise<{ photo: Photo; file: EnteFile } | undefined> => {
|
): Promise<EnteFile | undefined> => {
|
||||||
await lib.fresh();
|
// Only a refresh stores collections, so a cache holding any holds them all.
|
||||||
const photo = lib.photos.byID({ fileID });
|
if (lib.listCollections().length > 0) {
|
||||||
const file = lib.getFileByID(fileID);
|
await lib.fresh();
|
||||||
if (!photo || !file) return undefined;
|
return lib.getFileByID(fileID);
|
||||||
return { photo, file };
|
}
|
||||||
|
for (const collection of await lib.freshCollections()) {
|
||||||
|
const files = await lib.freshFiles(collection);
|
||||||
|
const file = files.find((f) => f.id === fileID);
|
||||||
|
if (file) return file;
|
||||||
|
}
|
||||||
|
return undefined;
|
||||||
|
};
|
||||||
|
|
||||||
|
// The most recently updated collection first, and the higher ID first on a
|
||||||
|
// tie. `Library.albums.list()` lists albums in this order.
|
||||||
|
const byNewestCollection = (a: Collection, b: Collection): number =>
|
||||||
|
b.updationTime - a.updationTime || b.id - a.id;
|
||||||
|
|
||||||
|
// Each file named by `fileIDs` that the cache holds, mapped to its record and
|
||||||
|
// the collection it was found in. Collections are searched in
|
||||||
|
// `byNewestCollection` order, and a file is found in the first that holds it.
|
||||||
|
// So `helper fix-missing-thumbnails` names a file after the same album as
|
||||||
|
// `helper list-missing-thumbnails`. It reads the cache alone, so the caller
|
||||||
|
// refreshes first.
|
||||||
|
export const cachedFilesByID = (
|
||||||
|
lib: FreshReadLibrary,
|
||||||
|
fileIDs: number[],
|
||||||
|
): Map<number, { file: EnteFile; collection: Collection }> => {
|
||||||
|
const wanted = new Set(fileIDs);
|
||||||
|
const found = new Map<number, { file: EnteFile; collection: Collection }>();
|
||||||
|
const collections = [...lib.listCollections()].sort(byNewestCollection);
|
||||||
|
for (const collection of collections) {
|
||||||
|
for (const file of lib.listFiles(collection.id)) {
|
||||||
|
if (wanted.has(file.id) && !found.has(file.id)) {
|
||||||
|
found.set(file.id, { file, collection });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return found;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Each file named by `fileIDs` as the server holds it now, mapped to its record
|
||||||
|
// and collection as `cachedFilesByID` maps it. When the cache holds the
|
||||||
|
// account's collections, it refreshes and looks the files up in the cache. On
|
||||||
|
// an empty cache, it reads the collection list, then one collection's files at
|
||||||
|
// a time, and stops once it has found every file. That search stores nothing.
|
||||||
|
export const freshFilesByID = async (
|
||||||
|
lib: FreshReadLibrary,
|
||||||
|
fileIDs: number[],
|
||||||
|
): Promise<Map<number, { file: EnteFile; collection: Collection }>> => {
|
||||||
|
// Only a refresh stores collections, so a cache holding any holds them all.
|
||||||
|
if (lib.listCollections().length > 0) {
|
||||||
|
await lib.fresh();
|
||||||
|
return cachedFilesByID(lib, fileIDs);
|
||||||
|
}
|
||||||
|
const wanted = new Set(fileIDs);
|
||||||
|
const found = new Map<number, { file: EnteFile; collection: Collection }>();
|
||||||
|
const collections = await lib.freshCollections();
|
||||||
|
for (const collection of [...collections].sort(byNewestCollection)) {
|
||||||
|
if (found.size === wanted.size) break;
|
||||||
|
for (const file of await lib.freshFiles(collection)) {
|
||||||
|
if (wanted.has(file.id) && !found.has(file.id)) {
|
||||||
|
found.set(file.id, { file, collection });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return found;
|
||||||
};
|
};
|
||||||
+38
-11
@@ -581,14 +581,34 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
fileID: number,
|
fileID: number,
|
||||||
opts?: ContentOptions,
|
opts?: ContentOptions,
|
||||||
): Promise<ContentResult> {
|
): Promise<ContentResult> {
|
||||||
return this.get(fileID, "original", "on-demand", opts?.onProgress);
|
const file = this.fileOrThrow(fileID);
|
||||||
|
return this.get(file, "original", "on-demand", opts?.onProgress);
|
||||||
}
|
}
|
||||||
|
|
||||||
async thumbnail(
|
async thumbnail(
|
||||||
fileID: number,
|
fileID: number,
|
||||||
opts?: ContentOptions,
|
opts?: ContentOptions,
|
||||||
): Promise<ContentResult> {
|
): Promise<ContentResult> {
|
||||||
return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
|
const file = this.fileOrThrow(fileID);
|
||||||
|
return this.get(file, "thumbnail", "on-demand", opts?.onProgress);
|
||||||
|
}
|
||||||
|
|
||||||
|
// The original of `file`, fetched and cached as `original` does. `file` is
|
||||||
|
// used as given, so `getFile` need not know it.
|
||||||
|
async originalOf(
|
||||||
|
file: EnteFile,
|
||||||
|
opts?: ContentOptions,
|
||||||
|
): Promise<ContentResult> {
|
||||||
|
return this.get(file, "original", "on-demand", opts?.onProgress);
|
||||||
|
}
|
||||||
|
|
||||||
|
// The thumbnail of `file`, fetched and cached as `thumbnail` does. `file`
|
||||||
|
// is used as given, so `getFile` need not know it.
|
||||||
|
async thumbnailOf(
|
||||||
|
file: EnteFile,
|
||||||
|
opts?: ContentOptions,
|
||||||
|
): Promise<ContentResult> {
|
||||||
|
return this.get(file, "thumbnail", "on-demand", opts?.onProgress);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Put the original at the save path of `file` under the download directory
|
// Put the original at the save path of `file` under the download directory
|
||||||
@@ -615,7 +635,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
destination: string,
|
destination: string,
|
||||||
): Promise<ContentResult> {
|
): Promise<ContentResult> {
|
||||||
const result = await this.acquire(
|
const result = await this.acquire(
|
||||||
fileID,
|
this.fileOrThrow(fileID),
|
||||||
"original",
|
"original",
|
||||||
"on-demand",
|
"on-demand",
|
||||||
undefined,
|
undefined,
|
||||||
@@ -638,8 +658,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
fileID: number,
|
fileID: number,
|
||||||
destination: string,
|
destination: string,
|
||||||
): Promise<{ path: string; videoPath?: string }> {
|
): Promise<{ path: string; videoPath?: string }> {
|
||||||
const file = this.getFile(fileID);
|
const file = this.fileOrThrow(fileID);
|
||||||
if (!file) throw new Error(`content cache: unknown file ${fileID}`);
|
|
||||||
return this.pools.content.run(
|
return this.pools.content.run(
|
||||||
async () => {
|
async () => {
|
||||||
const fetched = await this.fetchInto(
|
const fetched = await this.fetchInto(
|
||||||
@@ -721,7 +740,8 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
onProgress: ((event: EnsureEvent) => void) | undefined,
|
onProgress: ((event: EnsureEvent) => void) | undefined,
|
||||||
): Promise<EnsureResult> {
|
): Promise<EnsureResult> {
|
||||||
try {
|
try {
|
||||||
const result = await this.acquire(fileID, kind, priority, signal);
|
const file = this.fileOrThrow(fileID);
|
||||||
|
const result = await this.acquire(file, kind, priority, signal);
|
||||||
const status = result.cached ? "skipped" : "done";
|
const status = result.cached ? "skipped" : "done";
|
||||||
onProgress?.({ fileID, status, path: result.path });
|
onProgress?.({ fileID, status, path: result.path });
|
||||||
return { fileID, path: result.path };
|
return { fileID, path: result.path };
|
||||||
@@ -736,8 +756,16 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The record `getFile` gives for `fileID`. It throws for a file `getFile`
|
||||||
|
// does not know.
|
||||||
|
private fileOrThrow(fileID: number): EnteFile {
|
||||||
|
const file = this.getFile(fileID);
|
||||||
|
if (!file) throw new Error(`content cache: unknown file ${fileID}`);
|
||||||
|
return file;
|
||||||
|
}
|
||||||
|
|
||||||
private async get(
|
private async get(
|
||||||
fileID: number,
|
file: EnteFile,
|
||||||
kind: Kind,
|
kind: Kind,
|
||||||
priority: Priority,
|
priority: Priority,
|
||||||
onProgress: ((event: ContentEvent) => void) | undefined,
|
onProgress: ((event: ContentEvent) => void) | undefined,
|
||||||
@@ -745,7 +773,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
const onByte: ProgressCallback | undefined = onProgress
|
const onByte: ProgressCallback | undefined = onProgress
|
||||||
? (bytesDone) => onProgress({ status: "downloading", bytesDone })
|
? (bytesDone) => onProgress({ status: "downloading", bytesDone })
|
||||||
: undefined;
|
: undefined;
|
||||||
const result = await this.acquire(fileID, kind, priority, undefined, {
|
const result = await this.acquire(file, kind, priority, undefined, {
|
||||||
onByte,
|
onByte,
|
||||||
});
|
});
|
||||||
onProgress?.(
|
onProgress?.(
|
||||||
@@ -767,7 +795,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
// photo's original is present only with its video, and is returned with
|
// photo's original is present only with its video, and is returned with
|
||||||
// it.
|
// it.
|
||||||
private async acquire(
|
private async acquire(
|
||||||
fileID: number,
|
file: EnteFile,
|
||||||
kind: Kind,
|
kind: Kind,
|
||||||
priority: Priority,
|
priority: Priority,
|
||||||
signal: AbortSignal | undefined,
|
signal: AbortSignal | undefined,
|
||||||
@@ -778,8 +806,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
|
|||||||
videoPath?: string;
|
videoPath?: string;
|
||||||
cached: boolean;
|
cached: boolean;
|
||||||
}> {
|
}> {
|
||||||
const file = this.getFile(fileID);
|
const fileID = file.id;
|
||||||
if (!file) throw new Error(`content cache: unknown file ${fileID}`);
|
|
||||||
const isLivePhoto =
|
const isLivePhoto =
|
||||||
kind === "original" && file.metadata.fileType === "livePhoto";
|
kind === "original" && file.metadata.fileType === "livePhoto";
|
||||||
|
|
||||||
|
|||||||
+206
-282
@@ -1,37 +1,29 @@
|
|||||||
// The library surface over the local cache.
|
// The library surface over the local cache.
|
||||||
//
|
//
|
||||||
// `Library.open()` loads the on-disk metadata store (issue #41), then starts
|
// `Library.open()` loads the on-disk metadata store (issue #41), then starts
|
||||||
// the refresh loop. When the stored cursor is 0, `open()` awaits the first
|
// the refresh loop. When the cache loaded empty it awaits the first refresh,
|
||||||
// refresh. No refresh has then stored any collection: the cache is empty, or
|
// so the library never opens onto an empty store it could have filled; when an
|
||||||
// holds only what `refreshCollection()` stored. The library must not serve
|
// existing copy loaded, that first refresh runs in the background and `open()`
|
||||||
// that as the whole account. Otherwise that first refresh runs in the
|
// returns as soon as the cached data is ready to serve — a slow or unreachable
|
||||||
// background. `open()` returns as soon as the cached data is ready to serve,
|
// server no longer stalls opening. A background timer then refreshes every
|
||||||
// so a slow or unreachable server does not stall opening. A background timer
|
// `refreshIntervalSeconds`. Every default read is answered from RAM — no
|
||||||
// then takes a turn `refreshIntervalSeconds` after its last turn ends. Each
|
// default read touches the network. There is deliberately no `sync()`, no
|
||||||
// turn starts a refresh, unless a refresh or `refreshCollection()` is already
|
// `refresh()`, no `serverReachable` flag, and no "before each read" mode
|
||||||
// running. Every default read is answered from RAM — no default read touches
|
// (design #36).
|
||||||
// the network. There is deliberately no `sync()`, no `refresh()`, no
|
|
||||||
// `serverReachable` flag, and no "before each read" mode (design #36).
|
|
||||||
//
|
//
|
||||||
// `fresh()` is an exception (issue #75, an owner amendment to #36). It awaits
|
// `fresh()` is the one exception (issue #75, an owner amendment to #36): it
|
||||||
// a refresh, and only then hands back the read namespaces. A caller that needs
|
// forces a refresh, awaits it, and only then hands back the read namespaces, so
|
||||||
// server-current data uses it. Concurrent `fresh()` calls coalesce onto one
|
// a caller that needs server-current data can ask for it. Concurrent `fresh()`
|
||||||
// in-flight refresh. A refresh that fails rejects the caller. The default
|
// calls coalesce onto one in-flight refresh, and a refresh that fails rejects
|
||||||
// reads stay silent and serve the last good copy. The default methods and the
|
// the caller (the default reads stay silent and serve the last good copy). The
|
||||||
// background loop are unchanged.
|
// default methods and the background loop are unchanged.
|
||||||
//
|
//
|
||||||
// `freshCollections()` is the fresh read of the collection list alone (issue
|
// `freshCollections()` is the fresh read of the collection list alone (issue
|
||||||
// #210). It sends one collections request and stores nothing.
|
// #210): one collections request, no collection's files, and nothing stored,
|
||||||
// `quak collections` uses it. `refreshCollection()` brings one collection
|
// for a caller such as `quak collections` that needs no more. `freshFiles()`
|
||||||
// from that list up to date in the store (issue #217). It fetches that
|
// is the fresh read of one collection's files, and stores nothing either
|
||||||
// collection's files alone, as a refresh would. `quak files COLLECTION` uses
|
// (issue #217). A caller that asks for each refresh it needs can also open the
|
||||||
// it. It never runs at the same time as a refresh. Once the stored cursor is
|
// library with `refreshOnOpen: false`, so that `open()` starts none.
|
||||||
// past 0, it only updates a collection the store holds, and only a refresh
|
|
||||||
// adds one. A refresh from a stored cursor of 0 replaces all the store holds.
|
|
||||||
// So after a refresh, the store holds collections in the order the server
|
|
||||||
// first sent them. A caller that asks for
|
|
||||||
// each refresh it needs can also open the library with `refreshOnOpen: false`,
|
|
||||||
// so that `open()` starts none.
|
|
||||||
//
|
//
|
||||||
// A refresh reports through `onProgress` each collection whose files it has
|
// A refresh reports through `onProgress` each collection whose files it has
|
||||||
// fetched, so a large account, whose first refresh fetches the files of every
|
// fetched, so a large account, whose first refresh fetches the files of every
|
||||||
@@ -40,12 +32,10 @@
|
|||||||
// A refresh stages all of its network work first and only mutates the store
|
// A refresh stages all of its network work first and only mutates the store
|
||||||
// once every fetch has succeeded. A refresh that fails partway therefore never
|
// once every fetch has succeeded. A refresh that fails partway therefore never
|
||||||
// becomes visible to reads: the last good snapshot stays in place, and the
|
// becomes visible to reads: the last good snapshot stays in place, and the
|
||||||
// failure surfaces through `onProgress` and `status()` instead. A save can
|
// failure surfaces through `onProgress` and `status()` instead. A commit that
|
||||||
// still fail after the store has changed in RAM. The store then stays marked
|
// mutates RAM but then fails to persist keeps `status().lastError` set and the
|
||||||
// unsaved until a later save lands. Until then, every `commit()` retries the
|
// store marked unsaved until a later save actually lands, so a stuck disk is
|
||||||
// save. A refresh whose save fails sets `status().lastError`, which stays set
|
// never masked by a subsequent empty refresh.
|
||||||
// until a later refresh succeeds. `refreshCollection()` never sets or clears
|
|
||||||
// it. So a stuck disk is never hidden by a later refresh that changes nothing.
|
|
||||||
|
|
||||||
import { rm } from "node:fs/promises";
|
import { rm } from "node:fs/promises";
|
||||||
import { join, resolve } from "node:path";
|
import { join, resolve } from "node:path";
|
||||||
@@ -76,6 +66,8 @@ import {
|
|||||||
ContentCache,
|
ContentCache,
|
||||||
savePath,
|
savePath,
|
||||||
storedAtSavePath,
|
storedAtSavePath,
|
||||||
|
type ContentOptions,
|
||||||
|
type ContentResult,
|
||||||
type ContentSource,
|
type ContentSource,
|
||||||
type ThumbnailsAPI,
|
type ThumbnailsAPI,
|
||||||
type EnsureOptions,
|
type EnsureOptions,
|
||||||
@@ -170,23 +162,21 @@ export interface LibraryClient {
|
|||||||
payloads: Map<number, MLData>;
|
payloads: Map<number, MLData>;
|
||||||
errors: Map<number, unknown>;
|
errors: Map<number, unknown>;
|
||||||
}>;
|
}>;
|
||||||
// The byte source for the on-disk content cache. Optional, so a mock
|
// The byte source for the on-disk content cache. Optional so a mock client
|
||||||
// client that only serves metadata still satisfies the interface. Without
|
// that only serves metadata still satisfies the interface; when absent (and
|
||||||
// it, and with no `contentSource` passed to `open`, the content cache is
|
// no explicit `contentSource` is passed to `open`) the content cache is
|
||||||
// disabled. `Photo.original`/`thumbnail` and `thumbnails.ensure` then
|
// disabled and `Photo.original`/`thumbnail` and `thumbnails.ensure` throw.
|
||||||
// throw.
|
|
||||||
contentSource?(): ContentSource;
|
contentSource?(): ContentSource;
|
||||||
}
|
}
|
||||||
|
|
||||||
// A progress event for one unit of background work. A metadata "refresh" and
|
// A progress event for one unit of background work. A metadata "refresh" or an
|
||||||
// an ML "fetchMLData" pass each fire "started" before their network work. Each
|
// ML "fetchMLData" pass each fire "started" before their network work and then
|
||||||
// then fires exactly one of "done" or "failed". A "failed" event carries the
|
// exactly one of "done" or "failed"; "failed" carries the error message and
|
||||||
// error message. An ML "done" reports in `fetched` how many payloads it
|
// an ML "done" reports how many payloads it stored and, in `mlDataErrors`,
|
||||||
// stored. Its `mlDataErrors` lists each file whose ML data did not decrypt,
|
// each file whose ML data did not decrypt, gunzip or parse, with the reason.
|
||||||
// gunzip or parse, with the reason. In between, a refresh fires "progress"
|
// In between, a refresh fires "progress" after each collection whose files it
|
||||||
// after each collection whose files it fetched. Its `fetched` is how many
|
// fetched: `fetched` is how many collections it has fetched the files of so
|
||||||
// collections it has fetched the files of so far, and `total` how many it
|
// far, `total` how many it will. The precache fills
|
||||||
// will. The precache fills
|
|
||||||
// ("precacheThumbnails"/"precacheOriginals", #48) fire "started"/"done" around
|
// ("precacheThumbnails"/"precacheOriginals", #48) fire "started"/"done" around
|
||||||
// each sweep that has work, "done" reporting the count newly cached.
|
// each sweep that has work, "done" reporting the count newly cached.
|
||||||
export interface RefreshEvent {
|
export interface RefreshEvent {
|
||||||
@@ -214,10 +204,9 @@ export interface LibraryOptions {
|
|||||||
// directory at open. The content cache treats an original already stored
|
// directory at open. The content cache treats an original already stored
|
||||||
// at its save path as present.
|
// at its save path as present.
|
||||||
downloadDirectory?: string;
|
downloadDirectory?: string;
|
||||||
// How long after its last turn ends the background timer takes the next.
|
// How long after one refresh ends the background timer starts the next.
|
||||||
// A turn starts a refresh unless a refresh or `refreshCollection()` is
|
// `Infinity` starts none: the library then refreshes only as it opens and
|
||||||
// running. With `Infinity` the timer takes no turn: the library then
|
// when `fresh()` or `backup()` asks.
|
||||||
// refreshes only as it opens and when `fresh()` or `backup()` asks.
|
|
||||||
refreshIntervalSeconds?: number;
|
refreshIntervalSeconds?: number;
|
||||||
// Whether `open()` starts a refresh. Defaults to true. With false, `open()`
|
// Whether `open()` starts a refresh. Defaults to true. With false, `open()`
|
||||||
// returns at once onto what the cache holds, empty or not, and a refresh
|
// returns at once onto what the cache holds, empty or not, and a refresh
|
||||||
@@ -256,9 +245,8 @@ export interface LibraryStatus {
|
|||||||
// Wall-clock ms of the last refresh that succeeded, or undefined if none
|
// Wall-clock ms of the last refresh that succeeded, or undefined if none
|
||||||
// has yet.
|
// has yet.
|
||||||
lastRefreshAt?: number;
|
lastRefreshAt?: number;
|
||||||
// The error of the most recent refresh, set only when it failed. The next
|
// The message from the most recent refresh, set only while that refresh
|
||||||
// refresh that succeeds clears it. `refreshCollection()` never sets or
|
// failed; cleared by the next success.
|
||||||
// clears it.
|
|
||||||
lastError?: string;
|
lastError?: string;
|
||||||
// Wall-clock ms of the last ML fetch pass that succeeded, or undefined if
|
// Wall-clock ms of the last ML fetch pass that succeeded, or undefined if
|
||||||
// none has yet (or ML fetching is disabled).
|
// none has yet (or ML fetching is disabled).
|
||||||
@@ -321,17 +309,12 @@ export class Library {
|
|||||||
private readonly precache?: Precache;
|
private readonly precache?: Precache;
|
||||||
|
|
||||||
private timer?: ReturnType<typeof setTimeout>;
|
private timer?: ReturnType<typeof setTimeout>;
|
||||||
// The in-flight refresh cycle or `refreshCollection()`, or undefined when
|
// The in-flight refresh cycle, or undefined when none runs. One slot serves
|
||||||
// neither runs. One slot holds both, so they never run at the same time.
|
// both paths: the background loop skips when it is set, and a fresh read
|
||||||
// The background loop skips while it is set. A fresh read (issue #75)
|
// (issue #75) coalesces onto it or starts one. The promise carries the
|
||||||
// coalesces onto a running refresh. It waits for a running
|
// cycle's real outcome (it rejects on failure); the background loop ignores
|
||||||
// `refreshCollection()` to end, and then joins or starts a refresh. The
|
// that, a fresh read propagates it.
|
||||||
// promise carries the real outcome: it rejects on failure. The background
|
|
||||||
// loop ignores a failure. A fresh read passes on the failure of the
|
|
||||||
// refresh it joins or starts, and `refreshCollection()` passes on its own.
|
|
||||||
private cycle?: Promise<void>;
|
private cycle?: Promise<void>;
|
||||||
// Whether `cycle` is a refresh cycle rather than a `refreshCollection()`.
|
|
||||||
private cycleIsRefresh = false;
|
|
||||||
// Guards the ML fetch pass so a slow backfill never runs twice at once; a
|
// Guards the ML fetch pass so a slow backfill never runs twice at once; a
|
||||||
// refresh whose pass is still running kicks nothing new. Holds the running
|
// refresh whose pass is still running kicks nothing new. Holds the running
|
||||||
// pass, so `close()` and `backup()` can wait for it. It rejects when the
|
// pass, so `close()` and `backup()` can wait for it. It rejects when the
|
||||||
@@ -344,20 +327,19 @@ export class Library {
|
|||||||
private lastMLError?: string;
|
private lastMLError?: string;
|
||||||
// For each file whose ML data the last ML fetch pass that succeeded got
|
// For each file whose ML data the last ML fetch pass that succeeded got
|
||||||
// but could not decrypt, gunzip or parse, why not, by fileID. A pass
|
// but could not decrypt, gunzip or parse, why not, by fileID. A pass
|
||||||
// gathers its own and puts them here only when it ends without failing.
|
// gathers its own and puts them here when it ends without failing, so
|
||||||
// So `status()` and `backup()` never see the list of a pass still running
|
// `status()` and `backup()` never see the list of a pass that is still
|
||||||
// or one that failed. Kept in RAM only, never in the cache.
|
// running or that failed. Kept in RAM only, never in the cache.
|
||||||
private mlDataErrors = new Map<number, unknown>();
|
private mlDataErrors = new Map<number, unknown>();
|
||||||
// The plain-record projection as of the last store change, and the GUI
|
// The plain-record projection as of the last refresh, and the GUI change
|
||||||
// change subscribers. A refresh or `refreshCollection()` that alters the
|
// subscribers. A refresh that alters the projection notifies each with the
|
||||||
// projection notifies each subscriber with the delta. `lastRecords`
|
// delta; `lastRecords` is kept current every refresh so a subscriber that
|
||||||
// advances on each `commit()` that changes the store. A subscriber that
|
// joins later diffs against the state its own `snapshot()` already returned.
|
||||||
// joins later then diffs against the state its own `snapshot()` returned.
|
|
||||||
private readonly subscribers = new Set<(change: LibraryChange) => void>();
|
private readonly subscribers = new Set<(change: LibraryChange) => void>();
|
||||||
private lastRecords: DerivedRecords;
|
private lastRecords: DerivedRecords;
|
||||||
// RAM holds changes disk has not yet accepted (an earlier save failed).
|
// RAM holds changes disk has not yet accepted (an earlier save failed).
|
||||||
// Cleared only when a save succeeds. Until then, every `commit()` retries
|
// Cleared only when a save actually succeeds; keeps the store trying to
|
||||||
// the save.
|
// persist and the failure visible in `status()` until then.
|
||||||
private unsaved = false;
|
private unsaved = false;
|
||||||
|
|
||||||
private constructor(args: {
|
private constructor(args: {
|
||||||
@@ -387,7 +369,7 @@ export class Library {
|
|||||||
this.lastRecords = this.deriveNow();
|
this.lastRecords = this.deriveNow();
|
||||||
|
|
||||||
// The read namespaces derive fresh from the store on each call, so they
|
// The read namespaces derive fresh from the store on each call, so they
|
||||||
// always reflect what it holds now.
|
// always reflect the latest refresh.
|
||||||
const derive = (): DerivedRecords => this.deriveNow();
|
const derive = (): DerivedRecords => this.deriveNow();
|
||||||
const root = this.downloadDirectory;
|
const root = this.downloadDirectory;
|
||||||
const saves: SavePathLookup = {
|
const saves: SavePathLookup = {
|
||||||
@@ -414,16 +396,14 @@ export class Library {
|
|||||||
this.mldata = makeMLDataAPI(() => this.mlStore);
|
this.mldata = makeMLDataAPI(() => this.mlStore);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Load the cache and start the refresh loop. When the stored cursor is 0,
|
// Load the cache and start the refresh loop. With an empty cache the first
|
||||||
// the first refresh is awaited. Such a cache is empty, or holds only what
|
// refresh is awaited, so `open()` resolves onto populated data whenever the
|
||||||
// `refreshCollection()` stored. `open()` then resolves onto the whole
|
// server is reachable; that awaited refresh may still fail, and the library
|
||||||
// account whenever the server is reachable. If that refresh fails, the
|
// then opens empty with the failure recorded in `status()`. With an
|
||||||
// library opens onto what the cache holds, with the failure recorded in
|
// existing cache the first refresh runs in the background and `open()`
|
||||||
// `status()`. With any other cache the first refresh runs in the
|
// returns as soon as the cached data is ready — an unreachable server does
|
||||||
// background, and `open()` returns as soon as the cached data is ready.
|
// not block opening. With `refreshOnOpen: false` there is no first
|
||||||
// An unreachable server then does not block opening. With
|
// refresh: `open()` returns onto the cache as it is.
|
||||||
// `refreshOnOpen: false` there is no first refresh: `open()` returns onto
|
|
||||||
// the cache as it is.
|
|
||||||
static async open(opts: LibraryOptions): Promise<Library> {
|
static async open(opts: LibraryOptions): Promise<Library> {
|
||||||
const { userID } = opts.client.whoami();
|
const { userID } = opts.client.whoami();
|
||||||
const cacheDirectory =
|
const cacheDirectory =
|
||||||
@@ -439,8 +419,8 @@ export class Library {
|
|||||||
let store = await MetadataStore.load(metadataPath);
|
let store = await MetadataStore.load(metadataPath);
|
||||||
// A cache directory given explicitly can hold another account's cache.
|
// A cache directory given explicitly can hold another account's cache.
|
||||||
// Its records and cursor are not this account's, so delete it and the
|
// Its records and cursor are not this account's, so delete it and the
|
||||||
// ML data beside it and start empty. A user ID of 0 means nothing was
|
// ML data beside it and start empty. A user ID of 0 means the cache
|
||||||
// ever stored there, so there is nothing to discard.
|
// was never refreshed and so holds nothing to discard.
|
||||||
if (store.userID !== 0 && store.userID !== userID) {
|
if (store.userID !== 0 && store.userID !== userID) {
|
||||||
await rm(metadataPath, { force: true });
|
await rm(metadataPath, { force: true });
|
||||||
await rm(join(cacheDirectory, "mldata"), {
|
await rm(join(cacheDirectory, "mldata"), {
|
||||||
@@ -463,11 +443,10 @@ export class Library {
|
|||||||
? await MLDataStore.open(join(cacheDirectory, "mldata"))
|
? await MLDataStore.open(join(cacheDirectory, "mldata"))
|
||||||
: undefined;
|
: undefined;
|
||||||
|
|
||||||
// Build the content cache from an explicit source or the client's own.
|
// Build the content cache from an explicit source or the client's own,
|
||||||
// Take its record of what is already cached, and reap orphan temp
|
// and take its record of what is already cached (and reap orphan temp
|
||||||
// files, before the first projection. Cached paths are then present
|
// files) before the first projection, so cached paths are present from
|
||||||
// from the start, and the first `commit()` raises no spurious
|
// the start and the first refresh raises no spurious path-change diff.
|
||||||
// path-change diff.
|
|
||||||
const source = opts.contentSource ?? opts.client.contentSource?.();
|
const source = opts.contentSource ?? opts.client.contentSource?.();
|
||||||
let cache: ContentCache | undefined;
|
let cache: ContentCache | undefined;
|
||||||
let precache: Precache | undefined;
|
let precache: Precache | undefined;
|
||||||
@@ -521,26 +500,22 @@ export class Library {
|
|||||||
precache,
|
precache,
|
||||||
});
|
});
|
||||||
|
|
||||||
// Start filling from whatever the loaded store already holds. Each
|
// Start filling from whatever the loaded store already holds; each
|
||||||
// later `commit()` re-kicks the fills with its new files, and retries
|
// refresh below re-kicks with the new files (and retries any that
|
||||||
// any that failed. An empty store starts empty here, and fills once a
|
// failed). An empty store starts empty here and fills after its first
|
||||||
// `commit()` stores files.
|
// refresh.
|
||||||
precache?.start();
|
precache?.start();
|
||||||
|
|
||||||
if (opts.refreshOnOpen === false) {
|
if (opts.refreshOnOpen === false) {
|
||||||
// The caller asks for each refresh it needs.
|
// The caller asks for each refresh it needs.
|
||||||
lib.scheduleNext();
|
lib.scheduleNext();
|
||||||
} else if (store.collectionsSinceTime > 0) {
|
} else if (store.loadedFromDisk) {
|
||||||
// The stored cursor has moved, so a refresh has stored the
|
// An existing copy already answers reads; refresh in the background
|
||||||
// account's collections and the cache's copy answers reads.
|
// and start the interval once that first cycle settles.
|
||||||
// Refresh in the background, and start the interval once that
|
|
||||||
// first cycle settles.
|
|
||||||
void lib.runRefresh().then(() => lib.scheduleNext());
|
void lib.runRefresh().then(() => lib.scheduleNext());
|
||||||
} else {
|
} else {
|
||||||
// The stored cursor is still 0, so no refresh has stored any
|
// Nothing was cached: wait for the first refresh to fill the store
|
||||||
// collection: the cache is empty, or holds only what
|
// (or fail) rather than resolve onto an empty library.
|
||||||
// `refreshCollection()` stored. Wait for the first refresh to fill
|
|
||||||
// the store (or fail) rather than resolve onto part of the account.
|
|
||||||
await lib.runRefresh();
|
await lib.runRefresh();
|
||||||
lib.scheduleNext();
|
lib.scheduleNext();
|
||||||
}
|
}
|
||||||
@@ -578,9 +553,9 @@ export class Library {
|
|||||||
return snapshotFrom(this.deriveNow(), Date.now());
|
return snapshotFrom(this.deriveNow(), Date.now());
|
||||||
}
|
}
|
||||||
|
|
||||||
// Deliver a `LibraryChange` whenever a refresh or `refreshCollection()`
|
// Deliver a `LibraryChange` whenever a refresh alters the projection. A
|
||||||
// alters the projection. One that changes nothing delivers nothing. The
|
// refresh that changes nothing delivers nothing. The returned handle's
|
||||||
// returned handle's `unsubscribe` stops delivery.
|
// `unsubscribe` stops delivery.
|
||||||
subscribe(args: { onChange: (change: LibraryChange) => void }): {
|
subscribe(args: { onChange: (change: LibraryChange) => void }): {
|
||||||
unsubscribe: () => void;
|
unsubscribe: () => void;
|
||||||
} {
|
} {
|
||||||
@@ -623,13 +598,12 @@ export class Library {
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
// Fresh reads (issue #75, owner amendment to design #36). Wait for a
|
// Fresh reads (issue #75, owner amendment to design #36). Force a refresh,
|
||||||
// refresh to complete and persist, then hand back the same
|
// wait for it to complete and persist, then hand back the same
|
||||||
// `albums`/`photos`/`timeline` namespaces. They then reflect a completed
|
// `albums`/`photos`/`timeline` namespaces — now guaranteed to reflect a
|
||||||
// server round-trip. A running `refreshCollection()` is waited for first.
|
// completed server round-trip. Concurrent calls coalesce onto one refresh;
|
||||||
// Concurrent calls coalesce onto one refresh. A refresh that fails rejects
|
// a refresh that fails rejects here, where the default namespaces would
|
||||||
// here, where the default namespaces would stay silent and serve the last
|
// instead stay silent and serve the last good copy.
|
||||||
// good copy.
|
|
||||||
async fresh(): Promise<FreshReads> {
|
async fresh(): Promise<FreshReads> {
|
||||||
await this.refreshNow();
|
await this.refreshNow();
|
||||||
return {
|
return {
|
||||||
@@ -642,10 +616,10 @@ export class Library {
|
|||||||
// The fresh read of the collection list alone (issue #210): every live
|
// The fresh read of the collection list alone (issue #210): every live
|
||||||
// collection that decrypts, in the server's order, from one collections
|
// collection that decrypts, in the server's order, from one collections
|
||||||
// request. Unlike a refresh, it fetches no collection's files, which on a
|
// request. Unlike a refresh, it fetches no collection's files, which on a
|
||||||
// large account take long. It stores nothing. A refresh from a stored
|
// large account take long. Nothing is stored: a refresh fetches a
|
||||||
// cursor past 0 fetches a collection's files only when the collection is
|
// collection's files only when the collection is new or has changed since
|
||||||
// new or has changed since the stored copy. So does `refreshCollection()`.
|
// the stored copy, so a collection stored here without its files would
|
||||||
// Neither would fetch the files of a collection stored here without them.
|
// never have them fetched.
|
||||||
async freshCollections(): Promise<Collection[]> {
|
async freshCollections(): Promise<Collection[]> {
|
||||||
if (this.closed) throw new Error("the library is closed");
|
if (this.closed) throw new Error("the library is closed");
|
||||||
const { collections } = await this.client.collectionsSince({
|
const { collections } = await this.client.collectionsSince({
|
||||||
@@ -654,44 +628,67 @@ export class Library {
|
|||||||
return collections;
|
return collections;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Bring one collection up to date in the store (issue #217), given its
|
// The fresh read of one collection's files (issue #217): every live file
|
||||||
// record from `freshCollections()`. It does for that collection what a
|
// in it that decrypts, in the server's order. `collection` is a record
|
||||||
// refresh does for a changed one. It fetches no other collection's files.
|
// `freshCollections()` returned, whose key decrypts the files. It fetches
|
||||||
// It resolves to whether the store then holds the collection.
|
// no other collection's files. Like `freshCollections()`, it stores
|
||||||
//
|
// nothing. It returns each file once, as the latest copy the server sent,
|
||||||
// It adds a collection the store does not hold only while the stored
|
// where the first copy was.
|
||||||
// cursor is 0, before any refresh has received a collection. After that,
|
async freshFiles(collection: Collection): Promise<EnteFile[]> {
|
||||||
// it stores nothing for such a collection and resolves to false. Only a
|
|
||||||
// refresh may then add it, after the collections the store holds.
|
|
||||||
//
|
|
||||||
// It runs in the same slot as a refresh, so the two never run at the same
|
|
||||||
// time. It first waits for a running refresh or `refreshCollection()` to
|
|
||||||
// end. While its own work runs, a fresh read or `backup()` waits for it,
|
|
||||||
// and the background loop skips its turn.
|
|
||||||
async refreshCollection(collection: Collection): Promise<boolean> {
|
|
||||||
while (this.cycle) await this.cycle.catch(() => {});
|
|
||||||
if (this.closed) throw new Error("the library is closed");
|
if (this.closed) throw new Error("the library is closed");
|
||||||
let held = false;
|
const { files } = await this.client.filesSince({
|
||||||
await this.startCycle(false, async () => {
|
collectionID: collection.id,
|
||||||
held = await this.refreshCollectionOnce(collection);
|
collectionKey: collection.key,
|
||||||
|
sinceTime: 0,
|
||||||
});
|
});
|
||||||
return held;
|
const byID = new Map<number, EnteFile>();
|
||||||
|
for (const file of files) byID.set(file.id, file);
|
||||||
|
return [...byID.values()];
|
||||||
|
}
|
||||||
|
|
||||||
|
// The original of `file`, fetched and cached as `Photo.original()` does.
|
||||||
|
// `file` is used as given, so the store need not hold it. On an empty
|
||||||
|
// cache, `quak get` passes a file that `freshFiles()` returned.
|
||||||
|
async originalOf(
|
||||||
|
file: EnteFile,
|
||||||
|
opts?: ContentOptions,
|
||||||
|
): Promise<ContentResult> {
|
||||||
|
if (!this.cache) {
|
||||||
|
throw new Error(
|
||||||
|
"originalOf requires a library opened with a content cache",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return this.cache.originalOf(file, opts);
|
||||||
|
}
|
||||||
|
|
||||||
|
// The thumbnail of `file`, fetched and cached as `Photo.thumbnail()` does.
|
||||||
|
// `file` is used as given, so the store need not hold it. On an empty
|
||||||
|
// cache, `quak get-thumb` passes a file that `freshFiles()` returned.
|
||||||
|
async thumbnailOf(
|
||||||
|
file: EnteFile,
|
||||||
|
opts?: ContentOptions,
|
||||||
|
): Promise<ContentResult> {
|
||||||
|
if (!this.cache) {
|
||||||
|
throw new Error(
|
||||||
|
"thumbnailOf requires a library opened with a content cache",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
return this.cache.thumbnailOf(file, opts);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||||
// library's, each original at its save path, with a durable failure
|
// library's, each original at its save path, with a durable failure
|
||||||
// ledger (issue #51). It first takes the lock in that directory, unless
|
// ledger (issue #51). Takes the lock in that directory first, unless the
|
||||||
// the caller passes the lock it took as `opts.lock`. Taking it fails at
|
// caller passes the lock it took as `opts.lock`, failing at once while
|
||||||
// once while another backup of that directory runs. Once the lock is lost,
|
// another backup of it runs, and before it starts another file once the
|
||||||
// the backup fails before it starts another file (see
|
// lock is lost (see `lockBackupDirectory`). Waits for a
|
||||||
// `lockBackupDirectory`). It waits for a completed refresh as `fresh()`
|
// completed refresh, as `fresh()` does, joining one already running, and
|
||||||
// does, joining one already running. When that refresh fails, it rejects
|
// rejects before touching any file but the lock when it fails. Then puts
|
||||||
// before touching any file but the lock. It then puts pending originals
|
// pending originals at their save paths as `Photo.download()` does (and
|
||||||
// at their save paths as `Photo.download()` does, through the content
|
// optional thumbnails) through the content cache and pools, waits for an
|
||||||
// cache and pools. With `includeThumbnails` it also fetches thumbnails. It
|
// ML data fetch, and rebuilds the derived symlink/JSON views from the
|
||||||
// waits for an ML data fetch. It rebuilds the derived symlink/JSON views
|
// model, each file's JSON with its ML data, beside an `account.json` with
|
||||||
// from the model, each file's JSON with its ML data. An `account.json`
|
// the account's email and user ID.
|
||||||
// beside them holds the account's email and user ID.
|
|
||||||
// Throws before any network work when no content cache backs the
|
// Throws before any network work when no content cache backs the
|
||||||
// originals it must fetch.
|
// originals it must fetch.
|
||||||
backup(opts?: BackupOptions): Promise<BackupResult> {
|
backup(opts?: BackupOptions): Promise<BackupResult> {
|
||||||
@@ -728,15 +725,12 @@ export class Library {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Stop the background timer. Idempotent. A refresh or `refreshCollection()`
|
// Stop the background timer. Idempotent. An in-flight refresh is left to
|
||||||
// in flight is left to finish, and nothing starts after it. The returned
|
// finish; it will not schedule another cycle once closed. The returned
|
||||||
// promise resolves once all of these have finished:
|
// promise resolves once that refresh (including its cache write), the ML
|
||||||
// - that refresh or `refreshCollection()`, with its cache write;
|
// fetch pass and the precache fetches already running have all finished,
|
||||||
// - the ML fetch pass;
|
// so a caller can then remove the cache directory. A refresh failure is
|
||||||
// - the precache fetches already running.
|
// reported through `status()`, not thrown here.
|
||||||
// A caller can then remove the cache directory. A refresh failure is
|
|
||||||
// reported through `status()`, and a `refreshCollection()` failure to its
|
|
||||||
// own caller. Neither is thrown here.
|
|
||||||
async close(): Promise<void> {
|
async close(): Promise<void> {
|
||||||
this.closed = true;
|
this.closed = true;
|
||||||
const precacheClosed = this.precache?.close();
|
const precacheClosed = this.precache?.close();
|
||||||
@@ -759,41 +753,33 @@ export class Library {
|
|||||||
this.timer.unref?.();
|
this.timer.unref?.();
|
||||||
}
|
}
|
||||||
|
|
||||||
// The background loop's refresh: run a cycle unless a refresh or
|
// The background loop's refresh: run a cycle unless one is already in flight
|
||||||
// `refreshCollection()` is in flight (or the library is closed), and never
|
// (or the library is closed), and never let a failure escape — the
|
||||||
// let a failure escape. The background path reports errors through
|
// background path reports errors through `status()`/`onProgress`, it does
|
||||||
// `status()`/`onProgress`; it does not throw. It resolves once the cycle
|
// not throw. Resolves once the cycle it started (or skipped past) settles.
|
||||||
// it started settles, or at once when it starts none.
|
|
||||||
private runRefresh(): Promise<void> {
|
private runRefresh(): Promise<void> {
|
||||||
if (this.closed || this.cycle) return Promise.resolve();
|
if (this.closed || this.cycle) return Promise.resolve();
|
||||||
return this.startCycle(true, () => this.refreshCycle()).catch(() => {});
|
return this.startCycle().catch(() => {});
|
||||||
}
|
}
|
||||||
|
|
||||||
// A fresh read's refresh (issue #75): join the running refresh or start
|
// A fresh read's refresh (issue #75): force a cycle and await it, rejecting
|
||||||
// one, and await it, rejecting if it fails. Concurrent fresh reads
|
// if it fails. Concurrent fresh reads coalesce onto the one in-flight cycle
|
||||||
// coalesce onto the one in-flight cycle — the background loop's included —
|
// — the background loop's included — so they never fan out into redundant
|
||||||
// so they never fan out into redundant server round-trips. A running
|
// server round-trips.
|
||||||
// `refreshCollection()` is waited for first.
|
private refreshNow(): Promise<void> {
|
||||||
private async refreshNow(): Promise<void> {
|
if (this.closed) {
|
||||||
while (this.cycle && !this.cycleIsRefresh) {
|
return Promise.reject(new Error("the library is closed"));
|
||||||
await this.cycle.catch(() => {});
|
|
||||||
}
|
}
|
||||||
if (this.closed) throw new Error("the library is closed");
|
return this.cycle ?? this.startCycle();
|
||||||
return this.cycle ?? this.startCycle(true, () => this.refreshCycle());
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Start `work` and record it in the in-flight slot, so every other caller
|
// Start one refresh cycle and record it as the in-flight cycle so every
|
||||||
// sees it. `isRefresh` says whether it is a refresh cycle, which a fresh
|
// caller coalesces onto it. The returned promise carries the cycle's real
|
||||||
// read can coalesce onto. The returned promise carries the real outcome.
|
// outcome; each caller attaches the handling its own path needs, and the
|
||||||
// Each caller attaches the handling its own path needs, and the slot is
|
// slot is cleared once the cycle settles.
|
||||||
// cleared once the work settles.
|
private startCycle(): Promise<void> {
|
||||||
private startCycle(
|
const cycle = this.refreshCycle();
|
||||||
isRefresh: boolean,
|
|
||||||
work: () => Promise<void>,
|
|
||||||
): Promise<void> {
|
|
||||||
const cycle = work();
|
|
||||||
this.cycle = cycle;
|
this.cycle = cycle;
|
||||||
this.cycleIsRefresh = isRefresh;
|
|
||||||
void cycle.then(
|
void cycle.then(
|
||||||
() => {
|
() => {
|
||||||
if (this.cycle === cycle) this.cycle = undefined;
|
if (this.cycle === cycle) this.cycle = undefined;
|
||||||
@@ -816,9 +802,9 @@ export class Library {
|
|||||||
this.lastError = undefined;
|
this.lastError = undefined;
|
||||||
this.emit({ operation: "refresh", status: "done" });
|
this.emit({ operation: "refresh", status: "done" });
|
||||||
// Backfill ML data for the files this refresh knows about. It runs
|
// Backfill ML data for the files this refresh knows about. It runs
|
||||||
// outside the refresh's success or failure, so a fetch or disk
|
// outside the refresh's success/failure so a fetch or disk problem
|
||||||
// problem there never marks the metadata refresh failed. It is not
|
// there never marks the metadata refresh failed, and it is not
|
||||||
// awaited, so it never stalls the refresh interval. Its failure is
|
// awaited so it never stalls the refresh interval. Its failure is
|
||||||
// reported through `status()` and `onProgress`.
|
// reported through `status()` and `onProgress`.
|
||||||
void this.fetchMLDataNow().catch(() => {});
|
void this.fetchMLDataNow().catch(() => {});
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
@@ -831,14 +817,11 @@ export class Library {
|
|||||||
|
|
||||||
// Fetch every change since the stored cursor, then commit. All network
|
// Fetch every change since the stored cursor, then commit. All network
|
||||||
// reads happen before any store mutation, so a fetch that throws leaves the
|
// reads happen before any store mutation, so a fetch that throws leaves the
|
||||||
// store untouched and the previous snapshot intact.
|
// store untouched and the previous snapshot intact. A collection or file
|
||||||
//
|
// that does not decrypt is left out instead, as if deleted, with the reason
|
||||||
// From a stored cursor of 0, the server lists every live collection. The
|
// kept in the store: the cursor moves past it, so no later refresh sees it
|
||||||
// refresh then replaces all the store holds, and counts every collection
|
// again until the server's copy changes.
|
||||||
// as new. Only `refreshCollection()` can have stored anything before it.
|
|
||||||
// So the store ends as it would had `refreshCollection()` never run.
|
|
||||||
private async refreshOnce(): Promise<void> {
|
private async refreshOnce(): Promise<void> {
|
||||||
const replace = this.store.collectionsSinceTime === 0;
|
|
||||||
const page = await this.client.collectionsSince({
|
const page = await this.client.collectionsSince({
|
||||||
sinceTime: this.store.collectionsSinceTime,
|
sinceTime: this.store.collectionsSinceTime,
|
||||||
});
|
});
|
||||||
@@ -846,19 +829,16 @@ export class Library {
|
|||||||
// Stage per-collection file diffs. A collection's files are
|
// Stage per-collection file diffs. A collection's files are
|
||||||
// re-enumerated only when its updationTime has advanced past the cached
|
// re-enumerated only when its updationTime has advanced past the cached
|
||||||
// copy; an unchanged album's file list cannot have changed. New
|
// copy; an unchanged album's file list cannot have changed. New
|
||||||
// collections enumerate from the beginning of time. When the refresh
|
// collections enumerate from the beginning of time. Each collection
|
||||||
// replaces the store, every collection counts as new. Each collection
|
|
||||||
// whose files are fetched is reported, with how many there are to
|
// whose files are fetched is reported, with how many there are to
|
||||||
// fetch.
|
// fetch.
|
||||||
const storedCopy = (id: number): Collection | undefined =>
|
|
||||||
replace ? undefined : this.store.getCollection(id);
|
|
||||||
const changedCollections = page.collections.filter((collection) => {
|
const changedCollections = page.collections.filter((collection) => {
|
||||||
const known = storedCopy(collection.id);
|
const known = this.store.getCollection(collection.id);
|
||||||
return !known || collection.updationTime > known.updationTime;
|
return !known || collection.updationTime > known.updationTime;
|
||||||
});
|
});
|
||||||
const filePages: { collectionID: number; page: FilesPage }[] = [];
|
const filePages: { collectionID: number; page: FilesPage }[] = [];
|
||||||
for (const collection of changedCollections) {
|
for (const collection of changedCollections) {
|
||||||
const known = storedCopy(collection.id);
|
const known = this.store.getCollection(collection.id);
|
||||||
const filePage = await this.client.filesSince({
|
const filePage = await this.client.filesSince({
|
||||||
collectionID: collection.id,
|
collectionID: collection.id,
|
||||||
collectionKey: collection.key,
|
collectionKey: collection.key,
|
||||||
@@ -873,56 +853,8 @@ export class Library {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
await this.commit(page, filePages, replace);
|
// Network work done; commit to the store and persist only if something
|
||||||
}
|
// actually changed.
|
||||||
|
|
||||||
// `refreshCollection()`'s work, run in the slot: fetch the files of one
|
|
||||||
// collection changed since the stored copy, then commit. It resolves to
|
|
||||||
// whether the store then holds the collection.
|
|
||||||
private async refreshCollectionOnce(
|
|
||||||
collection: Collection,
|
|
||||||
): Promise<boolean> {
|
|
||||||
const known = this.store.getCollection(collection.id);
|
|
||||||
// A refresh has received the account's collections, and this one is
|
|
||||||
// not stored. It is new since then, or a refresh dropped it. Leave it
|
|
||||||
// to the next refresh, which adds new collections after the stored
|
|
||||||
// ones, in the server's order.
|
|
||||||
if (!known && this.store.collectionsSinceTime > 0) return false;
|
|
||||||
// The stored copy is as new or newer, so there is nothing to fetch.
|
|
||||||
if (known && collection.updationTime <= known.updationTime) return true;
|
|
||||||
const filePage = await this.client.filesSince({
|
|
||||||
collectionID: collection.id,
|
|
||||||
collectionKey: collection.key,
|
|
||||||
sinceTime: known ? known.updationTime : 0,
|
|
||||||
});
|
|
||||||
// The stored cursor stays where it is, so the next refresh still
|
|
||||||
// fetches the files of every other collection that changed.
|
|
||||||
await this.commit(
|
|
||||||
{
|
|
||||||
collections: [collection],
|
|
||||||
deleted: [],
|
|
||||||
cursor: this.store.collectionsSinceTime,
|
|
||||||
},
|
|
||||||
[{ collectionID: collection.id, page: filePage }],
|
|
||||||
false,
|
|
||||||
);
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Commit a collections page and the file pages fetched for its collections
|
|
||||||
// to the store. With `replace`, the page lists every live collection, so
|
|
||||||
// the commit first deletes every stored collection. A collection or file
|
|
||||||
// that does not decrypt is left out, as if deleted. The reason is kept in
|
|
||||||
// the store. Until the server's copy changes, only a refresh from a stored
|
|
||||||
// cursor of 0 fetches it again. Last, save the store when it holds changes
|
|
||||||
// not yet saved.
|
|
||||||
private async commit(
|
|
||||||
page: CollectionsPage,
|
|
||||||
filePages: { collectionID: number; page: FilesPage }[],
|
|
||||||
replace: boolean,
|
|
||||||
): Promise<void> {
|
|
||||||
// The network work is done, so change the store now. `changed` says
|
|
||||||
// whether this commit changed anything.
|
|
||||||
let changed = false;
|
let changed = false;
|
||||||
|
|
||||||
if (this.store.userID !== this.userID) {
|
if (this.store.userID !== this.userID) {
|
||||||
@@ -930,13 +862,6 @@ export class Library {
|
|||||||
changed = true;
|
changed = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (replace) {
|
|
||||||
for (const { id } of this.store.listCollections()) {
|
|
||||||
this.store.deleteCollection(id);
|
|
||||||
changed = true;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
for (const id of page.deleted) {
|
for (const id of page.deleted) {
|
||||||
if (this.store.getCollection(id)) {
|
if (this.store.getCollection(id)) {
|
||||||
this.store.deleteCollection(id);
|
this.store.deleteCollection(id);
|
||||||
@@ -990,10 +915,10 @@ export class Library {
|
|||||||
if (changed) this.unsaved = true;
|
if (changed) this.unsaved = true;
|
||||||
|
|
||||||
// Reproject and notify subscribers of the delta. This tracks RAM (what
|
// Reproject and notify subscribers of the delta. This tracks RAM (what
|
||||||
// reads see), so it fires whether or not the save below succeeds. A
|
// reads see), so it fires whether or not the save below succeeds; a
|
||||||
// failed save rejects the caller, and a refresh also records it in
|
// save failure surfaces separately through `status().lastError`.
|
||||||
// `status().lastError`. `lastRecords` advances on every commit that
|
// `lastRecords` advances every changed refresh so the next diff is
|
||||||
// changes something, so the next diff is against current state.
|
// against current state.
|
||||||
if (changed) {
|
if (changed) {
|
||||||
const next = this.deriveNow();
|
const next = this.deriveNow();
|
||||||
if (this.subscribers.size > 0) {
|
if (this.subscribers.size > 0) {
|
||||||
@@ -1004,16 +929,16 @@ export class Library {
|
|||||||
// Recompute the fill orders and pinned set against the new library.
|
// Recompute the fill orders and pinned set against the new library.
|
||||||
this.precache?.update(next);
|
this.precache?.update(next);
|
||||||
}
|
}
|
||||||
// Re-kick the fills on every commit. A finished sweep starts afresh to
|
// Re-kick the fills every cycle: a finished sweep starts afresh to pick
|
||||||
// pick up new files and retry any that failed. A running one is left
|
// up new files and retry any that failed, and a running one is left be.
|
||||||
// be.
|
|
||||||
this.precache?.start();
|
this.precache?.start();
|
||||||
|
|
||||||
// Persist whenever RAM holds changes disk has not accepted — including
|
// Persist whenever RAM holds changes disk has not accepted — including
|
||||||
// changes an earlier cycle staged whose save failed. `unsaved` clears
|
// changes an earlier cycle staged whose save failed. `unsaved` clears
|
||||||
// only once a save lands, so a later commit retries a failed save even
|
// only once a save lands, so a save failure both stays visible through
|
||||||
// when it changes nothing. A failed save rejects the caller, and a
|
// `status().lastError` (the throw below records it) and keeps being
|
||||||
// refresh also records it in `status().lastError`.
|
// retried, instead of a later empty refresh silently clearing it while
|
||||||
|
// the on-disk cache is still behind RAM.
|
||||||
if (this.unsaved) {
|
if (this.unsaved) {
|
||||||
await this.store.save();
|
await this.store.save();
|
||||||
this.unsaved = false;
|
this.unsaved = false;
|
||||||
@@ -1029,14 +954,13 @@ export class Library {
|
|||||||
return this.mlFetch;
|
return this.mlFetch;
|
||||||
}
|
}
|
||||||
|
|
||||||
// One ML fetch pass. It fetches, decrypts and stores the ML data of every
|
// One ML fetch pass: fetch, decrypt and store the ML data for every file
|
||||||
// file the store knows about that is not cached, or whose `updationTime`
|
// the store knows about that is not cached (or whose `updationTime` has
|
||||||
// has advanced. It fetches through the metadata pool, and updates the CLIP
|
// advanced), through the metadata pool, and update the CLIP index. For a
|
||||||
// index. For a file whose ML data does not decrypt, gunzip or parse, it
|
// file whose ML data does not decrypt, gunzip or parse, it gathers the
|
||||||
// gathers the reason instead. It puts the reasons it gathered in
|
// reason instead, and puts the reasons it gathered in `mlDataErrors` when
|
||||||
// `mlDataErrors` when it ends without failing. A failure leaves
|
// it ends without failing. A failure leaves `mlDataErrors` as it was and
|
||||||
// `mlDataErrors` as it was and is reported through `status()` and
|
// is reported through `status()` and `onProgress`, then thrown.
|
||||||
// `onProgress`, then thrown.
|
|
||||||
private async runMLFetch(): Promise<void> {
|
private async runMLFetch(): Promise<void> {
|
||||||
const mldata = this.mlStore;
|
const mldata = this.mlStore;
|
||||||
// Bind so the call keeps the client as its receiver when invoked
|
// Bind so the call keeps the client as its receiver when invoked
|
||||||
@@ -1104,9 +1028,9 @@ export class Library {
|
|||||||
}));
|
}));
|
||||||
}
|
}
|
||||||
|
|
||||||
// The distinct files the store holds, one entry per fileID, since a file
|
// The distinct files the store holds, one entry per fileID (a file in
|
||||||
// in several collections shares its ML data. Each entry carries the key
|
// several collections shares its ML data), each carrying the key and the
|
||||||
// and the newest `updationTime` seen across the file's memberships.
|
// newest `updationTime` seen across its memberships.
|
||||||
private uniqueFiles(): {
|
private uniqueFiles(): {
|
||||||
id: number;
|
id: number;
|
||||||
key: Uint8Array;
|
key: Uint8Array;
|
||||||
|
|||||||
@@ -122,7 +122,7 @@ export class Precache {
|
|||||||
}
|
}
|
||||||
|
|
||||||
// Recompute the fill orders and the pinned set from the current projection.
|
// Recompute the fill orders and the pinned set from the current projection.
|
||||||
// Called at open and after every change to the store.
|
// Called at open and after every refresh that changes the library.
|
||||||
update(records: DerivedRecords): void {
|
update(records: DerivedRecords): void {
|
||||||
const photos = [...records.photos.values()].sort(
|
const photos = [...records.photos.values()].sort(
|
||||||
(a, b) => b.takenAt - a.takenAt || b.fileID - a.fileID,
|
(a, b) => b.takenAt - a.takenAt || b.fileID - a.fileID,
|
||||||
|
|||||||
@@ -71,14 +71,13 @@ export interface LibrarySnapshot {
|
|||||||
takenAt: number;
|
takenAt: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
// What one refresh or `refreshCollection()` changed in the records.
|
|
||||||
export interface LibraryChange {
|
export interface LibraryChange {
|
||||||
// Full records for albums/photos added or changed.
|
// Full records for albums/photos added or changed by the refresh.
|
||||||
albumsChanged: AlbumRecord[];
|
albumsChanged: AlbumRecord[];
|
||||||
photosChanged: PhotoRecord[];
|
photosChanged: PhotoRecord[];
|
||||||
fileIDsRemoved: number[];
|
fileIDsRemoved: number[];
|
||||||
albumIDsRemoved: number[];
|
albumIDsRemoved: number[];
|
||||||
// Wall-clock milliseconds when the store changed.
|
// Wall-clock milliseconds of the refresh that produced this change.
|
||||||
refreshedAt: number;
|
refreshedAt: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+12
-13
@@ -3,7 +3,7 @@
|
|||||||
// The store keeps one `metadata.json` file holding the account's server
|
// The store keeps one `metadata.json` file holding the account's server
|
||||||
// state: the user id, a schema version, the cursor for the incremental
|
// state: the user id, a schema version, the cursor for the incremental
|
||||||
// collections listing, the decrypted collection and file records, and the
|
// collections listing, the decrypted collection and file records, and the
|
||||||
// collections and files that did not decrypt. The whole file is read
|
// collections and files the refresh could not decrypt. The whole file is read
|
||||||
// into RAM on load and rewritten as a whole on save; there is no partial
|
// into RAM on load and rewritten as a whole on save; there is no partial
|
||||||
// update and no lock file. A separate refresh unit populates the store from
|
// update and no lock file. A separate refresh unit populates the store from
|
||||||
// the server — this module only stores what it is given.
|
// the server — this module only stores what it is given.
|
||||||
@@ -34,11 +34,9 @@ const FILE_MODE = 0o600;
|
|||||||
type StoredCollection = Omit<Collection, "key"> & { key: string };
|
type StoredCollection = Omit<Collection, "key"> & { key: string };
|
||||||
type StoredFile = Omit<EnteFile, "key"> & { key: string };
|
type StoredFile = Omit<EnteFile, "key"> & { key: string };
|
||||||
|
|
||||||
// A collection, or with `fileID` a file in it, that did not decrypt or parse,
|
// A collection, or with `fileID` a file in it, that the last refresh to fetch
|
||||||
// and why. A refresh or `refreshCollection()` records it on fetching such a
|
// it could not decrypt or parse, and why. It is kept until a later refresh
|
||||||
// copy. A later fetch drops it on getting a copy that decrypts, or on learning
|
// fetches it again, because a refresh fetches only what changed.
|
||||||
// that the server deleted it. Each fetch asks only for what changed, so the
|
|
||||||
// store keeps the record between fetches.
|
|
||||||
export interface Undecryptable {
|
export interface Undecryptable {
|
||||||
collectionID: number;
|
collectionID: number;
|
||||||
fileID?: number;
|
fileID?: number;
|
||||||
@@ -72,6 +70,12 @@ export class MetadataStore {
|
|||||||
userID = 0;
|
userID = 0;
|
||||||
collectionsSinceTime: Microseconds = 0;
|
collectionsSinceTime: Microseconds = 0;
|
||||||
|
|
||||||
|
// True when `load` populated this store from a valid existing file; false
|
||||||
|
// on a first run or a missing/corrupt/wrong-version file that loaded empty.
|
||||||
|
// `Library.open` reads it to decide whether the first refresh may run in
|
||||||
|
// the background (an existing copy already serves reads) or must be awaited.
|
||||||
|
loadedFromDisk = false;
|
||||||
|
|
||||||
private readonly collections = new Map<number, Collection>();
|
private readonly collections = new Map<number, Collection>();
|
||||||
private readonly files = new Map<string, EnteFile>();
|
private readonly files = new Map<string, EnteFile>();
|
||||||
private undecryptable: Undecryptable[] = [];
|
private undecryptable: Undecryptable[] = [];
|
||||||
@@ -98,6 +102,7 @@ export class MetadataStore {
|
|||||||
if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) {
|
if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) {
|
||||||
return store;
|
return store;
|
||||||
}
|
}
|
||||||
|
store.loadedFromDisk = true;
|
||||||
store.userID = parsed.userID ?? 0;
|
store.userID = parsed.userID ?? 0;
|
||||||
store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0;
|
store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0;
|
||||||
for (const stored of parsed.collections ?? []) {
|
for (const stored of parsed.collections ?? []) {
|
||||||
@@ -224,14 +229,8 @@ export class MetadataStore {
|
|||||||
this.files.delete(fileKey(collectionID, fileID));
|
this.files.delete(fileKey(collectionID, fileID));
|
||||||
}
|
}
|
||||||
|
|
||||||
// Sorted by collection ID, then file ID, so the order does not depend on
|
|
||||||
// when each record was fetched.
|
|
||||||
listUndecryptable(): Undecryptable[] {
|
listUndecryptable(): Undecryptable[] {
|
||||||
return [...this.undecryptable].sort(
|
return [...this.undecryptable];
|
||||||
(a, b) =>
|
|
||||||
a.collectionID - b.collectionID ||
|
|
||||||
(a.fileID ?? 0) - (b.fileID ?? 0),
|
|
||||||
);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Replaces any earlier record of the same collection or file.
|
// Replaces any earlier record of the same collection or file.
|
||||||
|
|||||||
+11
-36
@@ -5,7 +5,7 @@ import type { Client } from "./client.js";
|
|||||||
import type { Library } from "./library/index.js";
|
import type { Library } from "./library/index.js";
|
||||||
import { ApiError } from "./api/client.js";
|
import { ApiError } from "./api/client.js";
|
||||||
import { encryptBlob, toBase64 } from "./crypto/index.js";
|
import { encryptBlob, toBase64 } from "./crypto/index.js";
|
||||||
import type { EnteFile } from "./model/types.js";
|
import type { Collection, EnteFile } from "./model/types.js";
|
||||||
|
|
||||||
// The server refuses a thumbnail larger than the one it already records for the
|
// The server refuses a thumbnail larger than the one it already records for the
|
||||||
// file (`thumbnail.size`, the encrypted size), so these encodings are tried
|
// file (`thumbnail.size`, the encrypted size), so these encodings are tried
|
||||||
@@ -226,16 +226,18 @@ const encryptThumbnailWithin = (
|
|||||||
return undefined;
|
return undefined;
|
||||||
};
|
};
|
||||||
|
|
||||||
// Regenerate and upload a thumbnail for each requested file. Originals are read
|
// Regenerate and upload a thumbnail for each requested file. `found` holds the
|
||||||
// through the library's content cache (`photo.original()`); the generated
|
// record of each requested file that a collection holds, and the collection it
|
||||||
// thumbnail is JPEG-encoded, encrypted under the file's own key, and registered
|
// was found in. Each original is read through the library's content cache, by
|
||||||
// with the server — the encrypt-and-upload path is unchanged. Each file is
|
// `lib.originalOf(file)`. The new thumbnail is a JPEG, encrypted under the
|
||||||
|
// file's own key. It is uploaded, then registered with the server. Each file is
|
||||||
// resolved to one outcome (fixed / skipped / failed) and a failure on one file
|
// resolved to one outcome (fixed / skipped / failed) and a failure on one file
|
||||||
// never stops the others.
|
// never stops the others.
|
||||||
export const fixMissingThumbnails = async (
|
export const fixMissingThumbnails = async (
|
||||||
lib: Library,
|
lib: Library,
|
||||||
client: Client,
|
client: Client,
|
||||||
fileIDs: number[],
|
fileIDs: number[],
|
||||||
|
found: Map<number, { file: EnteFile; collection: Collection }>,
|
||||||
onProgress?: ProgressCallback,
|
onProgress?: ProgressCallback,
|
||||||
): Promise<ThumbnailFixResult[]> => {
|
): Promise<ThumbnailFixResult[]> => {
|
||||||
const log = onProgress ?? (() => {});
|
const log = onProgress ?? (() => {});
|
||||||
@@ -243,31 +245,8 @@ export const fixMissingThumbnails = async (
|
|||||||
const api = client.getApiClient();
|
const api = client.getApiClient();
|
||||||
const { userID } = client.whoami();
|
const { userID } = client.whoami();
|
||||||
|
|
||||||
// Resolve each requested fileID to its file record and owning album by
|
|
||||||
// enumerating the library, each file taken from the first album that holds
|
|
||||||
// it. The raw `EnteFile` carries the per-file key the thumbnail is
|
|
||||||
// encrypted under, which the projected records deliberately do not.
|
|
||||||
const wanted = new Set(fileIDs);
|
|
||||||
const fileMap = new Map<
|
|
||||||
number,
|
|
||||||
{ file: EnteFile; collectionName: string }
|
|
||||||
>();
|
|
||||||
for (const album of lib.albums.list()) {
|
|
||||||
for (const photo of album.photos.list()) {
|
|
||||||
if (!wanted.has(photo.fileID) || fileMap.has(photo.fileID))
|
|
||||||
continue;
|
|
||||||
const file = lib.getFile(album.collectionID, photo.fileID);
|
|
||||||
if (file) {
|
|
||||||
fileMap.set(photo.fileID, {
|
|
||||||
file,
|
|
||||||
collectionName: album.name,
|
|
||||||
});
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
for (const fileID of fileIDs) {
|
for (const fileID of fileIDs) {
|
||||||
const entry = fileMap.get(fileID);
|
const entry = found.get(fileID);
|
||||||
if (!entry) {
|
if (!entry) {
|
||||||
results.push({
|
results.push({
|
||||||
fileID,
|
fileID,
|
||||||
@@ -279,7 +258,8 @@ export const fixMissingThumbnails = async (
|
|||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
const { file, collectionName } = entry;
|
const { file } = entry;
|
||||||
|
const collectionName = entry.collection.name;
|
||||||
const title = file.metadata.title;
|
const title = file.metadata.title;
|
||||||
|
|
||||||
const skipReason = reasonToSkip(file, userID);
|
const skipReason = reasonToSkip(file, userID);
|
||||||
@@ -297,15 +277,10 @@ export const fixMissingThumbnails = async (
|
|||||||
const maxSize = file.thumbnail.size!;
|
const maxSize = file.thumbnail.size!;
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const photo = lib.photos.byID({ fileID });
|
|
||||||
if (!photo) {
|
|
||||||
throw new Error("file not present in the library cache");
|
|
||||||
}
|
|
||||||
|
|
||||||
log(
|
log(
|
||||||
`[${collectionName}] Downloading ${title} for thumbnail generation...`,
|
`[${collectionName}] Downloading ${title} for thumbnail generation...`,
|
||||||
);
|
);
|
||||||
const { path } = await photo.original();
|
const { path } = await lib.originalOf(file);
|
||||||
const fileBytes = new Uint8Array(readFileSync(path));
|
const fileBytes = new Uint8Array(readFileSync(path));
|
||||||
|
|
||||||
if (!isJpeg(fileBytes)) {
|
if (!isJpeg(fileBytes)) {
|
||||||
|
|||||||
@@ -418,7 +418,7 @@ describe("the first backup of the account", () => {
|
|||||||
expect(downloads.sort()).toEqual([100, 200, 300]);
|
expect(downloads.sort()).toEqual([100, 200, 300]);
|
||||||
});
|
});
|
||||||
|
|
||||||
// The tree "Backup layout" in the README describes. Each original is at
|
// The tree under "The backup directory" in the README. Each original is at
|
||||||
// its save path, `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, with
|
// its save path, `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, with
|
||||||
// the extension of its title; the live photo is its image and its video,
|
// the extension of its title; the live photo is its image and its video,
|
||||||
// each with the extension it had inside the ZIP, and a `.livephoto.json`
|
// each with the extension it had inside the ZIP, and a `.livephoto.json`
|
||||||
|
|||||||
@@ -7,6 +7,11 @@
|
|||||||
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
|
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
|
||||||
* replaced by one that records the options it is given and reports no session,
|
* replaced by one that records the options it is given and reports no session,
|
||||||
* so the command stops before it makes any request.
|
* so the command stops before it makes any request.
|
||||||
|
*
|
||||||
|
* The first load of the sources `bin/quak.ts` imports compiles them, and the
|
||||||
|
* first `init()` starts libsodium. On a loaded host that takes longer than a
|
||||||
|
* test's timeout, so both happen here, before the tests run. Each test's
|
||||||
|
* `vi.resetModules()` then only runs the compiled sources again.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { mkdtempSync, rmSync } from "node:fs";
|
import { mkdtempSync, rmSync } from "node:fs";
|
||||||
@@ -14,8 +19,13 @@ import { tmpdir } from "node:os";
|
|||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
import { afterAll, afterEach, describe, expect, it, vi } from "vitest";
|
import { afterAll, afterEach, describe, expect, it, vi } from "vitest";
|
||||||
import type { ApiClientOptions } from "../../src/api/client.js";
|
import type { ApiClientOptions } from "../../src/api/client.js";
|
||||||
|
import "../../src/cli-commands.js";
|
||||||
|
import { init } from "../../src/crypto/index.js";
|
||||||
|
import "../../src/index.js";
|
||||||
import { UNATTENDED_RETRY_OPTIONS } from "../../src/retry.js";
|
import { UNATTENDED_RETRY_OPTIONS } from "../../src/retry.js";
|
||||||
|
|
||||||
|
await init();
|
||||||
|
|
||||||
const loaded = vi.hoisted(() => [] as (ApiClientOptions | undefined)[]);
|
const loaded = vi.hoisted(() => [] as (ApiClientOptions | undefined)[]);
|
||||||
|
|
||||||
vi.mock("../../src/cli-session.js", () => ({
|
vi.mock("../../src/cli-session.js", () => ({
|
||||||
|
|||||||
+307
-174
@@ -124,35 +124,20 @@ const FILES: Record<number, EnteFile[]> = {
|
|||||||
// thumbnail as empty. `withNewFile` adds new.jpg (102) to Vacation, advancing
|
// thumbnail as empty. `withNewFile` adds new.jpg (102) to Vacation, advancing
|
||||||
// the collection's updationTime as the server does. `beachInWork` adds
|
// the collection's updationTime as the server does. `beachInWork` adds
|
||||||
// beach.jpg (100) to Work as well, so that one file is in both albums.
|
// beach.jpg (100) to Work as well, so that one file is in both albums.
|
||||||
// `newAlbums` adds the albums A (3) and B (4), in that order, which both hold
|
// `refreshError` makes listing collections fail with that message.
|
||||||
// shared.jpg (300). `refreshError` makes listing collections fail with that
|
|
||||||
// message.
|
|
||||||
const fakeClient = (
|
const fakeClient = (
|
||||||
opts: {
|
opts: {
|
||||||
failID?: number;
|
failID?: number;
|
||||||
emptyThumbID?: number;
|
emptyThumbID?: number;
|
||||||
withNewFile?: boolean;
|
withNewFile?: boolean;
|
||||||
beachInWork?: boolean;
|
beachInWork?: boolean;
|
||||||
newAlbums?: boolean;
|
|
||||||
refreshError?: string;
|
refreshError?: string;
|
||||||
} = {},
|
} = {},
|
||||||
) => {
|
) => {
|
||||||
let collections = opts.withNewFile
|
const collections = opts.withNewFile
|
||||||
? [{ ...COLLECTIONS[0], updationTime: 2 }, COLLECTIONS[1]]
|
? [{ ...COLLECTIONS[0], updationTime: 2 }, COLLECTIONS[1]]
|
||||||
: COLLECTIONS;
|
: COLLECTIONS;
|
||||||
let files = FILES;
|
let files = FILES;
|
||||||
if (opts.newAlbums) {
|
|
||||||
collections = [
|
|
||||||
...collections,
|
|
||||||
{ ...collection(3, "A"), updationTime: 2 },
|
|
||||||
{ ...collection(4, "B"), updationTime: 2 },
|
|
||||||
];
|
|
||||||
files = {
|
|
||||||
...files,
|
|
||||||
3: [file(300, 3, "shared.jpg")],
|
|
||||||
4: [file(300, 4, "shared.jpg")],
|
|
||||||
};
|
|
||||||
}
|
|
||||||
if (opts.withNewFile) {
|
if (opts.withNewFile) {
|
||||||
files = {
|
files = {
|
||||||
...files,
|
...files,
|
||||||
@@ -209,9 +194,6 @@ const FETCHED_BOTH_ALBUMS =
|
|||||||
"Fetched the files of 1 of 2 collections\n" +
|
"Fetched the files of 1 of 2 collections\n" +
|
||||||
"Fetched the files of 2 of 2 collections\n";
|
"Fetched the files of 2 of 2 collections\n";
|
||||||
|
|
||||||
// The same for an account with one album.
|
|
||||||
const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n";
|
|
||||||
|
|
||||||
// Collects everything written to it.
|
// Collects everything written to it.
|
||||||
class Output {
|
class Output {
|
||||||
text = "";
|
text = "";
|
||||||
@@ -584,57 +566,6 @@ describe("files", () => {
|
|||||||
]);
|
]);
|
||||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS);
|
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS);
|
||||||
});
|
});
|
||||||
|
|
||||||
// `files 2` on an empty cache stores Work alone. The refresh that `files`
|
|
||||||
// then waits for must still store Vacation first, as the server lists it.
|
|
||||||
it("without a collection ID lists in the server's order after files 2 on an empty cache", async () => {
|
|
||||||
const ctx = context(fakeClient({ beachInWork: true }));
|
|
||||||
expect(await filesCommand(ctx, "2", {})).toBe(0);
|
|
||||||
stdout.text = "";
|
|
||||||
stderr.text = "";
|
|
||||||
|
|
||||||
expect(await filesCommand(ctx, undefined, { json: true })).toBe(0);
|
|
||||||
expect(
|
|
||||||
JSON.parse(stdout.text).map(
|
|
||||||
(row: { id: number; collectionID: number }) => [
|
|
||||||
row.id,
|
|
||||||
row.collectionID,
|
|
||||||
],
|
|
||||||
),
|
|
||||||
).toEqual([
|
|
||||||
[100, 1],
|
|
||||||
[101, 1],
|
|
||||||
[200, 2],
|
|
||||||
]);
|
|
||||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS);
|
|
||||||
});
|
|
||||||
|
|
||||||
// After a refresh, the server adds A and then B. `files 4` must not store B
|
|
||||||
// ahead of A, so it waits for a refresh, which stores both in that order.
|
|
||||||
it("without a collection ID lists new albums in the server's order after files with the second one's ID", async () => {
|
|
||||||
expect(await filesCommand(context(), undefined, {})).toBe(0);
|
|
||||||
const ctx = context(fakeClient({ newAlbums: true }));
|
|
||||||
stdout.text = "";
|
|
||||||
|
|
||||||
expect(await filesCommand(ctx, "4", {})).toBe(0);
|
|
||||||
expect(stdout.text).toBe("300\timage\tshared.jpg\n");
|
|
||||||
stdout.text = "";
|
|
||||||
|
|
||||||
expect(await filesCommand(ctx, undefined, { json: true })).toBe(0);
|
|
||||||
expect(
|
|
||||||
JSON.parse(stdout.text).map(
|
|
||||||
(row: { id: number; collectionID: number }) => [
|
|
||||||
row.id,
|
|
||||||
row.collectionID,
|
|
||||||
],
|
|
||||||
),
|
|
||||||
).toEqual([
|
|
||||||
[100, 1],
|
|
||||||
[101, 1],
|
|
||||||
[200, 2],
|
|
||||||
[300, 3],
|
|
||||||
]);
|
|
||||||
});
|
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("get and get-thumb", () => {
|
describe("get and get-thumb", () => {
|
||||||
@@ -642,27 +573,27 @@ describe("get and get-thumb", () => {
|
|||||||
const out = join(root, "diagram.png");
|
const out = join(root, "diagram.png");
|
||||||
expect(await getCommand(context(), "200", { out })).toBe(0);
|
expect(await getCommand(context(), "200", { out })).toBe(0);
|
||||||
expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
|
expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
|
||||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + `7 bytes -> ${out}\n`);
|
expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("get-thumb finds a file in any album without --collection", async () => {
|
it("get-thumb finds a file in any album without --collection", async () => {
|
||||||
const out = join(root, "thumb.jpg");
|
const out = join(root, "thumb.jpg");
|
||||||
expect(await getThumbCommand(context(), "200", { out })).toBe(0);
|
expect(await getThumbCommand(context(), "200", { out })).toBe(0);
|
||||||
expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
|
expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
|
||||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + `3 bytes -> ${out}\n`);
|
expect(stderr.text).toBe(`3 bytes -> ${out}\n`);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("get exits 1 when no album has the file", async () => {
|
it("get exits 1 when no album has the file", async () => {
|
||||||
const out = join(root, "x");
|
const out = join(root, "x");
|
||||||
expect(await getCommand(context(), "999", { out })).toBe(1);
|
expect(await getCommand(context(), "999", { out })).toBe(1);
|
||||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + "File 999 not found\n");
|
expect(stderr.text).toBe("File 999 not found\n");
|
||||||
expect(existsSync(out)).toBe(false);
|
expect(existsSync(out)).toBe(false);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("get-thumb exits 1 when no album has the file", async () => {
|
it("get-thumb exits 1 when no album has the file", async () => {
|
||||||
const out = join(root, "x");
|
const out = join(root, "x");
|
||||||
expect(await getThumbCommand(context(), "999", { out })).toBe(1);
|
expect(await getThumbCommand(context(), "999", { out })).toBe(1);
|
||||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + "File 999 not found\n");
|
expect(stderr.text).toBe("File 999 not found\n");
|
||||||
expect(existsSync(out)).toBe(false);
|
expect(existsSync(out)).toBe(false);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
@@ -710,7 +641,7 @@ describe("names from an album name or a title over 255 bytes", () => {
|
|||||||
expect(Buffer.byteLength(name)).toBe(255);
|
expect(Buffer.byteLength(name)).toBe(255);
|
||||||
expect(code).toBe(0);
|
expect(code).toBe(0);
|
||||||
expect(readdirSync(dir)).toEqual([name]);
|
expect(readdirSync(dir)).toEqual([name]);
|
||||||
expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `7 bytes -> ${name}\n`);
|
expect(stderr.text).toBe(`7 bytes -> ${name}\n`);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("get-thumb cuts the title short so that thumb_ and it fit in 255 bytes, and exits 0", async () => {
|
it("get-thumb cuts the title short so that thumb_ and it fit in 255 bytes, and exits 0", async () => {
|
||||||
@@ -722,7 +653,7 @@ describe("names from an album name or a title over 255 bytes", () => {
|
|||||||
expect(Buffer.byteLength(name)).toBe(255);
|
expect(Buffer.byteLength(name)).toBe(255);
|
||||||
expect(code).toBe(0);
|
expect(code).toBe(0);
|
||||||
expect(readdirSync(dir)).toEqual([name]);
|
expect(readdirSync(dir)).toEqual([name]);
|
||||||
expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `3 bytes -> ${name}\n`);
|
expect(stderr.text).toBe(`3 bytes -> ${name}\n`);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("backup-metadata cuts the album name short so that its folder's name fits in 255 bytes, and exits 0", async () => {
|
it("backup-metadata cuts the album name short so that its folder's name fits in 255 bytes, and exits 0", async () => {
|
||||||
@@ -811,8 +742,7 @@ describe("a live photo", () => {
|
|||||||
Buffer.from(VIDEO),
|
Buffer.from(VIDEO),
|
||||||
);
|
);
|
||||||
expect(stderr.text).toBe(
|
expect(stderr.text).toBe(
|
||||||
FETCHED_ONE_ALBUM +
|
`${IMAGE.length} bytes -> IMG_0300.heic\n` +
|
||||||
`${IMAGE.length} bytes -> IMG_0300.heic\n` +
|
|
||||||
`${VIDEO.length} bytes -> IMG_0300.mov\n`,
|
`${VIDEO.length} bytes -> IMG_0300.mov\n`,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
@@ -866,8 +796,7 @@ describe("a live photo", () => {
|
|||||||
expect(readFileSync(out)).toEqual(Buffer.from(IMAGE));
|
expect(readFileSync(out)).toEqual(Buffer.from(IMAGE));
|
||||||
expect(readFileSync(video)).toEqual(Buffer.from(VIDEO));
|
expect(readFileSync(video)).toEqual(Buffer.from(VIDEO));
|
||||||
expect(stderr.text).toBe(
|
expect(stderr.text).toBe(
|
||||||
FETCHED_ONE_ALBUM +
|
`${IMAGE.length} bytes -> ${out}\n${VIDEO.length} bytes -> ${video}\n`,
|
||||||
`${IMAGE.length} bytes -> ${out}\n${VIDEO.length} bytes -> ${video}\n`,
|
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -881,8 +810,7 @@ describe("a live photo", () => {
|
|||||||
expect(existsSync(out)).toBe(false);
|
expect(existsSync(out)).toBe(false);
|
||||||
expect(existsSync(join(root, "photo.mov"))).toBe(false);
|
expect(existsSync(join(root, "photo.mov"))).toBe(false);
|
||||||
expect(stderr.text).toBe(
|
expect(stderr.text).toBe(
|
||||||
FETCHED_ONE_ALBUM +
|
`File 300 is a live photo, and its video would also be written to ${out}\n`,
|
||||||
`File 300 is a live photo, and its video would also be written to ${out}\n`,
|
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -1425,6 +1353,23 @@ describe("helper fix-missing-thumbnails", () => {
|
|||||||
).toBe(1);
|
).toBe(1);
|
||||||
expect(stderr.text).toBe(last);
|
expect(stderr.text).toBe(last);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// beach.jpg (100) is in Vacation (1) and Work (2), both updated at the same
|
||||||
|
// time. The fixer names a file after the first album holding it in the
|
||||||
|
// order of `Library.albums.list()`. On a tie, the album with the higher ID
|
||||||
|
// comes first, so beach.jpg is named after Work.
|
||||||
|
it("names a photo in two albums after the same album on an empty cache as on a cache that holds the account", async () => {
|
||||||
|
const ctx = context(fakeClient({ beachInWork: true }));
|
||||||
|
const opts = { file: ["100"], json: true };
|
||||||
|
const inWork = [{ fileID: 100, collection: "Work", status: "skipped" }];
|
||||||
|
expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
|
||||||
|
expect(JSON.parse(stdout.text)).toMatchObject(inWork);
|
||||||
|
|
||||||
|
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||||
|
stdout.text = "";
|
||||||
|
expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
|
||||||
|
expect(JSON.parse(stdout.text)).toMatchObject(inWork);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("backup-metadata --exif", () => {
|
describe("backup-metadata --exif", () => {
|
||||||
@@ -1454,8 +1399,8 @@ describe("backup-metadata --exif", () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
// Each test first runs `files` so the cache holds the account as it was, then
|
// Each test first runs `files` so the cache holds the account as it was, then
|
||||||
// changes the server under it. (`collections` would not do: it reads the
|
// changes the server under it. (`collections` and `files` with a collection ID
|
||||||
// collection list alone and stores nothing.)
|
// would not do: they store nothing.)
|
||||||
describe("backup-metadata and the thumbnail helpers refresh first", () => {
|
describe("backup-metadata and the thumbnail helpers refresh first", () => {
|
||||||
beforeEach(async () => {
|
beforeEach(async () => {
|
||||||
expect(await filesCommand(context(), undefined, {})).toBe(0);
|
expect(await filesCommand(context(), undefined, {})).toBe(0);
|
||||||
@@ -1641,22 +1586,83 @@ describe("an ID given on the command line", () => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
// An account with many albums (issue #210). On an empty cache `collections`
|
// An account with many albums (issues #210, #217 and #222). On an empty cache
|
||||||
// used to wait for a refresh that fetched and decrypted the files of every
|
// `collections`, `files` with a collection ID and `get` used to wait for a
|
||||||
// album, one album after another. It printed nothing until then, so on such an
|
// refresh that fetched and decrypted the files of every album, one album after
|
||||||
// account it seemed to hang. `files` with an album's ID did the same before it
|
// another. So did `helper fix-missing-thumbnails --file` (issue #223). On such
|
||||||
// listed that album (issue #217). The account here has 40 albums of one photo
|
// an account they seemed to hang. The account here has 40 albums of one photo
|
||||||
// each, encrypted as Ente's clients encrypt them. The fake Ente server of
|
// each, encrypted as Ente's clients encrypt them, on the fake Ente server of
|
||||||
// `test/ente-server.ts` serves them. A wrapper around it records the path and
|
// `test/ente-server.ts`. The tests record the path and query of each request
|
||||||
// query of each request quak sends. The session is the one logging in to that
|
// quak sends. The session is the one logging in to that server saves.
|
||||||
// server saves.
|
|
||||||
describe("an account with many albums", () => {
|
describe("an account with many albums", () => {
|
||||||
const ALBUMS = 40;
|
const ALBUMS = 40;
|
||||||
const PASSWORD = "many-albums";
|
const PASSWORD = "many-albums";
|
||||||
const paths: string[] = [];
|
const requests: string[] = [];
|
||||||
let server: typeof globalThis.fetch;
|
let server: typeof globalThis.fetch;
|
||||||
let snapshot: ClientSnapshot;
|
let snapshot: ClientSnapshot;
|
||||||
|
|
||||||
|
// What `files` without a collection ID prints on an empty cache: every
|
||||||
|
// album's photo on stdout, and on stderr each album whose files its
|
||||||
|
// refresh fetched.
|
||||||
|
let everyPhoto = "";
|
||||||
|
let progress = "";
|
||||||
|
for (let n = 1; n <= ALBUMS; n++) {
|
||||||
|
everyPhoto += `${n * 100}\timage\tphoto-${n}.jpg\n`;
|
||||||
|
progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The key of each album, by album ID.
|
||||||
|
const albumKeys = new Map<number, Uint8Array>();
|
||||||
|
|
||||||
|
// A photo in album `collectionID` as the server sends it, and its original
|
||||||
|
// encrypted as the server stores it. The original's bytes are the title.
|
||||||
|
// These tests fetch no thumbnail. The server records a thumbnail size, so
|
||||||
|
// `helper fix-missing-thumbnails` downloads the original before it skips
|
||||||
|
// the photo as not a JPEG.
|
||||||
|
const photoRow = (photo: {
|
||||||
|
id: number;
|
||||||
|
collectionID: number;
|
||||||
|
title: string;
|
||||||
|
updationTime: number;
|
||||||
|
}): { row: Record<string, unknown>; original: Uint8Array } => {
|
||||||
|
const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||||
|
const photoKeyBox = encryptSecretbox(
|
||||||
|
photoKey,
|
||||||
|
albumKeys.get(photo.collectionID)!,
|
||||||
|
);
|
||||||
|
const metadata = encryptStreamBlob(
|
||||||
|
new TextEncoder().encode(
|
||||||
|
JSON.stringify({
|
||||||
|
title: photo.title,
|
||||||
|
fileType: 0,
|
||||||
|
creationTime: 1000,
|
||||||
|
modificationTime: 1000,
|
||||||
|
}),
|
||||||
|
),
|
||||||
|
photoKey,
|
||||||
|
);
|
||||||
|
const original = encryptStreamBlob(
|
||||||
|
new TextEncoder().encode(photo.title),
|
||||||
|
photoKey,
|
||||||
|
);
|
||||||
|
const row = {
|
||||||
|
id: photo.id,
|
||||||
|
collectionID: photo.collectionID,
|
||||||
|
ownerID: USER_ID,
|
||||||
|
encryptedKey: toBase64(photoKeyBox.ciphertext),
|
||||||
|
keyDecryptionNonce: toBase64(photoKeyBox.nonce),
|
||||||
|
metadata: {
|
||||||
|
encryptedData: toBase64(metadata.ciphertext),
|
||||||
|
decryptionHeader: toBase64(metadata.header),
|
||||||
|
},
|
||||||
|
file: { decryptionHeader: toBase64(original.header) },
|
||||||
|
thumbnail: { decryptionHeader: "" },
|
||||||
|
info: { thumbSize: 1000 },
|
||||||
|
updationTime: photo.updationTime,
|
||||||
|
};
|
||||||
|
return { row, original: original.ciphertext };
|
||||||
|
};
|
||||||
|
|
||||||
// Album n is named `Album n` and holds one photo, n * 100, titled
|
// Album n is named `Album n` and holds one photo, n * 100, titled
|
||||||
// `photo-n.jpg`.
|
// `photo-n.jpg`.
|
||||||
beforeAll(async () => {
|
beforeAll(async () => {
|
||||||
@@ -1664,8 +1670,10 @@ describe("an account with many albums", () => {
|
|||||||
const account = await buildAccount(PASSWORD);
|
const account = await buildAccount(PASSWORD);
|
||||||
const collections: Record<string, unknown>[] = [];
|
const collections: Record<string, unknown>[] = [];
|
||||||
const filesByCollection: Record<number, Record<string, unknown>[]> = {};
|
const filesByCollection: Record<number, Record<string, unknown>[]> = {};
|
||||||
|
const fileCiphertexts: Record<number, Uint8Array> = {};
|
||||||
for (let id = 1; id <= ALBUMS; id++) {
|
for (let id = 1; id <= ALBUMS; id++) {
|
||||||
const albumKey = sodium.crypto_secretbox_keygen();
|
const albumKey = sodium.crypto_secretbox_keygen();
|
||||||
|
albumKeys.set(id, albumKey);
|
||||||
const albumKeyBox = encryptSecretbox(albumKey, account.masterKey);
|
const albumKeyBox = encryptSecretbox(albumKey, account.masterKey);
|
||||||
const name = encryptSecretbox(
|
const name = encryptSecretbox(
|
||||||
new TextEncoder().encode(`Album ${id}`),
|
new TextEncoder().encode(`Album ${id}`),
|
||||||
@@ -1681,37 +1689,14 @@ describe("an account with many albums", () => {
|
|||||||
type: "album",
|
type: "album",
|
||||||
updationTime: 1,
|
updationTime: 1,
|
||||||
});
|
});
|
||||||
const photoKey =
|
const { row, original } = photoRow({
|
||||||
sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
id: id * 100,
|
||||||
const photoKeyBox = encryptSecretbox(photoKey, albumKey);
|
collectionID: id,
|
||||||
const metadata = encryptStreamBlob(
|
title: `photo-${id}.jpg`,
|
||||||
new TextEncoder().encode(
|
updationTime: 1,
|
||||||
JSON.stringify({
|
});
|
||||||
title: `photo-${id}.jpg`,
|
filesByCollection[id] = [row];
|
||||||
fileType: 0,
|
fileCiphertexts[id * 100] = original;
|
||||||
creationTime: 1000,
|
|
||||||
modificationTime: 1000,
|
|
||||||
}),
|
|
||||||
),
|
|
||||||
photoKey,
|
|
||||||
);
|
|
||||||
filesByCollection[id] = [
|
|
||||||
{
|
|
||||||
id: id * 100,
|
|
||||||
collectionID: id,
|
|
||||||
ownerID: USER_ID,
|
|
||||||
encryptedKey: toBase64(photoKeyBox.ciphertext),
|
|
||||||
keyDecryptionNonce: toBase64(photoKeyBox.nonce),
|
|
||||||
metadata: {
|
|
||||||
encryptedData: toBase64(metadata.ciphertext),
|
|
||||||
decryptionHeader: toBase64(metadata.header),
|
|
||||||
},
|
|
||||||
// These tests fetch no original and no thumbnail.
|
|
||||||
file: { decryptionHeader: "" },
|
|
||||||
thumbnail: { decryptionHeader: "" },
|
|
||||||
updationTime: 1,
|
|
||||||
},
|
|
||||||
];
|
|
||||||
}
|
}
|
||||||
const ente = buildMetaFetch({
|
const ente = buildMetaFetch({
|
||||||
verifier: account.verifier,
|
verifier: account.verifier,
|
||||||
@@ -1721,7 +1706,7 @@ describe("an account with many albums", () => {
|
|||||||
collections,
|
collections,
|
||||||
filesByCollection,
|
filesByCollection,
|
||||||
encryptedMLData: {},
|
encryptedMLData: {},
|
||||||
fileCiphertexts: {},
|
fileCiphertexts,
|
||||||
});
|
});
|
||||||
server = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
server = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||||
const url =
|
const url =
|
||||||
@@ -1731,7 +1716,7 @@ describe("an account with many albums", () => {
|
|||||||
? input.href
|
? input.href
|
||||||
: input.url;
|
: input.url;
|
||||||
const { pathname, search } = new URL(url);
|
const { pathname, search } = new URL(url);
|
||||||
paths.push(pathname + search);
|
requests.push(pathname + search);
|
||||||
return ente(input, init);
|
return ente(input, init);
|
||||||
}) as typeof globalThis.fetch;
|
}) as typeof globalThis.fetch;
|
||||||
const client = await Client.login({
|
const client = await Client.login({
|
||||||
@@ -1743,16 +1728,17 @@ describe("an account with many albums", () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
beforeEach(() => {
|
beforeEach(() => {
|
||||||
paths.length = 0;
|
requests.length = 0;
|
||||||
});
|
});
|
||||||
|
|
||||||
// A context whose saved session sends its requests to the fake server.
|
// A context whose saved session sends its requests to `fetch`, by default
|
||||||
const accountContext = (): CliContext => {
|
// the fake server.
|
||||||
|
const accountContext = (fetch = server): CliContext => {
|
||||||
const ctx = context();
|
const ctx = context();
|
||||||
saveSession(ctx.sessionDir, snapshot);
|
saveSession(ctx.sessionDir, snapshot);
|
||||||
return {
|
return {
|
||||||
...ctx,
|
...ctx,
|
||||||
loadSession: (path) => loadSession(path, { fetch: server }),
|
loadSession: (path) => loadSession(path, { fetch }),
|
||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -1765,72 +1751,219 @@ describe("an account with many albums", () => {
|
|||||||
albums += `${id}\talbum\tAlbum ${id}\n`;
|
albums += `${id}\talbum\tAlbum ${id}\n`;
|
||||||
expect(stdout.text).toBe(albums);
|
expect(stdout.text).toBe(albums);
|
||||||
expect(stderr.text).toBe("");
|
expect(stderr.text).toBe("");
|
||||||
expect(paths).toEqual(["/collections/v2?sinceTime=0"]);
|
expect(requests).toEqual(["/collections/v2?sinceTime=0"]);
|
||||||
|
|
||||||
// It stored nothing that would make `refreshCollection()` skip an
|
// It stored nothing that would make a refresh skip an album's files:
|
||||||
// album's files: `files` after it still finds the album's photo.
|
// `files` after it still lists every album's photo.
|
||||||
stdout.text = "";
|
stdout.text = "";
|
||||||
expect(await filesCommand(ctx, "7", {})).toBe(0);
|
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||||
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
expect(stdout.text).toBe(everyPhoto);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("files fetches the files of the album it lists and of no other", async () => {
|
it("files with a collection ID asks for the album list and that album's files alone", async () => {
|
||||||
expect(await filesCommand(accountContext(), "7", {})).toBe(0);
|
expect(await filesCommand(accountContext(), "7", {})).toBe(0);
|
||||||
|
|
||||||
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
||||||
expect(stderr.text).toBe("");
|
expect(stderr.text).toBe("");
|
||||||
expect(paths).toEqual([
|
expect(requests).toEqual([
|
||||||
"/collections/v2?sinceTime=0",
|
"/collections/v2?sinceTime=0",
|
||||||
"/collections/v2/diff?collectionID=7&sinceTime=0",
|
"/collections/v2/diff?collectionID=7&sinceTime=0",
|
||||||
]);
|
]);
|
||||||
});
|
});
|
||||||
|
|
||||||
// The first listing stored the album, so the second finds its stored copy
|
// Here the server sends album 7's files in two pages. Photo 700 changed
|
||||||
// current and asks for no album's files.
|
// between them, so both pages hold a copy of it.
|
||||||
it("files run again for the same album sends no request for its files", async () => {
|
it("files with a collection ID lists only the later copy of a photo sent in two pages", async () => {
|
||||||
const ctx = accountContext();
|
const firstPage = [
|
||||||
expect(await filesCommand(ctx, "7", {})).toBe(0);
|
photoRow({
|
||||||
stdout.text = "";
|
id: 700,
|
||||||
paths.length = 0;
|
collectionID: 7,
|
||||||
|
title: "photo-7.jpg",
|
||||||
|
updationTime: 1,
|
||||||
|
}).row,
|
||||||
|
photoRow({
|
||||||
|
id: 701,
|
||||||
|
collectionID: 7,
|
||||||
|
title: "photo-701.jpg",
|
||||||
|
updationTime: 1,
|
||||||
|
}).row,
|
||||||
|
];
|
||||||
|
const secondPage = [
|
||||||
|
photoRow({
|
||||||
|
id: 700,
|
||||||
|
collectionID: 7,
|
||||||
|
title: "photo-7-edited.jpg",
|
||||||
|
updationTime: 2,
|
||||||
|
}).row,
|
||||||
|
];
|
||||||
|
const paged = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||||
|
const url = new URL(input instanceof Request ? input.url : input);
|
||||||
|
if (
|
||||||
|
url.pathname !== "/collections/v2/diff" ||
|
||||||
|
url.searchParams.get("collectionID") !== "7"
|
||||||
|
) {
|
||||||
|
return server(input, init);
|
||||||
|
}
|
||||||
|
const first = url.searchParams.get("sinceTime") === "0";
|
||||||
|
return new Response(
|
||||||
|
JSON.stringify({
|
||||||
|
diff: first ? firstPage : secondPage,
|
||||||
|
hasMore: first,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}) as typeof globalThis.fetch;
|
||||||
|
|
||||||
expect(await filesCommand(ctx, "7", {})).toBe(0);
|
expect(await filesCommand(accountContext(paged), "7", {})).toBe(0);
|
||||||
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
|
||||||
expect(paths).toEqual(["/collections/v2?sinceTime=0"]);
|
// Photo 700 is listed once, with the later title, before photo 701.
|
||||||
|
expect(stdout.text).toBe(
|
||||||
|
"700\timage\tphoto-7-edited.jpg\n701\timage\tphoto-701.jpg\n",
|
||||||
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("files with no album's ID counts on stderr each album whose files its refresh fetched", async () => {
|
// The cache stays empty, so a later `files` without a collection ID still
|
||||||
|
// fetches the files of all 40 albums.
|
||||||
|
it("files with a collection ID stores nothing in the cache", async () => {
|
||||||
|
const ctx = accountContext();
|
||||||
|
expect(await filesCommand(ctx, "7", {})).toBe(0);
|
||||||
|
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
|
||||||
|
|
||||||
|
stdout.text = "";
|
||||||
|
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||||
|
expect(stdout.text).toBe(everyPhoto);
|
||||||
|
expect(stderr.text).toBe(progress);
|
||||||
|
});
|
||||||
|
|
||||||
|
// Photo 200 is in album 2, so `get` asks for no file of album 3 or later.
|
||||||
|
// The last request downloads the photo's original.
|
||||||
|
it("get on an empty cache asks for the albums' files only until one holds the photo", async () => {
|
||||||
|
const out = join(root, "photo.jpg");
|
||||||
|
expect(await getCommand(accountContext(), "200", { out })).toBe(0);
|
||||||
|
|
||||||
|
expect(readFileSync(out, "utf-8")).toBe("photo-2.jpg");
|
||||||
|
expect(stderr.text).toBe(`11 bytes -> ${out}\n`);
|
||||||
|
expect(requests).toEqual([
|
||||||
|
"/collections/v2?sinceTime=0",
|
||||||
|
"/collections/v2/diff?collectionID=1&sinceTime=0",
|
||||||
|
"/collections/v2/diff?collectionID=2&sinceTime=0",
|
||||||
|
"/?fileID=200",
|
||||||
|
]);
|
||||||
|
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
// `files` first fills the cache, so `get` waits for a refresh. No album has
|
||||||
|
// changed since, so the refresh asks for no album's files.
|
||||||
|
it("get on a cache that holds the account asks for no unchanged album's files, also for a photo in no album", async () => {
|
||||||
|
const ctx = accountContext();
|
||||||
|
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||||
|
requests.length = 0;
|
||||||
|
stderr.text = "";
|
||||||
|
|
||||||
|
const out = join(root, "photo.jpg");
|
||||||
|
expect(await getCommand(ctx, "4000", { out })).toBe(0);
|
||||||
|
expect(readFileSync(out, "utf-8")).toBe("photo-40.jpg");
|
||||||
|
expect(stderr.text).toBe(`12 bytes -> ${out}\n`);
|
||||||
|
expect(requests).toEqual([
|
||||||
|
"/collections/v2?sinceTime=1",
|
||||||
|
"/?fileID=4000",
|
||||||
|
]);
|
||||||
|
|
||||||
|
requests.length = 0;
|
||||||
|
stderr.text = "";
|
||||||
|
expect(await getCommand(ctx, "999", { out })).toBe(1);
|
||||||
|
expect(stderr.text).toBe("File 999 not found\n");
|
||||||
|
expect(requests).toEqual(["/collections/v2?sinceTime=1"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
// The helper searches the most recently updated album first. All 40 were
|
||||||
|
// updated at the same time, and on a tie the higher ID comes first. Photo
|
||||||
|
// 3900 is in album 39, so the helper asks for no file of album 38 or
|
||||||
|
// earlier. The last request downloads the photo's original.
|
||||||
|
it("fix-missing-thumbnails --file on an empty cache asks for the albums' files only until one holds the photo", async () => {
|
||||||
|
const opts = { file: ["3900"], json: true };
|
||||||
|
expect(await fixMissingThumbnailsCommand(accountContext(), opts)).toBe(
|
||||||
|
0,
|
||||||
|
);
|
||||||
|
|
||||||
|
const reason =
|
||||||
|
"unsupported image format (only baseline JPEG can be regenerated)";
|
||||||
|
expect(JSON.parse(stdout.text)).toEqual([
|
||||||
|
{
|
||||||
|
fileID: 3900,
|
||||||
|
title: "photo-39.jpg",
|
||||||
|
collection: "Album 39",
|
||||||
|
status: "skipped",
|
||||||
|
reason,
|
||||||
|
},
|
||||||
|
]);
|
||||||
|
expect(stderr.text).toBe(
|
||||||
|
"[Album 39] Downloading photo-39.jpg for thumbnail generation...\n" +
|
||||||
|
`[Album 39] Skipping photo-39.jpg: ${reason}\n`,
|
||||||
|
);
|
||||||
|
expect(requests).toEqual([
|
||||||
|
"/collections/v2?sinceTime=0",
|
||||||
|
"/collections/v2/diff?collectionID=40&sinceTime=0",
|
||||||
|
"/collections/v2/diff?collectionID=39&sinceTime=0",
|
||||||
|
"/?fileID=3900",
|
||||||
|
]);
|
||||||
|
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
// `files` first fills the cache, so the helper waits for a refresh. No
|
||||||
|
// album has changed since, so the refresh asks for no album's files.
|
||||||
|
it("fix-missing-thumbnails --file on a cache that holds the account asks for no unchanged album's files, also for a photo in no album", async () => {
|
||||||
|
const ctx = accountContext();
|
||||||
|
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||||
|
requests.length = 0;
|
||||||
|
stdout.text = "";
|
||||||
|
stderr.text = "";
|
||||||
|
|
||||||
|
const opts = { file: ["3900"], json: true };
|
||||||
|
expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
|
||||||
|
const reason =
|
||||||
|
"unsupported image format (only baseline JPEG can be regenerated)";
|
||||||
|
expect(JSON.parse(stdout.text)).toEqual([
|
||||||
|
{
|
||||||
|
fileID: 3900,
|
||||||
|
title: "photo-39.jpg",
|
||||||
|
collection: "Album 39",
|
||||||
|
status: "skipped",
|
||||||
|
reason,
|
||||||
|
},
|
||||||
|
]);
|
||||||
|
expect(stderr.text).toBe(
|
||||||
|
"[Album 39] Downloading photo-39.jpg for thumbnail generation...\n" +
|
||||||
|
`[Album 39] Skipping photo-39.jpg: ${reason}\n`,
|
||||||
|
);
|
||||||
|
expect(requests).toEqual([
|
||||||
|
"/collections/v2?sinceTime=1",
|
||||||
|
"/?fileID=3900",
|
||||||
|
]);
|
||||||
|
|
||||||
|
requests.length = 0;
|
||||||
|
stdout.text = "";
|
||||||
|
const notFound = { file: ["999"], json: true };
|
||||||
|
expect(await fixMissingThumbnailsCommand(ctx, notFound)).toBe(1);
|
||||||
|
expect(JSON.parse(stdout.text)).toMatchObject([
|
||||||
|
{ fileID: 999, reason: "file not found in any collection" },
|
||||||
|
]);
|
||||||
|
expect(requests).toEqual(["/collections/v2?sinceTime=1"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("files counts on stderr each album whose files its refresh fetched", async () => {
|
||||||
expect(await filesCommand(accountContext(), undefined, {})).toBe(0);
|
expect(await filesCommand(accountContext(), undefined, {})).toBe(0);
|
||||||
|
|
||||||
let photos = "";
|
expect(stdout.text).toBe(everyPhoto);
|
||||||
let progress = "";
|
|
||||||
for (let n = 1; n <= ALBUMS; n++) {
|
|
||||||
photos += `${n * 100}\timage\tphoto-${n}.jpg\n`;
|
|
||||||
progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`;
|
|
||||||
}
|
|
||||||
expect(stdout.text).toBe(photos);
|
|
||||||
expect(stderr.text).toBe(progress);
|
expect(stderr.text).toBe(progress);
|
||||||
});
|
});
|
||||||
|
|
||||||
// `--json` shapes stdout only, so the refresh's progress is still counted
|
// `--json` shapes stdout only, so the refresh's progress is still counted
|
||||||
// on stderr.
|
// on stderr.
|
||||||
it("files --json with no album's ID counts on stderr each album whose files its refresh fetched", async () => {
|
it("files --json counts on stderr each album whose files its refresh fetched", async () => {
|
||||||
expect(
|
const opts = { json: true };
|
||||||
await filesCommand(accountContext(), undefined, { json: true }),
|
expect(await filesCommand(accountContext(), undefined, opts)).toBe(0);
|
||||||
).toBe(0);
|
|
||||||
|
|
||||||
const rows = JSON.parse(stdout.text);
|
expect(JSON.parse(stdout.text)).toHaveLength(ALBUMS);
|
||||||
expect(rows).toHaveLength(ALBUMS);
|
|
||||||
expect(rows[6]).toEqual({
|
|
||||||
id: 700,
|
|
||||||
title: "photo-7.jpg",
|
|
||||||
fileType: "image",
|
|
||||||
creationTime: 1000,
|
|
||||||
collectionID: 7,
|
|
||||||
});
|
|
||||||
let progress = "";
|
|
||||||
for (let n = 1; n <= ALBUMS; n++) {
|
|
||||||
progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`;
|
|
||||||
}
|
|
||||||
expect(stderr.text).toBe(progress);
|
expect(stderr.text).toBe(progress);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
@@ -234,13 +234,15 @@ describe("-q", () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// The line saying why the command failed is the fatal one, and it is
|
// No album holds file 999, so fixing it fails. The summary from
|
||||||
// printed.
|
// `--- Done ---` to the end of the `Failed files:` list is left out. The
|
||||||
|
// line saying why the command failed is the fatal one, and it is printed.
|
||||||
it("prints only the line a failing command exits with", async () => {
|
it("prints only the line a failing command exits with", async () => {
|
||||||
expect(await quak("-q", "files", "9")).toEqual({
|
const fix = ["helper", "fix-missing-thumbnails", "--file", "999"];
|
||||||
|
expect(await quak("-q", ...fix)).toEqual({
|
||||||
code: 1,
|
code: 1,
|
||||||
stdout: "",
|
stdout: "",
|
||||||
stderr: "Collection 9 not found\n",
|
stderr: `quak: 1 file(s) failed; see the "Failed files:" list, or the JSON with --json\n`,
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -326,7 +328,7 @@ describe("what the library reports", () => {
|
|||||||
]);
|
]);
|
||||||
});
|
});
|
||||||
|
|
||||||
// Each operation is named in plain words, as the README names it.
|
// Each operation is named in plain words.
|
||||||
it("logs a refresh's progress at info, and its start and end at debug", () => {
|
it("logs a refresh's progress at info, and its start and end at debug", () => {
|
||||||
const { log, lines } = collect();
|
const { log, lines } = collect();
|
||||||
const onProgress = logRefresh(log);
|
const onProgress = logRefresh(log);
|
||||||
|
|||||||
+139
-87
@@ -3,25 +3,25 @@
|
|||||||
* issue #36, issue #52).
|
* issue #36, issue #52).
|
||||||
*
|
*
|
||||||
* The `collections`, `files`, `get`, and `get-thumb` commands must answer for
|
* The `collections`, `files`, `get`, and `get-thumb` commands must answer for
|
||||||
* current server state, not the local cache. `files` without a collection ID,
|
* current server state, not the local cache. `get`, `get-thumb` and `files`
|
||||||
* `get` and `get-thumb` force a `Library.fresh()` round-trip before they read.
|
* without a collection ID force a `Library.fresh()` round-trip before they
|
||||||
* The stand-in library below serves nothing until `fresh()` has been awaited.
|
* read. The stand-in library below serves no cached record until `fresh()` has
|
||||||
* A helper that read without refreshing would come back empty and fail here.
|
* been awaited, so a helper that read without refreshing would come back empty
|
||||||
* `collections` asks the library for the collection list alone
|
* and fail here. `collections` asks the library for the collection list alone
|
||||||
* (`freshCollections()`, issue #210). It must not wait for a refresh, which
|
* (`freshCollections()`, issue #210). `files` with a collection ID asks for
|
||||||
* would fetch the files of every collection. `files COLLECTION` reads that list
|
* that list, then for that one collection's files (`freshFiles()`, issue
|
||||||
* too, then brings that one collection up to date (`refreshCollection()`,
|
* #217). Neither may wait for a refresh, which would fetch the files of every
|
||||||
* issue #217). The stand-in serves a collection's files once `fresh()` has run,
|
* collection. On an empty cache, `get` and `get-thumb` ask for that list, then
|
||||||
* or once `refreshCollection()` has resolved to `true` for that collection.
|
* for one collection's files at a time, until they find the file (issue #222).
|
||||||
* When `refreshCollection()` resolves to `false`, `files COLLECTION` forces a
|
* `helper fix-missing-thumbnails --file` does the same until it finds every
|
||||||
* refresh instead.
|
* file it names, reading the most recently updated collection first (issue
|
||||||
|
* #223).
|
||||||
*
|
*
|
||||||
* `collections` and `files` also list in the library's enumeration order
|
* `collections` and `files` also list in the order the library gives them, not
|
||||||
* (`listCollections`/`listFiles`) — the order the pre-library CLI printed — not
|
|
||||||
* the albums/photos projection's newest-first order. The fixtures are seeded in
|
* the albums/photos projection's newest-first order. The fixtures are seeded in
|
||||||
* an enumeration order that a newest-first sort would rearrange, so a
|
* an order that a newest-first sort would rearrange, so a regression to the
|
||||||
* regression to the projection order would fail here too. Field values still
|
* projection order would fail here too. Field values still come from the raw
|
||||||
* come from the raw metadata via `cli-output.ts`.
|
* metadata via `cli-output.ts`.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { describe, it, expect } from "vitest";
|
import { describe, it, expect } from "vitest";
|
||||||
@@ -30,10 +30,10 @@ import {
|
|||||||
freshCollections,
|
freshCollections,
|
||||||
freshFiles,
|
freshFiles,
|
||||||
freshFile,
|
freshFile,
|
||||||
|
freshFilesByID,
|
||||||
type FreshReadLibrary,
|
type FreshReadLibrary,
|
||||||
} from "../../src/cli-read.js";
|
} from "../../src/cli-read.js";
|
||||||
import { fileListRow } from "../../src/cli-output.js";
|
import { fileListRow } from "../../src/cli-output.js";
|
||||||
import type { Photo } from "../../src/library/index.js";
|
|
||||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||||
|
|
||||||
const collection = (id: number, updationTime: number): Collection => ({
|
const collection = (id: number, updationTime: number): Collection => ({
|
||||||
@@ -67,20 +67,15 @@ const file = (
|
|||||||
updationTime: creationTime,
|
updationTime: creationTime,
|
||||||
});
|
});
|
||||||
|
|
||||||
// A library that reveals its records only after `fresh()` has been awaited, and
|
// A library that reveals its cached records only after `fresh()` has been
|
||||||
// serves them in the enumeration order it was given. `freshCollections()`
|
// awaited, and serves them in the enumeration order it was given.
|
||||||
// answers the collection list in that order without a refresh. When
|
// `freshCollections()` and `freshFiles()` answer in that order without a
|
||||||
// `refreshCollection()` resolves to true, it reveals the files of that one
|
// refresh.
|
||||||
// collection.
|
|
||||||
// `photos.byID` returns a stand-in `Photo` carrying just the fileID the helper
|
|
||||||
// passes through.
|
|
||||||
class FakeLibrary implements FreshReadLibrary {
|
class FakeLibrary implements FreshReadLibrary {
|
||||||
freshCalls = 0;
|
freshCalls = 0;
|
||||||
freshCollectionsCalls = 0;
|
freshCollectionsCalls = 0;
|
||||||
refreshCollectionIDs: number[] = [];
|
// The ID of each collection `freshFiles()` was asked for.
|
||||||
// What `refreshCollection()` resolves to. With false it reveals nothing,
|
freshFilesCalls: number[] = [];
|
||||||
// as a library that has had a refresh does for a collection it lacks.
|
|
||||||
refreshCollectionHolds = true;
|
|
||||||
private refreshed = false;
|
private refreshed = false;
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
@@ -99,14 +94,9 @@ class FakeLibrary implements FreshReadLibrary {
|
|||||||
return this.collections;
|
return this.collections;
|
||||||
}
|
}
|
||||||
|
|
||||||
async refreshCollection(collection: Collection): Promise<boolean> {
|
async freshFiles(collection: Collection): Promise<EnteFile[]> {
|
||||||
this.refreshCollectionIDs.push(collection.id);
|
this.freshFilesCalls.push(collection.id);
|
||||||
return this.refreshCollectionHolds;
|
return this.files.filter((f) => f.collectionID === collection.id);
|
||||||
}
|
|
||||||
|
|
||||||
getCollection(id: number): Collection | undefined {
|
|
||||||
if (!this.refreshed) return undefined;
|
|
||||||
return this.collections.find((c) => c.id === id);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
listCollections(): Collection[] {
|
listCollections(): Collection[] {
|
||||||
@@ -114,11 +104,7 @@ class FakeLibrary implements FreshReadLibrary {
|
|||||||
}
|
}
|
||||||
|
|
||||||
listFiles(collectionID: number): EnteFile[] {
|
listFiles(collectionID: number): EnteFile[] {
|
||||||
const current =
|
return this.refreshed
|
||||||
this.refreshed ||
|
|
||||||
(this.refreshCollectionHolds &&
|
|
||||||
this.refreshCollectionIDs.includes(collectionID));
|
|
||||||
return current
|
|
||||||
? this.files.filter((f) => f.collectionID === collectionID)
|
? this.files.filter((f) => f.collectionID === collectionID)
|
||||||
: [];
|
: [];
|
||||||
}
|
}
|
||||||
@@ -127,14 +113,6 @@ class FakeLibrary implements FreshReadLibrary {
|
|||||||
if (!this.refreshed) return undefined;
|
if (!this.refreshed) return undefined;
|
||||||
return this.files.find((f) => f.id === fileID);
|
return this.files.find((f) => f.id === fileID);
|
||||||
}
|
}
|
||||||
|
|
||||||
photos = {
|
|
||||||
byID: ({ fileID }: { fileID: number }): Photo | undefined => {
|
|
||||||
if (!this.refreshed) return undefined;
|
|
||||||
if (!this.files.some((f) => f.id === fileID)) return undefined;
|
|
||||||
return { fileID } as unknown as Photo;
|
|
||||||
},
|
|
||||||
};
|
|
||||||
}
|
}
|
||||||
|
|
||||||
describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
||||||
@@ -158,10 +136,9 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
|||||||
expect(rows.map((c) => c.id)).not.toEqual(newestFirst);
|
expect(rows.map((c) => c.id)).not.toEqual(newestFirst);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("freshFiles brings the collection alone up to date, lists in enumeration order, keeps raw fields", async () => {
|
it("freshFiles with a collection ID reads that collection's files alone and keeps their order and raw fields", async () => {
|
||||||
// Enumeration order by id 10, 11, 12; creationTimes ascending, so a
|
// Given in the order 10, 11, 12; creationTimes ascending, so a
|
||||||
// newest-first sort would reverse them. Collection 2's file is not
|
// newest-first sort would reverse them.
|
||||||
// listed.
|
|
||||||
const files = [
|
const files = [
|
||||||
file(10, 1, 1_700_000_000_000_000),
|
file(10, 1, 1_700_000_000_000_000),
|
||||||
file(11, 1, 1_700_000_000_000_001),
|
file(11, 1, 1_700_000_000_000_001),
|
||||||
@@ -169,14 +146,14 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
|||||||
file(20, 2, 1_700_000_000_000_003),
|
file(20, 2, 1_700_000_000_000_003),
|
||||||
];
|
];
|
||||||
const lib = new FakeLibrary(
|
const lib = new FakeLibrary(
|
||||||
[collection(1, 100), collection(2, 100)],
|
[collection(1, 100), collection(2, 200)],
|
||||||
files,
|
files,
|
||||||
);
|
);
|
||||||
|
|
||||||
const rows = await freshFiles(lib, 1);
|
const rows = await freshFiles(lib, 1);
|
||||||
expect(lib.freshCollectionsCalls).toBe(1);
|
|
||||||
expect(lib.refreshCollectionIDs).toEqual([1]);
|
|
||||||
expect(lib.freshCalls).toBe(0);
|
expect(lib.freshCalls).toBe(0);
|
||||||
|
expect(lib.freshCollectionsCalls).toBe(1);
|
||||||
|
expect(lib.freshFilesCalls).toEqual([1]);
|
||||||
expect(rows?.map((f) => f.id)).toEqual([10, 11, 12]);
|
expect(rows?.map((f) => f.id)).toEqual([10, 11, 12]);
|
||||||
|
|
||||||
// Field values come from raw metadata: microsecond creationTime and the
|
// Field values come from raw metadata: microsecond creationTime and the
|
||||||
@@ -230,45 +207,120 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
|||||||
]);
|
]);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("freshFiles forces a refresh when refreshCollection resolves to false", async () => {
|
|
||||||
const lib = new FakeLibrary(
|
|
||||||
[collection(1, 100)],
|
|
||||||
[file(10, 1, 1_700_000_000_000_000)],
|
|
||||||
);
|
|
||||||
lib.refreshCollectionHolds = false;
|
|
||||||
|
|
||||||
const rows = await freshFiles(lib, 1);
|
|
||||||
expect(lib.refreshCollectionIDs).toEqual([1]);
|
|
||||||
expect(lib.freshCalls).toBe(1);
|
|
||||||
expect(rows?.map((f) => f.id)).toEqual([10]);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("freshFiles returns undefined for an unknown collection", async () => {
|
it("freshFiles returns undefined for an unknown collection", async () => {
|
||||||
const lib = new FakeLibrary([collection(1, 100)], []);
|
const lib = new FakeLibrary([collection(1, 100)], []);
|
||||||
const rows = await freshFiles(lib, 999);
|
const rows = await freshFiles(lib, 999);
|
||||||
expect(lib.freshCollectionsCalls).toBe(1);
|
|
||||||
expect(lib.refreshCollectionIDs).toEqual([]);
|
|
||||||
expect(lib.freshCalls).toBe(0);
|
expect(lib.freshCalls).toBe(0);
|
||||||
|
expect(lib.freshFilesCalls).toEqual([]);
|
||||||
expect(rows).toBeUndefined();
|
expect(rows).toBeUndefined();
|
||||||
});
|
});
|
||||||
|
|
||||||
it("freshFile refreshes first, then resolves the photo and its raw record", async () => {
|
// Three collections, each holding one file.
|
||||||
const f = file(10, 1, 1_700_000_000_000_000);
|
const threeCollections = (): FakeLibrary =>
|
||||||
const lib = new FakeLibrary([collection(1, 100)], [f]);
|
new FakeLibrary(
|
||||||
|
[collection(1, 100), collection(2, 200), collection(3, 300)],
|
||||||
const resolved = await freshFile(lib, 10);
|
[
|
||||||
expect(lib.freshCalls).toBe(1);
|
file(10, 1, 1_700_000_000_000_000),
|
||||||
expect(resolved?.photo.fileID).toBe(10);
|
file(20, 2, 1_700_000_000_000_001),
|
||||||
expect(resolved?.file.metadata.title).toBe("file-10.jpg");
|
file(30, 3, 1_700_000_000_000_002),
|
||||||
expect(resolved?.file.metadata.creationTime).toBe(
|
],
|
||||||
1_700_000_000_000_000,
|
|
||||||
);
|
);
|
||||||
|
|
||||||
|
it("freshFile on an empty cache reads each collection's files in the server's order until one holds the file", async () => {
|
||||||
|
const lib = threeCollections();
|
||||||
|
|
||||||
|
const found = await freshFile(lib, 20);
|
||||||
|
expect(lib.freshCalls).toBe(0);
|
||||||
|
expect(lib.freshCollectionsCalls).toBe(1);
|
||||||
|
expect(lib.freshFilesCalls).toEqual([1, 2]);
|
||||||
|
expect(found?.metadata.title).toBe("file-20.jpg");
|
||||||
|
expect(found?.metadata.creationTime).toBe(1_700_000_000_000_001);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("freshFile returns undefined for an unknown file", async () => {
|
// The stand-in's cache holds what the server holds once `fresh()` has run.
|
||||||
const lib = new FakeLibrary([collection(1, 100)], []);
|
it("freshFile on a cache that holds the collections refreshes, then looks the file up in the cache", async () => {
|
||||||
const resolved = await freshFile(lib, 404);
|
const lib = threeCollections();
|
||||||
expect(lib.freshCalls).toBe(1);
|
await lib.fresh();
|
||||||
expect(resolved).toBeUndefined();
|
|
||||||
|
const found = await freshFile(lib, 30);
|
||||||
|
expect(lib.freshCalls).toBe(2);
|
||||||
|
expect(lib.freshCollectionsCalls).toBe(0);
|
||||||
|
expect(lib.freshFilesCalls).toEqual([]);
|
||||||
|
expect(found?.collectionID).toBe(3);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("freshFile on an empty cache returns undefined for an unknown file, after reading every collection's files", async () => {
|
||||||
|
const lib = threeCollections();
|
||||||
|
|
||||||
|
const found = await freshFile(lib, 404);
|
||||||
|
expect(lib.freshCalls).toBe(0);
|
||||||
|
expect(lib.freshFilesCalls).toEqual([1, 2, 3]);
|
||||||
|
expect(found).toBeUndefined();
|
||||||
|
});
|
||||||
|
|
||||||
|
// The server lists collections 1, 2 and 3, and collection 3 was updated
|
||||||
|
// most recently. File 11 is in collections 1 and 2.
|
||||||
|
const fileInTwoCollections = (): FakeLibrary =>
|
||||||
|
new FakeLibrary(
|
||||||
|
[collection(1, 100), collection(2, 200), collection(3, 300)],
|
||||||
|
[
|
||||||
|
file(10, 1, 1_700_000_000_000_000),
|
||||||
|
file(11, 1, 1_700_000_000_000_001),
|
||||||
|
file(11, 2, 1_700_000_000_000_001),
|
||||||
|
file(20, 2, 1_700_000_000_000_002),
|
||||||
|
file(30, 3, 1_700_000_000_000_003),
|
||||||
|
],
|
||||||
|
);
|
||||||
|
|
||||||
|
it("freshFilesByID on an empty cache reads the most recently updated collection's files first, until it has found every file", async () => {
|
||||||
|
const lib = fileInTwoCollections();
|
||||||
|
|
||||||
|
const found = await freshFilesByID(lib, [11, 20]);
|
||||||
|
expect(lib.freshCalls).toBe(0);
|
||||||
|
expect(lib.freshCollectionsCalls).toBe(1);
|
||||||
|
expect(lib.freshFilesCalls).toEqual([3, 2]);
|
||||||
|
expect(found.get(11)?.collection.id).toBe(2);
|
||||||
|
expect(found.get(11)?.file.collectionID).toBe(2);
|
||||||
|
expect(found.get(20)?.collection.id).toBe(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
// Collection 1 was updated last, though collections 2 and 3 have higher
|
||||||
|
// IDs. File 11 is in collections 1 and 2.
|
||||||
|
it("freshFilesByID on an empty cache reads the most recently updated collection first, though its ID is the lowest", async () => {
|
||||||
|
const lib = new FakeLibrary(
|
||||||
|
[collection(1, 400), collection(2, 200), collection(3, 300)],
|
||||||
|
[
|
||||||
|
file(11, 1, 1_700_000_000_000_001),
|
||||||
|
file(11, 2, 1_700_000_000_000_001),
|
||||||
|
],
|
||||||
|
);
|
||||||
|
|
||||||
|
const found = await freshFilesByID(lib, [11]);
|
||||||
|
expect(lib.freshFilesCalls).toEqual([1]);
|
||||||
|
expect(found.get(11)?.collection.id).toBe(1);
|
||||||
|
expect(found.get(11)?.file.collectionID).toBe(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
// The stand-in's cache holds what the server holds once `fresh()` has run.
|
||||||
|
it("freshFilesByID on a cache that holds the collections refreshes, then finds each file in the most recently updated collection that holds it", async () => {
|
||||||
|
const lib = fileInTwoCollections();
|
||||||
|
await lib.fresh();
|
||||||
|
|
||||||
|
const found = await freshFilesByID(lib, [11, 404]);
|
||||||
|
expect(lib.freshCalls).toBe(2);
|
||||||
|
expect(lib.freshCollectionsCalls).toBe(0);
|
||||||
|
expect(lib.freshFilesCalls).toEqual([]);
|
||||||
|
expect(found.get(11)?.collection.id).toBe(2);
|
||||||
|
expect(found.get(11)?.file.collectionID).toBe(2);
|
||||||
|
expect(found.has(404)).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("freshFilesByID on an empty cache reads every collection's files when a file is in none of them", async () => {
|
||||||
|
const lib = fileInTwoCollections();
|
||||||
|
|
||||||
|
const found = await freshFilesByID(lib, [10, 404]);
|
||||||
|
expect(lib.freshFilesCalls).toEqual([3, 2, 1]);
|
||||||
|
expect(found.get(10)?.collection.id).toBe(1);
|
||||||
|
expect(found.has(404)).toBe(false);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
+5
-2
@@ -19,8 +19,11 @@ import { SRP, SrpServer } from "fast-srp-hap";
|
|||||||
import { toBase64, deriveKEK, deriveLoginSubkey } from "../src/crypto/index.js";
|
import { toBase64, deriveKEK, deriveLoginSubkey } from "../src/crypto/index.js";
|
||||||
import type { KeyAttributes } from "../src/auth/types.js";
|
import type { KeyAttributes } from "../src/auth/types.js";
|
||||||
|
|
||||||
const TEST_OPS = 2;
|
// The account's password is stretched at libsodium's lowest Argon2id cost.
|
||||||
const TEST_MEM = 64 * 1024 * 1024;
|
// Every login stretches it again, and at a realistic cost that takes seconds on
|
||||||
|
// a loaded host.
|
||||||
|
const TEST_OPS = 1;
|
||||||
|
const TEST_MEM = 8192;
|
||||||
|
|
||||||
export interface MetaMockState {
|
export interface MetaMockState {
|
||||||
verifier: Buffer;
|
verifier: Buffer;
|
||||||
|
|||||||
@@ -253,6 +253,12 @@ describe("Library content wiring", () => {
|
|||||||
await expect(
|
await expect(
|
||||||
lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
|
lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
|
||||||
).rejects.toThrow(/content cache/i);
|
).rejects.toThrow(/content cache/i);
|
||||||
|
await expect(lib.originalOf(file(1, 1))).rejects.toThrow(
|
||||||
|
/content cache/i,
|
||||||
|
);
|
||||||
|
await expect(lib.thumbnailOf(file(1, 1))).rejects.toThrow(
|
||||||
|
/content cache/i,
|
||||||
|
);
|
||||||
await lib.close();
|
await lib.close();
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -375,6 +375,25 @@ describe("ContentCache.original / thumbnail", () => {
|
|||||||
await cache.open();
|
await cache.open();
|
||||||
await expect(cache.original(999)).rejects.toThrow(/unknown file/i);
|
await expect(cache.original(999)).rejects.toThrow(/unknown file/i);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// `getFile` knows no file here, so the cache can fetch only the file it is
|
||||||
|
// given.
|
||||||
|
it("originalOf and thumbnailOf fetch and cache the file they are given", async () => {
|
||||||
|
const { cache, source } = buildCache({ files: [] });
|
||||||
|
await cache.open();
|
||||||
|
|
||||||
|
const original = await cache.originalOf(file(5));
|
||||||
|
const thumbnail = await cache.thumbnailOf(file(5));
|
||||||
|
|
||||||
|
expect(source.originalCalls).toEqual([5]);
|
||||||
|
expect(source.thumbnailCalls).toEqual([5]);
|
||||||
|
expect(original.path).toBe(join(cacheDir, "originals", "5.jpg"));
|
||||||
|
expect(thumbnail.path).toBe(join(cacheDir, "thumbnails", "5.jpg"));
|
||||||
|
expect(cache.pathsFor(5)).toEqual({
|
||||||
|
originalPath: original.path,
|
||||||
|
thumbnailPath: thumbnail.path,
|
||||||
|
});
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("ContentCache.ensureThumbnails", () => {
|
describe("ContentCache.ensureThumbnails", () => {
|
||||||
|
|||||||
+4
-273
@@ -17,14 +17,6 @@
|
|||||||
* 4. `freshCollections()` reads the collection list alone, with one request
|
* 4. `freshCollections()` reads the collection list alone, with one request
|
||||||
* and no collection's files, and stores nothing, so the next refresh still
|
* and no collection's files, and stores nothing, so the next refresh still
|
||||||
* fetches the files of every collection (issue #210).
|
* fetches the files of every collection (issue #210).
|
||||||
* 5. `refreshCollection()` brings one collection up to date in the store
|
|
||||||
* (issue #217). It leaves the stored cursor where it was, so the next
|
|
||||||
* refresh still fetches the files of every other collection that changed.
|
|
||||||
* It never runs at the same time as a refresh. After a refresh, it stores
|
|
||||||
* no collection the store does not hold: not a new one, and not one a
|
|
||||||
* refresh deleted. `close()` waits for it. A library opened on a cache that
|
|
||||||
* only it wrote awaits its first refresh. That refresh replaces what it
|
|
||||||
* stored, in the server's order.
|
|
||||||
*
|
*
|
||||||
* The client is the same metadata-only mock the background-refresh tests use:
|
* The client is the same metadata-only mock the background-refresh tests use:
|
||||||
* no crypto, no network, scripted pages, and a record of each call. A long
|
* no crypto, no network, scripted pages, and a record of each call. A long
|
||||||
@@ -38,7 +30,6 @@ import { tmpdir } from "node:os";
|
|||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
|
|
||||||
import { Library } from "../../src/library/index.js";
|
import { Library } from "../../src/library/index.js";
|
||||||
import { MetadataStore } from "../../src/library/store.js";
|
|
||||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||||
|
|
||||||
@@ -153,26 +144,6 @@ const openSeeded = async (
|
|||||||
return { client, lib };
|
return { client, lib };
|
||||||
};
|
};
|
||||||
|
|
||||||
// Hold the client's next files request until `release()` is called. `started`
|
|
||||||
// resolves once that request has been sent.
|
|
||||||
const holdNextFilesRequest = (
|
|
||||||
client: MockClient,
|
|
||||||
): { started: Promise<void>; release: () => void } => {
|
|
||||||
let release!: () => void;
|
|
||||||
const held = new Promise<void>((r) => (release = r));
|
|
||||||
let sent!: () => void;
|
|
||||||
const started = new Promise<void>((r) => (sent = r));
|
|
||||||
const realFilesSince = client.filesSince.bind(client);
|
|
||||||
client.filesSince = async (args) => {
|
|
||||||
client.filesSince = realFilesSince;
|
|
||||||
const page = await realFilesSince(args);
|
|
||||||
sent();
|
|
||||||
await held;
|
|
||||||
return page;
|
|
||||||
};
|
|
||||||
return { started, release };
|
|
||||||
};
|
|
||||||
|
|
||||||
describe("Library.fresh", () => {
|
describe("Library.fresh", () => {
|
||||||
let dir: string;
|
let dir: string;
|
||||||
let cacheDirectory: string;
|
let cacheDirectory: string;
|
||||||
@@ -316,10 +287,10 @@ describe("Library.freshCollections", () => {
|
|||||||
rmSync(dir, { recursive: true, force: true });
|
rmSync(dir, { recursive: true, force: true });
|
||||||
});
|
});
|
||||||
|
|
||||||
// A refresh from a stored cursor past 0 fetches a collection's files only
|
// A refresh fetches a collection's files only when the collection is new
|
||||||
// when the collection is new or its updationTime has advanced past the
|
// or its updationTime has advanced past the stored copy's. Had the
|
||||||
// stored copy's. So does `refreshCollection()`. Neither would fetch the
|
// collection list been stored, the next refresh would take every
|
||||||
// files of a collection stored here without them.
|
// collection for one it already holds the files of, and fetch none.
|
||||||
it("asks for the collection list alone and stores nothing", async () => {
|
it("asks for the collection list alone and stores nothing", async () => {
|
||||||
const client = new MockClient();
|
const client = new MockClient();
|
||||||
const page = {
|
const page = {
|
||||||
@@ -369,243 +340,3 @@ describe("Library.freshCollections", () => {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("Library.refreshCollection", () => {
|
|
||||||
let dir: string;
|
|
||||||
let cacheDirectory: string;
|
|
||||||
|
|
||||||
beforeEach(() => {
|
|
||||||
dir = mkdtempSync(join(tmpdir(), "quak-refresh-collection-"));
|
|
||||||
cacheDirectory = join(dir, "cache");
|
|
||||||
});
|
|
||||||
|
|
||||||
afterEach(() => {
|
|
||||||
rmSync(dir, { recursive: true, force: true });
|
|
||||||
});
|
|
||||||
|
|
||||||
// Collections 1 and 2 are stored at updationTime 100, with a stored cursor
|
|
||||||
// of 100, and then both change on the server.
|
|
||||||
it("fetches the files of one changed collection and leaves the others to the next refresh", async () => {
|
|
||||||
const client = new MockClient();
|
|
||||||
client.collectionsQueue.push({
|
|
||||||
collections: [collection(1, 100), collection(2, 100)],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 100,
|
|
||||||
});
|
|
||||||
client.filesFor(1, {
|
|
||||||
files: [file(1001, 1, 90)],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 90,
|
|
||||||
});
|
|
||||||
client.filesFor(2, {
|
|
||||||
files: [file(2001, 2, 90)],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 90,
|
|
||||||
});
|
|
||||||
const lib = await Library.open({
|
|
||||||
client,
|
|
||||||
cacheDirectory,
|
|
||||||
refreshIntervalSeconds: SLOW_INTERVAL,
|
|
||||||
});
|
|
||||||
try {
|
|
||||||
client.filesCalls.length = 0;
|
|
||||||
client.filesFor(1, {
|
|
||||||
files: [file(1002, 1, 110)],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 110,
|
|
||||||
});
|
|
||||||
client.filesFor(2, {
|
|
||||||
files: [file(2002, 2, 120)],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 120,
|
|
||||||
});
|
|
||||||
|
|
||||||
// Collection 1's files alone, since the stored copy.
|
|
||||||
expect(await lib.refreshCollection(collection(1, 110))).toBe(true);
|
|
||||||
expect(client.filesCalls).toEqual([
|
|
||||||
{ collectionID: 1, sinceTime: 100 },
|
|
||||||
]);
|
|
||||||
expect(lib.listFiles(1).map((f) => f.id)).toEqual([1001, 1002]);
|
|
||||||
expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001]);
|
|
||||||
|
|
||||||
// The stored copy is now current, so no request.
|
|
||||||
await lib.refreshCollection(collection(1, 110));
|
|
||||||
expect(client.filesCalls).toHaveLength(1);
|
|
||||||
|
|
||||||
// The next refresh asks from the stored cursor, 100, so it still
|
|
||||||
// sees collection 2 changed and fetches its files.
|
|
||||||
client.collectionsQueue.push({
|
|
||||||
collections: [collection(1, 110), collection(2, 120)],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 120,
|
|
||||||
});
|
|
||||||
await lib.fresh();
|
|
||||||
expect(client.collectionsSinceTimes).toEqual([0, 100]);
|
|
||||||
expect(client.filesCalls).toEqual([
|
|
||||||
{ collectionID: 1, sinceTime: 100 },
|
|
||||||
{ collectionID: 2, sinceTime: 100 },
|
|
||||||
]);
|
|
||||||
expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001, 2002]);
|
|
||||||
} finally {
|
|
||||||
await lib.close();
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
// Collection 1 is stored at 100. `refreshCollection()` asks for it at 110,
|
|
||||||
// which adds file 1002. While that files request waits, a fresh read
|
|
||||||
// starts a refresh. The server tells that refresh that collection 1 was
|
|
||||||
// deleted at 130.
|
|
||||||
it("holds off a refresh that starts during the files request, which then deletes the collection", async () => {
|
|
||||||
const { client, lib } = await openSeeded(cacheDirectory);
|
|
||||||
const files = holdNextFilesRequest(client);
|
|
||||||
try {
|
|
||||||
client.filesFor(1, {
|
|
||||||
files: [file(1002, 1, 110)],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 110,
|
|
||||||
});
|
|
||||||
client.collectionsQueue.push({
|
|
||||||
collections: [],
|
|
||||||
deleted: [1],
|
|
||||||
cursor: 130,
|
|
||||||
});
|
|
||||||
|
|
||||||
const refreshing = lib.refreshCollection(collection(1, 110));
|
|
||||||
await files.started;
|
|
||||||
const reading = lib.fresh();
|
|
||||||
// The refresh has not sent its collections request yet.
|
|
||||||
await new Promise((r) => setImmediate(r));
|
|
||||||
expect(client.collectionsSinceTimes).toEqual([0]);
|
|
||||||
|
|
||||||
files.release();
|
|
||||||
await refreshing;
|
|
||||||
await reading;
|
|
||||||
expect(client.collectionsSinceTimes).toEqual([0, 100]);
|
|
||||||
expect(lib.getCollection(1)).toBeUndefined();
|
|
||||||
expect(lib.listFiles(1)).toEqual([]);
|
|
||||||
} finally {
|
|
||||||
files.release();
|
|
||||||
await lib.close();
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
// `freshCollections()` lists collection 1 at 110. Before
|
|
||||||
// `refreshCollection()` is called, a refresh learns that it was deleted at
|
|
||||||
// 130.
|
|
||||||
it("stores nothing for a collection that a refresh deleted before the call", async () => {
|
|
||||||
const { client, lib } = await openSeeded(cacheDirectory);
|
|
||||||
try {
|
|
||||||
client.collectionsQueue.push(
|
|
||||||
{ collections: [collection(1, 110)], deleted: [], cursor: 110 },
|
|
||||||
{ collections: [], deleted: [1], cursor: 130 },
|
|
||||||
);
|
|
||||||
const [listed] = await lib.freshCollections();
|
|
||||||
await lib.fresh();
|
|
||||||
client.filesCalls.length = 0;
|
|
||||||
|
|
||||||
expect(await lib.refreshCollection(listed)).toBe(false);
|
|
||||||
expect(client.filesCalls).toEqual([]);
|
|
||||||
expect(lib.getCollection(1)).toBeUndefined();
|
|
||||||
} finally {
|
|
||||||
await lib.close();
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
// A refresh has stored collection 1. Collection 2 is new on the server
|
|
||||||
// since then. Only the next refresh may add it, after collection 1.
|
|
||||||
it("stores nothing for a collection that is new since the last refresh", async () => {
|
|
||||||
const { client, lib } = await openSeeded(cacheDirectory);
|
|
||||||
try {
|
|
||||||
client.filesCalls.length = 0;
|
|
||||||
|
|
||||||
expect(await lib.refreshCollection(collection(2, 110))).toBe(false);
|
|
||||||
expect(client.filesCalls).toEqual([]);
|
|
||||||
expect(lib.getCollection(2)).toBeUndefined();
|
|
||||||
} finally {
|
|
||||||
await lib.close();
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
// The caller may remove the cache directory once `close()` resolves, so
|
|
||||||
// the cache write must have landed by then.
|
|
||||||
it("close() waits for a refreshCollection() in flight", async () => {
|
|
||||||
const { client, lib } = await openSeeded(cacheDirectory);
|
|
||||||
const files = holdNextFilesRequest(client);
|
|
||||||
client.filesFor(1, {
|
|
||||||
files: [file(1002, 1, 110)],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 110,
|
|
||||||
});
|
|
||||||
|
|
||||||
const refreshing = lib.refreshCollection(collection(1, 110));
|
|
||||||
await files.started;
|
|
||||||
let closed = false;
|
|
||||||
const closing = lib.close().then(() => {
|
|
||||||
closed = true;
|
|
||||||
});
|
|
||||||
await new Promise((r) => setImmediate(r));
|
|
||||||
expect(closed).toBe(false);
|
|
||||||
|
|
||||||
files.release();
|
|
||||||
await closing;
|
|
||||||
const saved = await MetadataStore.load(
|
|
||||||
join(cacheDirectory, "metadata.json"),
|
|
||||||
);
|
|
||||||
expect(saved.listFiles(1).map((f) => f.id)).toEqual([1001, 1002]);
|
|
||||||
await refreshing;
|
|
||||||
});
|
|
||||||
|
|
||||||
// `quak files 2` on an empty cache stores collection 2 alone. A library
|
|
||||||
// opened later on that cache must not serve it as the whole account. Its
|
|
||||||
// first refresh replaces what the cache holds, in the server's order.
|
|
||||||
it("open() awaits the first refresh on a cache that only refreshCollection() wrote", async () => {
|
|
||||||
const client = new MockClient();
|
|
||||||
client.filesFor(2, {
|
|
||||||
files: [file(2001, 2, 90)],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 90,
|
|
||||||
});
|
|
||||||
const first = await Library.open({
|
|
||||||
client,
|
|
||||||
cacheDirectory,
|
|
||||||
refreshOnOpen: false,
|
|
||||||
refreshIntervalSeconds: SLOW_INTERVAL,
|
|
||||||
});
|
|
||||||
expect(await first.refreshCollection(collection(2, 100))).toBe(true);
|
|
||||||
await first.close();
|
|
||||||
|
|
||||||
client.collectionsQueue.push({
|
|
||||||
collections: [collection(1, 100), collection(2, 100)],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 100,
|
|
||||||
});
|
|
||||||
client.filesFor(1, {
|
|
||||||
files: [file(1001, 1, 90)],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 90,
|
|
||||||
});
|
|
||||||
client.filesFor(2, {
|
|
||||||
files: [file(2001, 2, 90)],
|
|
||||||
deleted: [],
|
|
||||||
cursor: 90,
|
|
||||||
});
|
|
||||||
client.filesCalls.length = 0;
|
|
||||||
const lib = await Library.open({
|
|
||||||
client,
|
|
||||||
cacheDirectory,
|
|
||||||
refreshIntervalSeconds: SLOW_INTERVAL,
|
|
||||||
});
|
|
||||||
try {
|
|
||||||
expect(lib.status().lastRefreshAt).toBeDefined();
|
|
||||||
expect(lib.listCollections().map((c) => c.id)).toEqual([1, 2]);
|
|
||||||
// It fetches collection 2's files again, from the beginning.
|
|
||||||
expect(client.filesCalls).toEqual([
|
|
||||||
{ collectionID: 1, sinceTime: 0 },
|
|
||||||
{ collectionID: 2, sinceTime: 0 },
|
|
||||||
]);
|
|
||||||
expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001]);
|
|
||||||
} finally {
|
|
||||||
await lib.close();
|
|
||||||
}
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -20,9 +20,9 @@
|
|||||||
* 5. `close()` stops the timer and is idempotent, and its promise resolves
|
* 5. `close()` stops the timer and is idempotent, and its promise resolves
|
||||||
* only once an in-flight refresh has written the cache file.
|
* only once an in-flight refresh has written the cache file.
|
||||||
* 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id.
|
* 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id.
|
||||||
* 7. `open()` branches on the stored cursor. At 0, as on an empty cache, it
|
* 7. `open()` branches on the cache: an empty cache awaits the first refresh
|
||||||
* awaits the first refresh. Past 0 it serves the cache's copy at once and
|
* (it has nothing to serve yet); an existing cache serves its copy at once
|
||||||
* refreshes in the background, so a slow or dead server never stalls
|
* and refreshes in the background, so a slow or dead server never stalls
|
||||||
* opening.
|
* opening.
|
||||||
* 8. A save failure that leaves RAM ahead of disk keeps `status().lastError`
|
* 8. A save failure that leaves RAM ahead of disk keeps `status().lastError`
|
||||||
* set and keeps retrying the write; a later empty refresh does not clear it.
|
* set and keeps retrying the write; a later empty refresh does not clear it.
|
||||||
|
|||||||
@@ -4,8 +4,7 @@
|
|||||||
* These are the surface the GUI consumes across Electron IPC. `snapshot()` is
|
* These are the surface the GUI consumes across Electron IPC. `snapshot()` is
|
||||||
* synchronous — it reads the in-RAM store and projects it into plain records
|
* synchronous — it reads the in-RAM store and projects it into plain records
|
||||||
* (no keys) — and `subscribe({ onChange })` delivers a `LibraryChange` whenever
|
* (no keys) — and `subscribe({ onChange })` delivers a `LibraryChange` whenever
|
||||||
* a refresh or `refreshCollection()` changes the derived records. The
|
* a background refresh actually changes the derived records. The contracts:
|
||||||
* contracts:
|
|
||||||
*
|
*
|
||||||
* 1. `snapshot()` deduplicates a file across memberships into one record with
|
* 1. `snapshot()` deduplicates a file across memberships into one record with
|
||||||
* every album id, orders photos newest first, and carries no key material.
|
* every album id, orders photos newest first, and carries no key material.
|
||||||
|
|||||||
@@ -172,7 +172,7 @@ describe("MetadataStore", () => {
|
|||||||
expect(store.listCollections()).toEqual([]);
|
expect(store.listCollections()).toEqual([]);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("loads an empty store when the list of what did not decrypt is damaged", async () => {
|
it("loads an empty store when the list of what the refresh could not decrypt is damaged", async () => {
|
||||||
// A backup reads that list on every run, so a damaged one that loaded
|
// A backup reads that list on every run, so a damaged one that loaded
|
||||||
// would fail every backup until the cache was deleted. A string is
|
// would fail every backup until the cache was deleted. A string is
|
||||||
// damaged too, though JavaScript can loop over its characters.
|
// damaged too, though JavaScript can loop over its characters.
|
||||||
@@ -266,23 +266,4 @@ describe("MetadataStore", () => {
|
|||||||
expect(store.getFile(12345, 67890)).toBeUndefined();
|
expect(store.getFile(12345, 67890)).toBeUndefined();
|
||||||
expect(store.getFile(55555, 67890)?.key).toEqual(inB.key);
|
expect(store.getFile(55555, 67890)?.key).toEqual(inB.key);
|
||||||
});
|
});
|
||||||
|
|
||||||
// A backup reports these in this order. It must not depend on whether a
|
|
||||||
// refresh or `Library.refreshCollection()` fetched each one first.
|
|
||||||
it("lists what did not decrypt by collection ID, then file ID", async () => {
|
|
||||||
const store = await MetadataStore.load(path);
|
|
||||||
store.putUndecryptable({ collectionID: 2, fileID: 7, error: "a" });
|
|
||||||
store.putUndecryptable({ collectionID: 3, error: "b" });
|
|
||||||
store.putUndecryptable({ collectionID: 1, fileID: 9, error: "c" });
|
|
||||||
store.putUndecryptable({ collectionID: 2, fileID: 5, error: "d" });
|
|
||||||
|
|
||||||
expect(
|
|
||||||
store.listUndecryptable().map((r) => [r.collectionID, r.fileID]),
|
|
||||||
).toEqual([
|
|
||||||
[1, 9],
|
|
||||||
[2, 5],
|
|
||||||
[2, 7],
|
|
||||||
[3, undefined],
|
|
||||||
]);
|
|
||||||
});
|
|
||||||
});
|
});
|
||||||
@@ -6,10 +6,12 @@
|
|||||||
* working thumbnails, others return 404 or empty bodies. The tests
|
* working thumbnails, others return 404 or empty bodies. The tests
|
||||||
* verify that the detection and repair logic handles each case correctly.
|
* verify that the detection and repair logic handles each case correctly.
|
||||||
*
|
*
|
||||||
* As of issue #52 both helpers take an open `Library` for enumeration and the
|
* `listMissingThumbnails` takes an open `Library` to enumerate the files. Both
|
||||||
* `Client` for the API operations that stay unchanged (the thumbnail existence
|
* helpers take the `Client` for the API operations: the thumbnail existence
|
||||||
* check, and the encrypt-and-upload path). `fixMissingThumbnails` reads each
|
* check, and the encrypt-and-upload path. `fixMissingThumbnails` takes the
|
||||||
* original through the library's content cache (`photo.original()`).
|
* record of each file, which these tests look up with `cachedFilesByID`. It
|
||||||
|
* reads each original through the library's content cache, by
|
||||||
|
* `lib.originalOf(file)`.
|
||||||
*
|
*
|
||||||
* `fixMissingThumbnails` is the most complex function in quak: it
|
* `fixMissingThumbnails` is the most complex function in quak: it
|
||||||
* downloads the original file, generates a JPEG thumbnail with jpeg-js,
|
* downloads the original file, generates a JPEG thumbnail with jpeg-js,
|
||||||
@@ -41,6 +43,7 @@ import {
|
|||||||
import { SRP, SrpServer } from "fast-srp-hap";
|
import { SRP, SrpServer } from "fast-srp-hap";
|
||||||
import { Client } from "../../src/client.js";
|
import { Client } from "../../src/client.js";
|
||||||
import { Library } from "../../src/library/index.js";
|
import { Library } from "../../src/library/index.js";
|
||||||
|
import { cachedFilesByID } from "../../src/cli-read.js";
|
||||||
import {
|
import {
|
||||||
listMissingThumbnails,
|
listMissingThumbnails,
|
||||||
fixMissingThumbnails,
|
fixMissingThumbnails,
|
||||||
@@ -54,8 +57,11 @@ import type { RetryOptions } from "../../src/retry.js";
|
|||||||
|
|
||||||
const TEST_EMAIL = "thumb@example.com";
|
const TEST_EMAIL = "thumb@example.com";
|
||||||
const TEST_PASSWORD = "thumbpass";
|
const TEST_PASSWORD = "thumbpass";
|
||||||
const TEST_OPS = 2;
|
// The password is stretched at libsodium's lowest Argon2id cost. Each test
|
||||||
const TEST_MEM = 64 * 1024 * 1024;
|
// builds an account and logs in, and at a realistic cost that takes seconds on
|
||||||
|
// a loaded host.
|
||||||
|
const TEST_OPS = 1;
|
||||||
|
const TEST_MEM = 8192;
|
||||||
const TEST_TIME = 1700000000000000;
|
const TEST_TIME = 1700000000000000;
|
||||||
|
|
||||||
interface ThumbMockState {
|
interface ThumbMockState {
|
||||||
@@ -656,7 +662,12 @@ describe("fixMissingThumbnails", () => {
|
|||||||
const client = await login(buildThumbFetch(fixMock));
|
const client = await login(buildThumbFetch(fixMock));
|
||||||
const lib = await openLib(client);
|
const lib = await openLib(client);
|
||||||
|
|
||||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
const results = await fixMissingThumbnails(
|
||||||
|
lib,
|
||||||
|
client,
|
||||||
|
[101],
|
||||||
|
cachedFilesByID(lib, [101]),
|
||||||
|
);
|
||||||
lib.close();
|
lib.close();
|
||||||
|
|
||||||
expect(results.length).toBe(1);
|
expect(results.length).toBe(1);
|
||||||
@@ -693,7 +704,12 @@ describe("fixMissingThumbnails", () => {
|
|||||||
const client = await login(buildThumbFetch(fixMock));
|
const client = await login(buildThumbFetch(fixMock));
|
||||||
const lib = await openLib(client);
|
const lib = await openLib(client);
|
||||||
|
|
||||||
const results = await fixMissingThumbnails(lib, client, [999]);
|
const results = await fixMissingThumbnails(
|
||||||
|
lib,
|
||||||
|
client,
|
||||||
|
[999],
|
||||||
|
cachedFilesByID(lib, [999]),
|
||||||
|
);
|
||||||
lib.close();
|
lib.close();
|
||||||
|
|
||||||
expect(results.length).toBe(1);
|
expect(results.length).toBe(1);
|
||||||
@@ -715,6 +731,7 @@ describe("fixMissingThumbnails", () => {
|
|||||||
lib,
|
lib,
|
||||||
client,
|
client,
|
||||||
[101, 102],
|
[101, 102],
|
||||||
|
cachedFilesByID(lib, [101, 102]),
|
||||||
(msg, level) => {
|
(msg, level) => {
|
||||||
if (level === "error") errors.push(msg);
|
if (level === "error") errors.push(msg);
|
||||||
},
|
},
|
||||||
@@ -745,6 +762,7 @@ describe("fixMissingThumbnails", () => {
|
|||||||
lib,
|
lib,
|
||||||
client,
|
client,
|
||||||
[101, 103, 104],
|
[101, 103, 104],
|
||||||
|
cachedFilesByID(lib, [101, 103, 104]),
|
||||||
);
|
);
|
||||||
lib.close();
|
lib.close();
|
||||||
|
|
||||||
@@ -779,7 +797,12 @@ describe("fixMissingThumbnails", () => {
|
|||||||
const client = await login(counted.fetch);
|
const client = await login(counted.fetch);
|
||||||
const lib = await openLib(client);
|
const lib = await openLib(client);
|
||||||
|
|
||||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
const results = await fixMissingThumbnails(
|
||||||
|
lib,
|
||||||
|
client,
|
||||||
|
[101],
|
||||||
|
cachedFilesByID(lib, [101]),
|
||||||
|
);
|
||||||
lib.close();
|
lib.close();
|
||||||
|
|
||||||
expect(results[0]!.status).toBe("skipped");
|
expect(results[0]!.status).toBe("skipped");
|
||||||
@@ -800,7 +823,12 @@ describe("fixMissingThumbnails", () => {
|
|||||||
const client = await login(counted.fetch);
|
const client = await login(counted.fetch);
|
||||||
const lib = await openLib(client);
|
const lib = await openLib(client);
|
||||||
|
|
||||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
const results = await fixMissingThumbnails(
|
||||||
|
lib,
|
||||||
|
client,
|
||||||
|
[101],
|
||||||
|
cachedFilesByID(lib, [101]),
|
||||||
|
);
|
||||||
lib.close();
|
lib.close();
|
||||||
|
|
||||||
expect(results[0]!.status).toBe("skipped");
|
expect(results[0]!.status).toBe("skipped");
|
||||||
@@ -841,7 +869,12 @@ describe("fixMissingThumbnails", () => {
|
|||||||
const client = await login(buildThumbFetch(fixMock));
|
const client = await login(buildThumbFetch(fixMock));
|
||||||
const lib = await openLib(client);
|
const lib = await openLib(client);
|
||||||
|
|
||||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
const results = await fixMissingThumbnails(
|
||||||
|
lib,
|
||||||
|
client,
|
||||||
|
[101],
|
||||||
|
cachedFilesByID(lib, [101]),
|
||||||
|
);
|
||||||
lib.close();
|
lib.close();
|
||||||
|
|
||||||
expect(results[0]!.status).toBe("fixed");
|
expect(results[0]!.status).toBe("fixed");
|
||||||
|
|||||||
@@ -3,8 +3,15 @@ import { configDefaults, defineConfig } from "vitest/config";
|
|||||||
// vitest does not read .gitignore when looking for tests. A checkout nested
|
// vitest does not read .gitignore when looking for tests. A checkout nested
|
||||||
// under .claude/ has its own test/ tree, and without this exclude the suite
|
// under .claude/ has its own test/ tree, and without this exclude the suite
|
||||||
// runs once per nested checkout and still reports success.
|
// runs once per nested checkout and still reports success.
|
||||||
|
//
|
||||||
|
// vitest starts one worker per CPU core by default, and a shared host has
|
||||||
|
// dozens. Four keep the suite well inside the Dockerfile's time limit. vitest
|
||||||
|
// 2 rejects a maxWorkers below its default minWorkers, so minWorkers is set
|
||||||
|
// too.
|
||||||
export default defineConfig({
|
export default defineConfig({
|
||||||
test: {
|
test: {
|
||||||
exclude: [...configDefaults.exclude, ".claude/**"],
|
exclude: [...configDefaults.exclude, ".claude/**"],
|
||||||
|
minWorkers: 1,
|
||||||
|
maxWorkers: 4,
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
Reference in new issue
Block a user