From ec07fd1dc99f54f65f6d0fab31fe95d050882e02 Mon Sep 17 00:00:00 2001 From: sneak Date: Sun, 25 Oct 2026 23:24:16 +0000 Subject: [PATCH] quak files COLLECTION fetches that collection's files alone (closes #217) `quak files COLLECTION` used to wait for a full refresh, which on an empty cache fetched the files of every collection first. It now reads the collection list, then calls the new `Library.refreshCollection()`. That brings one collection up to date in the store as a refresh would. Once the stored cursor is past 0, it only updates a collection the store holds. For a new one, `quak files COLLECTION` waits for a refresh. A refresh from a cursor of 0 replaces all the store holds, so collections then sit in the server's order. `open()` awaits the first refresh while the cursor is 0. The store lists what did not decrypt in a fixed order. Model: opus-5-5 --- README.md | 348 ++++++++++++++++---------- TODO.md | 15 ++ src/cli-commands.ts | 14 +- src/cli-read.ts | 71 +++--- src/library/index.ts | 445 ++++++++++++++++++++++------------ src/library/precache.ts | 2 +- src/library/records.ts | 5 +- src/library/store.ts | 25 +- test/cli/commands.test.ts | 151 +++++++++--- test/cli/log.test.ts | 4 +- test/cli/read.test.ts | 74 ++++-- test/library/fresh.test.ts | 277 ++++++++++++++++++++- test/library/library.test.ts | 6 +- test/library/snapshot.test.ts | 3 +- test/library/store.test.ts | 21 +- 15 files changed, 1074 insertions(+), 387 deletions(-) diff --git a/README.md b/README.md index 8e34535..e11c0b5 100644 --- a/README.md +++ b/README.md @@ -2,21 +2,31 @@ quak is a WTFPL-licensed TypeScript client library and CLI by [@sneak](https://sneak.berlin) for the [Ente](https://ente.io) end-to-end -encrypted photo hosting service. It logs in, enumerates collections and files, -and downloads individual images while decrypting them on the way to disk. +encrypted photo hosting service. It logs in, lists collections and their files, +and downloads and decrypts single files. -quak also includes a resilient backup command that downloads every file in the -account into a deduplicated local directory tree, skipping files that already -exist on disk and continuing past individual download failures instead of -crashing. For each file it persists the basic metadata fields quak keeps (title, -file type, creation and modification time, latitude, longitude, content hash), -its update time, and the private and public magic metadata in full. It keeps any -ML data Ente has for the file, except the CLIP embedding, and writes the file's -keywords in place of the embedding. For an image (for a live photo, its image), -it keeps its original's EXIF and XMP, with its dimensions for a JPEG only. For a -video it keeps none of these three. It runs unattended from cron (see "Running -the backup from cron"). A helper subcommand can detect and regenerate missing -thumbnails, encrypting and uploading them back to the server. +`quak backup` downloads every file in the account into a local directory tree. +It downloads and stores each file once, however many collections hold it. It +skips files already on disk. It logs a failed download and moves on to the next +file. Beside each file it writes a JSON file that holds: + +- the file's title and file type +- its creation and modification times +- its latitude and longitude +- its content hash +- its update time +- its private and public magic metadata, in full +- the ML data that Ente has for it, without the CLIP embedding +- its keywords in place of that embedding (see "Backup layout") +- for an image, the EXIF and XMP of its original, and its dimensions if it is a + JPEG +- for a live photo, the same for its image + +The backup runs unattended from cron (see "Running the backup from cron"). + +`quak helper list-missing-thumbnails` finds files whose thumbnail is missing. +`quak helper fix-missing-thumbnails` makes a new thumbnail for such a file, then +encrypts it and uploads it to the server. ## Getting Started @@ -60,10 +70,11 @@ const client = await Client.login({ password: "your-password", }); -// Open a cache-backed library. On an empty cache this awaits one server -// refresh; on an existing cache it returns immediately and refreshes in the -// background. Later refreshes start `refreshIntervalSeconds` (default 3) after -// the previous one ends. +// Open a cache-backed library. On an empty cache, or one that only +// `lib.refreshCollection()` wrote, this awaits one server refresh. Otherwise +// it returns immediately and refreshes in the background. A background timer +// then refreshes again `refreshIntervalSeconds` (default 3) after its previous +// refresh ends. const lib = await Library.open({ client }); // Default reads answer synchronously from the local cache — no network. @@ -640,17 +651,17 @@ The CLI stores the snapshot at the platform-appropriate data directory via `0600`. The key material is stored in cleartext in the JSON; treat this file as you would treat the password itself. A missing file is reported as "not logged in"; a file that exists but is corrupt is reported as such, naming the bad -field. When the refresh a command starts with gets HTTP 401 from the server, -because it no longer accepts the saved session's token, the command prints one -line, `quak: the saved session is no longer valid; run "quak login"`, with no -stack trace. All three exit with status 3, which means the user must run -`quak login` again. `quak logout` is the exception: with no file it says there -is no session and exits 0, and it handles a corrupt file or a failed server call -as described below. `quak backup` meets an expired session on the refresh that -starts every run, before it touches any file but its lock (see "Backup layout"). -A session that stops working partway through a backup instead fails each -remaining file into `failures.json`, so that run exits 1 and the next one stops -at its refresh with status 3. No command but `quak login` ever prompts. +field. The server answers HTTP 401 when it no longer accepts the saved session's +token. A command that gets that answer prints one line, +`quak: the saved session is no longer valid; run "quak login"`, with no stack +trace. All three exit with status 3, which means the user must run `quak login` +again. `quak logout` is the exception: with no file it says there is no session +and exits 0, and it handles a corrupt file or a failed server call as described +below. `quak backup` meets an expired session on the refresh that starts every +run, before it touches any file but its lock (see "Backup layout"). A session +that stops working partway through a backup instead fails each remaining file +into `failures.json`, so that run exits 1 and the next one stops at its refresh +with status 3. No command but `quak login` ever prompts. `quak logout` ends the session on the server, so the token in `session.json` stops working even in a copy of the file, and then deletes the file. If the @@ -680,22 +691,40 @@ quak helper fix-missing-thumbnails [--file ids] [--json] generate + upload miss ``` Every command except `login`, `whoami` and `logout` runs on the cache-backed -library. The read commands — `collections`, `files`, `get`, `get-thumb`, -`backup-metadata`, `helper list-missing-thumbnails` and -`helper fix-missing-thumbnails` — force a fresh server round-trip before they -answer, so they report current account state rather than whatever the cache last -held. `collections` asks the server for the collection list alone, in one -request, and neither fetches any collection's files nor changes the cache. The -others wait for a refresh. With an empty cache, as on a first run, that refresh -fetches the files of every collection, one collection after another, and stores -nothing until it has them all, which on a large account takes a while. Meanwhile -the command logs a line on stderr for each collection whose files it has -fetched, such as `Fetched the files of 3 of 120 collections`, with `--json` too; -`backup` does the same while the refresh it starts with runs. stdout holds only -the command's output. If that round-trip fails, the command prints the error on -one line and exits 1, or 3 when the server no longer accepts the saved session -(see "Session handling"). An album or file quak cannot decrypt does not fail it; -the command answers from what did decrypt (see "Default reads vs. fresh reads"). +library. Each read command makes a fresh server round-trip before it answers. It +then reports the account's current state, not what the cache last held. The read +commands are: + +- `collections` +- `files` +- `get` +- `get-thumb` +- `backup-metadata` +- `helper list-missing-thumbnails` +- `helper fix-missing-thumbnails` + +`collections` asks the server for the collection list alone, in one request. It +fetches no collection's files and leaves the cache as it is. `files` with a +collection ID asks for the collection list the same way. It then asks for the +files of that collection alone, and stores them in the cache. It asks for no +files when the cache's copy of that collection is current. The other read +commands, and `files` without a collection ID, wait for a refresh. So does +`files` with the ID of a collection that is new since the last refresh. + +The first refresh fetches the files of every collection, one collection after +another. That includes any collection that `files` with a collection ID stored +before it. The refresh stores nothing until it has them all, and on a large +account that takes a while. Meanwhile the command logs a line on stderr for each +collection whose files it has fetched, such as +`Fetched the files of 3 of 120 collections`. It logs these lines with `--json` +too. `backup` logs them during the refresh it starts with. stdout holds only the +command's output. + +If the server round-trip fails, the command prints the error on one line. It +exits 1, or 3 when the server no longer accepts the saved session (see "Session +handling"). An album or file that quak cannot decrypt does not fail the command. +The command answers from what did decrypt (see "Default reads vs. fresh reads"). + `--cache-dir` overrides where the cache lives; without it each account gets its own directory under the per-user cache path. @@ -750,9 +779,10 @@ its `Failed files:` list. Every other line is logged as it is without `--json`. exit status 1. `files ` lists the files in that collection. `files` with no -argument lists every file in the account, collection by collection, each file -once: a file in several collections is listed with the first of them, which is -its `collectionID` in `--json`. +argument lists every file in the account once, collection by collection. The +collections come in the order the server first sent them in a refresh. A file in +several collections is listed with the first of them, which is its +`collectionID` in `--json`. An ID given to `files`, `get`, `get-thumb` or `helper fix-missing-thumbnails --file` is written in decimal digits, no larger @@ -1092,15 +1122,17 @@ against a fake of the Ente HTTP API, and `yarn test` verifies them. ### Opening a library `Library.open(options)` loads the on-disk cache, starts the background refresh -loop, and resolves to a `Library`. On an empty cache it awaits the first -refresh, so it opens onto the account's data whenever the server is reachable; -if that refresh fails, it opens with no data and records the error in -`lib.status()`. On an existing cache it returns immediately and refreshes in the -background, so an unreachable server does not block opening. With -`refreshOnOpen: false` it starts no refresh and returns at once onto what the -cache holds, empty or not; a refresh then starts only when `fresh()` or -`lib.backup()` asks, or the background timer fires. The CLI's read commands open -it so (see "CLI surface"). +loop, and resolves to a `Library`. On an empty cache, or one that only +`lib.refreshCollection()` wrote, it awaits one refresh. It then opens onto the +account's data whenever the server is reachable. If that refresh fails, it opens +onto what the cache holds, and records the error in `lib.status()`. On any other +cache it returns immediately and refreshes in the background. An unreachable +server then does not block opening. + +With `refreshOnOpen: false`, `open()` starts no refresh. It returns at once onto +what the cache holds, even when the cache is empty. A refresh then starts only +when `fresh()` or `lib.backup()` asks, or the background timer fires. The CLI's +read commands open the library this way (see "CLI surface"). `LibraryOptions`: @@ -1127,22 +1159,35 @@ when the library opens; `lib.downloadDirectory` holds the result. `onProgress` receives a `RefreshEvent` with status `"started"` as a refresh, an ML data fetch or a precache sweep starts, and `"done"` or `"failed"` as it ends. In between, a refresh sends `"progress"` after each collection whose files it -fetched, with `fetched`, how many collections it has fetched the files of so -far, and `total`, how many it will. A refresh fetches the files of each -collection that is new or changed since the cache's copy, so on an empty cache -it fetches those of every collection. +fetched. That event's `fetched` is the number of collections whose files the +refresh has fetched so far. Its `total` is the number of collections whose files +it will fetch in all. A refresh fetches the files of each collection that is new +or changed since the cache's copy. On the first refresh, that is every +collection. Concurrency is set through `pools`: construct `new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })` and pass it. The three pools default to 10 / 5 / 25 (see Request pools below). -`lib.status()` returns a `LibraryStatus` (collection/file counts, last -refresh/ML times and errors, originals usage and effective limit, precache -progress, and `closed`). `lib.close()` stops the background timer; it is -idempotent, and an in-flight refresh is left to finish. The promise it returns -resolves once that refresh (including its cache write), the ML data fetch and -the precache fetches already running have all finished, so the cache directory -can then be removed. +`lib.status()` returns a `LibraryStatus`, which holds: + +- the collection and file counts +- the time of the last refresh that succeeded, and the error of the latest + refresh if it failed +- the same two for the ML data fetch +- the originals cache's usage and its effective limit +- the precache progress +- `closed` + +`lib.close()` stops the background timer, and is idempotent. A refresh or +`lib.refreshCollection()` in flight is left to finish. `close()` returns a +promise that resolves once all of these have finished: + +- that refresh or `lib.refreshCollection()`, with its cache write +- the ML data fetch +- the precache fetches already running + +The cache directory can then be removed. `lib.status().mlDataErrors` lists, as `{ fileID, error }`, each file whose ML data the last ML data fetch that succeeded got but could not decrypt, gunzip or @@ -1156,39 +1201,72 @@ cache. Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer synchronously from the copy held in RAM and never touch the network. A refresh -changes that copy only once all its server requests have succeeded, and the -background timer starts the next refresh `refreshIntervalSeconds` after the -previous one ends. So a default read is immediate, but only as current as the -last refresh whose requests all succeeded; right after opening an existing -cache, it is the copy on disk. +changes that copy only once all its server requests have succeeded. +`lib.refreshCollection()` changes it the same way for one collection. So a +default read is immediate, but only as current as the last refresh or +`lib.refreshCollection()` whose requests all succeeded. Right after `open()` +returns without awaiting a refresh, that copy is the one loaded from disk. -`await lib.fresh()` waits for a refresh to complete and persist, and returns the -same `{ albums, photos, timeline }` namespaces, which then reflect a completed -server round-trip. When a refresh is already running, background or not, -`fresh()` waits for that one, so its answer can come from requests made before -the call; only when none is running does it start one. A refresh that fails +The background timer takes a turn `refreshIntervalSeconds` after its last turn +ends. Each turn starts a refresh, unless a refresh or `lib.refreshCollection()` +is already running. + +`await lib.fresh()` waits for a refresh to complete and persist. It then returns +the same `{ albums, photos, timeline }` namespaces, which reflect a completed +server round-trip. If a `lib.refreshCollection()` is running, `fresh()` first +waits for it to end. When a refresh is already running, background or not, +`fresh()` waits for that one. Its answer can then come from requests made before +the call. Only when none is running does it start one. A refresh that fails rejects the caller (default reads stay silent and keep serving the last good -copy). The CLI's read commands use fresh reads (issue -https://git.eeqj.de/sneak/quak/issues/75). +copy). Every CLI read command calls `fresh()`, except `quak collections`. +`quak files` with a collection ID calls it only when `lib.refreshCollection()` +resolves to `false`. -`await lib.freshCollections()` is the fresh read of the collection list alone: -one collections request, answered with every live collection that decrypts, as -the `Collection` objects `listCollections()` returns, in the server's order. It -fetches no collection's files and stores nothing, not even the list: a refresh -fetches a collection's files only when the collection is new or has changed -since the stored copy, so a collection stored without its files would never get -them. `quak collections` uses it (issue -https://git.eeqj.de/sneak/quak/issues/210). +`await lib.freshCollections()` is the fresh read of the collection list alone. +It sends one collections request. It answers with every live collection that +decrypts, in the server's order, as the `Collection` objects that +`listCollections()` returns. It fetches no collection's files and stores +nothing, not even the list. A refresh from a cursor past 0 fetches a +collection's files only when the collection is new or has changed since the +stored copy. So does `lib.refreshCollection()`. Neither would fetch the files of +a collection stored without them. `quak collections` uses `freshCollections()`. -An album or file in the server's answer that quak cannot decrypt, or whose -metadata is not JSON, does not fail the refresh. The refresh leaves it out, an -album with all its files, and drops any older copy the cache held, so reads -answer from what did decrypt. The library keeps a list of what it left out, with -the reason, in `metadata.json`: the next refresh asks the server only for what -changed, so it would not see the record again. A record stays on that list until -a later refresh gets a copy that decrypts or learns that the server deleted it. -`lib.backup()` reports each such album, and each such file in its scope that has -no copy that decrypted (see "Backup layout"). +`await lib.refreshCollection(collection)` brings one collection up to date in +the cache, given its `Collection` from `freshCollections()`. It does for that +collection what a refresh does for a changed one, and fetches no other +collection's files. It fetches the collection's files changed since the cache's +copy, or all of them when the cache has no copy. It then stores and saves them. +It sends no request when the cache's copy has the same `updationTime` or a later +one. It resolves to `true` when the cache then holds the collection. + +`lib.refreshCollection()` adds a collection the cache does not hold only while +the refresh cursor is 0. The cursor moves past 0 once a refresh receives the +account's collections. After that, for a collection the cache does not hold, it +sends no request and resolves to `false`. Such a collection is new since that +refresh, or the refresh dropped it. Only a refresh may then add it, after the +collections the cache holds. + +A refresh from a cursor of 0 replaces everything the cache holds. It fetches the +files of every collection, even those `lib.refreshCollection()` stored. It +stores the collections in the server's order. The cache then holds what it would +had `lib.refreshCollection()` never run. + +`lib.refreshCollection()` leaves the refresh cursor where it was, so the next +refresh still fetches the files of every other collection that changed. It never +runs at the same time as a refresh. It first waits for a running refresh or +`lib.refreshCollection()` to end. While it runs, a `fresh()` or `lib.backup()` +waits for it to end. `quak files` with a collection ID uses it. + +A refresh does not fail on an album or file that quak cannot decrypt, or whose +metadata is not JSON. The refresh leaves out such a file, and such an album +together with all its files. It also drops any older copy that the cache held, +so reads answer from what did decrypt. The library keeps a list of what it left +out, with the reason, in `metadata.json`. The next refresh asks the server only +for what changed, so it would not see them again. An entry stays on that list +until a later refresh or `lib.refreshCollection()` stores a copy that decrypts, +or learns that the server deleted it. `lib.backup()` reports each such album, +and each such file in its scope that has no copy that decrypted (see "Backup +layout"). ### Read surface @@ -1282,10 +1360,17 @@ The GUI-facing records hold no key material and no binary, so they survive - `LibrarySnapshot`: `{ albums, photos, takenAt }`. `lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first, -photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange` -(`albumsChanged`, `photosChanged`, `fileIDsRemoved`, `albumIDsRemoved`, -`refreshedAt`) whenever a refresh alters the projection, and returns -`{ unsubscribe }`; a refresh that changes nothing delivers nothing. +photos newest first). `lib.subscribe({ onChange })` returns `{ unsubscribe }`. +It calls `onChange` with a `LibraryChange` whenever a refresh or +`lib.refreshCollection()` changes these records. A refresh or +`lib.refreshCollection()` that changes nothing does not call it. A +`LibraryChange` holds: + +- `albumsChanged` +- `photosChanged` +- `fileIDsRemoved` +- `albumIDsRemoved` +- `refreshedAt` ### Thumbnails, ML search, and backup @@ -1301,29 +1386,44 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange` `SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first, default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes a query vector the caller produced elsewhere. -- `await lib.backup(opts?)` → `BackupResult`. It takes the lock in the download - directory, and fails at once with an error whose `code` is `ELOCKED` while - another backup of it runs. A caller can instead take the lock itself, before - it opens its library, as `quak backup` does: `await lockBackupDirectory(dir)` - takes it, failing the same way, and returns a `BackupLock`, whose `release()` - removes the lock unless it was lost and whose `lost` holds why it was lost. - The caller passes it to the backup as `lock`, and releases it once the backup - ends. It waits for a refresh as `fresh()` does, puts every in-scope original - not already at its save path there as `photo.download()` does (and, with - `includeThumbnails`, fetches thumbnails) through the content cache, waits for - an ML data fetch, and rebuilds the on-disk backup tree with a durable failure - ledger. Each file's JSON holds its ML data and keywords. An image's JSON also - holds the EXIF and XMP of its original, and its dimensions if it is a JPEG. A - live photo's JSON holds these for its image. A video's JSON holds no EXIF, XMP - or dimensions. A fetched original is written straight to its save path and not - into the cache, which then counts it as present; one the cache already held is - copied from there. `BackupOptions`: `downloadDirectory` (falls back to the - library's), `includeOriginals` (default `true`), `includeThumbnails` (default - `false`), `onlyAlbumNames`, `verify` (default `false`), `onProgress`, and - `lock`. See Backup layout above for the tree it writes, and for the albums - `BackupResult` lists in `undecryptableAlbums` and `failedAlbums`. Whether the - backup took the lock or its caller did, once the lock is lost the backup - rejects with `lost the lock on : ` before it starts another file. +- `await lib.backup(opts?)` → `BackupResult`. It takes the lock, `backup.lock` + in the download directory. It fails at once with an error whose `code` is + `ELOCKED` while another backup of that directory runs. A caller can instead + take the lock itself before it opens its library, as `quak backup` does. + `await lockBackupDirectory(dir)` takes it, failing the same way, and returns a + `BackupLock`. Its `release()` removes the lock unless it was lost, and its + `lost` holds why it was lost. The caller passes it to the backup as `lock`, + and releases it once the backup ends. + + The backup waits for a refresh as `fresh()` does. It then fetches each + in-scope original that is not yet at its save path, as `photo.download()` + does, through the content cache. With `includeThumbnails` it also fetches + thumbnails. It waits for an ML data fetch, then rebuilds the backup tree. It + records each file that failed in `failures.json`. + + Each file's JSON holds its ML data and keywords. An image's JSON also holds + the EXIF and XMP of its original, and its dimensions if it is a JPEG. A live + photo's JSON holds these for its image. A video's JSON holds no EXIF, XMP or + dimensions. + + A fetched original is written straight to its save path, not into the cache. + The cache then counts it as present. An original that the cache already held + is copied from there. + + `BackupOptions` holds: + - `downloadDirectory`, which falls back to the library's + - `includeOriginals`, `true` by default + - `includeThumbnails`, `false` by default + - `onlyAlbumNames` + - `verify`, `false` by default + - `onProgress` + - `lock` + + "Backup layout" above describes the tree that the backup writes. It also + describes the albums that `BackupResult` lists in `undecryptableAlbums` and + `failedAlbums`. Once the lock is lost, the backup rejects with + `lost the lock on : ` before it starts another file. This holds + whether the backup or its caller took the lock. ### Request pools diff --git a/TODO.md b/TODO.md index 99ae803..6596cf3 100644 --- a/TODO.md +++ b/TODO.md @@ -26,6 +26,21 @@ declares one. # Completed Steps +- 2026-10-25: `quak files COLLECTION` fetches the files of that collection alone + (https://git.eeqj.de/sneak/quak/issues/217). It used to wait for a refresh, + which on an empty cache fetched the files of every collection first. It now + reads the collection list through `lib.freshCollections()`, then brings that + one collection up to date through the new `lib.refreshCollection()`. That + stores and saves its files as a refresh would, and leaves the refresh cursor + where it was. When the stored copy is current, it asks for none of the files. + It never runs at the same time as a refresh. Once a refresh has received the + account's collections, it stores no collection the cache does not hold, and + `quak files COLLECTION` waits for a refresh instead. A refresh from a cursor + of 0 replaces all the cache holds, so collections keep the server's order. A + library opened on a cache that only `lib.refreshCollection()` wrote awaits its + first refresh. `quak files` with no argument still waits for a refresh. The + store lists what did not decrypt by collection ID and file ID. + - 2026-10-22: The JSON `quak backup` and `quak backup-metadata` write for a file holds `keywords` in place of the CLIP embedding in its ML data (https://git.eeqj.de/sneak/quak/issues/218). The keywords are the titles of diff --git a/src/cli-commands.ts b/src/cli-commands.ts index 52f1b05..7c4d205 100644 --- a/src/cli-commands.ts +++ b/src/cli-commands.ts @@ -128,12 +128,14 @@ const readLibraryClient = (client: Client): LibraryClient => ({ contentSource: () => client.contentSource(), }); -// Open a library for a single point command: the aggressive background precache -// (issue #48) is off — a one-shot `collections` or `get` must not start -// downloading the whole account — and no refresh starts as it opens or on a -// timer: each command asks for the one it needs, and `collections` needs none -// (issue #210). While a refresh runs, each collection whose files it has -// fetched is logged, so a large account shows progress rather than nothing. +// Open a library for a single point command. The aggressive background +// precache (issue #48) is off, because a one-shot `collections` or `get` must +// not start downloading the whole account. No refresh starts as it opens or on +// a timer. Each command asks for the refresh it needs. `collections` needs none +// (issue #210). `files COLLECTION` needs one only when `refreshCollection()` +// resolves to `false` (issue #217). While a refresh runs, +// each collection whose files it has fetched is logged, so a large account +// shows progress rather than nothing. const openReadLibrary = (ctx: CliContext, client: Client): Promise => Library.open({ client: readLibraryClient(client), diff --git a/src/cli-read.ts b/src/cli-read.ts index 70b1f21..af6b72c 100644 --- a/src/cli-read.ts +++ b/src/cli-read.ts @@ -2,33 +2,42 @@ // // `collections`, `files`, `get`, and `get-thumb` must answer for // the account's state at the moment the command runs, not for whatever the -// local cache last happened to hold (owner amendment, issue #36). `files`, -// `get` and `get-thumb` therefore force a server round-trip through -// `Library.fresh()` and only then read — so a collection, file, or metadata -// change made elsewhere is visible. `collections` asks the library for the -// collection list alone through `Library.freshCollections()` (issue #210): a -// refresh would also fetch the files of every collection, which on a large -// account takes long and which `collections` does not print. +// local cache last happened to hold (owner amendment, issue #36). `files` +// without a collection ID, `get` and `get-thumb` therefore force a server +// round-trip through `Library.fresh()`, and only then read. A collection, file, +// or metadata change made elsewhere is then visible. `collections` asks the +// library for the collection list alone through `Library.freshCollections()` +// (issue #210). A refresh would also fetch the files of every collection, which +// `collections` does not print, and on a large account that takes long. +// `files COLLECTION` reads the collection list the same way, for the same +// reason. It then brings that one collection up to date through +// `Library.refreshCollection()` (issue #217), and fetches no other collection's +// files. Once a refresh has received the account's collections, +// `refreshCollection()` stores no collection the library does not hold. +// `files COLLECTION` then forces a refresh instead. // -// `collections` lists in the server's order and `files` in the library's own -// enumeration order (`listFiles()`), the orders the pre-library CLI printed, -// rather than the `albums`/`photos` projection's newest-first order, which -// re-sorts the rows. The field values still come from each record's raw -// metadata via `cli-output.ts`. `files` without a collection ID lists every -// file in the account once (issue #211): the library keeps a record of a file -// for each collection that holds it, and only the first collection's record is -// listed. +// `collections` lists in the server's order. `files` lists in the library's +// own enumeration order: `listCollections()`, then `listFiles()` for each. +// Those are the orders the pre-library CLI printed. The `albums`/`photos` +// projection would re-sort the rows newest first. After a refresh, the +// library holds collections in the order the server first sent them. The +// field values still come from each record's raw metadata via +// `cli-output.ts`. `files` without a collection ID lists every file in the +// account once (issue #211). +// The library keeps a record of a file for each collection that holds it, and +// only the first collection's record is listed. import type { Collection, EnteFile } from "./model/types.js"; import type { Photo, PhotosAPI } from "./library/index.js"; // The slice of `Library` these helpers read. `Library` satisfies it -// structurally; a test can drive them with a stand-in that records the -// `fresh()` and `freshCollections()` calls and serves records in a known -// enumeration order. +// structurally. A test can drive them with a stand-in that records the +// `fresh()`, `freshCollections()` and `refreshCollection()` calls. The +// stand-in serves records in a known enumeration order. export interface FreshReadLibrary { fresh(): Promise; freshCollections(): Promise; + refreshCollection(collection: Collection): Promise; getCollection(id: number): Collection | undefined; listCollections(): Collection[]; listFiles(collectionID: number): EnteFile[]; @@ -42,21 +51,29 @@ export const freshCollections = ( lib: FreshReadLibrary, ): Promise => lib.freshCollections(); -// The files of one collection, current as of a forced refresh, in enumeration -// order. `undefined` (not an empty list) when the collection does not exist, so -// the caller can tell "no such collection" from "an empty collection". Without -// a collection ID, every file in the account, each once, as held by the first -// collection that holds it. +// The files of one collection, in enumeration order. `undefined` (not an empty +// list) when the collection does not exist, so the caller can tell "no such +// collection" from "an empty collection". They are current as of one +// collections request and the requests that bring that collection alone up to +// date. When `refreshCollection()` resolves to `false`, they are current as of +// a forced refresh instead. Without a collection ID, every file in the account, +// current as of a forced refresh. Each file is listed once, as held by the +// first collection that holds it. export const freshFiles = async ( lib: FreshReadLibrary, collectionID?: number, ): Promise => { - await lib.fresh(); if (collectionID !== undefined) { - return lib.getCollection(collectionID) - ? lib.listFiles(collectionID) - : undefined; + const collections = await lib.freshCollections(); + const collection = collections.find((c) => c.id === collectionID); + if (!collection) return undefined; + if (!(await lib.refreshCollection(collection))) { + await lib.fresh(); + if (!lib.getCollection(collectionID)) return undefined; + } + return lib.listFiles(collectionID); } + await lib.fresh(); const listed = new Set(); const files: EnteFile[] = []; for (const collection of lib.listCollections()) { diff --git a/src/library/index.ts b/src/library/index.ts index a3bed63..8ea68dc 100644 --- a/src/library/index.ts +++ b/src/library/index.ts @@ -1,28 +1,37 @@ // The library surface over the local cache. // // `Library.open()` loads the on-disk metadata store (issue #41), then starts -// the refresh loop. When the cache loaded empty it awaits the first refresh, -// so the library never opens onto an empty store it could have filled; when an -// existing copy loaded, that first refresh runs in the background and `open()` -// returns as soon as the cached data is ready to serve — a slow or unreachable -// server no longer stalls opening. A background timer then refreshes every -// `refreshIntervalSeconds`. Every default read is answered from RAM — no -// default read touches the network. There is deliberately no `sync()`, no -// `refresh()`, no `serverReachable` flag, and no "before each read" mode -// (design #36). +// the refresh loop. When the stored cursor is 0, `open()` awaits the first +// refresh. No refresh has then stored any collection: the cache is empty, or +// holds only what `refreshCollection()` stored. The library must not serve +// that as the whole account. Otherwise that first refresh runs in the +// background. `open()` returns as soon as the cached data is ready to serve, +// so a slow or unreachable server does not stall opening. A background timer +// then takes a turn `refreshIntervalSeconds` after its last turn ends. Each +// turn starts a refresh, unless a refresh or `refreshCollection()` is already +// running. Every default read is answered from RAM — no default read touches +// the network. There is deliberately no `sync()`, no `refresh()`, no +// `serverReachable` flag, and no "before each read" mode (design #36). // -// `fresh()` is the one exception (issue #75, an owner amendment to #36): it -// forces a refresh, awaits it, and only then hands back the read namespaces, so -// a caller that needs server-current data can ask for it. Concurrent `fresh()` -// calls coalesce onto one in-flight refresh, and a refresh that fails rejects -// the caller (the default reads stay silent and serve the last good copy). The -// default methods and the background loop are unchanged. +// `fresh()` is an exception (issue #75, an owner amendment to #36). It awaits +// a refresh, and only then hands back the read namespaces. A caller that needs +// server-current data uses it. Concurrent `fresh()` calls coalesce onto one +// in-flight refresh. A refresh that fails rejects the caller. The default +// reads stay silent and serve the last good copy. The default methods and the +// background loop are unchanged. // // `freshCollections()` is the fresh read of the collection list alone (issue -// #210): one collections request, no collection's files, and nothing stored, -// for a caller such as `quak collections` that needs no more. A caller that -// asks for each refresh it needs can also open the library with -// `refreshOnOpen: false`, so that `open()` starts none. +// #210). It sends one collections request and stores nothing. +// `quak collections` uses it. `refreshCollection()` brings one collection +// from that list up to date in the store (issue #217). It fetches that +// collection's files alone, as a refresh would. `quak files COLLECTION` uses +// it. It never runs at the same time as a refresh. Once the stored cursor is +// past 0, it only updates a collection the store holds, and only a refresh +// adds one. A refresh from a stored cursor of 0 replaces all the store holds. +// So after a refresh, the store holds collections in the order the server +// first sent them. A caller that asks for +// each refresh it needs can also open the library with `refreshOnOpen: false`, +// so that `open()` starts none. // // A refresh reports through `onProgress` each collection whose files it has // fetched, so a large account, whose first refresh fetches the files of every @@ -31,10 +40,12 @@ // A refresh stages all of its network work first and only mutates the store // once every fetch has succeeded. A refresh that fails partway therefore never // becomes visible to reads: the last good snapshot stays in place, and the -// failure surfaces through `onProgress` and `status()` instead. A commit that -// mutates RAM but then fails to persist keeps `status().lastError` set and the -// store marked unsaved until a later save actually lands, so a stuck disk is -// never masked by a subsequent empty refresh. +// failure surfaces through `onProgress` and `status()` instead. A save can +// still fail after the store has changed in RAM. The store then stays marked +// unsaved until a later save lands. Until then, every `commit()` retries the +// save. A refresh whose save fails sets `status().lastError`, which stays set +// until a later refresh succeeds. `refreshCollection()` never sets or clears +// it. So a stuck disk is never hidden by a later refresh that changes nothing. import { rm } from "node:fs/promises"; import { join, resolve } from "node:path"; @@ -159,21 +170,23 @@ export interface LibraryClient { payloads: Map; errors: Map; }>; - // The byte source for the on-disk content cache. Optional so a mock client - // that only serves metadata still satisfies the interface; when absent (and - // no explicit `contentSource` is passed to `open`) the content cache is - // disabled and `Photo.original`/`thumbnail` and `thumbnails.ensure` throw. + // The byte source for the on-disk content cache. Optional, so a mock + // client that only serves metadata still satisfies the interface. Without + // it, and with no `contentSource` passed to `open`, the content cache is + // disabled. `Photo.original`/`thumbnail` and `thumbnails.ensure` then + // throw. contentSource?(): ContentSource; } -// A progress event for one unit of background work. A metadata "refresh" or an -// ML "fetchMLData" pass each fire "started" before their network work and then -// exactly one of "done" or "failed"; "failed" carries the error message and -// an ML "done" reports how many payloads it stored and, in `mlDataErrors`, -// each file whose ML data did not decrypt, gunzip or parse, with the reason. -// In between, a refresh fires "progress" after each collection whose files it -// fetched: `fetched` is how many collections it has fetched the files of so -// far, `total` how many it will. The precache fills +// A progress event for one unit of background work. A metadata "refresh" and +// an ML "fetchMLData" pass each fire "started" before their network work. Each +// then fires exactly one of "done" or "failed". A "failed" event carries the +// error message. An ML "done" reports in `fetched` how many payloads it +// stored. Its `mlDataErrors` lists each file whose ML data did not decrypt, +// gunzip or parse, with the reason. In between, a refresh fires "progress" +// after each collection whose files it fetched. Its `fetched` is how many +// collections it has fetched the files of so far, and `total` how many it +// will. The precache fills // ("precacheThumbnails"/"precacheOriginals", #48) fire "started"/"done" around // each sweep that has work, "done" reporting the count newly cached. export interface RefreshEvent { @@ -201,9 +214,10 @@ export interface LibraryOptions { // directory at open. The content cache treats an original already stored // at its save path as present. downloadDirectory?: string; - // How long after one refresh ends the background timer starts the next. - // `Infinity` starts none: the library then refreshes only as it opens and - // when `fresh()` or `backup()` asks. + // How long after its last turn ends the background timer takes the next. + // A turn starts a refresh unless a refresh or `refreshCollection()` is + // running. With `Infinity` the timer takes no turn: the library then + // refreshes only as it opens and when `fresh()` or `backup()` asks. refreshIntervalSeconds?: number; // Whether `open()` starts a refresh. Defaults to true. With false, `open()` // returns at once onto what the cache holds, empty or not, and a refresh @@ -242,8 +256,9 @@ export interface LibraryStatus { // Wall-clock ms of the last refresh that succeeded, or undefined if none // has yet. lastRefreshAt?: number; - // The message from the most recent refresh, set only while that refresh - // failed; cleared by the next success. + // The error of the most recent refresh, set only when it failed. The next + // refresh that succeeds clears it. `refreshCollection()` never sets or + // clears it. lastError?: string; // Wall-clock ms of the last ML fetch pass that succeeded, or undefined if // none has yet (or ML fetching is disabled). @@ -306,12 +321,17 @@ export class Library { private readonly precache?: Precache; private timer?: ReturnType; - // The in-flight refresh cycle, or undefined when none runs. One slot serves - // both paths: the background loop skips when it is set, and a fresh read - // (issue #75) coalesces onto it or starts one. The promise carries the - // cycle's real outcome (it rejects on failure); the background loop ignores - // that, a fresh read propagates it. + // The in-flight refresh cycle or `refreshCollection()`, or undefined when + // neither runs. One slot holds both, so they never run at the same time. + // The background loop skips while it is set. A fresh read (issue #75) + // coalesces onto a running refresh. It waits for a running + // `refreshCollection()` to end, and then joins or starts a refresh. The + // promise carries the real outcome: it rejects on failure. The background + // loop ignores a failure. A fresh read passes on the failure of the + // refresh it joins or starts, and `refreshCollection()` passes on its own. private cycle?: Promise; + // Whether `cycle` is a refresh cycle rather than a `refreshCollection()`. + private cycleIsRefresh = false; // Guards the ML fetch pass so a slow backfill never runs twice at once; a // refresh whose pass is still running kicks nothing new. Holds the running // pass, so `close()` and `backup()` can wait for it. It rejects when the @@ -324,19 +344,20 @@ export class Library { private lastMLError?: string; // For each file whose ML data the last ML fetch pass that succeeded got // but could not decrypt, gunzip or parse, why not, by fileID. A pass - // gathers its own and puts them here when it ends without failing, so - // `status()` and `backup()` never see the list of a pass that is still - // running or that failed. Kept in RAM only, never in the cache. + // gathers its own and puts them here only when it ends without failing. + // So `status()` and `backup()` never see the list of a pass still running + // or one that failed. Kept in RAM only, never in the cache. private mlDataErrors = new Map(); - // The plain-record projection as of the last refresh, and the GUI change - // subscribers. A refresh that alters the projection notifies each with the - // delta; `lastRecords` is kept current every refresh so a subscriber that - // joins later diffs against the state its own `snapshot()` already returned. + // The plain-record projection as of the last store change, and the GUI + // change subscribers. A refresh or `refreshCollection()` that alters the + // projection notifies each subscriber with the delta. `lastRecords` + // advances on each `commit()` that changes the store. A subscriber that + // joins later then diffs against the state its own `snapshot()` returned. private readonly subscribers = new Set<(change: LibraryChange) => void>(); private lastRecords: DerivedRecords; // RAM holds changes disk has not yet accepted (an earlier save failed). - // Cleared only when a save actually succeeds; keeps the store trying to - // persist and the failure visible in `status()` until then. + // Cleared only when a save succeeds. Until then, every `commit()` retries + // the save. private unsaved = false; private constructor(args: { @@ -366,7 +387,7 @@ export class Library { this.lastRecords = this.deriveNow(); // The read namespaces derive fresh from the store on each call, so they - // always reflect the latest refresh. + // always reflect what it holds now. const derive = (): DerivedRecords => this.deriveNow(); const root = this.downloadDirectory; const saves: SavePathLookup = { @@ -393,14 +414,16 @@ export class Library { this.mldata = makeMLDataAPI(() => this.mlStore); } - // Load the cache and start the refresh loop. With an empty cache the first - // refresh is awaited, so `open()` resolves onto populated data whenever the - // server is reachable; that awaited refresh may still fail, and the library - // then opens empty with the failure recorded in `status()`. With an - // existing cache the first refresh runs in the background and `open()` - // returns as soon as the cached data is ready — an unreachable server does - // not block opening. With `refreshOnOpen: false` there is no first - // refresh: `open()` returns onto the cache as it is. + // Load the cache and start the refresh loop. When the stored cursor is 0, + // the first refresh is awaited. Such a cache is empty, or holds only what + // `refreshCollection()` stored. `open()` then resolves onto the whole + // account whenever the server is reachable. If that refresh fails, the + // library opens onto what the cache holds, with the failure recorded in + // `status()`. With any other cache the first refresh runs in the + // background, and `open()` returns as soon as the cached data is ready. + // An unreachable server then does not block opening. With + // `refreshOnOpen: false` there is no first refresh: `open()` returns onto + // the cache as it is. static async open(opts: LibraryOptions): Promise { const { userID } = opts.client.whoami(); const cacheDirectory = @@ -416,8 +439,8 @@ export class Library { let store = await MetadataStore.load(metadataPath); // A cache directory given explicitly can hold another account's cache. // Its records and cursor are not this account's, so delete it and the - // ML data beside it and start empty. A user ID of 0 means the cache - // was never refreshed and so holds nothing to discard. + // ML data beside it and start empty. A user ID of 0 means nothing was + // ever stored there, so there is nothing to discard. if (store.userID !== 0 && store.userID !== userID) { await rm(metadataPath, { force: true }); await rm(join(cacheDirectory, "mldata"), { @@ -440,10 +463,11 @@ export class Library { ? await MLDataStore.open(join(cacheDirectory, "mldata")) : undefined; - // Build the content cache from an explicit source or the client's own, - // and take its record of what is already cached (and reap orphan temp - // files) before the first projection, so cached paths are present from - // the start and the first refresh raises no spurious path-change diff. + // Build the content cache from an explicit source or the client's own. + // Take its record of what is already cached, and reap orphan temp + // files, before the first projection. Cached paths are then present + // from the start, and the first `commit()` raises no spurious + // path-change diff. const source = opts.contentSource ?? opts.client.contentSource?.(); let cache: ContentCache | undefined; let precache: Precache | undefined; @@ -497,22 +521,26 @@ export class Library { precache, }); - // Start filling from whatever the loaded store already holds; each - // refresh below re-kicks with the new files (and retries any that - // failed). An empty store starts empty here and fills after its first - // refresh. + // Start filling from whatever the loaded store already holds. Each + // later `commit()` re-kicks the fills with its new files, and retries + // any that failed. An empty store starts empty here, and fills once a + // `commit()` stores files. precache?.start(); if (opts.refreshOnOpen === false) { // The caller asks for each refresh it needs. lib.scheduleNext(); - } else if (store.loadedFromDisk) { - // An existing copy already answers reads; refresh in the background - // and start the interval once that first cycle settles. + } else if (store.collectionsSinceTime > 0) { + // The stored cursor has moved, so a refresh has stored the + // account's collections and the cache's copy answers reads. + // Refresh in the background, and start the interval once that + // first cycle settles. void lib.runRefresh().then(() => lib.scheduleNext()); } else { - // Nothing was cached: wait for the first refresh to fill the store - // (or fail) rather than resolve onto an empty library. + // The stored cursor is still 0, so no refresh has stored any + // collection: the cache is empty, or holds only what + // `refreshCollection()` stored. Wait for the first refresh to fill + // the store (or fail) rather than resolve onto part of the account. await lib.runRefresh(); lib.scheduleNext(); } @@ -550,9 +578,9 @@ export class Library { return snapshotFrom(this.deriveNow(), Date.now()); } - // Deliver a `LibraryChange` whenever a refresh alters the projection. A - // refresh that changes nothing delivers nothing. The returned handle's - // `unsubscribe` stops delivery. + // Deliver a `LibraryChange` whenever a refresh or `refreshCollection()` + // alters the projection. One that changes nothing delivers nothing. The + // returned handle's `unsubscribe` stops delivery. subscribe(args: { onChange: (change: LibraryChange) => void }): { unsubscribe: () => void; } { @@ -595,12 +623,13 @@ export class Library { }; } - // Fresh reads (issue #75, owner amendment to design #36). Force a refresh, - // wait for it to complete and persist, then hand back the same - // `albums`/`photos`/`timeline` namespaces — now guaranteed to reflect a - // completed server round-trip. Concurrent calls coalesce onto one refresh; - // a refresh that fails rejects here, where the default namespaces would - // instead stay silent and serve the last good copy. + // Fresh reads (issue #75, owner amendment to design #36). Wait for a + // refresh to complete and persist, then hand back the same + // `albums`/`photos`/`timeline` namespaces. They then reflect a completed + // server round-trip. A running `refreshCollection()` is waited for first. + // Concurrent calls coalesce onto one refresh. A refresh that fails rejects + // here, where the default namespaces would stay silent and serve the last + // good copy. async fresh(): Promise { await this.refreshNow(); return { @@ -613,10 +642,10 @@ export class Library { // The fresh read of the collection list alone (issue #210): every live // collection that decrypts, in the server's order, from one collections // request. Unlike a refresh, it fetches no collection's files, which on a - // large account take long. Nothing is stored: a refresh fetches a - // collection's files only when the collection is new or has changed since - // the stored copy, so a collection stored here without its files would - // never have them fetched. + // large account take long. It stores nothing. A refresh from a stored + // cursor past 0 fetches a collection's files only when the collection is + // new or has changed since the stored copy. So does `refreshCollection()`. + // Neither would fetch the files of a collection stored here without them. async freshCollections(): Promise { if (this.closed) throw new Error("the library is closed"); const { collections } = await this.client.collectionsSince({ @@ -625,19 +654,44 @@ export class Library { return collections; } + // Bring one collection up to date in the store (issue #217), given its + // record from `freshCollections()`. It does for that collection what a + // refresh does for a changed one. It fetches no other collection's files. + // It resolves to whether the store then holds the collection. + // + // It adds a collection the store does not hold only while the stored + // cursor is 0, before any refresh has received a collection. After that, + // it stores nothing for such a collection and resolves to false. Only a + // refresh may then add it, after the collections the store holds. + // + // It runs in the same slot as a refresh, so the two never run at the same + // time. It first waits for a running refresh or `refreshCollection()` to + // end. While its own work runs, a fresh read or `backup()` waits for it, + // and the background loop skips its turn. + async refreshCollection(collection: Collection): Promise { + while (this.cycle) await this.cycle.catch(() => {}); + if (this.closed) throw new Error("the library is closed"); + let held = false; + await this.startCycle(false, async () => { + held = await this.refreshCollectionOnce(collection); + }); + return held; + } + // Back up every in-scope file to `opts.downloadDirectory`, or else the // library's, each original at its save path, with a durable failure - // ledger (issue #51). Takes the lock in that directory first, unless the - // caller passes the lock it took as `opts.lock`, failing at once while - // another backup of it runs, and before it starts another file once the - // lock is lost (see `lockBackupDirectory`). Waits for a - // completed refresh, as `fresh()` does, joining one already running, and - // rejects before touching any file but the lock when it fails. Then puts - // pending originals at their save paths as `Photo.download()` does (and - // optional thumbnails) through the content cache and pools, waits for an - // ML data fetch, and rebuilds the derived symlink/JSON views from the - // model, each file's JSON with its ML data, beside an `account.json` with - // the account's email and user ID. + // ledger (issue #51). It first takes the lock in that directory, unless + // the caller passes the lock it took as `opts.lock`. Taking it fails at + // once while another backup of that directory runs. Once the lock is lost, + // the backup fails before it starts another file (see + // `lockBackupDirectory`). It waits for a completed refresh as `fresh()` + // does, joining one already running. When that refresh fails, it rejects + // before touching any file but the lock. It then puts pending originals + // at their save paths as `Photo.download()` does, through the content + // cache and pools. With `includeThumbnails` it also fetches thumbnails. It + // waits for an ML data fetch. It rebuilds the derived symlink/JSON views + // from the model, each file's JSON with its ML data. An `account.json` + // beside them holds the account's email and user ID. // Throws before any network work when no content cache backs the // originals it must fetch. backup(opts?: BackupOptions): Promise { @@ -674,12 +728,15 @@ export class Library { ); } - // Stop the background timer. Idempotent. An in-flight refresh is left to - // finish; it will not schedule another cycle once closed. The returned - // promise resolves once that refresh (including its cache write), the ML - // fetch pass and the precache fetches already running have all finished, - // so a caller can then remove the cache directory. A refresh failure is - // reported through `status()`, not thrown here. + // Stop the background timer. Idempotent. A refresh or `refreshCollection()` + // in flight is left to finish, and nothing starts after it. The returned + // promise resolves once all of these have finished: + // - that refresh or `refreshCollection()`, with its cache write; + // - the ML fetch pass; + // - the precache fetches already running. + // A caller can then remove the cache directory. A refresh failure is + // reported through `status()`, and a `refreshCollection()` failure to its + // own caller. Neither is thrown here. async close(): Promise { this.closed = true; const precacheClosed = this.precache?.close(); @@ -702,33 +759,41 @@ export class Library { this.timer.unref?.(); } - // The background loop's refresh: run a cycle unless one is already in flight - // (or the library is closed), and never let a failure escape — the - // background path reports errors through `status()`/`onProgress`, it does - // not throw. Resolves once the cycle it started (or skipped past) settles. + // The background loop's refresh: run a cycle unless a refresh or + // `refreshCollection()` is in flight (or the library is closed), and never + // let a failure escape. The background path reports errors through + // `status()`/`onProgress`; it does not throw. It resolves once the cycle + // it started settles, or at once when it starts none. private runRefresh(): Promise { if (this.closed || this.cycle) return Promise.resolve(); - return this.startCycle().catch(() => {}); + return this.startCycle(true, () => this.refreshCycle()).catch(() => {}); } - // A fresh read's refresh (issue #75): force a cycle and await it, rejecting - // if it fails. Concurrent fresh reads coalesce onto the one in-flight cycle - // — the background loop's included — so they never fan out into redundant - // server round-trips. - private refreshNow(): Promise { - if (this.closed) { - return Promise.reject(new Error("the library is closed")); + // A fresh read's refresh (issue #75): join the running refresh or start + // one, and await it, rejecting if it fails. Concurrent fresh reads + // coalesce onto the one in-flight cycle — the background loop's included — + // so they never fan out into redundant server round-trips. A running + // `refreshCollection()` is waited for first. + private async refreshNow(): Promise { + while (this.cycle && !this.cycleIsRefresh) { + await this.cycle.catch(() => {}); } - return this.cycle ?? this.startCycle(); + if (this.closed) throw new Error("the library is closed"); + return this.cycle ?? this.startCycle(true, () => this.refreshCycle()); } - // Start one refresh cycle and record it as the in-flight cycle so every - // caller coalesces onto it. The returned promise carries the cycle's real - // outcome; each caller attaches the handling its own path needs, and the - // slot is cleared once the cycle settles. - private startCycle(): Promise { - const cycle = this.refreshCycle(); + // Start `work` and record it in the in-flight slot, so every other caller + // sees it. `isRefresh` says whether it is a refresh cycle, which a fresh + // read can coalesce onto. The returned promise carries the real outcome. + // Each caller attaches the handling its own path needs, and the slot is + // cleared once the work settles. + private startCycle( + isRefresh: boolean, + work: () => Promise, + ): Promise { + const cycle = work(); this.cycle = cycle; + this.cycleIsRefresh = isRefresh; void cycle.then( () => { if (this.cycle === cycle) this.cycle = undefined; @@ -751,9 +816,9 @@ export class Library { this.lastError = undefined; this.emit({ operation: "refresh", status: "done" }); // Backfill ML data for the files this refresh knows about. It runs - // outside the refresh's success/failure so a fetch or disk problem - // there never marks the metadata refresh failed, and it is not - // awaited so it never stalls the refresh interval. Its failure is + // outside the refresh's success or failure, so a fetch or disk + // problem there never marks the metadata refresh failed. It is not + // awaited, so it never stalls the refresh interval. Its failure is // reported through `status()` and `onProgress`. void this.fetchMLDataNow().catch(() => {}); } catch (err) { @@ -766,11 +831,14 @@ export class Library { // Fetch every change since the stored cursor, then commit. All network // reads happen before any store mutation, so a fetch that throws leaves the - // store untouched and the previous snapshot intact. A collection or file - // that does not decrypt is left out instead, as if deleted, with the reason - // kept in the store: the cursor moves past it, so no later refresh sees it - // again until the server's copy changes. + // store untouched and the previous snapshot intact. + // + // From a stored cursor of 0, the server lists every live collection. The + // refresh then replaces all the store holds, and counts every collection + // as new. Only `refreshCollection()` can have stored anything before it. + // So the store ends as it would had `refreshCollection()` never run. private async refreshOnce(): Promise { + const replace = this.store.collectionsSinceTime === 0; const page = await this.client.collectionsSince({ sinceTime: this.store.collectionsSinceTime, }); @@ -778,16 +846,19 @@ export class Library { // Stage per-collection file diffs. A collection's files are // re-enumerated only when its updationTime has advanced past the cached // copy; an unchanged album's file list cannot have changed. New - // collections enumerate from the beginning of time. Each collection + // collections enumerate from the beginning of time. When the refresh + // replaces the store, every collection counts as new. Each collection // whose files are fetched is reported, with how many there are to // fetch. + const storedCopy = (id: number): Collection | undefined => + replace ? undefined : this.store.getCollection(id); const changedCollections = page.collections.filter((collection) => { - const known = this.store.getCollection(collection.id); + const known = storedCopy(collection.id); return !known || collection.updationTime > known.updationTime; }); const filePages: { collectionID: number; page: FilesPage }[] = []; for (const collection of changedCollections) { - const known = this.store.getCollection(collection.id); + const known = storedCopy(collection.id); const filePage = await this.client.filesSince({ collectionID: collection.id, collectionKey: collection.key, @@ -802,8 +873,56 @@ export class Library { }); } - // Network work done; commit to the store and persist only if something - // actually changed. + await this.commit(page, filePages, replace); + } + + // `refreshCollection()`'s work, run in the slot: fetch the files of one + // collection changed since the stored copy, then commit. It resolves to + // whether the store then holds the collection. + private async refreshCollectionOnce( + collection: Collection, + ): Promise { + const known = this.store.getCollection(collection.id); + // A refresh has received the account's collections, and this one is + // not stored. It is new since then, or a refresh dropped it. Leave it + // to the next refresh, which adds new collections after the stored + // ones, in the server's order. + if (!known && this.store.collectionsSinceTime > 0) return false; + // The stored copy is as new or newer, so there is nothing to fetch. + if (known && collection.updationTime <= known.updationTime) return true; + const filePage = await this.client.filesSince({ + collectionID: collection.id, + collectionKey: collection.key, + sinceTime: known ? known.updationTime : 0, + }); + // The stored cursor stays where it is, so the next refresh still + // fetches the files of every other collection that changed. + await this.commit( + { + collections: [collection], + deleted: [], + cursor: this.store.collectionsSinceTime, + }, + [{ collectionID: collection.id, page: filePage }], + false, + ); + return true; + } + + // Commit a collections page and the file pages fetched for its collections + // to the store. With `replace`, the page lists every live collection, so + // the commit first deletes every stored collection. A collection or file + // that does not decrypt is left out, as if deleted. The reason is kept in + // the store. Until the server's copy changes, only a refresh from a stored + // cursor of 0 fetches it again. Last, save the store when it holds changes + // not yet saved. + private async commit( + page: CollectionsPage, + filePages: { collectionID: number; page: FilesPage }[], + replace: boolean, + ): Promise { + // The network work is done, so change the store now. `changed` says + // whether this commit changed anything. let changed = false; if (this.store.userID !== this.userID) { @@ -811,6 +930,13 @@ export class Library { changed = true; } + if (replace) { + for (const { id } of this.store.listCollections()) { + this.store.deleteCollection(id); + changed = true; + } + } + for (const id of page.deleted) { if (this.store.getCollection(id)) { this.store.deleteCollection(id); @@ -864,10 +990,10 @@ export class Library { if (changed) this.unsaved = true; // Reproject and notify subscribers of the delta. This tracks RAM (what - // reads see), so it fires whether or not the save below succeeds; a - // save failure surfaces separately through `status().lastError`. - // `lastRecords` advances every changed refresh so the next diff is - // against current state. + // reads see), so it fires whether or not the save below succeeds. A + // failed save rejects the caller, and a refresh also records it in + // `status().lastError`. `lastRecords` advances on every commit that + // changes something, so the next diff is against current state. if (changed) { const next = this.deriveNow(); if (this.subscribers.size > 0) { @@ -878,16 +1004,16 @@ export class Library { // Recompute the fill orders and pinned set against the new library. this.precache?.update(next); } - // Re-kick the fills every cycle: a finished sweep starts afresh to pick - // up new files and retry any that failed, and a running one is left be. + // Re-kick the fills on every commit. A finished sweep starts afresh to + // pick up new files and retry any that failed. A running one is left + // be. this.precache?.start(); // Persist whenever RAM holds changes disk has not accepted — including // changes an earlier cycle staged whose save failed. `unsaved` clears - // only once a save lands, so a save failure both stays visible through - // `status().lastError` (the throw below records it) and keeps being - // retried, instead of a later empty refresh silently clearing it while - // the on-disk cache is still behind RAM. + // only once a save lands, so a later commit retries a failed save even + // when it changes nothing. A failed save rejects the caller, and a + // refresh also records it in `status().lastError`. if (this.unsaved) { await this.store.save(); this.unsaved = false; @@ -903,13 +1029,14 @@ export class Library { return this.mlFetch; } - // One ML fetch pass: fetch, decrypt and store the ML data for every file - // the store knows about that is not cached (or whose `updationTime` has - // advanced), through the metadata pool, and update the CLIP index. For a - // file whose ML data does not decrypt, gunzip or parse, it gathers the - // reason instead, and puts the reasons it gathered in `mlDataErrors` when - // it ends without failing. A failure leaves `mlDataErrors` as it was and - // is reported through `status()` and `onProgress`, then thrown. + // One ML fetch pass. It fetches, decrypts and stores the ML data of every + // file the store knows about that is not cached, or whose `updationTime` + // has advanced. It fetches through the metadata pool, and updates the CLIP + // index. For a file whose ML data does not decrypt, gunzip or parse, it + // gathers the reason instead. It puts the reasons it gathered in + // `mlDataErrors` when it ends without failing. A failure leaves + // `mlDataErrors` as it was and is reported through `status()` and + // `onProgress`, then thrown. private async runMLFetch(): Promise { const mldata = this.mlStore; // Bind so the call keeps the client as its receiver when invoked @@ -977,9 +1104,9 @@ export class Library { })); } - // The distinct files the store holds, one entry per fileID (a file in - // several collections shares its ML data), each carrying the key and the - // newest `updationTime` seen across its memberships. + // The distinct files the store holds, one entry per fileID, since a file + // in several collections shares its ML data. Each entry carries the key + // and the newest `updationTime` seen across the file's memberships. private uniqueFiles(): { id: number; key: Uint8Array; diff --git a/src/library/precache.ts b/src/library/precache.ts index 472d627..164e186 100644 --- a/src/library/precache.ts +++ b/src/library/precache.ts @@ -122,7 +122,7 @@ export class Precache { } // Recompute the fill orders and the pinned set from the current projection. - // Called at open and after every refresh that changes the library. + // Called at open and after every change to the store. update(records: DerivedRecords): void { const photos = [...records.photos.values()].sort( (a, b) => b.takenAt - a.takenAt || b.fileID - a.fileID, diff --git a/src/library/records.ts b/src/library/records.ts index 4a59e92..04cb07d 100644 --- a/src/library/records.ts +++ b/src/library/records.ts @@ -71,13 +71,14 @@ export interface LibrarySnapshot { takenAt: number; } +// What one refresh or `refreshCollection()` changed in the records. export interface LibraryChange { - // Full records for albums/photos added or changed by the refresh. + // Full records for albums/photos added or changed. albumsChanged: AlbumRecord[]; photosChanged: PhotoRecord[]; fileIDsRemoved: number[]; albumIDsRemoved: number[]; - // Wall-clock milliseconds of the refresh that produced this change. + // Wall-clock milliseconds when the store changed. refreshedAt: number; } diff --git a/src/library/store.ts b/src/library/store.ts index fc59b78..fd5248a 100644 --- a/src/library/store.ts +++ b/src/library/store.ts @@ -3,7 +3,7 @@ // The store keeps one `metadata.json` file holding the account's server // state: the user id, a schema version, the cursor for the incremental // collections listing, the decrypted collection and file records, and the -// collections and files the refresh could not decrypt. The whole file is read +// collections and files that did not decrypt. The whole file is read // into RAM on load and rewritten as a whole on save; there is no partial // update and no lock file. A separate refresh unit populates the store from // the server — this module only stores what it is given. @@ -34,9 +34,11 @@ const FILE_MODE = 0o600; type StoredCollection = Omit & { key: string }; type StoredFile = Omit & { key: string }; -// A collection, or with `fileID` a file in it, that the last refresh to fetch -// it could not decrypt or parse, and why. It is kept until a later refresh -// fetches it again, because a refresh fetches only what changed. +// A collection, or with `fileID` a file in it, that did not decrypt or parse, +// and why. A refresh or `refreshCollection()` records it on fetching such a +// copy. A later fetch drops it on getting a copy that decrypts, or on learning +// that the server deleted it. Each fetch asks only for what changed, so the +// store keeps the record between fetches. export interface Undecryptable { collectionID: number; fileID?: number; @@ -70,12 +72,6 @@ export class MetadataStore { userID = 0; collectionsSinceTime: Microseconds = 0; - // True when `load` populated this store from a valid existing file; false - // on a first run or a missing/corrupt/wrong-version file that loaded empty. - // `Library.open` reads it to decide whether the first refresh may run in - // the background (an existing copy already serves reads) or must be awaited. - loadedFromDisk = false; - private readonly collections = new Map(); private readonly files = new Map(); private undecryptable: Undecryptable[] = []; @@ -102,7 +98,6 @@ export class MetadataStore { if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) { return store; } - store.loadedFromDisk = true; store.userID = parsed.userID ?? 0; store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0; for (const stored of parsed.collections ?? []) { @@ -229,8 +224,14 @@ export class MetadataStore { this.files.delete(fileKey(collectionID, fileID)); } + // Sorted by collection ID, then file ID, so the order does not depend on + // when each record was fetched. listUndecryptable(): Undecryptable[] { - return [...this.undecryptable]; + return [...this.undecryptable].sort( + (a, b) => + a.collectionID - b.collectionID || + (a.fileID ?? 0) - (b.fileID ?? 0), + ); } // Replaces any earlier record of the same collection or file. diff --git a/test/cli/commands.test.ts b/test/cli/commands.test.ts index 3269fdb..6a00c8c 100644 --- a/test/cli/commands.test.ts +++ b/test/cli/commands.test.ts @@ -124,20 +124,35 @@ const FILES: Record = { // thumbnail as empty. `withNewFile` adds new.jpg (102) to Vacation, advancing // the collection's updationTime as the server does. `beachInWork` adds // beach.jpg (100) to Work as well, so that one file is in both albums. -// `refreshError` makes listing collections fail with that message. +// `newAlbums` adds the albums A (3) and B (4), in that order, which both hold +// shared.jpg (300). `refreshError` makes listing collections fail with that +// message. const fakeClient = ( opts: { failID?: number; emptyThumbID?: number; withNewFile?: boolean; beachInWork?: boolean; + newAlbums?: boolean; refreshError?: string; } = {}, ) => { - const collections = opts.withNewFile + let collections = opts.withNewFile ? [{ ...COLLECTIONS[0], updationTime: 2 }, COLLECTIONS[1]] : COLLECTIONS; let files = FILES; + if (opts.newAlbums) { + collections = [ + ...collections, + { ...collection(3, "A"), updationTime: 2 }, + { ...collection(4, "B"), updationTime: 2 }, + ]; + files = { + ...files, + 3: [file(300, 3, "shared.jpg")], + 4: [file(300, 4, "shared.jpg")], + }; + } if (opts.withNewFile) { files = { ...files, @@ -535,9 +550,7 @@ describe("files", () => { it("exits 1 for an unknown collection", async () => { expect(await filesCommand(context(), "9", {})).toBe(1); - expect(stderr.text).toBe( - FETCHED_BOTH_ALBUMS + "Collection 9 not found\n", - ); + expect(stderr.text).toBe("Collection 9 not found\n"); }); // Without a collection ID it lists every file in the account once (issue @@ -571,6 +584,57 @@ describe("files", () => { ]); expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS); }); + + // `files 2` on an empty cache stores Work alone. The refresh that `files` + // then waits for must still store Vacation first, as the server lists it. + it("without a collection ID lists in the server's order after files 2 on an empty cache", async () => { + const ctx = context(fakeClient({ beachInWork: true })); + expect(await filesCommand(ctx, "2", {})).toBe(0); + stdout.text = ""; + stderr.text = ""; + + expect(await filesCommand(ctx, undefined, { json: true })).toBe(0); + expect( + JSON.parse(stdout.text).map( + (row: { id: number; collectionID: number }) => [ + row.id, + row.collectionID, + ], + ), + ).toEqual([ + [100, 1], + [101, 1], + [200, 2], + ]); + expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS); + }); + + // After a refresh, the server adds A and then B. `files 4` must not store B + // ahead of A, so it waits for a refresh, which stores both in that order. + it("without a collection ID lists new albums in the server's order after files with the second one's ID", async () => { + expect(await filesCommand(context(), undefined, {})).toBe(0); + const ctx = context(fakeClient({ newAlbums: true })); + stdout.text = ""; + + expect(await filesCommand(ctx, "4", {})).toBe(0); + expect(stdout.text).toBe("300\timage\tshared.jpg\n"); + stdout.text = ""; + + expect(await filesCommand(ctx, undefined, { json: true })).toBe(0); + expect( + JSON.parse(stdout.text).map( + (row: { id: number; collectionID: number }) => [ + row.id, + row.collectionID, + ], + ), + ).toEqual([ + [100, 1], + [101, 1], + [200, 2], + [300, 3], + ]); + }); }); describe("get and get-thumb", () => { @@ -1394,7 +1458,7 @@ describe("backup-metadata --exif", () => { // collection list alone and stores nothing.) describe("backup-metadata and the thumbnail helpers refresh first", () => { beforeEach(async () => { - expect(await filesCommand(context(), "1", {})).toBe(0); + expect(await filesCommand(context(), undefined, {})).toBe(0); stdout.text = ""; stderr.text = ""; }); @@ -1579,11 +1643,13 @@ describe("an ID given on the command line", () => { // An account with many albums (issue #210). On an empty cache `collections` // used to wait for a refresh that fetched and decrypted the files of every -// album, one album after another, before it printed anything, so on such an -// account it seemed to hang. The account here has 40 albums of one photo each, -// encrypted as Ente's clients encrypt them, on the fake Ente server of -// `test/ente-server.ts`, which also records the path of each request quak -// sends. The session is the one logging in to that server saves. +// album, one album after another. It printed nothing until then, so on such an +// account it seemed to hang. `files` with an album's ID did the same before it +// listed that album (issue #217). The account here has 40 albums of one photo +// each, encrypted as Ente's clients encrypt them. The fake Ente server of +// `test/ente-server.ts` serves them. A wrapper around it records the path and +// query of each request quak sends. The session is the one logging in to that +// server saves. describe("an account with many albums", () => { const ALBUMS = 40; const PASSWORD = "many-albums"; @@ -1664,7 +1730,8 @@ describe("an account with many albums", () => { : input instanceof URL ? input.href : input.url; - paths.push(new URL(url).pathname); + const { pathname, search } = new URL(url); + paths.push(pathname + search); return ente(input, init); }) as typeof globalThis.fetch; const client = await Client.login({ @@ -1698,42 +1765,68 @@ describe("an account with many albums", () => { albums += `${id}\talbum\tAlbum ${id}\n`; expect(stdout.text).toBe(albums); expect(stderr.text).toBe(""); - expect(paths).toEqual(["/collections/v2"]); + expect(paths).toEqual(["/collections/v2?sinceTime=0"]); - // It stored nothing that would make a refresh skip an album's files: - // `files` after it still finds the album's photo. + // It stored nothing that would make `refreshCollection()` skip an + // album's files: `files` after it still finds the album's photo. stdout.text = ""; expect(await filesCommand(ctx, "7", {})).toBe(0); expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n"); }); - it("files counts on stderr each album whose files its refresh fetched", async () => { + it("files fetches the files of the album it lists and of no other", async () => { expect(await filesCommand(accountContext(), "7", {})).toBe(0); expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n"); + expect(stderr.text).toBe(""); + expect(paths).toEqual([ + "/collections/v2?sinceTime=0", + "/collections/v2/diff?collectionID=7&sinceTime=0", + ]); + }); + + // The first listing stored the album, so the second finds its stored copy + // current and asks for no album's files. + it("files run again for the same album sends no request for its files", async () => { + const ctx = accountContext(); + expect(await filesCommand(ctx, "7", {})).toBe(0); + stdout.text = ""; + paths.length = 0; + + expect(await filesCommand(ctx, "7", {})).toBe(0); + expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n"); + expect(paths).toEqual(["/collections/v2?sinceTime=0"]); + }); + + it("files with no album's ID counts on stderr each album whose files its refresh fetched", async () => { + expect(await filesCommand(accountContext(), undefined, {})).toBe(0); + + let photos = ""; let progress = ""; for (let n = 1; n <= ALBUMS; n++) { + photos += `${n * 100}\timage\tphoto-${n}.jpg\n`; progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`; } + expect(stdout.text).toBe(photos); expect(stderr.text).toBe(progress); }); // `--json` shapes stdout only, so the refresh's progress is still counted // on stderr. - it("files --json counts on stderr each album whose files its refresh fetched", async () => { - expect(await filesCommand(accountContext(), "7", { json: true })).toBe( - 0, - ); + it("files --json with no album's ID counts on stderr each album whose files its refresh fetched", async () => { + expect( + await filesCommand(accountContext(), undefined, { json: true }), + ).toBe(0); - expect(JSON.parse(stdout.text)).toEqual([ - { - id: 700, - title: "photo-7.jpg", - fileType: "image", - creationTime: 1000, - collectionID: 7, - }, - ]); + const rows = JSON.parse(stdout.text); + expect(rows).toHaveLength(ALBUMS); + expect(rows[6]).toEqual({ + id: 700, + title: "photo-7.jpg", + fileType: "image", + creationTime: 1000, + collectionID: 7, + }); let progress = ""; for (let n = 1; n <= ALBUMS; n++) { progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`; diff --git a/test/cli/log.test.ts b/test/cli/log.test.ts index 137c983..e2cfc68 100644 --- a/test/cli/log.test.ts +++ b/test/cli/log.test.ts @@ -234,8 +234,8 @@ describe("-q", () => { }); }); - // The refresh's progress is left out; the line saying why the command - // failed is the fatal one, and it is printed. + // The line saying why the command failed is the fatal one, and it is + // printed. it("prints only the line a failing command exits with", async () => { expect(await quak("-q", "files", "9")).toEqual({ code: 1, diff --git a/test/cli/read.test.ts b/test/cli/read.test.ts index 66ff369..051606f 100644 --- a/test/cli/read.test.ts +++ b/test/cli/read.test.ts @@ -3,13 +3,18 @@ * issue #36, issue #52). * * The `collections`, `files`, `get`, and `get-thumb` commands must answer for - * current server state, not the local cache. `files`, `get` and `get-thumb` - * force a `Library.fresh()` round-trip before they read; the stand-in library - * below serves nothing until `fresh()` has been awaited, so a helper that read - * without refreshing would come back empty and fail here. `collections` asks - * the library for the collection list alone (`freshCollections()`, issue - * #210), and must not wait for a refresh, which would fetch the files of every - * collection. + * current server state, not the local cache. `files` without a collection ID, + * `get` and `get-thumb` force a `Library.fresh()` round-trip before they read. + * The stand-in library below serves nothing until `fresh()` has been awaited. + * A helper that read without refreshing would come back empty and fail here. + * `collections` asks the library for the collection list alone + * (`freshCollections()`, issue #210). It must not wait for a refresh, which + * would fetch the files of every collection. `files COLLECTION` reads that list + * too, then brings that one collection up to date (`refreshCollection()`, + * issue #217). The stand-in serves a collection's files once `fresh()` has run, + * or once `refreshCollection()` has resolved to `true` for that collection. + * When `refreshCollection()` resolves to `false`, `files COLLECTION` forces a + * refresh instead. * * `collections` and `files` also list in the library's enumeration order * (`listCollections`/`listFiles`) — the order the pre-library CLI printed — not @@ -64,12 +69,18 @@ const file = ( // A library that reveals its records only after `fresh()` has been awaited, and // serves them in the enumeration order it was given. `freshCollections()` -// answers the collection list in that order without a refresh. `photos.byID` -// returns a stand-in `Photo` carrying just the fileID the helper passes -// through. +// answers the collection list in that order without a refresh. When +// `refreshCollection()` resolves to true, it reveals the files of that one +// collection. +// `photos.byID` returns a stand-in `Photo` carrying just the fileID the helper +// passes through. class FakeLibrary implements FreshReadLibrary { freshCalls = 0; freshCollectionsCalls = 0; + refreshCollectionIDs: number[] = []; + // What `refreshCollection()` resolves to. With false it reveals nothing, + // as a library that has had a refresh does for a collection it lacks. + refreshCollectionHolds = true; private refreshed = false; constructor( @@ -88,6 +99,11 @@ class FakeLibrary implements FreshReadLibrary { return this.collections; } + async refreshCollection(collection: Collection): Promise { + this.refreshCollectionIDs.push(collection.id); + return this.refreshCollectionHolds; + } + getCollection(id: number): Collection | undefined { if (!this.refreshed) return undefined; return this.collections.find((c) => c.id === id); @@ -98,7 +114,11 @@ class FakeLibrary implements FreshReadLibrary { } listFiles(collectionID: number): EnteFile[] { - return this.refreshed + const current = + this.refreshed || + (this.refreshCollectionHolds && + this.refreshCollectionIDs.includes(collectionID)); + return current ? this.files.filter((f) => f.collectionID === collectionID) : []; } @@ -138,18 +158,25 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => { expect(rows.map((c) => c.id)).not.toEqual(newestFirst); }); - it("freshFiles refreshes first, lists in enumeration order, keeps raw fields", async () => { + it("freshFiles brings the collection alone up to date, lists in enumeration order, keeps raw fields", async () => { // Enumeration order by id 10, 11, 12; creationTimes ascending, so a - // newest-first sort would reverse them. + // newest-first sort would reverse them. Collection 2's file is not + // listed. const files = [ file(10, 1, 1_700_000_000_000_000), file(11, 1, 1_700_000_000_000_001), file(12, 1, 1_700_000_000_000_002), + file(20, 2, 1_700_000_000_000_003), ]; - const lib = new FakeLibrary([collection(1, 100)], files); + const lib = new FakeLibrary( + [collection(1, 100), collection(2, 100)], + files, + ); const rows = await freshFiles(lib, 1); - expect(lib.freshCalls).toBe(1); + expect(lib.freshCollectionsCalls).toBe(1); + expect(lib.refreshCollectionIDs).toEqual([1]); + expect(lib.freshCalls).toBe(0); expect(rows?.map((f) => f.id)).toEqual([10, 11, 12]); // Field values come from raw metadata: microsecond creationTime and the @@ -203,10 +230,25 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => { ]); }); + it("freshFiles forces a refresh when refreshCollection resolves to false", async () => { + const lib = new FakeLibrary( + [collection(1, 100)], + [file(10, 1, 1_700_000_000_000_000)], + ); + lib.refreshCollectionHolds = false; + + const rows = await freshFiles(lib, 1); + expect(lib.refreshCollectionIDs).toEqual([1]); + expect(lib.freshCalls).toBe(1); + expect(rows?.map((f) => f.id)).toEqual([10]); + }); + it("freshFiles returns undefined for an unknown collection", async () => { const lib = new FakeLibrary([collection(1, 100)], []); const rows = await freshFiles(lib, 999); - expect(lib.freshCalls).toBe(1); + expect(lib.freshCollectionsCalls).toBe(1); + expect(lib.refreshCollectionIDs).toEqual([]); + expect(lib.freshCalls).toBe(0); expect(rows).toBeUndefined(); }); diff --git a/test/library/fresh.test.ts b/test/library/fresh.test.ts index 859a8da..540c537 100644 --- a/test/library/fresh.test.ts +++ b/test/library/fresh.test.ts @@ -17,6 +17,14 @@ * 4. `freshCollections()` reads the collection list alone, with one request * and no collection's files, and stores nothing, so the next refresh still * fetches the files of every collection (issue #210). + * 5. `refreshCollection()` brings one collection up to date in the store + * (issue #217). It leaves the stored cursor where it was, so the next + * refresh still fetches the files of every other collection that changed. + * It never runs at the same time as a refresh. After a refresh, it stores + * no collection the store does not hold: not a new one, and not one a + * refresh deleted. `close()` waits for it. A library opened on a cache that + * only it wrote awaits its first refresh. That refresh replaces what it + * stored, in the server's order. * * The client is the same metadata-only mock the background-refresh tests use: * no crypto, no network, scripted pages, and a record of each call. A long @@ -30,6 +38,7 @@ import { tmpdir } from "node:os"; import { join } from "node:path"; import { Library } from "../../src/library/index.js"; +import { MetadataStore } from "../../src/library/store.js"; import type { CollectionsPage, FilesPage } from "../../src/client.js"; import type { Collection, EnteFile } from "../../src/model/types.js"; @@ -144,6 +153,26 @@ const openSeeded = async ( return { client, lib }; }; +// Hold the client's next files request until `release()` is called. `started` +// resolves once that request has been sent. +const holdNextFilesRequest = ( + client: MockClient, +): { started: Promise; release: () => void } => { + let release!: () => void; + const held = new Promise((r) => (release = r)); + let sent!: () => void; + const started = new Promise((r) => (sent = r)); + const realFilesSince = client.filesSince.bind(client); + client.filesSince = async (args) => { + client.filesSince = realFilesSince; + const page = await realFilesSince(args); + sent(); + await held; + return page; + }; + return { started, release }; +}; + describe("Library.fresh", () => { let dir: string; let cacheDirectory: string; @@ -287,10 +316,10 @@ describe("Library.freshCollections", () => { rmSync(dir, { recursive: true, force: true }); }); - // A refresh fetches a collection's files only when the collection is new - // or its updationTime has advanced past the stored copy's. Had the - // collection list been stored, the next refresh would take every - // collection for one it already holds the files of, and fetch none. + // A refresh from a stored cursor past 0 fetches a collection's files only + // when the collection is new or its updationTime has advanced past the + // stored copy's. So does `refreshCollection()`. Neither would fetch the + // files of a collection stored here without them. it("asks for the collection list alone and stores nothing", async () => { const client = new MockClient(); const page = { @@ -340,3 +369,243 @@ describe("Library.freshCollections", () => { } }); }); + +describe("Library.refreshCollection", () => { + let dir: string; + let cacheDirectory: string; + + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), "quak-refresh-collection-")); + cacheDirectory = join(dir, "cache"); + }); + + afterEach(() => { + rmSync(dir, { recursive: true, force: true }); + }); + + // Collections 1 and 2 are stored at updationTime 100, with a stored cursor + // of 100, and then both change on the server. + it("fetches the files of one changed collection and leaves the others to the next refresh", async () => { + const client = new MockClient(); + client.collectionsQueue.push({ + collections: [collection(1, 100), collection(2, 100)], + deleted: [], + cursor: 100, + }); + client.filesFor(1, { + files: [file(1001, 1, 90)], + deleted: [], + cursor: 90, + }); + client.filesFor(2, { + files: [file(2001, 2, 90)], + deleted: [], + cursor: 90, + }); + const lib = await Library.open({ + client, + cacheDirectory, + refreshIntervalSeconds: SLOW_INTERVAL, + }); + try { + client.filesCalls.length = 0; + client.filesFor(1, { + files: [file(1002, 1, 110)], + deleted: [], + cursor: 110, + }); + client.filesFor(2, { + files: [file(2002, 2, 120)], + deleted: [], + cursor: 120, + }); + + // Collection 1's files alone, since the stored copy. + expect(await lib.refreshCollection(collection(1, 110))).toBe(true); + expect(client.filesCalls).toEqual([ + { collectionID: 1, sinceTime: 100 }, + ]); + expect(lib.listFiles(1).map((f) => f.id)).toEqual([1001, 1002]); + expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001]); + + // The stored copy is now current, so no request. + await lib.refreshCollection(collection(1, 110)); + expect(client.filesCalls).toHaveLength(1); + + // The next refresh asks from the stored cursor, 100, so it still + // sees collection 2 changed and fetches its files. + client.collectionsQueue.push({ + collections: [collection(1, 110), collection(2, 120)], + deleted: [], + cursor: 120, + }); + await lib.fresh(); + expect(client.collectionsSinceTimes).toEqual([0, 100]); + expect(client.filesCalls).toEqual([ + { collectionID: 1, sinceTime: 100 }, + { collectionID: 2, sinceTime: 100 }, + ]); + expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001, 2002]); + } finally { + await lib.close(); + } + }); + + // Collection 1 is stored at 100. `refreshCollection()` asks for it at 110, + // which adds file 1002. While that files request waits, a fresh read + // starts a refresh. The server tells that refresh that collection 1 was + // deleted at 130. + it("holds off a refresh that starts during the files request, which then deletes the collection", async () => { + const { client, lib } = await openSeeded(cacheDirectory); + const files = holdNextFilesRequest(client); + try { + client.filesFor(1, { + files: [file(1002, 1, 110)], + deleted: [], + cursor: 110, + }); + client.collectionsQueue.push({ + collections: [], + deleted: [1], + cursor: 130, + }); + + const refreshing = lib.refreshCollection(collection(1, 110)); + await files.started; + const reading = lib.fresh(); + // The refresh has not sent its collections request yet. + await new Promise((r) => setImmediate(r)); + expect(client.collectionsSinceTimes).toEqual([0]); + + files.release(); + await refreshing; + await reading; + expect(client.collectionsSinceTimes).toEqual([0, 100]); + expect(lib.getCollection(1)).toBeUndefined(); + expect(lib.listFiles(1)).toEqual([]); + } finally { + files.release(); + await lib.close(); + } + }); + + // `freshCollections()` lists collection 1 at 110. Before + // `refreshCollection()` is called, a refresh learns that it was deleted at + // 130. + it("stores nothing for a collection that a refresh deleted before the call", async () => { + const { client, lib } = await openSeeded(cacheDirectory); + try { + client.collectionsQueue.push( + { collections: [collection(1, 110)], deleted: [], cursor: 110 }, + { collections: [], deleted: [1], cursor: 130 }, + ); + const [listed] = await lib.freshCollections(); + await lib.fresh(); + client.filesCalls.length = 0; + + expect(await lib.refreshCollection(listed)).toBe(false); + expect(client.filesCalls).toEqual([]); + expect(lib.getCollection(1)).toBeUndefined(); + } finally { + await lib.close(); + } + }); + + // A refresh has stored collection 1. Collection 2 is new on the server + // since then. Only the next refresh may add it, after collection 1. + it("stores nothing for a collection that is new since the last refresh", async () => { + const { client, lib } = await openSeeded(cacheDirectory); + try { + client.filesCalls.length = 0; + + expect(await lib.refreshCollection(collection(2, 110))).toBe(false); + expect(client.filesCalls).toEqual([]); + expect(lib.getCollection(2)).toBeUndefined(); + } finally { + await lib.close(); + } + }); + + // The caller may remove the cache directory once `close()` resolves, so + // the cache write must have landed by then. + it("close() waits for a refreshCollection() in flight", async () => { + const { client, lib } = await openSeeded(cacheDirectory); + const files = holdNextFilesRequest(client); + client.filesFor(1, { + files: [file(1002, 1, 110)], + deleted: [], + cursor: 110, + }); + + const refreshing = lib.refreshCollection(collection(1, 110)); + await files.started; + let closed = false; + const closing = lib.close().then(() => { + closed = true; + }); + await new Promise((r) => setImmediate(r)); + expect(closed).toBe(false); + + files.release(); + await closing; + const saved = await MetadataStore.load( + join(cacheDirectory, "metadata.json"), + ); + expect(saved.listFiles(1).map((f) => f.id)).toEqual([1001, 1002]); + await refreshing; + }); + + // `quak files 2` on an empty cache stores collection 2 alone. A library + // opened later on that cache must not serve it as the whole account. Its + // first refresh replaces what the cache holds, in the server's order. + it("open() awaits the first refresh on a cache that only refreshCollection() wrote", async () => { + const client = new MockClient(); + client.filesFor(2, { + files: [file(2001, 2, 90)], + deleted: [], + cursor: 90, + }); + const first = await Library.open({ + client, + cacheDirectory, + refreshOnOpen: false, + refreshIntervalSeconds: SLOW_INTERVAL, + }); + expect(await first.refreshCollection(collection(2, 100))).toBe(true); + await first.close(); + + client.collectionsQueue.push({ + collections: [collection(1, 100), collection(2, 100)], + deleted: [], + cursor: 100, + }); + client.filesFor(1, { + files: [file(1001, 1, 90)], + deleted: [], + cursor: 90, + }); + client.filesFor(2, { + files: [file(2001, 2, 90)], + deleted: [], + cursor: 90, + }); + client.filesCalls.length = 0; + const lib = await Library.open({ + client, + cacheDirectory, + refreshIntervalSeconds: SLOW_INTERVAL, + }); + try { + expect(lib.status().lastRefreshAt).toBeDefined(); + expect(lib.listCollections().map((c) => c.id)).toEqual([1, 2]); + // It fetches collection 2's files again, from the beginning. + expect(client.filesCalls).toEqual([ + { collectionID: 1, sinceTime: 0 }, + { collectionID: 2, sinceTime: 0 }, + ]); + expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001]); + } finally { + await lib.close(); + } + }); +}); diff --git a/test/library/library.test.ts b/test/library/library.test.ts index def8c33..308de42 100644 --- a/test/library/library.test.ts +++ b/test/library/library.test.ts @@ -20,9 +20,9 @@ * 5. `close()` stops the timer and is idempotent, and its promise resolves * only once an in-flight refresh has written the cache file. * 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id. - * 7. `open()` branches on the cache: an empty cache awaits the first refresh - * (it has nothing to serve yet); an existing cache serves its copy at once - * and refreshes in the background, so a slow or dead server never stalls + * 7. `open()` branches on the stored cursor. At 0, as on an empty cache, it + * awaits the first refresh. Past 0 it serves the cache's copy at once and + * refreshes in the background, so a slow or dead server never stalls * opening. * 8. A save failure that leaves RAM ahead of disk keeps `status().lastError` * set and keeps retrying the write; a later empty refresh does not clear it. diff --git a/test/library/snapshot.test.ts b/test/library/snapshot.test.ts index c0f306c..a40e88a 100644 --- a/test/library/snapshot.test.ts +++ b/test/library/snapshot.test.ts @@ -4,7 +4,8 @@ * These are the surface the GUI consumes across Electron IPC. `snapshot()` is * synchronous — it reads the in-RAM store and projects it into plain records * (no keys) — and `subscribe({ onChange })` delivers a `LibraryChange` whenever - * a background refresh actually changes the derived records. The contracts: + * a refresh or `refreshCollection()` changes the derived records. The + * contracts: * * 1. `snapshot()` deduplicates a file across memberships into one record with * every album id, orders photos newest first, and carries no key material. diff --git a/test/library/store.test.ts b/test/library/store.test.ts index 38e7e25..378fa30 100644 --- a/test/library/store.test.ts +++ b/test/library/store.test.ts @@ -172,7 +172,7 @@ describe("MetadataStore", () => { expect(store.listCollections()).toEqual([]); }); - it("loads an empty store when the list of what the refresh could not decrypt is damaged", async () => { + it("loads an empty store when the list of what did not decrypt is damaged", async () => { // A backup reads that list on every run, so a damaged one that loaded // would fail every backup until the cache was deleted. A string is // damaged too, though JavaScript can loop over its characters. @@ -266,4 +266,23 @@ describe("MetadataStore", () => { expect(store.getFile(12345, 67890)).toBeUndefined(); expect(store.getFile(55555, 67890)?.key).toEqual(inB.key); }); + + // A backup reports these in this order. It must not depend on whether a + // refresh or `Library.refreshCollection()` fetched each one first. + it("lists what did not decrypt by collection ID, then file ID", async () => { + const store = await MetadataStore.load(path); + store.putUndecryptable({ collectionID: 2, fileID: 7, error: "a" }); + store.putUndecryptable({ collectionID: 3, error: "b" }); + store.putUndecryptable({ collectionID: 1, fileID: 9, error: "c" }); + store.putUndecryptable({ collectionID: 2, fileID: 5, error: "d" }); + + expect( + store.listUndecryptable().map((r) => [r.collectionID, r.fileID]), + ).toEqual([ + [1, 9], + [2, 5], + [2, 7], + [3, undefined], + ]); + }); }); -- 2.54.0