quak collections reads the collection list alone (closes #210)

On an empty cache the read commands opened their library with a refresh that fetched and decrypted the files of every collection in turn, printing nothing and storing nothing until it had them all, so `quak collections` seemed to hang on a large account. The new `lib.freshCollections()` is the fresh read of the collection list alone: one request, and nothing stored, so a later refresh still fetches every collection's files. The read commands open their library with the new `refreshOnOpen: false` and make only the refresh they ask for. A refresh sends a "progress" event after each collection whose files it fetched; the commands that wait on one print it on stderr, except with `--json`.

Model: opus-5-5

src/library/index.ts

@@ -18,6 +18,16 @@
 // the caller (the default reads stay silent and serve the last good copy). The
 // default methods and the background loop are unchanged.
 //
+// `freshCollections()` is the fresh read of the collection list alone (issue
+// #210): one collections request, no collection's files, and nothing stored,
+// for a caller such as `quak collections` that needs no more. A caller that
+// asks for each refresh it needs can also open the library with
+// `refreshOnOpen: false`, so that `open()` starts none.
+//
+// A refresh reports through `onProgress` each collection whose files it has
+// fetched, so a large account, whose first refresh fetches the files of every
+// collection, shows progress.
+//
 // A refresh stages all of its network work first and only mutates the store
 // once every fetch has succeeded. A refresh that fails partway therefore never
 // becomes visible to reads: the last good snapshot stays in place, and the
@@ -161,18 +171,21 @@ export interface LibraryClient {
 // exactly one of "done" or "failed"; "failed" carries the error message and
 // an ML "done" reports how many payloads it stored and, in `mlDataErrors`,
 // each file whose ML data did not decrypt, gunzip or parse, with the reason.
-// The precache fills ("precacheThumbnails"/"precacheOriginals", #48) fire
-// "started"/"done" around each sweep that has work, "done" reporting the count
-// newly cached.
+// In between, a refresh fires "progress" after each collection whose files it
+// fetched: `fetched` is how many collections it has fetched the files of so
+// far, `total` how many it will. The precache fills
+// ("precacheThumbnails"/"precacheOriginals", #48) fire "started"/"done" around
+// each sweep that has work, "done" reporting the count newly cached.
 export interface RefreshEvent {
     operation:
         | "refresh"
         | "fetchMLData"
         | "precacheThumbnails"
         | "precacheOriginals";
-    status: "started" | "done" | "failed";
+    status: "started" | "progress" | "done" | "failed";
     error?: string;
     fetched?: number;
+    total?: number;
     mlDataErrors?: { fileID: number; error: string }[];
 }
 
@@ -192,6 +205,10 @@ export interface LibraryOptions {
     // `Infinity` starts none: the library then refreshes only as it opens and
     // when `fresh()` or `backup()` asks.
     refreshIntervalSeconds?: number;
+    // Whether `open()` starts a refresh. Defaults to true. With false, `open()`
+    // returns at once onto what the cache holds, empty or not, and a refresh
+    // starts only when `fresh()` or `backup()` asks or the timer fires.
+    refreshOnOpen?: boolean;
     onProgress?: RefreshProgressCallback;
     // The bounded request pools (issue #45), shared by the ML-data fetch (the
     // metadata pool) and the content cache. Defaults to a fresh set at the
@@ -382,7 +399,8 @@ export class Library {
     // then opens empty with the failure recorded in `status()`. With an
     // existing cache the first refresh runs in the background and `open()`
     // returns as soon as the cached data is ready — an unreachable server does
-    // not block opening.
+    // not block opening. With `refreshOnOpen: false` there is no first
+    // refresh: `open()` returns onto the cache as it is.
     static async open(opts: LibraryOptions): Promise<Library> {
         const { userID } = opts.client.whoami();
         const cacheDirectory =
@@ -485,7 +503,10 @@ export class Library {
         // refresh.
         precache?.start();
 
-        if (store.loadedFromDisk) {
+        if (opts.refreshOnOpen === false) {
+            // The caller asks for each refresh it needs.
+            lib.scheduleNext();
+        } else if (store.loadedFromDisk) {
             // An existing copy already answers reads; refresh in the background
             // and start the interval once that first cycle settles.
             void lib.runRefresh().then(() => lib.scheduleNext());
@@ -589,6 +610,21 @@ export class Library {
         };
     }
 
+    // The fresh read of the collection list alone (issue #210): every live
+    // collection that decrypts, in the server's order, from one collections
+    // request. Unlike a refresh, it fetches no collection's files, which on a
+    // large account take long. Nothing is stored: a refresh fetches a
+    // collection's files only when the collection is new or has changed since
+    // the stored copy, so a collection stored here without its files would
+    // never have them fetched.
+    async freshCollections(): Promise<Collection[]> {
+        if (this.closed) throw new Error("the library is closed");
+        const { collections } = await this.client.collectionsSince({
+            sinceTime: 0,
+        });
+        return collections;
+    }
+
     // Back up every in-scope file to `opts.downloadDirectory`, or else the
     // library's, each original at its save path, with a durable failure
     // ledger (issue #51). Takes the lock in that directory first, unless the
@@ -742,18 +778,28 @@ export class Library {
         // Stage per-collection file diffs. A collection's files are
         // re-enumerated only when its updationTime has advanced past the cached
         // copy; an unchanged album's file list cannot have changed. New
-        // collections enumerate from the beginning of time.
+        // collections enumerate from the beginning of time. Each collection
+        // whose files are fetched is reported, with how many there are to
+        // fetch.
+        const changedCollections = page.collections.filter((collection) => {
+            const known = this.store.getCollection(collection.id);
+            return !known || collection.updationTime > known.updationTime;
+        });
         const filePages: { collectionID: number; page: FilesPage }[] = [];
-        for (const collection of page.collections) {
+        for (const collection of changedCollections) {
             const known = this.store.getCollection(collection.id);
-            if (known && collection.updationTime <= known.updationTime)
-                continue;
             const filePage = await this.client.filesSince({
                 collectionID: collection.id,
                 collectionKey: collection.key,
                 sinceTime: known ? known.updationTime : 0,
             });
             filePages.push({ collectionID: collection.id, page: filePage });
+            this.emit({
+                operation: "refresh",
+                status: "progress",
+                fetched: filePages.length,
+                total: changedCollections.length,
+            });
         }
 
         // Network work done; commit to the store and persist only if something