quak collections reads the collection list alone (closes #210)

On an empty cache the read commands opened their library with a refresh that fetched and decrypted the files of every collection in turn, printing nothing and storing nothing until it had them all, so `quak collections` seemed to hang on a large account. The new `lib.freshCollections()` is the fresh read of the collection list alone: one request, and nothing stored, so a later refresh still fetches every collection's files. The read commands open their library with the new `refreshOnOpen: false` and make only the refresh they ask for. A refresh sends a "progress" event after each collection whose files it fetched; the commands that wait on one print it on stderr, except with `--json`.

Model: opus-5-5

src/cli-commands.ts

@@ -125,15 +125,29 @@ const readLibraryClient = (client: Client): LibraryClient => ({
 
 // Open a library for a single point command: the aggressive background precache
 // (issue #48) is off — a one-shot `collections` or `get` must not start
-// downloading the whole account — and the refresh interval is long so no second
-// refresh fires mid-command.
-const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
+// downloading the whole account — and no refresh starts as it opens or on a
+// timer: each command asks for the one it needs, and `collections` needs none
+// (issue #210). While a refresh runs, each collection whose files it has
+// fetched is counted on stderr, unless `json` is set, so a large account shows
+// progress rather than nothing.
+const openReadLibrary = (
+    ctx: CliContext,
+    client: Client,
+    opts: { json?: boolean } = {},
+): Promise<Library> =>
     Library.open({
         client: readLibraryClient(client),
         cacheDirectory: ctx.cacheDir,
-        refreshIntervalSeconds: 3600,
+        refreshOnOpen: false,
+        refreshIntervalSeconds: Infinity,
         precacheThumbnails: false,
         precacheOriginals: false,
+        onProgress: (event) => {
+            if (opts.json || event.status !== "progress") return;
+            ctx.stderr.write(
+                `Fetched the files of ${event.fetched} of ${event.total} collections\n`,
+            );
+        },
     });
 
 export const loginCommand = async (ctx: CliContext): Promise<number> => {
@@ -221,9 +235,10 @@ export const collectionsCommand = async (
     if (!client) return 3;
     const lib = await openReadLibrary(ctx, client);
     try {
-        // Force a server round-trip and list in enumeration order (issue #36
-        // amendment, issue #52): the pre-library CLI printed current state in
-        // this order, not the albums projection's newest-first order.
+        // Ask the server for the collection list alone and list it in its
+        // order (issue #36 amendment, issues #52 and #210): the pre-library
+        // CLI printed current state in this order, not the albums
+        // projection's newest-first order.
         const collections = await freshCollections(lib);
 
         if (opts.json) {
@@ -264,7 +279,7 @@ export const filesCommand = async (
     const collectionID = parseID(ctx, "collection", opts.collection);
     if (collectionID === undefined) return 1;
 
-    const lib = await openReadLibrary(ctx, client);
+    const lib = await openReadLibrary(ctx, client, opts);
     try {
         // Force a server round-trip and list in enumeration order (issue #36
         // amendment, issue #52). Each file prints from its own decrypted
@@ -490,7 +505,7 @@ export const listMissingThumbnailsCommand = async (
     await init();
     const client = requireSession(ctx);
     if (!client) return 3;
-    const lib = await openReadLibrary(ctx, client);
+    const lib = await openReadLibrary(ctx, client, opts);
     try {
         // Refresh first so files added since the cache was written are
         // checked; a failed refresh throws.
@@ -535,7 +550,7 @@ export const fixMissingThumbnailsCommand = async (
         if (fileID === undefined) return 1;
         fileIDs.push(fileID);
     }
-    const lib = await openReadLibrary(ctx, client);
+    const lib = await openReadLibrary(ctx, client, opts);
     try {
         // Refresh first so files added since the cache was written are found;
         // a failed refresh throws.

src/cli-read.ts

@@ -2,12 +2,16 @@
 //
 // `collections`, `files --collection`, `get`, and `get-thumb` must answer for
 // the account's state at the moment the command runs, not for whatever the
-// local cache last happened to hold (owner amendment, issue #36). Each helper
-// therefore forces a server round-trip through `Library.fresh()` and only then
-// reads — so a collection, file, or metadata change made elsewhere is visible.
+// local cache last happened to hold (owner amendment, issue #36). `files`,
+// `get` and `get-thumb` therefore force a server round-trip through
+// `Library.fresh()` and only then read — so a collection, file, or metadata
+// change made elsewhere is visible. `collections` asks the library for the
+// collection list alone through `Library.freshCollections()` (issue #210): a
+// refresh would also fetch the files of every collection, which on a large
+// account takes long and which `collections` does not print.
 //
-// `collections` and `files` also list in the library's own enumeration order —
-// `listCollections()`/`listFiles()`, the order the pre-library CLI printed —
+// `collections` lists in the server's order and `files` in the library's own
+// enumeration order (`listFiles()`), the orders the pre-library CLI printed,
 // rather than the `albums`/`photos` projection's newest-first order, which
 // re-sorts the rows. The field values still come from each record's raw
 // metadata via `cli-output.ts`.
@@ -17,23 +21,22 @@ import type { Photo, PhotosAPI } from "./library/index.js";
 
 // The slice of `Library` these helpers read. `Library` satisfies it
 // structurally; a test can drive them with a stand-in that records the
-// `fresh()` call and serves records in a known enumeration order.
+// `fresh()` and `freshCollections()` calls and serves records in a known
+// enumeration order.
 export interface FreshReadLibrary {
     fresh(): Promise<unknown>;
-    listCollections(): Collection[];
+    freshCollections(): Promise<Collection[]>;
     getCollection(id: number): Collection | undefined;
     listFiles(collectionID: number): EnteFile[];
     getFileByID(fileID: number): EnteFile | undefined;
     photos: Pick<PhotosAPI, "byID">;
 }
 
-// Every live collection, current as of a forced refresh, in enumeration order.
-export const freshCollections = async (
+// Every live collection, current as of one collections request, in the
+// server's order.
+export const freshCollections = (
     lib: FreshReadLibrary,
-): Promise<Collection[]> => {
-    await lib.fresh();
-    return lib.listCollections();
-};
+): Promise<Collection[]> => lib.freshCollections();
 
 // The files of one collection, current as of a forced refresh, in enumeration
 // order. `undefined` (not an empty list) when the collection does not exist, so

src/library/index.ts

@@ -18,6 +18,16 @@
 // the caller (the default reads stay silent and serve the last good copy). The
 // default methods and the background loop are unchanged.
 //
+// `freshCollections()` is the fresh read of the collection list alone (issue
+// #210): one collections request, no collection's files, and nothing stored,
+// for a caller such as `quak collections` that needs no more. A caller that
+// asks for each refresh it needs can also open the library with
+// `refreshOnOpen: false`, so that `open()` starts none.
+//
+// A refresh reports through `onProgress` each collection whose files it has
+// fetched, so a large account, whose first refresh fetches the files of every
+// collection, shows progress.
+//
 // A refresh stages all of its network work first and only mutates the store
 // once every fetch has succeeded. A refresh that fails partway therefore never
 // becomes visible to reads: the last good snapshot stays in place, and the
@@ -161,18 +171,21 @@ export interface LibraryClient {
 // exactly one of "done" or "failed"; "failed" carries the error message and
 // an ML "done" reports how many payloads it stored and, in `mlDataErrors`,
 // each file whose ML data did not decrypt, gunzip or parse, with the reason.
-// The precache fills ("precacheThumbnails"/"precacheOriginals", #48) fire
-// "started"/"done" around each sweep that has work, "done" reporting the count
-// newly cached.
+// In between, a refresh fires "progress" after each collection whose files it
+// fetched: `fetched` is how many collections it has fetched the files of so
+// far, `total` how many it will. The precache fills
+// ("precacheThumbnails"/"precacheOriginals", #48) fire "started"/"done" around
+// each sweep that has work, "done" reporting the count newly cached.
 export interface RefreshEvent {
     operation:
         | "refresh"
         | "fetchMLData"
         | "precacheThumbnails"
         | "precacheOriginals";
-    status: "started" | "done" | "failed";
+    status: "started" | "progress" | "done" | "failed";
     error?: string;
     fetched?: number;
+    total?: number;
     mlDataErrors?: { fileID: number; error: string }[];
 }
 
@@ -192,6 +205,10 @@ export interface LibraryOptions {
     // `Infinity` starts none: the library then refreshes only as it opens and
     // when `fresh()` or `backup()` asks.
     refreshIntervalSeconds?: number;
+    // Whether `open()` starts a refresh. Defaults to true. With false, `open()`
+    // returns at once onto what the cache holds, empty or not, and a refresh
+    // starts only when `fresh()` or `backup()` asks or the timer fires.
+    refreshOnOpen?: boolean;
     onProgress?: RefreshProgressCallback;
     // The bounded request pools (issue #45), shared by the ML-data fetch (the
     // metadata pool) and the content cache. Defaults to a fresh set at the
@@ -382,7 +399,8 @@ export class Library {
     // then opens empty with the failure recorded in `status()`. With an
     // existing cache the first refresh runs in the background and `open()`
     // returns as soon as the cached data is ready — an unreachable server does
-    // not block opening.
+    // not block opening. With `refreshOnOpen: false` there is no first
+    // refresh: `open()` returns onto the cache as it is.
     static async open(opts: LibraryOptions): Promise<Library> {
         const { userID } = opts.client.whoami();
         const cacheDirectory =
@@ -485,7 +503,10 @@ export class Library {
         // refresh.
         precache?.start();
 
-        if (store.loadedFromDisk) {
+        if (opts.refreshOnOpen === false) {
+            // The caller asks for each refresh it needs.
+            lib.scheduleNext();
+        } else if (store.loadedFromDisk) {
             // An existing copy already answers reads; refresh in the background
             // and start the interval once that first cycle settles.
             void lib.runRefresh().then(() => lib.scheduleNext());
@@ -589,6 +610,21 @@ export class Library {
         };
     }
 
+    // The fresh read of the collection list alone (issue #210): every live
+    // collection that decrypts, in the server's order, from one collections
+    // request. Unlike a refresh, it fetches no collection's files, which on a
+    // large account take long. Nothing is stored: a refresh fetches a
+    // collection's files only when the collection is new or has changed since
+    // the stored copy, so a collection stored here without its files would
+    // never have them fetched.
+    async freshCollections(): Promise<Collection[]> {
+        if (this.closed) throw new Error("the library is closed");
+        const { collections } = await this.client.collectionsSince({
+            sinceTime: 0,
+        });
+        return collections;
+    }
+
     // Back up every in-scope file to `opts.downloadDirectory`, or else the
     // library's, each original at its save path, with a durable failure
     // ledger (issue #51). Takes the lock in that directory first, unless the
@@ -742,18 +778,28 @@ export class Library {
         // Stage per-collection file diffs. A collection's files are
         // re-enumerated only when its updationTime has advanced past the cached
         // copy; an unchanged album's file list cannot have changed. New
-        // collections enumerate from the beginning of time.
+        // collections enumerate from the beginning of time. Each collection
+        // whose files are fetched is reported, with how many there are to
+        // fetch.
+        const changedCollections = page.collections.filter((collection) => {
+            const known = this.store.getCollection(collection.id);
+            return !known || collection.updationTime > known.updationTime;
+        });
         const filePages: { collectionID: number; page: FilesPage }[] = [];
-        for (const collection of page.collections) {
+        for (const collection of changedCollections) {
             const known = this.store.getCollection(collection.id);
-            if (known && collection.updationTime <= known.updationTime)
-                continue;
             const filePage = await this.client.filesSince({
                 collectionID: collection.id,
                 collectionKey: collection.key,
                 sinceTime: known ? known.updationTime : 0,
             });
             filePages.push({ collectionID: collection.id, page: filePage });
+            this.emit({
+                operation: "refresh",
+                status: "progress",
+                fetched: filePages.length,
+                total: changedCollections.length,
+            });
         }
 
         // Network work done; commit to the store and persist only if something