quak get and get-thumb stop fetching collections once they find the file (closes #222)

`quak get` and `quak get-thumb` used to wait for a full refresh, which on
an empty cache fetched the files of every collection. `freshFile` now
reads the collection list, then one collection's files at a time, and
stops at the first that holds the file. It tries first the collections
the cache says hold the file, and stores nothing. The commands fetch the
content through the new `Library.originalOf()` and
`Library.thumbnailOf()`, which take the file record itself, since the
store may not hold it.

They no longer print a refresh's progress lines. The `-q` test that
relied on `get` refreshing now runs `helper fix-missing-thumbnails`.

Model: opus-5-5

src/cli-commands.ts

@@ -132,9 +132,10 @@ const readLibraryClient = (client: Client): LibraryClient => ({
 // precache (issue #48) is off, because a one-shot `collections` or `get` must
 // not start downloading the whole account. No refresh starts as the library
 // opens or on a timer. Each command asks for the refresh it needs.
-// `collections` and `files` with a collection ID need none (issues #210 and
-// #217). While a refresh runs, each collection whose files it has fetched is
-// logged, so a large account shows progress rather than nothing.
+// `collections` needs none (issue #210). Nor do `get`, `get-thumb` and `files`
+// with a collection ID (issues #217 and #222). While a refresh runs, each
+// collection whose files it has fetched is logged, so a large account shows
+// progress rather than nothing.
 const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
     Library.open({
         client: readLibraryClient(client),
@@ -322,16 +323,15 @@ export const getCommand = async (
 
     const lib = await openReadLibrary(ctx, client);
     try {
-        // Force a server round-trip so the file resolves against current state
-        // (issue #36 amendment, issue #52).
-        const resolved = await freshFile(lib, fileID);
-        if (!resolved) {
+        // Ask the server for the file as it is now (issue #36 amendment,
+        // issues #52 and #222).
+        const file = await freshFile(lib, fileID);
+        if (!file) {
             ctx.log("fatal", `File ${fileID} not found`);
             return 1;
         }
-        const { photo, file } = resolved;
 
-        const result = await photo.original();
+        const result = await lib.originalOf(file);
         // Default name is the file's own title, as the pre-library CLI used
         // (not the editedName-preferring projection title) (issue #52).
         if (result.videoPath === undefined) {
@@ -380,16 +380,15 @@ export const getThumbCommand = async (
 
     const lib = await openReadLibrary(ctx, client);
     try {
-        // Force a server round-trip so the file resolves against current state
-        // (issue #36 amendment, issue #52).
-        const resolved = await freshFile(lib, fileID);
-        if (!resolved) {
+        // Ask the server for the file as it is now (issue #36 amendment,
+        // issues #52 and #222).
+        const file = await freshFile(lib, fileID);
+        if (!file) {
             ctx.log("fatal", `File ${fileID} not found`);
             return 1;
         }
-        const { photo, file } = resolved;
 
-        const result = await photo.thumbnail();
+        const result = await lib.thumbnailOf(file);
         // Default name is thumb_<file's own title>, as the pre-library CLI
         // used (not the projection title) (issue #52).
         const outPath = opts.out ?? thumbnailName(file);

src/cli-read.ts

@@ -2,15 +2,16 @@
 //
 // `collections`, `files`, `get`, and `get-thumb` must answer for
 // the account's state at the moment the command runs, not for whatever the
-// local cache last happened to hold (owner amendment, issue #36). `get`,
-// `get-thumb` and `files` without a collection ID therefore force a server
-// round-trip through `Library.fresh()` and only then read, so a change made
-// elsewhere is visible. `collections` asks the library for the collection list
-// alone through `Library.freshCollections()` (issue #210): a refresh would also
-// fetch the files of every collection, which on a large account takes long and
-// which `collections` does not print. `files` with a collection ID reads that
-// list too, then that one collection's files through `Library.freshFiles()`
-// (issue #217).
+// local cache last happened to hold (owner amendment, issue #36). `files`
+// without a collection ID therefore forces a server round-trip through
+// `Library.fresh()` and only then reads, so a change made elsewhere is visible.
+// `collections` asks the library for the collection list alone through
+// `Library.freshCollections()` (issue #210): a refresh would also fetch the
+// files of every collection, which on a large account takes long and which
+// `collections` does not print. `files` with a collection ID reads that list
+// too, then that one collection's files through `Library.freshFiles()` (issue
+// #217). `get` and `get-thumb` read that list, then one collection's files at a
+// time, until they find the file (issue #222).
 //
 // `collections` and `files` with a collection ID list in the server's order.
 // `files` without one lists in the library's own enumeration order
@@ -22,7 +23,6 @@
 // first collection's record is listed.
 
 import type { Collection, EnteFile } from "./model/types.js";
-import type { Photo, PhotosAPI } from "./library/index.js";
 
 // The slice of `Library` these helpers read. `Library` satisfies it
 // structurally; a test can drive them with a stand-in that records each fresh
@@ -33,8 +33,7 @@ export interface FreshReadLibrary {
     freshFiles(collection: Collection): Promise<EnteFile[]>;
     listCollections(): Collection[];
     listFiles(collectionID: number): EnteFile[];
-    getFileByID(fileID: number): EnteFile | undefined;
-    photos: Pick<PhotosAPI, "byID">;
+    getFile(collectionID: number, fileID: number): EnteFile | undefined;
 }
 
 // Every live collection, current as of one collections request, in the
@@ -70,16 +69,27 @@ export const freshFiles = async (
     return files;
 };
 
-// One file, current as of a forced refresh, resolved to both its content
-// handle (`Photo`, for fetching bytes) and its raw record (`EnteFile`, for the
-// default output name and field values). `undefined` when the file is unknown.
+// One file as the server holds it now, or `undefined` when no collection holds
+// it. It reads the collection list, then one collection's files at a time, and
+// stops at the first collection that holds the file. It tries first the
+// collections the cache says hold the file, then the rest in the server's
+// order. It stores nothing.
 export const freshFile = async (
     lib: FreshReadLibrary,
     fileID: number,
-): Promise<{ photo: Photo; file: EnteFile } | undefined> => {
-    await lib.fresh();
-    const photo = lib.photos.byID({ fileID });
-    const file = lib.getFileByID(fileID);
-    if (!photo || !file) return undefined;
-    return { photo, file };
+): Promise<EnteFile | undefined> => {
+    const collections = await lib.freshCollections();
+    // Whether the cache holds the file in collection `c`.
+    const inCache = (c: Collection): boolean =>
+        lib.getFile(c.id, fileID) !== undefined;
+    const ordered = [
+        ...collections.filter(inCache),
+        ...collections.filter((c) => !inCache(c)),
+    ];
+    for (const collection of ordered) {
+        const files = await lib.freshFiles(collection);
+        const file = files.find((f) => f.id === fileID);
+        if (file) return file;
+    }
+    return undefined;
 };

src/library/content.ts

@@ -581,14 +581,34 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         fileID: number,
         opts?: ContentOptions,
     ): Promise<ContentResult> {
-        return this.get(fileID, "original", "on-demand", opts?.onProgress);
+        const file = this.fileOrThrow(fileID);
+        return this.get(file, "original", "on-demand", opts?.onProgress);
     }
 
     async thumbnail(
         fileID: number,
         opts?: ContentOptions,
     ): Promise<ContentResult> {
-        return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
+        const file = this.fileOrThrow(fileID);
+        return this.get(file, "thumbnail", "on-demand", opts?.onProgress);
+    }
+
+    // The original of `file`, fetched and cached as `original` does. `file` is
+    // used as given, so `getFile` need not know it.
+    async originalOf(
+        file: EnteFile,
+        opts?: ContentOptions,
+    ): Promise<ContentResult> {
+        return this.get(file, "original", "on-demand", opts?.onProgress);
+    }
+
+    // The thumbnail of `file`, fetched and cached as `thumbnail` does. `file`
+    // is used as given, so `getFile` need not know it.
+    async thumbnailOf(
+        file: EnteFile,
+        opts?: ContentOptions,
+    ): Promise<ContentResult> {
+        return this.get(file, "thumbnail", "on-demand", opts?.onProgress);
     }
 
     // Put the original at the save path of `file` under the download directory
@@ -615,7 +635,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         destination: string,
     ): Promise<ContentResult> {
         const result = await this.acquire(
-            fileID,
+            this.fileOrThrow(fileID),
             "original",
             "on-demand",
             undefined,
@@ -638,8 +658,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         fileID: number,
         destination: string,
     ): Promise<{ path: string; videoPath?: string }> {
-        const file = this.getFile(fileID);
-        if (!file) throw new Error(`content cache: unknown file ${fileID}`);
+        const file = this.fileOrThrow(fileID);
         return this.pools.content.run(
             async () => {
                 const fetched = await this.fetchInto(
@@ -721,7 +740,8 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         onProgress: ((event: EnsureEvent) => void) | undefined,
     ): Promise<EnsureResult> {
         try {
-            const result = await this.acquire(fileID, kind, priority, signal);
+            const file = this.fileOrThrow(fileID);
+            const result = await this.acquire(file, kind, priority, signal);
             const status = result.cached ? "skipped" : "done";
             onProgress?.({ fileID, status, path: result.path });
             return { fileID, path: result.path };
@@ -736,8 +756,16 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         }
     }
 
+    // The record `getFile` gives for `fileID`. It throws for a file `getFile`
+    // does not know.
+    private fileOrThrow(fileID: number): EnteFile {
+        const file = this.getFile(fileID);
+        if (!file) throw new Error(`content cache: unknown file ${fileID}`);
+        return file;
+    }
+
     private async get(
-        fileID: number,
+        file: EnteFile,
         kind: Kind,
         priority: Priority,
         onProgress: ((event: ContentEvent) => void) | undefined,
@@ -745,7 +773,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         const onByte: ProgressCallback | undefined = onProgress
             ? (bytesDone) => onProgress({ status: "downloading", bytesDone })
             : undefined;
-        const result = await this.acquire(fileID, kind, priority, undefined, {
+        const result = await this.acquire(file, kind, priority, undefined, {
             onByte,
         });
         onProgress?.(
@@ -767,7 +795,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
     // photo's original is present only with its video, and is returned with
     // it.
     private async acquire(
-        fileID: number,
+        file: EnteFile,
         kind: Kind,
         priority: Priority,
         signal: AbortSignal | undefined,
@@ -778,8 +806,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         videoPath?: string;
         cached: boolean;
     }> {
-        const file = this.getFile(fileID);
-        if (!file) throw new Error(`content cache: unknown file ${fileID}`);
+        const fileID = file.id;
         const isLivePhoto =
             kind === "original" && file.metadata.fileType === "livePhoto";
 

src/library/index.ts

@@ -66,6 +66,8 @@ import {
     ContentCache,
     savePath,
     storedAtSavePath,
+    type ContentOptions,
+    type ContentResult,
     type ContentSource,
     type ThumbnailsAPI,
     type EnsureOptions,
@@ -644,6 +646,36 @@ export class Library {
         return [...byID.values()];
     }
 
+    // The original of `file`, fetched and cached as `Photo.original()` does.
+    // `file` is used as given, so the store need not hold it. `quak get`
+    // passes a file that `freshFiles()` returned.
+    async originalOf(
+        file: EnteFile,
+        opts?: ContentOptions,
+    ): Promise<ContentResult> {
+        if (!this.cache) {
+            throw new Error(
+                "originalOf requires a library opened with a content cache",
+            );
+        }
+        return this.cache.originalOf(file, opts);
+    }
+
+    // The thumbnail of `file`, fetched and cached as `Photo.thumbnail()` does.
+    // `file` is used as given, so the store need not hold it. `quak get-thumb`
+    // passes a file that `freshFiles()` returned.
+    async thumbnailOf(
+        file: EnteFile,
+        opts?: ContentOptions,
+    ): Promise<ContentResult> {
+        if (!this.cache) {
+            throw new Error(
+                "thumbnailOf requires a library opened with a content cache",
+            );
+        }
+        return this.cache.thumbnailOf(file, opts);
+    }
+
     // Back up every in-scope file to `opts.downloadDirectory`, or else the
     // library's, each original at its save path, with a durable failure
     // ledger (issue #51). Takes the lock in that directory first, unless the