quak files COLLECTION fetches that collection's files alone (closes #217)

`quak files COLLECTION` used to wait for a full refresh, which on an
empty cache fetched the files of every collection first. It now reads
the collection list, then calls the new `Library.refreshCollection()`.
That brings one collection up to date in the store as a refresh would.
Once the stored cursor is past 0, it only updates a collection the store
holds. For a new one, `quak files COLLECTION` waits for a refresh. A
refresh from a cursor of 0 replaces all the store holds, so collections
keep the server's order. `open()` awaits the first refresh while the
cursor is 0. The store lists what did not decrypt in a fixed order.

Model: opus-5-5

src/cli-commands.ts

@@ -128,12 +128,13 @@ const readLibraryClient = (client: Client): LibraryClient => ({
     contentSource: () => client.contentSource(),
 });
 
-// Open a library for a single point command: the aggressive background precache
-// (issue #48) is off — a one-shot `collections` or `get` must not start
-// downloading the whole account — and no refresh starts as it opens or on a
-// timer: each command asks for the one it needs, and `collections` needs none
-// (issue #210). While a refresh runs, each collection whose files it has
-// fetched is logged, so a large account shows progress rather than nothing.
+// Open a library for a single point command. The aggressive background
+// precache (issue #48) is off, because a one-shot `collections` or `get` must
+// not start downloading the whole account. No refresh starts as it opens or on
+// a timer. Each command asks for the refresh it needs, and `collections` and
+// `files COLLECTION` need none (issues #210 and #217). While a refresh runs,
+// each collection whose files it has fetched is logged, so a large account
+// shows progress rather than nothing.
 const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
     Library.open({
         client: readLibraryClient(client),

src/cli-read.ts

@@ -2,33 +2,42 @@
 //
 // `collections`, `files`, `get`, and `get-thumb` must answer for
 // the account's state at the moment the command runs, not for whatever the
-// local cache last happened to hold (owner amendment, issue #36). `files`,
-// `get` and `get-thumb` therefore force a server round-trip through
-// `Library.fresh()` and only then read — so a collection, file, or metadata
-// change made elsewhere is visible. `collections` asks the library for the
-// collection list alone through `Library.freshCollections()` (issue #210): a
-// refresh would also fetch the files of every collection, which on a large
-// account takes long and which `collections` does not print.
+// local cache last happened to hold (owner amendment, issue #36). `files`
+// without a collection ID, `get` and `get-thumb` therefore force a server
+// round-trip through `Library.fresh()`, and only then read. A collection, file,
+// or metadata change made elsewhere is then visible. `collections` asks the
+// library for the collection list alone through `Library.freshCollections()`
+// (issue #210). A refresh would also fetch the files of every collection, which
+// `collections` does not print, and on a large account that takes long.
+// `files COLLECTION` reads the collection list the same way, for the same
+// reason. It then brings that one collection up to date through
+// `Library.refreshCollection()` (issue #217), and fetches no other collection's
+// files. Once a refresh has received the account's collections,
+// `refreshCollection()` stores no collection the library does not hold.
+// `files COLLECTION` then forces a refresh instead.
 //
-// `collections` lists in the server's order and `files` in the library's own
-// enumeration order (`listFiles()`), the orders the pre-library CLI printed,
-// rather than the `albums`/`photos` projection's newest-first order, which
-// re-sorts the rows. The field values still come from each record's raw
-// metadata via `cli-output.ts`. `files` without a collection ID lists every
-// file in the account once (issue #211): the library keeps a record of a file
-// for each collection that holds it, and only the first collection's record is
-// listed.
+// `collections` lists in the server's order. `files` lists in the library's
+// own enumeration order: `listCollections()`, then `listFiles()` for each.
+// Those are the orders the pre-library CLI printed. The `albums`/`photos`
+// projection would re-sort the rows newest first. After a refresh, the
+// library holds collections in the order the server first sent them. The
+// field values still come from each record's raw metadata via
+// `cli-output.ts`. `files` without a collection ID lists every file in the
+// account once (issue #211).
+// The library keeps a record of a file for each collection that holds it, and
+// only the first collection's record is listed.
 
 import type { Collection, EnteFile } from "./model/types.js";
 import type { Photo, PhotosAPI } from "./library/index.js";
 
 // The slice of `Library` these helpers read. `Library` satisfies it
-// structurally; a test can drive them with a stand-in that records the
-// `fresh()` and `freshCollections()` calls and serves records in a known
-// enumeration order.
+// structurally. A test can drive them with a stand-in that records the
+// `fresh()`, `freshCollections()` and `refreshCollection()` calls. The
+// stand-in serves records in a known enumeration order.
 export interface FreshReadLibrary {
     fresh(): Promise<unknown>;
     freshCollections(): Promise<Collection[]>;
+    refreshCollection(collection: Collection): Promise<boolean>;
     getCollection(id: number): Collection | undefined;
     listCollections(): Collection[];
     listFiles(collectionID: number): EnteFile[];
@@ -42,21 +51,29 @@ export const freshCollections = (
     lib: FreshReadLibrary,
 ): Promise<Collection[]> => lib.freshCollections();
 
-// The files of one collection, current as of a forced refresh, in enumeration
-// order. `undefined` (not an empty list) when the collection does not exist, so
-// the caller can tell "no such collection" from "an empty collection". Without
-// a collection ID, every file in the account, each once, as held by the first
-// collection that holds it.
+// The files of one collection, in enumeration order. `undefined` (not an empty
+// list) when the collection does not exist, so the caller can tell "no such
+// collection" from "an empty collection". They are current as of one
+// collections request and the requests that bring that collection alone up to
+// date. When `refreshCollection()` stores nothing, they are current as of a
+// forced refresh instead. Without a collection ID, every file in the account,
+// current as of a forced refresh. Each file is listed once, as held by the
+// first collection that holds it.
 export const freshFiles = async (
     lib: FreshReadLibrary,
     collectionID?: number,
 ): Promise<EnteFile[] | undefined> => {
-    await lib.fresh();
     if (collectionID !== undefined) {
-        return lib.getCollection(collectionID)
-            ? lib.listFiles(collectionID)
-            : undefined;
+        const collections = await lib.freshCollections();
+        const collection = collections.find((c) => c.id === collectionID);
+        if (!collection) return undefined;
+        if (!(await lib.refreshCollection(collection))) {
+            await lib.fresh();
+            if (!lib.getCollection(collectionID)) return undefined;
+        }
+        return lib.listFiles(collectionID);
     }
+    await lib.fresh();
     const listed = new Set<number>();
     const files: EnteFile[] = [];
     for (const collection of lib.listCollections()) {

src/library/index.ts

@@ -1,28 +1,36 @@
 // The library surface over the local cache.
 //
 // `Library.open()` loads the on-disk metadata store (issue #41), then starts
-// the refresh loop. When the cache loaded empty it awaits the first refresh,
-// so the library never opens onto an empty store it could have filled; when an
-// existing copy loaded, that first refresh runs in the background and `open()`
-// returns as soon as the cached data is ready to serve — a slow or unreachable
-// server no longer stalls opening. A background timer then refreshes every
-// `refreshIntervalSeconds`. Every default read is answered from RAM — no
-// default read touches the network. There is deliberately no `sync()`, no
-// `refresh()`, no `serverReachable` flag, and no "before each read" mode
-// (design #36).
+// the refresh loop. When the stored cursor is 0, `open()` awaits the first
+// refresh. No refresh has then stored any collection: the cache is empty, or
+// holds only what `refreshCollection()` stored. The library must not serve
+// that as the whole account. Otherwise that first refresh runs in the
+// background. `open()` returns as soon as the cached data is ready to serve,
+// so a slow or unreachable server does not stall opening. A background timer
+// then takes a turn `refreshIntervalSeconds` after its last turn ends. Each
+// turn starts a refresh, unless a refresh or `refreshCollection()` is already
+// running. Every default read is answered from RAM — no default read touches
+// the network. There is deliberately no `sync()`, no `refresh()`, no
+// `serverReachable` flag, and no "before each read" mode (design #36).
 //
-// `fresh()` is the one exception (issue #75, an owner amendment to #36): it
-// forces a refresh, awaits it, and only then hands back the read namespaces, so
-// a caller that needs server-current data can ask for it. Concurrent `fresh()`
-// calls coalesce onto one in-flight refresh, and a refresh that fails rejects
-// the caller (the default reads stay silent and serve the last good copy). The
-// default methods and the background loop are unchanged.
+// `fresh()` is an exception (issue #75, an owner amendment to #36). It awaits
+// a refresh, and only then hands back the read namespaces. A caller that needs
+// server-current data uses it. Concurrent `fresh()` calls coalesce onto one
+// in-flight refresh. A refresh that fails rejects the caller. The default
+// reads stay silent and serve the last good copy. The default methods and the
+// background loop are unchanged.
 //
 // `freshCollections()` is the fresh read of the collection list alone (issue
-// #210): one collections request, no collection's files, and nothing stored,
-// for a caller such as `quak collections` that needs no more. A caller that
-// asks for each refresh it needs can also open the library with
-// `refreshOnOpen: false`, so that `open()` starts none.
+// #210). It sends one collections request and stores nothing.
+// `quak collections` uses it. `refreshCollection()` brings one collection
+// from that list up to date in the store (issue #217). It fetches that
+// collection's files alone, as a refresh would. `quak files COLLECTION` uses
+// it. It never runs at the same time as a refresh. Once the stored cursor is
+// past 0, it only updates a collection the store holds, and only a refresh
+// adds one. A refresh from a stored cursor of 0 replaces all the store holds.
+// So the store keeps collections in the server's order. A caller that asks for
+// each refresh it needs can also open the library with `refreshOnOpen: false`,
+// so that `open()` starts none.
 //
 // A refresh reports through `onProgress` each collection whose files it has
 // fetched, so a large account, whose first refresh fetches the files of every
@@ -31,10 +39,12 @@
 // A refresh stages all of its network work first and only mutates the store
 // once every fetch has succeeded. A refresh that fails partway therefore never
 // becomes visible to reads: the last good snapshot stays in place, and the
-// failure surfaces through `onProgress` and `status()` instead. A commit that
-// mutates RAM but then fails to persist keeps `status().lastError` set and the
-// store marked unsaved until a later save actually lands, so a stuck disk is
-// never masked by a subsequent empty refresh.
+// failure surfaces through `onProgress` and `status()` instead. A save can
+// still fail after the store has changed in RAM. The store then stays marked
+// unsaved until a later save lands. Until then, every `commit()` retries the
+// save. A refresh whose save fails sets `status().lastError`, which stays set
+// until a later refresh succeeds. `refreshCollection()` never sets or clears
+// it. So a stuck disk is never hidden by a later refresh that changes nothing.
 
 import { rm } from "node:fs/promises";
 import { join, resolve } from "node:path";
@@ -159,21 +169,23 @@ export interface LibraryClient {
         payloads: Map<number, MLData>;
         errors: Map<number, unknown>;
     }>;
-    // The byte source for the on-disk content cache. Optional so a mock client
-    // that only serves metadata still satisfies the interface; when absent (and
-    // no explicit `contentSource` is passed to `open`) the content cache is
-    // disabled and `Photo.original`/`thumbnail` and `thumbnails.ensure` throw.
+    // The byte source for the on-disk content cache. Optional, so a mock
+    // client that only serves metadata still satisfies the interface. Without
+    // it, and with no `contentSource` passed to `open`, the content cache is
+    // disabled. `Photo.original`/`thumbnail` and `thumbnails.ensure` then
+    // throw.
     contentSource?(): ContentSource;
 }
 
-// A progress event for one unit of background work. A metadata "refresh" or an
-// ML "fetchMLData" pass each fire "started" before their network work and then
-// exactly one of "done" or "failed"; "failed" carries the error message and
-// an ML "done" reports how many payloads it stored and, in `mlDataErrors`,
-// each file whose ML data did not decrypt, gunzip or parse, with the reason.
-// In between, a refresh fires "progress" after each collection whose files it
-// fetched: `fetched` is how many collections it has fetched the files of so
-// far, `total` how many it will. The precache fills
+// A progress event for one unit of background work. A metadata "refresh" and
+// an ML "fetchMLData" pass each fire "started" before their network work. Each
+// then fires exactly one of "done" or "failed". A "failed" event carries the
+// error message. An ML "done" reports in `fetched` how many payloads it
+// stored. Its `mlDataErrors` lists each file whose ML data did not decrypt,
+// gunzip or parse, with the reason. In between, a refresh fires "progress"
+// after each collection whose files it fetched. Its `fetched` is how many
+// collections it has fetched the files of so far, and `total` how many it
+// will. The precache fills
 // ("precacheThumbnails"/"precacheOriginals", #48) fire "started"/"done" around
 // each sweep that has work, "done" reporting the count newly cached.
 export interface RefreshEvent {
@@ -201,9 +213,10 @@ export interface LibraryOptions {
     // directory at open. The content cache treats an original already stored
     // at its save path as present.
     downloadDirectory?: string;
-    // How long after one refresh ends the background timer starts the next.
-    // `Infinity` starts none: the library then refreshes only as it opens and
-    // when `fresh()` or `backup()` asks.
+    // How long after its last turn ends the background timer takes the next.
+    // A turn starts a refresh unless a refresh or `refreshCollection()` is
+    // running. With `Infinity` the timer takes no turn: the library then
+    // refreshes only as it opens and when `fresh()` or `backup()` asks.
     refreshIntervalSeconds?: number;
     // Whether `open()` starts a refresh. Defaults to true. With false, `open()`
     // returns at once onto what the cache holds, empty or not, and a refresh
@@ -242,8 +255,9 @@ export interface LibraryStatus {
     // Wall-clock ms of the last refresh that succeeded, or undefined if none
     // has yet.
     lastRefreshAt?: number;
-    // The message from the most recent refresh, set only while that refresh
-    // failed; cleared by the next success.
+    // The error of the most recent refresh, set only when it failed. The next
+    // refresh that succeeds clears it. `refreshCollection()` never sets or
+    // clears it.
     lastError?: string;
     // Wall-clock ms of the last ML fetch pass that succeeded, or undefined if
     // none has yet (or ML fetching is disabled).
@@ -306,12 +320,17 @@ export class Library {
     private readonly precache?: Precache;
 
     private timer?: ReturnType<typeof setTimeout>;
-    // The in-flight refresh cycle, or undefined when none runs. One slot serves
-    // both paths: the background loop skips when it is set, and a fresh read
-    // (issue #75) coalesces onto it or starts one. The promise carries the
-    // cycle's real outcome (it rejects on failure); the background loop ignores
-    // that, a fresh read propagates it.
+    // The in-flight refresh cycle or `refreshCollection()`, or undefined when
+    // neither runs. One slot holds both, so they never run at the same time.
+    // The background loop skips while it is set. A fresh read (issue #75)
+    // coalesces onto a running refresh. It waits for a running
+    // `refreshCollection()` to end, and then joins or starts a refresh. The
+    // promise carries the real outcome: it rejects on failure. The background
+    // loop ignores a failure. A fresh read passes on the failure of the
+    // refresh it joins or starts, and `refreshCollection()` passes on its own.
     private cycle?: Promise<void>;
+    // Whether `cycle` is a refresh cycle rather than a `refreshCollection()`.
+    private cycleIsRefresh = false;
     // Guards the ML fetch pass so a slow backfill never runs twice at once; a
     // refresh whose pass is still running kicks nothing new. Holds the running
     // pass, so `close()` and `backup()` can wait for it. It rejects when the
@@ -324,19 +343,20 @@ export class Library {
     private lastMLError?: string;
     // For each file whose ML data the last ML fetch pass that succeeded got
     // but could not decrypt, gunzip or parse, why not, by fileID. A pass
-    // gathers its own and puts them here when it ends without failing, so
-    // `status()` and `backup()` never see the list of a pass that is still
-    // running or that failed. Kept in RAM only, never in the cache.
+    // gathers its own and puts them here only when it ends without failing.
+    // So `status()` and `backup()` never see the list of a pass still running
+    // or one that failed. Kept in RAM only, never in the cache.
     private mlDataErrors = new Map<number, unknown>();
-    // The plain-record projection as of the last refresh, and the GUI change
-    // subscribers. A refresh that alters the projection notifies each with the
-    // delta; `lastRecords` is kept current every refresh so a subscriber that
-    // joins later diffs against the state its own `snapshot()` already returned.
+    // The plain-record projection as of the last store change, and the GUI
+    // change subscribers. A refresh or `refreshCollection()` that alters the
+    // projection notifies each subscriber with the delta. `lastRecords`
+    // advances on each `commit()` that changes the store. A subscriber that
+    // joins later then diffs against the state its own `snapshot()` returned.
     private readonly subscribers = new Set<(change: LibraryChange) => void>();
     private lastRecords: DerivedRecords;
     // RAM holds changes disk has not yet accepted (an earlier save failed).
-    // Cleared only when a save actually succeeds; keeps the store trying to
-    // persist and the failure visible in `status()` until then.
+    // Cleared only when a save succeeds. Until then, every `commit()` retries
+    // the save.
     private unsaved = false;
 
     private constructor(args: {
@@ -366,7 +386,7 @@ export class Library {
         this.lastRecords = this.deriveNow();
 
         // The read namespaces derive fresh from the store on each call, so they
-        // always reflect the latest refresh.
+        // always reflect what it holds now.
         const derive = (): DerivedRecords => this.deriveNow();
         const root = this.downloadDirectory;
         const saves: SavePathLookup = {
@@ -393,14 +413,16 @@ export class Library {
         this.mldata = makeMLDataAPI(() => this.mlStore);
     }
 
-    // Load the cache and start the refresh loop. With an empty cache the first
-    // refresh is awaited, so `open()` resolves onto populated data whenever the
-    // server is reachable; that awaited refresh may still fail, and the library
-    // then opens empty with the failure recorded in `status()`. With an
-    // existing cache the first refresh runs in the background and `open()`
-    // returns as soon as the cached data is ready — an unreachable server does
-    // not block opening. With `refreshOnOpen: false` there is no first
-    // refresh: `open()` returns onto the cache as it is.
+    // Load the cache and start the refresh loop. When the stored cursor is 0,
+    // the first refresh is awaited. Such a cache is empty, or holds only what
+    // `refreshCollection()` stored. `open()` then resolves onto the whole
+    // account whenever the server is reachable. If that refresh fails, the
+    // library opens onto what the cache holds, with the failure recorded in
+    // `status()`. With any other cache the first refresh runs in the
+    // background, and `open()` returns as soon as the cached data is ready.
+    // An unreachable server then does not block opening. With
+    // `refreshOnOpen: false` there is no first refresh: `open()` returns onto
+    // the cache as it is.
     static async open(opts: LibraryOptions): Promise<Library> {
         const { userID } = opts.client.whoami();
         const cacheDirectory =
@@ -416,8 +438,8 @@ export class Library {
         let store = await MetadataStore.load(metadataPath);
         // A cache directory given explicitly can hold another account's cache.
         // Its records and cursor are not this account's, so delete it and the
-        // ML data beside it and start empty. A user ID of 0 means the cache
-        // was never refreshed and so holds nothing to discard.
+        // ML data beside it and start empty. A user ID of 0 means nothing was
+        // ever stored there, so there is nothing to discard.
         if (store.userID !== 0 && store.userID !== userID) {
             await rm(metadataPath, { force: true });
             await rm(join(cacheDirectory, "mldata"), {
@@ -440,10 +462,11 @@ export class Library {
             ? await MLDataStore.open(join(cacheDirectory, "mldata"))
             : undefined;
 
-        // Build the content cache from an explicit source or the client's own,
-        // and take its record of what is already cached (and reap orphan temp
-        // files) before the first projection, so cached paths are present from
-        // the start and the first refresh raises no spurious path-change diff.
+        // Build the content cache from an explicit source or the client's own.
+        // Take its record of what is already cached, and reap orphan temp
+        // files, before the first projection. Cached paths are then present
+        // from the start, and the first `commit()` raises no spurious
+        // path-change diff.
         const source = opts.contentSource ?? opts.client.contentSource?.();
         let cache: ContentCache | undefined;
         let precache: Precache | undefined;
@@ -497,22 +520,26 @@ export class Library {
             precache,
         });
 
-        // Start filling from whatever the loaded store already holds; each
-        // refresh below re-kicks with the new files (and retries any that
-        // failed). An empty store starts empty here and fills after its first
-        // refresh.
+        // Start filling from whatever the loaded store already holds. Each
+        // later `commit()` re-kicks the fills with its new files, and retries
+        // any that failed. An empty store starts empty here, and fills once a
+        // `commit()` stores files.
         precache?.start();
 
         if (opts.refreshOnOpen === false) {
             // The caller asks for each refresh it needs.
             lib.scheduleNext();
-        } else if (store.loadedFromDisk) {
-            // An existing copy already answers reads; refresh in the background
-            // and start the interval once that first cycle settles.
+        } else if (store.collectionsSinceTime > 0) {
+            // The stored cursor has moved, so a refresh has stored the
+            // account's collections and the cache's copy answers reads.
+            // Refresh in the background, and start the interval once that
+            // first cycle settles.
             void lib.runRefresh().then(() => lib.scheduleNext());
         } else {
-            // Nothing was cached: wait for the first refresh to fill the store
-            // (or fail) rather than resolve onto an empty library.
+            // The stored cursor is still 0, so no refresh has stored any
+            // collection: the cache is empty, or holds only what
+            // `refreshCollection()` stored. Wait for the first refresh to fill
+            // the store (or fail) rather than resolve onto part of the account.
             await lib.runRefresh();
             lib.scheduleNext();
         }
@@ -550,9 +577,9 @@ export class Library {
         return snapshotFrom(this.deriveNow(), Date.now());
     }
 
-    // Deliver a `LibraryChange` whenever a refresh alters the projection. A
-    // refresh that changes nothing delivers nothing. The returned handle's
-    // `unsubscribe` stops delivery.
+    // Deliver a `LibraryChange` whenever a refresh or `refreshCollection()`
+    // alters the projection. One that changes nothing delivers nothing. The
+    // returned handle's `unsubscribe` stops delivery.
     subscribe(args: { onChange: (change: LibraryChange) => void }): {
         unsubscribe: () => void;
     } {
@@ -595,12 +622,13 @@ export class Library {
         };
     }
 
-    // Fresh reads (issue #75, owner amendment to design #36). Force a refresh,
-    // wait for it to complete and persist, then hand back the same
-    // `albums`/`photos`/`timeline` namespaces — now guaranteed to reflect a
-    // completed server round-trip. Concurrent calls coalesce onto one refresh;
-    // a refresh that fails rejects here, where the default namespaces would
-    // instead stay silent and serve the last good copy.
+    // Fresh reads (issue #75, owner amendment to design #36). Wait for a
+    // refresh to complete and persist, then hand back the same
+    // `albums`/`photos`/`timeline` namespaces. They then reflect a completed
+    // server round-trip. A running `refreshCollection()` is waited for first.
+    // Concurrent calls coalesce onto one refresh. A refresh that fails rejects
+    // here, where the default namespaces would stay silent and serve the last
+    // good copy.
     async fresh(): Promise<FreshReads> {
         await this.refreshNow();
         return {
@@ -613,10 +641,10 @@ export class Library {
     // The fresh read of the collection list alone (issue #210): every live
     // collection that decrypts, in the server's order, from one collections
     // request. Unlike a refresh, it fetches no collection's files, which on a
-    // large account take long. Nothing is stored: a refresh fetches a
+    // large account take long. It stores nothing. A refresh fetches a
     // collection's files only when the collection is new or has changed since
-    // the stored copy, so a collection stored here without its files would
-    // never have them fetched.
+    // the stored copy. A collection stored here without its files would never
+    // have them fetched.
     async freshCollections(): Promise<Collection[]> {
         if (this.closed) throw new Error("the library is closed");
         const { collections } = await this.client.collectionsSince({
@@ -625,19 +653,44 @@ export class Library {
         return collections;
     }
 
+    // Bring one collection up to date in the store (issue #217), given its
+    // record from `freshCollections()`. It does for that collection what a
+    // refresh does for a changed one. It fetches no other collection's files.
+    // It resolves to whether the store then holds the collection.
+    //
+    // It adds a collection the store does not hold only while the stored
+    // cursor is 0, before any refresh has received a collection. After that,
+    // it stores nothing for such a collection and resolves to false. Only a
+    // refresh may then add it, after the collections the store holds.
+    //
+    // It runs in the same slot as a refresh, so the two never run at the same
+    // time. It first waits for a running refresh or `refreshCollection()` to
+    // end. While its own work runs, a fresh read or `backup()` waits for it,
+    // and the background loop skips its turn.
+    async refreshCollection(collection: Collection): Promise<boolean> {
+        while (this.cycle) await this.cycle.catch(() => {});
+        if (this.closed) throw new Error("the library is closed");
+        let held = false;
+        await this.startCycle(false, async () => {
+            held = await this.refreshCollectionOnce(collection);
+        });
+        return held;
+    }
+
     // Back up every in-scope file to `opts.downloadDirectory`, or else the
     // library's, each original at its save path, with a durable failure
-    // ledger (issue #51). Takes the lock in that directory first, unless the
-    // caller passes the lock it took as `opts.lock`, failing at once while
-    // another backup of it runs, and before it starts another file once the
-    // lock is lost (see `lockBackupDirectory`). Waits for a
-    // completed refresh, as `fresh()` does, joining one already running, and
-    // rejects before touching any file but the lock when it fails. Then puts
-    // pending originals at their save paths as `Photo.download()` does (and
-    // optional thumbnails) through the content cache and pools, waits for an
-    // ML data fetch, and rebuilds the derived symlink/JSON views from the
-    // model, each file's JSON with its ML data, beside an `account.json` with
-    // the account's email and user ID.
+    // ledger (issue #51). It first takes the lock in that directory, unless
+    // the caller passes the lock it took as `opts.lock`. Taking it fails at
+    // once while another backup of that directory runs. Once the lock is lost,
+    // the backup fails before it starts another file (see
+    // `lockBackupDirectory`). It waits for a completed refresh as `fresh()`
+    // does, joining one already running. When that refresh fails, it rejects
+    // before touching any file but the lock. It then puts pending originals
+    // at their save paths as `Photo.download()` does, through the content
+    // cache and pools. With `includeThumbnails` it also fetches thumbnails. It
+    // waits for an ML data fetch. It rebuilds the derived symlink/JSON views
+    // from the model, each file's JSON with its ML data. An `account.json`
+    // beside them holds the account's email and user ID.
     // Throws before any network work when no content cache backs the
     // originals it must fetch.
     backup(opts?: BackupOptions): Promise<BackupResult> {
@@ -674,12 +727,15 @@ export class Library {
         );
     }
 
-    // Stop the background timer. Idempotent. An in-flight refresh is left to
-    // finish; it will not schedule another cycle once closed. The returned
-    // promise resolves once that refresh (including its cache write), the ML
-    // fetch pass and the precache fetches already running have all finished,
-    // so a caller can then remove the cache directory. A refresh failure is
-    // reported through `status()`, not thrown here.
+    // Stop the background timer. Idempotent. A refresh or `refreshCollection()`
+    // in flight is left to finish, and nothing starts after it. The returned
+    // promise resolves once all of these have finished:
+    //  - that refresh or `refreshCollection()`, with its cache write;
+    //  - the ML fetch pass;
+    //  - the precache fetches already running.
+    // A caller can then remove the cache directory. A refresh failure is
+    // reported through `status()`, and a `refreshCollection()` failure to its
+    // own caller. Neither is thrown here.
     async close(): Promise<void> {
         this.closed = true;
         const precacheClosed = this.precache?.close();
@@ -702,33 +758,41 @@ export class Library {
         this.timer.unref?.();
     }
 
-    // The background loop's refresh: run a cycle unless one is already in flight
-    // (or the library is closed), and never let a failure escape — the
-    // background path reports errors through `status()`/`onProgress`, it does
-    // not throw. Resolves once the cycle it started (or skipped past) settles.
+    // The background loop's refresh: run a cycle unless a refresh or
+    // `refreshCollection()` is in flight (or the library is closed), and never
+    // let a failure escape. The background path reports errors through
+    // `status()`/`onProgress`; it does not throw. It resolves once the cycle
+    // it started settles, or at once when it starts none.
     private runRefresh(): Promise<void> {
         if (this.closed || this.cycle) return Promise.resolve();
-        return this.startCycle().catch(() => {});
+        return this.startCycle(true, () => this.refreshCycle()).catch(() => {});
     }
 
-    // A fresh read's refresh (issue #75): force a cycle and await it, rejecting
-    // if it fails. Concurrent fresh reads coalesce onto the one in-flight cycle
-    // — the background loop's included — so they never fan out into redundant
-    // server round-trips.
-    private refreshNow(): Promise<void> {
-        if (this.closed) {
-            return Promise.reject(new Error("the library is closed"));
+    // A fresh read's refresh (issue #75): join the running refresh or start
+    // one, and await it, rejecting if it fails. Concurrent fresh reads
+    // coalesce onto the one in-flight cycle — the background loop's included —
+    // so they never fan out into redundant server round-trips. A running
+    // `refreshCollection()` is waited for first.
+    private async refreshNow(): Promise<void> {
+        while (this.cycle && !this.cycleIsRefresh) {
+            await this.cycle.catch(() => {});
         }
-        return this.cycle ?? this.startCycle();
+        if (this.closed) throw new Error("the library is closed");
+        return this.cycle ?? this.startCycle(true, () => this.refreshCycle());
     }
 
-    // Start one refresh cycle and record it as the in-flight cycle so every
-    // caller coalesces onto it. The returned promise carries the cycle's real
-    // outcome; each caller attaches the handling its own path needs, and the
-    // slot is cleared once the cycle settles.
-    private startCycle(): Promise<void> {
-        const cycle = this.refreshCycle();
+    // Start `work` and record it in the in-flight slot, so every other caller
+    // sees it. `isRefresh` says whether it is a refresh cycle, which a fresh
+    // read can coalesce onto. The returned promise carries the real outcome.
+    // Each caller attaches the handling its own path needs, and the slot is
+    // cleared once the work settles.
+    private startCycle(
+        isRefresh: boolean,
+        work: () => Promise<void>,
+    ): Promise<void> {
+        const cycle = work();
         this.cycle = cycle;
+        this.cycleIsRefresh = isRefresh;
         void cycle.then(
             () => {
                 if (this.cycle === cycle) this.cycle = undefined;
@@ -751,9 +815,9 @@ export class Library {
             this.lastError = undefined;
             this.emit({ operation: "refresh", status: "done" });
             // Backfill ML data for the files this refresh knows about. It runs
-            // outside the refresh's success/failure so a fetch or disk problem
-            // there never marks the metadata refresh failed, and it is not
-            // awaited so it never stalls the refresh interval. Its failure is
+            // outside the refresh's success or failure, so a fetch or disk
+            // problem there never marks the metadata refresh failed. It is not
+            // awaited, so it never stalls the refresh interval. Its failure is
             // reported through `status()` and `onProgress`.
             void this.fetchMLDataNow().catch(() => {});
         } catch (err) {
@@ -766,11 +830,14 @@ export class Library {
 
     // Fetch every change since the stored cursor, then commit. All network
     // reads happen before any store mutation, so a fetch that throws leaves the
-    // store untouched and the previous snapshot intact. A collection or file
-    // that does not decrypt is left out instead, as if deleted, with the reason
-    // kept in the store: the cursor moves past it, so no later refresh sees it
-    // again until the server's copy changes.
+    // store untouched and the previous snapshot intact.
+    //
+    // From a stored cursor of 0, the server lists every live collection. The
+    // refresh then replaces all the store holds, and counts every collection
+    // as new. Only `refreshCollection()` can have stored anything before it.
+    // So the store ends as it would had `refreshCollection()` never run.
     private async refreshOnce(): Promise<void> {
+        const replace = this.store.collectionsSinceTime === 0;
         const page = await this.client.collectionsSince({
             sinceTime: this.store.collectionsSinceTime,
         });
@@ -778,16 +845,19 @@ export class Library {
         // Stage per-collection file diffs. A collection's files are
         // re-enumerated only when its updationTime has advanced past the cached
         // copy; an unchanged album's file list cannot have changed. New
-        // collections enumerate from the beginning of time. Each collection
+        // collections enumerate from the beginning of time. When the refresh
+        // replaces the store, every collection counts as new. Each collection
         // whose files are fetched is reported, with how many there are to
         // fetch.
+        const storedCopy = (id: number): Collection | undefined =>
+            replace ? undefined : this.store.getCollection(id);
         const changedCollections = page.collections.filter((collection) => {
-            const known = this.store.getCollection(collection.id);
+            const known = storedCopy(collection.id);
             return !known || collection.updationTime > known.updationTime;
         });
         const filePages: { collectionID: number; page: FilesPage }[] = [];
         for (const collection of changedCollections) {
-            const known = this.store.getCollection(collection.id);
+            const known = storedCopy(collection.id);
             const filePage = await this.client.filesSince({
                 collectionID: collection.id,
                 collectionKey: collection.key,
@@ -802,8 +872,55 @@ export class Library {
             });
         }
 
-        // Network work done; commit to the store and persist only if something
-        // actually changed.
+        await this.commit(page, filePages, replace);
+    }
+
+    // `refreshCollection()`'s work, run in the slot: fetch the files of one
+    // collection changed since the stored copy, then commit. It resolves to
+    // whether the store then holds the collection.
+    private async refreshCollectionOnce(
+        collection: Collection,
+    ): Promise<boolean> {
+        const known = this.store.getCollection(collection.id);
+        // A refresh has received the account's collections, and this one is
+        // not stored. It is new since then, or a refresh dropped it. Leave it
+        // to the next refresh, which adds new collections after the stored
+        // ones, in the server's order.
+        if (!known && this.store.collectionsSinceTime > 0) return false;
+        // The stored copy is as new or newer, so there is nothing to fetch.
+        if (known && collection.updationTime <= known.updationTime) return true;
+        const filePage = await this.client.filesSince({
+            collectionID: collection.id,
+            collectionKey: collection.key,
+            sinceTime: known ? known.updationTime : 0,
+        });
+        // The stored cursor stays where it is, so the next refresh still
+        // fetches the files of every other collection that changed.
+        await this.commit(
+            {
+                collections: [collection],
+                deleted: [],
+                cursor: this.store.collectionsSinceTime,
+            },
+            [{ collectionID: collection.id, page: filePage }],
+            false,
+        );
+        return true;
+    }
+
+    // Commit a collections page and the file pages fetched for its collections
+    // to the store. With `replace`, the page lists every live collection, so
+    // the commit first deletes every stored collection. A collection or file
+    // that does not decrypt is left out, as if deleted. The reason is kept in
+    // the store. No later refresh sees it again until the server's copy
+    // changes. Last, save the store when it holds changes not yet saved.
+    private async commit(
+        page: CollectionsPage,
+        filePages: { collectionID: number; page: FilesPage }[],
+        replace: boolean,
+    ): Promise<void> {
+        // The network work is done, so change the store now. `changed` says
+        // whether this commit changed anything.
         let changed = false;
 
         if (this.store.userID !== this.userID) {
@@ -811,6 +928,13 @@ export class Library {
             changed = true;
         }
 
+        if (replace) {
+            for (const { id } of this.store.listCollections()) {
+                this.store.deleteCollection(id);
+                changed = true;
+            }
+        }
+
         for (const id of page.deleted) {
             if (this.store.getCollection(id)) {
                 this.store.deleteCollection(id);
@@ -864,10 +988,10 @@ export class Library {
         if (changed) this.unsaved = true;
 
         // Reproject and notify subscribers of the delta. This tracks RAM (what
-        // reads see), so it fires whether or not the save below succeeds; a
-        // save failure surfaces separately through `status().lastError`.
-        // `lastRecords` advances every changed refresh so the next diff is
-        // against current state.
+        // reads see), so it fires whether or not the save below succeeds. A
+        // failed save rejects the caller, and a refresh also records it in
+        // `status().lastError`. `lastRecords` advances on every commit that
+        // changes something, so the next diff is against current state.
         if (changed) {
             const next = this.deriveNow();
             if (this.subscribers.size > 0) {
@@ -878,16 +1002,16 @@ export class Library {
             // Recompute the fill orders and pinned set against the new library.
             this.precache?.update(next);
         }
-        // Re-kick the fills every cycle: a finished sweep starts afresh to pick
-        // up new files and retry any that failed, and a running one is left be.
+        // Re-kick the fills on every commit. A finished sweep starts afresh to
+        // pick up new files and retry any that failed. A running one is left
+        // be.
         this.precache?.start();
 
         // Persist whenever RAM holds changes disk has not accepted — including
         // changes an earlier cycle staged whose save failed. `unsaved` clears
-        // only once a save lands, so a save failure both stays visible through
-        // `status().lastError` (the throw below records it) and keeps being
-        // retried, instead of a later empty refresh silently clearing it while
-        // the on-disk cache is still behind RAM.
+        // only once a save lands, so a later commit retries a failed save even
+        // when it changes nothing. A failed save rejects the caller, and a
+        // refresh also records it in `status().lastError`.
         if (this.unsaved) {
             await this.store.save();
             this.unsaved = false;
@@ -903,13 +1027,14 @@ export class Library {
         return this.mlFetch;
     }
 
-    // One ML fetch pass: fetch, decrypt and store the ML data for every file
-    // the store knows about that is not cached (or whose `updationTime` has
-    // advanced), through the metadata pool, and update the CLIP index. For a
-    // file whose ML data does not decrypt, gunzip or parse, it gathers the
-    // reason instead, and puts the reasons it gathered in `mlDataErrors` when
-    // it ends without failing. A failure leaves `mlDataErrors` as it was and
-    // is reported through `status()` and `onProgress`, then thrown.
+    // One ML fetch pass. It fetches, decrypts and stores the ML data of every
+    // file the store knows about that is not cached, or whose `updationTime`
+    // has advanced. It fetches through the metadata pool, and updates the CLIP
+    // index. For a file whose ML data does not decrypt, gunzip or parse, it
+    // gathers the reason instead. It puts the reasons it gathered in
+    // `mlDataErrors` when it ends without failing. A failure leaves
+    // `mlDataErrors` as it was and is reported through `status()` and
+    // `onProgress`, then thrown.
     private async runMLFetch(): Promise<void> {
         const mldata = this.mlStore;
         // Bind so the call keeps the client as its receiver when invoked
@@ -977,9 +1102,9 @@ export class Library {
         }));
     }
 
-    // The distinct files the store holds, one entry per fileID (a file in
-    // several collections shares its ML data), each carrying the key and the
-    // newest `updationTime` seen across its memberships.
+    // The distinct files the store holds, one entry per fileID, since a file
+    // in several collections shares its ML data. Each entry carries the key
+    // and the newest `updationTime` seen across the file's memberships.
     private uniqueFiles(): {
         id: number;
         key: Uint8Array;

src/library/store.ts

@@ -70,12 +70,6 @@ export class MetadataStore {
     userID = 0;
     collectionsSinceTime: Microseconds = 0;
 
-    // True when `load` populated this store from a valid existing file; false
-    // on a first run or a missing/corrupt/wrong-version file that loaded empty.
-    // `Library.open` reads it to decide whether the first refresh may run in
-    // the background (an existing copy already serves reads) or must be awaited.
-    loadedFromDisk = false;
-
     private readonly collections = new Map<number, Collection>();
     private readonly files = new Map<string, EnteFile>();
     private undecryptable: Undecryptable[] = [];
@@ -102,7 +96,6 @@ export class MetadataStore {
             if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) {
                 return store;
             }
-            store.loadedFromDisk = true;
             store.userID = parsed.userID ?? 0;
             store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0;
             for (const stored of parsed.collections ?? []) {
@@ -229,8 +222,14 @@ export class MetadataStore {
         this.files.delete(fileKey(collectionID, fileID));
     }
 
+    // Sorted by collection ID, then file ID, so the order does not depend on
+    // when each record was fetched.
     listUndecryptable(): Undecryptable[] {
-        return [...this.undecryptable];
+        return [...this.undecryptable].sort(
+            (a, b) =>
+                a.collectionID - b.collectionID ||
+                (a.fileID ?? 0) - (b.fileID ?? 0),
+        );
     }
 
     // Replaces any earlier record of the same collection or file.