quak files COLLECTION fetches that collection's files alone (closes #217)

`quak files COLLECTION` used to wait for a full refresh, which on an
empty cache fetched the files of every collection first. It now reads
the collection list, then calls the new `Library.refreshCollection()`.
That brings one collection up to date in the store as a refresh would.
Once the stored cursor is past 0, it only updates a collection the store
holds. For a new one, `quak files COLLECTION` waits for a refresh. A
refresh from a cursor of 0 replaces all the store holds, so collections
then sit in the server's order. `open()` awaits the first refresh while
the cursor is 0. The store lists what did not decrypt in a fixed order.

Model: opus-5-5

README.md

@@ -2,21 +2,31 @@
 
 quak is a WTFPL-licensed TypeScript client library and CLI by
 [@sneak](https://sneak.berlin) for the [Ente](https://ente.io) end-to-end
-encrypted photo hosting service. It logs in, enumerates collections and files,
-and downloads individual images while decrypting them on the way to disk.
+encrypted photo hosting service. It logs in, lists collections and their files,
+and downloads and decrypts single files.
 
-quak also includes a resilient backup command that downloads every file in the
-account into a deduplicated local directory tree, skipping files that already
-exist on disk and continuing past individual download failures instead of
-crashing. For each file it persists the basic metadata fields quak keeps (title,
-file type, creation and modification time, latitude, longitude, content hash),
-its update time, and the private and public magic metadata in full. It keeps any
-ML data Ente has for the file, except the CLIP embedding, and writes the file's
-keywords in place of the embedding. For an image (for a live photo, its image),
-it keeps its original's EXIF and XMP, with its dimensions for a JPEG only. For a
-video it keeps none of these three. It runs unattended from cron (see "Running
-the backup from cron"). A helper subcommand can detect and regenerate missing
-thumbnails, encrypting and uploading them back to the server.
+`quak backup` downloads every file in the account into a local directory tree.
+It downloads and stores each file once, however many collections hold it. It
+skips files already on disk. It logs a failed download and moves on to the next
+file. Beside each file it writes a JSON file that holds:
+
+- the file's title and file type
+- its creation and modification times
+- its latitude and longitude
+- its content hash
+- its update time
+- its private and public magic metadata, in full
+- the ML data that Ente has for it, without the CLIP embedding
+- its keywords in place of that embedding (see "Backup layout")
+- for an image, the EXIF and XMP of its original, and its dimensions if it is a
+  JPEG
+- for a live photo, the same for its image
+
+The backup runs unattended from cron (see "Running the backup from cron").
+
+`quak helper list-missing-thumbnails` finds files whose thumbnail is missing.
+`quak helper fix-missing-thumbnails` makes a new thumbnail for such a file, then
+encrypts it and uploads it to the server.
 
 ## Getting Started
 
@@ -60,10 +70,11 @@ const client = await Client.login({
     password: "your-password",
 });
 
-// Open a cache-backed library. On an empty cache this awaits one server
-// refresh; on an existing cache it returns immediately and refreshes in the
-// background. Later refreshes start `refreshIntervalSeconds` (default 3) after
-// the previous one ends.
+// Open a cache-backed library. On an empty cache, or one that only
+// `lib.refreshCollection()` wrote, this awaits one server refresh. Otherwise
+// it returns immediately and refreshes in the background. A background timer
+// then refreshes again `refreshIntervalSeconds` (default 3) after its previous
+// refresh ends.
 const lib = await Library.open({ client });
 
 // Default reads answer synchronously from the local cache — no network.
@@ -640,17 +651,17 @@ The CLI stores the snapshot at the platform-appropriate data directory via
 `0600`. The key material is stored in cleartext in the JSON; treat this file as
 you would treat the password itself. A missing file is reported as "not logged
 in"; a file that exists but is corrupt is reported as such, naming the bad
-field. When the refresh a command starts with gets HTTP 401 from the server,
-because it no longer accepts the saved session's token, the command prints one
-line, `quak: the saved session is no longer valid; run "quak login"`, with no
-stack trace. All three exit with status 3, which means the user must run
-`quak login` again. `quak logout` is the exception: with no file it says there
-is no session and exits 0, and it handles a corrupt file or a failed server call
-as described below. `quak backup` meets an expired session on the refresh that
-starts every run, before it touches any file but its lock (see "Backup layout").
-A session that stops working partway through a backup instead fails each
-remaining file into `failures.json`, so that run exits 1 and the next one stops
-at its refresh with status 3. No command but `quak login` ever prompts.
+field. The server answers HTTP 401 when it no longer accepts the saved session's
+token. A command that gets that answer prints one line,
+`quak: the saved session is no longer valid; run "quak login"`, with no stack
+trace. All three exit with status 3, which means the user must run `quak login`
+again. `quak logout` is the exception: with no file it says there is no session
+and exits 0, and it handles a corrupt file or a failed server call as described
+below. `quak backup` meets an expired session on the refresh that starts every
+run, before it touches any file but its lock (see "Backup layout"). A session
+that stops working partway through a backup instead fails each remaining file
+into `failures.json`, so that run exits 1 and the next one stops at its refresh
+with status 3. No command but `quak login` ever prompts.
 
 `quak logout` ends the session on the server, so the token in `session.json`
 stops working even in a copy of the file, and then deletes the file. If the
@@ -680,22 +691,40 @@ quak helper fix-missing-thumbnails [--file ids] [--json]  generate + upload miss
 ```
 
 Every command except `login`, `whoami` and `logout` runs on the cache-backed
-library. The read commands — `collections`, `files`, `get`, `get-thumb`,
-`backup-metadata`, `helper list-missing-thumbnails` and
-`helper fix-missing-thumbnails` — force a fresh server round-trip before they
-answer, so they report current account state rather than whatever the cache last
-held. `collections` asks the server for the collection list alone, in one
-request, and neither fetches any collection's files nor changes the cache. The
-others wait for a refresh. With an empty cache, as on a first run, that refresh
-fetches the files of every collection, one collection after another, and stores
-nothing until it has them all, which on a large account takes a while. Meanwhile
-the command logs a line on stderr for each collection whose files it has
-fetched, such as `Fetched the files of 3 of 120 collections`, with `--json` too;
-`backup` does the same while the refresh it starts with runs. stdout holds only
-the command's output. If that round-trip fails, the command prints the error on
-one line and exits 1, or 3 when the server no longer accepts the saved session
-(see "Session handling"). An album or file quak cannot decrypt does not fail it;
-the command answers from what did decrypt (see "Default reads vs. fresh reads").
+library. Each read command makes a fresh server round-trip before it answers. It
+then reports the account's current state, not what the cache last held. The read
+commands are:
+
+- `collections`
+- `files`
+- `get`
+- `get-thumb`
+- `backup-metadata`
+- `helper list-missing-thumbnails`
+- `helper fix-missing-thumbnails`
+
+`collections` asks the server for the collection list alone, in one request. It
+fetches no collection's files and leaves the cache as it is. `files` with a
+collection ID asks for the collection list the same way. It then asks for the
+files of that collection alone, and stores them in the cache. It asks for no
+files when the cache's copy of that collection is current. The other read
+commands, and `files` without a collection ID, wait for a refresh. So does
+`files` with the ID of a collection that is new since the last refresh.
+
+The first refresh fetches the files of every collection, one collection after
+another. That includes any collection that `files` with a collection ID stored
+before it. The refresh stores nothing until it has them all, and on a large
+account that takes a while. Meanwhile the command logs a line on stderr for each
+collection whose files it has fetched, such as
+`Fetched the files of 3 of 120 collections`. It logs these lines with `--json`
+too. `backup` logs them during the refresh it starts with. stdout holds only the
+command's output.
+
+If the server round-trip fails, the command prints the error on one line. It
+exits 1, or 3 when the server no longer accepts the saved session (see "Session
+handling"). An album or file that quak cannot decrypt does not fail the command.
+The command answers from what did decrypt (see "Default reads vs. fresh reads").
+
 `--cache-dir` overrides where the cache lives; without it each account gets its
 own directory under the per-user cache path.
 
@@ -750,9 +779,10 @@ its `Failed files:` list. Every other line is logged as it is without `--json`.
 exit status 1.
 
 `files <collectionID>` lists the files in that collection. `files` with no
-argument lists every file in the account, collection by collection, each file
-once: a file in several collections is listed with the first of them, which is
-its `collectionID` in `--json`.
+argument lists every file in the account once, collection by collection. The
+collections come in the order the server first sent them in a refresh. A file in
+several collections is listed with the first of them, which is its
+`collectionID` in `--json`.
 
 An ID given to `files`, `get`, `get-thumb` or
 `helper fix-missing-thumbnails --file` is written in decimal digits, no larger
@@ -1092,15 +1122,17 @@ against a fake of the Ente HTTP API, and `yarn test` verifies them.
 ### Opening a library
 
 `Library.open(options)` loads the on-disk cache, starts the background refresh
-loop, and resolves to a `Library`. On an empty cache it awaits the first
-refresh, so it opens onto the account's data whenever the server is reachable;
-if that refresh fails, it opens with no data and records the error in
-`lib.status()`. On an existing cache it returns immediately and refreshes in the
-background, so an unreachable server does not block opening. With
-`refreshOnOpen: false` it starts no refresh and returns at once onto what the
-cache holds, empty or not; a refresh then starts only when `fresh()` or
-`lib.backup()` asks, or the background timer fires. The CLI's read commands open
-it so (see "CLI surface").
+loop, and resolves to a `Library`. On an empty cache, or one that only
+`lib.refreshCollection()` wrote, it awaits one refresh. It then opens onto the
+account's data whenever the server is reachable. If that refresh fails, it opens
+onto what the cache holds, and records the error in `lib.status()`. On any other
+cache it returns immediately and refreshes in the background. An unreachable
+server then does not block opening.
+
+With `refreshOnOpen: false`, `open()` starts no refresh. It returns at once onto
+what the cache holds, even when the cache is empty. A refresh then starts only
+when `fresh()` or `lib.backup()` asks, or the background timer fires. The CLI's
+read commands open the library this way (see "CLI surface").
 
 `LibraryOptions`:
 
@@ -1127,22 +1159,35 @@ when the library opens; `lib.downloadDirectory` holds the result.
 `onProgress` receives a `RefreshEvent` with status `"started"` as a refresh, an
 ML data fetch or a precache sweep starts, and `"done"` or `"failed"` as it ends.
 In between, a refresh sends `"progress"` after each collection whose files it
-fetched, with `fetched`, how many collections it has fetched the files of so
-far, and `total`, how many it will. A refresh fetches the files of each
-collection that is new or changed since the cache's copy, so on an empty cache
-it fetches those of every collection.
+fetched. That event's `fetched` is the number of collections whose files the
+refresh has fetched so far. Its `total` is the number of collections whose files
+it will fetch in all. A refresh fetches the files of each collection that is new
+or changed since the cache's copy. On the first refresh, that is every
+collection.
 
 Concurrency is set through `pools`: construct
 `new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
 and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).
 
-`lib.status()` returns a `LibraryStatus` (collection/file counts, last
-refresh/ML times and errors, originals usage and effective limit, precache
-progress, and `closed`). `lib.close()` stops the background timer; it is
-idempotent, and an in-flight refresh is left to finish. The promise it returns
-resolves once that refresh (including its cache write), the ML data fetch and
-the precache fetches already running have all finished, so the cache directory
-can then be removed.
+`lib.status()` returns a `LibraryStatus`, which holds:
+
+- the collection and file counts
+- the time of the last refresh that succeeded, and the error of the latest
+  refresh if it failed
+- the same two for the ML data fetch
+- the originals cache's usage and its effective limit
+- the precache progress
+- `closed`
+
+`lib.close()` stops the background timer, and is idempotent. A refresh or
+`lib.refreshCollection()` in flight is left to finish. `close()` returns a
+promise that resolves once all of these have finished:
+
+- that refresh or `lib.refreshCollection()`, with its cache write
+- the ML data fetch
+- the precache fetches already running
+
+The cache directory can then be removed.
 
 `lib.status().mlDataErrors` lists, as `{ fileID, error }`, each file whose ML
 data the last ML data fetch that succeeded got but could not decrypt, gunzip or
@@ -1156,39 +1201,72 @@ cache.
 
 Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
 synchronously from the copy held in RAM and never touch the network. A refresh
-changes that copy only once all its server requests have succeeded, and the
-background timer starts the next refresh `refreshIntervalSeconds` after the
-previous one ends. So a default read is immediate, but only as current as the
-last refresh whose requests all succeeded; right after opening an existing
-cache, it is the copy on disk.
+changes that copy only once all its server requests have succeeded.
+`lib.refreshCollection()` changes it the same way for one collection. So a
+default read is immediate, but only as current as the last refresh or
+`lib.refreshCollection()` whose requests all succeeded. Right after `open()`
+returns without awaiting a refresh, that copy is the one loaded from disk.
 
-`await lib.fresh()` waits for a refresh to complete and persist, and returns the
-same `{ albums, photos, timeline }` namespaces, which then reflect a completed
-server round-trip. When a refresh is already running, background or not,
-`fresh()` waits for that one, so its answer can come from requests made before
-the call; only when none is running does it start one. A refresh that fails
+The background timer takes a turn `refreshIntervalSeconds` after its last turn
+ends. Each turn starts a refresh, unless a refresh or `lib.refreshCollection()`
+is already running.
+
+`await lib.fresh()` waits for a refresh to complete and persist. It then returns
+the same `{ albums, photos, timeline }` namespaces, which reflect a completed
+server round-trip. If a `lib.refreshCollection()` is running, `fresh()` first
+waits for it to end. When a refresh is already running, background or not,
+`fresh()` waits for that one. Its answer can then come from requests made before
+the call. Only when none is running does it start one. A refresh that fails
 rejects the caller (default reads stay silent and keep serving the last good
-copy). The CLI's read commands use fresh reads (issue
-https://git.eeqj.de/sneak/quak/issues/75).
+copy). Every CLI read command calls `fresh()`, except `quak collections`.
+`quak files` with a collection ID calls it only when `lib.refreshCollection()`
+resolves to `false`.
 
-`await lib.freshCollections()` is the fresh read of the collection list alone:
-one collections request, answered with every live collection that decrypts, as
-the `Collection` objects `listCollections()` returns, in the server's order. It
-fetches no collection's files and stores nothing, not even the list: a refresh
-fetches a collection's files only when the collection is new or has changed
-since the stored copy, so a collection stored without its files would never get
-them. `quak collections` uses it (issue
-https://git.eeqj.de/sneak/quak/issues/210).
+`await lib.freshCollections()` is the fresh read of the collection list alone.
+It sends one collections request. It answers with every live collection that
+decrypts, in the server's order, as the `Collection` objects that
+`listCollections()` returns. It fetches no collection's files and stores
+nothing, not even the list. A refresh from a cursor past 0 fetches a
+collection's files only when the collection is new or has changed since the
+stored copy. So does `lib.refreshCollection()`. Neither would fetch the files of
+a collection stored without them. `quak collections` uses `freshCollections()`.
 
-An album or file in the server's answer that quak cannot decrypt, or whose
-metadata is not JSON, does not fail the refresh. The refresh leaves it out, an
-album with all its files, and drops any older copy the cache held, so reads
-answer from what did decrypt. The library keeps a list of what it left out, with
-the reason, in `metadata.json`: the next refresh asks the server only for what
-changed, so it would not see the record again. A record stays on that list until
-a later refresh gets a copy that decrypts or learns that the server deleted it.
-`lib.backup()` reports each such album, and each such file in its scope that has
-no copy that decrypted (see "Backup layout").
+`await lib.refreshCollection(collection)` brings one collection up to date in
+the cache, given its `Collection` from `freshCollections()`. It does for that
+collection what a refresh does for a changed one, and fetches no other
+collection's files. It fetches the collection's files changed since the cache's
+copy, or all of them when the cache has no copy. It then stores and saves them.
+It sends no request when the cache's copy has the same `updationTime` or a later
+one. It resolves to `true` when the cache then holds the collection.
+
+`lib.refreshCollection()` adds a collection the cache does not hold only while
+the refresh cursor is 0. The cursor moves past 0 once a refresh receives the
+account's collections. After that, for a collection the cache does not hold, it
+sends no request and resolves to `false`. Such a collection is new since that
+refresh, or the refresh dropped it. Only a refresh may then add it, after the
+collections the cache holds.
+
+A refresh from a cursor of 0 replaces everything the cache holds. It fetches the
+files of every collection, even those `lib.refreshCollection()` stored. It
+stores the collections in the server's order. The cache then holds what it would
+had `lib.refreshCollection()` never run.
+
+`lib.refreshCollection()` leaves the refresh cursor where it was, so the next
+refresh still fetches the files of every other collection that changed. It never
+runs at the same time as a refresh. It first waits for a running refresh or
+`lib.refreshCollection()` to end. While it runs, a `fresh()` or `lib.backup()`
+waits for it to end. `quak files` with a collection ID uses it.
+
+A refresh does not fail on an album or file that quak cannot decrypt, or whose
+metadata is not JSON. The refresh leaves out such a file, and such an album
+together with all its files. It also drops any older copy that the cache held,
+so reads answer from what did decrypt. The library keeps a list of what it left
+out, with the reason, in `metadata.json`. The next refresh asks the server only
+for what changed, so it would not see them again. An entry stays on that list
+until a later refresh or `lib.refreshCollection()` stores a copy that decrypts,
+or learns that the server deleted it. `lib.backup()` reports each such album,
+and each such file in its scope that has no copy that decrypted (see "Backup
+layout").
 
 ### Read surface
 
@@ -1282,10 +1360,17 @@ The GUI-facing records hold no key material and no binary, so they survive
 - `LibrarySnapshot`: `{ albums, photos, takenAt }`.
 
 `lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first,
-photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
-(`albumsChanged`, `photosChanged`, `fileIDsRemoved`, `albumIDsRemoved`,
-`refreshedAt`) whenever a refresh alters the projection, and returns
-`{ unsubscribe }`; a refresh that changes nothing delivers nothing.
+photos newest first). `lib.subscribe({ onChange })` returns `{ unsubscribe }`.
+It calls `onChange` with a `LibraryChange` whenever a refresh or
+`lib.refreshCollection()` changes these records. A refresh or
+`lib.refreshCollection()` that changes nothing does not call it. A
+`LibraryChange` holds:
+
+- `albumsChanged`
+- `photosChanged`
+- `fileIDsRemoved`
+- `albumIDsRemoved`
+- `refreshedAt`
 
 ### Thumbnails, ML search, and backup
 
@@ -1301,29 +1386,44 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
   `SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
   default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
   a query vector the caller produced elsewhere.
-- `await lib.backup(opts?)` → `BackupResult`. It takes the lock in the download
-  directory, and fails at once with an error whose `code` is `ELOCKED` while
-  another backup of it runs. A caller can instead take the lock itself, before
-  it opens its library, as `quak backup` does: `await lockBackupDirectory(dir)`
-  takes it, failing the same way, and returns a `BackupLock`, whose `release()`
-  removes the lock unless it was lost and whose `lost` holds why it was lost.
-  The caller passes it to the backup as `lock`, and releases it once the backup
-  ends. It waits for a refresh as `fresh()` does, puts every in-scope original
-  not already at its save path there as `photo.download()` does (and, with
-  `includeThumbnails`, fetches thumbnails) through the content cache, waits for
-  an ML data fetch, and rebuilds the on-disk backup tree with a durable failure
-  ledger. Each file's JSON holds its ML data and keywords. An image's JSON also
-  holds the EXIF and XMP of its original, and its dimensions if it is a JPEG. A
-  live photo's JSON holds these for its image. A video's JSON holds no EXIF, XMP
-  or dimensions. A fetched original is written straight to its save path and not
-  into the cache, which then counts it as present; one the cache already held is
-  copied from there. `BackupOptions`: `downloadDirectory` (falls back to the
-  library's), `includeOriginals` (default `true`), `includeThumbnails` (default
-  `false`), `onlyAlbumNames`, `verify` (default `false`), `onProgress`, and
-  `lock`. See Backup layout above for the tree it writes, and for the albums
-  `BackupResult` lists in `undecryptableAlbums` and `failedAlbums`. Whether the
-  backup took the lock or its caller did, once the lock is lost the backup
-  rejects with `lost the lock on <dir>: <reason>` before it starts another file.
+- `await lib.backup(opts?)` → `BackupResult`. It takes the lock, `backup.lock`
+  in the download directory. It fails at once with an error whose `code` is
+  `ELOCKED` while another backup of that directory runs. A caller can instead
+  take the lock itself before it opens its library, as `quak backup` does.
+  `await lockBackupDirectory(dir)` takes it, failing the same way, and returns a
+  `BackupLock`. Its `release()` removes the lock unless it was lost, and its
+  `lost` holds why it was lost. The caller passes it to the backup as `lock`,
+  and releases it once the backup ends.
+
+    The backup waits for a refresh as `fresh()` does. It then fetches each
+    in-scope original that is not yet at its save path, as `photo.download()`
+    does, through the content cache. With `includeThumbnails` it also fetches
+    thumbnails. It waits for an ML data fetch, then rebuilds the backup tree. It
+    records each file that failed in `failures.json`.
+
+    Each file's JSON holds its ML data and keywords. An image's JSON also holds
+    the EXIF and XMP of its original, and its dimensions if it is a JPEG. A live
+    photo's JSON holds these for its image. A video's JSON holds no EXIF, XMP or
+    dimensions.
+
+    A fetched original is written straight to its save path, not into the cache.
+    The cache then counts it as present. An original that the cache already held
+    is copied from there.
+
+    `BackupOptions` holds:
+    - `downloadDirectory`, which falls back to the library's
+    - `includeOriginals`, `true` by default
+    - `includeThumbnails`, `false` by default
+    - `onlyAlbumNames`
+    - `verify`, `false` by default
+    - `onProgress`
+    - `lock`
+
+    "Backup layout" above describes the tree that the backup writes. It also
+    describes the albums that `BackupResult` lists in `undecryptableAlbums` and
+    `failedAlbums`. Once the lock is lost, the backup rejects with
+    `lost the lock on <dir>: <reason>` before it starts another file. This holds
+    whether the backup or its caller took the lock.
 
 ### Request pools