quak files COLLECTION fetches that collection's files alone (closes #217)

`quak files COLLECTION` used to wait for a full refresh, which on an
empty cache fetched the files of every collection before it listed one.
It now reads the collection list, then calls the new
`Library.refreshCollection()`. That brings one collection up to date in
the store as a refresh would, and saves it. It runs in the same slot as
a refresh, so the two never run at the same time, and `close()` waits
for it. It never stores a collection that a refresh has already dropped.
It leaves the refresh cursor alone. `open()` therefore awaits the first
refresh on a cache whose cursor is still 0. `quak files` with no
argument still waits for a refresh.

Model: opus-5-5

README.md

@@ -2,21 +2,31 @@
 
 quak is a WTFPL-licensed TypeScript client library and CLI by
 [@sneak](https://sneak.berlin) for the [Ente](https://ente.io) end-to-end
-encrypted photo hosting service. It logs in, enumerates collections and files,
+encrypted photo hosting service. It logs in, lists collections and their files,
-and downloads individual images while decrypting them on the way to disk.
+and downloads and decrypts single files.
 
-quak also includes a resilient backup command that downloads every file in the
+`quak backup` downloads every file in the account into a local directory tree.
-account into a deduplicated local directory tree, skipping files that already
+It downloads and stores each file once, however many collections hold it. It
-exist on disk and continuing past individual download failures instead of
+skips files already on disk. It logs a failed download and moves on to the next
-crashing. For each file it persists the basic metadata fields quak keeps (title,
+file. Beside each file it writes a JSON file that holds:
-file type, creation and modification time, latitude, longitude, content hash),
+
-its update time, and the private and public magic metadata in full. It keeps any
+- the file's title and file type
-ML data Ente has for the file, except the CLIP embedding, and writes the file's
+- its creation and modification times
-keywords in place of the embedding. For an image (for a live photo, its image),
+- its latitude and longitude
-it keeps its original's EXIF and XMP, with its dimensions for a JPEG only. For a
+- its content hash
-video it keeps none of these three. It runs unattended from cron (see "Running
+- its update time
-the backup from cron"). A helper subcommand can detect and regenerate missing
+- its private and public magic metadata, in full
-thumbnails, encrypting and uploading them back to the server.
+- the ML data that Ente has for it, without the CLIP embedding
+- its keywords in place of that embedding (see "Backup layout")
+- for an image, the EXIF and XMP of its original, and its dimensions if it is a
+  JPEG
+- for a live photo, the same for its image
+
+The backup runs unattended from cron (see "Running the backup from cron").
+
+`quak helper list-missing-thumbnails` finds files whose thumbnail is missing.
+`quak helper fix-missing-thumbnails` makes a new thumbnail for such a file, then
+encrypts it and uploads it to the server.
 
 ## Getting Started
 
@@ -60,10 +70,11 @@ const client = await Client.login({
     password: "your-password",
 });
 
-// Open a cache-backed library. On an empty cache this awaits one server
+// Open a cache-backed library. On an empty cache, or one that only
-// refresh; on an existing cache it returns immediately and refreshes in the
+// `lib.refreshCollection()` wrote, this awaits one server refresh. Otherwise
-// background. Later refreshes start `refreshIntervalSeconds` (default 3) after
+// it returns immediately and refreshes in the background. A background timer
-// the previous one ends.
+// then refreshes again `refreshIntervalSeconds` (default 3) after its previous
+// refresh ends.
 const lib = await Library.open({ client });
 
 // Default reads answer synchronously from the local cache — no network.
@@ -680,26 +691,38 @@ quak helper fix-missing-thumbnails [--file ids] [--json]  generate + upload miss
 ```
 
 Every command except `login`, `whoami` and `logout` runs on the cache-backed
-library. The read commands — `collections`, `files`, `get`, `get-thumb`,
+library. Each read command makes a fresh server round-trip before it answers. It
-`backup-metadata`, `helper list-missing-thumbnails` and
+then reports the account's current state, not what the cache last held. The read
-`helper fix-missing-thumbnails` — force a fresh server round-trip before they
+commands are:
-answer, so they report current account state rather than whatever the cache last
+
-held. `collections` asks the server for the collection list alone, in one
+- `collections`
-request, and neither fetches any collection's files nor changes the cache.
+- `files`
-`files` with a collection ID asks for the collection list the same way, then for
+- `get`
-the files of that collection alone, and stores them in the cache. It asks for
+- `get-thumb`
-none of them when the cache's copy of that collection is current. `files`
+- `backup-metadata`
-without a collection ID and the other read commands wait for a refresh. With an
+- `helper list-missing-thumbnails`
-empty cache, as on a first run, that refresh fetches the files of every
+- `helper fix-missing-thumbnails`
-collection, one collection after another, and stores nothing until it has them
+
-all, which on a large account takes a while. Meanwhile the command logs a line
+`collections` asks the server for the collection list alone, in one request. It
-on stderr for each collection whose files it has fetched, such as
+fetches no collection's files and leaves the cache as it is. `files` with a
-`Fetched the files of 3 of 120 collections`, with `--json` too; `backup` does
+collection ID asks for the collection list the same way. It then asks for the
-the same while the refresh it starts with runs. stdout holds only the command's
+files of that collection alone, and stores them in the cache. It asks for no
-output. If that round-trip fails, the command prints the error on one line and
+files when the cache's copy of that collection is current. The other read
+commands, and `files` without a collection ID, wait for a refresh.
+
+On an empty cache, as on a first run, that refresh fetches the files of every
+collection, one collection after another. It stores nothing until it has them
+all, and on a large account that takes a while. Meanwhile the command logs a
+line on stderr for each collection whose files it has fetched, such as
+`Fetched the files of 3 of 120 collections`. It logs these lines with `--json`
+too. `backup` logs them during the refresh it starts with. stdout holds only the
+command's output.
+
+If the server round-trip fails, the command prints the error on one line. It
 exits 1, or 3 when the server no longer accepts the saved session (see "Session
-handling"). An album or file quak cannot decrypt does not fail it; the command
+handling"). An album or file that quak cannot decrypt does not fail the command.
-answers from what did decrypt (see "Default reads vs. fresh reads").
+The command answers from what did decrypt (see "Default reads vs. fresh reads").
+
 `--cache-dir` overrides where the cache lives; without it each account gets its
 own directory under the per-user cache path.
 
@@ -1096,16 +1119,17 @@ against a fake of the Ente HTTP API, and `yarn test` verifies them.
 ### Opening a library
 
 `Library.open(options)` loads the on-disk cache, starts the background refresh
-loop, and resolves to a `Library`. On an empty cache it awaits the first
+loop, and resolves to a `Library`. On an empty cache, or one that only
-refresh, so it opens onto the account's data whenever the server is reachable.
+`lib.refreshCollection()` wrote, it awaits one refresh. It then opens onto the
-It also awaits it on a cache that holds only what `lib.refreshCollection()`
+account's data whenever the server is reachable. If that refresh fails, it opens
-stored. If that refresh fails, it opens onto what the cache holds and records
+onto what the cache holds, and records the error in `lib.status()`. On any other
-the error in `lib.status()`. On any other cache it returns immediately and
+cache it returns immediately and refreshes in the background. An unreachable
-refreshes in the background, so an unreachable server does not block opening.
+server then does not block opening.
-With `refreshOnOpen: false` it starts no refresh, and returns at once onto what
+
-the cache holds, empty or not. A refresh then starts only when `fresh()` or
+With `refreshOnOpen: false`, `open()` starts no refresh. It returns at once onto
-`lib.backup()` asks, or the background timer fires. The CLI's read commands open
+what the cache holds, even when the cache is empty. A refresh then starts only
-it so (see "CLI surface").
+when `fresh()` or `lib.backup()` asks, or the background timer fires. The CLI's
+read commands open the library this way (see "CLI surface").
 
 `LibraryOptions`:
 
@@ -1132,28 +1156,35 @@ when the library opens; `lib.downloadDirectory` holds the result.
 `onProgress` receives a `RefreshEvent` with status `"started"` as a refresh, an
 ML data fetch or a precache sweep starts, and `"done"` or `"failed"` as it ends.
 In between, a refresh sends `"progress"` after each collection whose files it
-fetched. That event's `fetched` is how many collections it has fetched the files
+fetched. That event's `fetched` is the number of collections whose files the
-of so far, and its `total` how many it will. A refresh fetches the files of each
+refresh has fetched so far. Its `total` is the number of collections whose files
-collection that is new or changed since the cache's copy, so on an empty cache
+it will fetch in all. A refresh fetches the files of each collection that is new
-it fetches those of every collection.
+or changed since the cache's copy. On an empty cache, that is every collection.
 
 Concurrency is set through `pools`: construct
 `new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
 and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).
 
-`lib.status()` returns a `LibraryStatus` (collection/file counts, last
+`lib.status()` returns a `LibraryStatus`, which holds:
-refresh/ML times and errors, originals usage and effective limit, precache
+
-progress, and `closed`).
+- the collection and file counts
+- the time of the last refresh that succeeded, and the error of the latest
+  refresh if it failed
+- the same two for the ML data fetch
+- the originals cache's usage and its effective limit
+- the precache progress
+- `closed`
 
 `lib.close()` stops the background timer, and is idempotent. A refresh or
-`lib.refreshCollection()` in flight is left to finish. The promise `close()`
+`lib.refreshCollection()` in flight is left to finish. `close()` returns a
-returns resolves once all of these have finished, so the cache directory can
+promise that resolves once all of these have finished:
-then be removed:
 
 - that refresh or `lib.refreshCollection()`, with its cache write
 - the ML data fetch
 - the precache fetches already running
 
+The cache directory can then be removed.
+
 `lib.status().mlDataErrors` lists, as `{ fileID, error }`, each file whose ML
 data the last ML data fetch that succeeded got but could not decrypt, gunzip or
 parse; the `fetchMLData` event with status `"done"` carries the same list. A
@@ -1169,54 +1200,56 @@ synchronously from the copy held in RAM and never touch the network. A refresh
 changes that copy only once all its server requests have succeeded.
 `lib.refreshCollection()` changes it the same way for one collection. So a
 default read is immediate, but only as current as the last refresh or
-`lib.refreshCollection()` whose requests all succeeded. Right after opening an
+`lib.refreshCollection()` whose requests all succeeded. Right after `open()`
-existing cache, it is the copy on disk.
+returns without awaiting a refresh, that copy is the one loaded from disk.
 
 The background timer takes a turn `refreshIntervalSeconds` after its last turn
 ends. Each turn starts a refresh, unless a refresh or `lib.refreshCollection()`
 is already running.
 
-`await lib.fresh()` waits for a refresh to complete and persist, and returns the
+`await lib.fresh()` waits for a refresh to complete and persist. It then returns
-same `{ albums, photos, timeline }` namespaces, which then reflect a completed
+the same `{ albums, photos, timeline }` namespaces, which reflect a completed
-server round-trip. It first waits for a running `lib.refreshCollection()` to
+server round-trip. If a `lib.refreshCollection()` is running, `fresh()` first
-end. When a refresh is already running, background or not, `fresh()` waits for
+waits for it to end. When a refresh is already running, background or not,
-that one. Its answer can then come from requests made before the call. Only when
+`fresh()` waits for that one. Its answer can then come from requests made before
-none is running does it start one. A refresh that fails rejects the caller
+the call. Only when none is running does it start one. A refresh that fails
-(default reads stay silent and keep serving the last good copy). The CLI's read
+rejects the caller (default reads stay silent and keep serving the last good
-commands use fresh reads (issue https://git.eeqj.de/sneak/quak/issues/75).
+copy). Every CLI read command calls `fresh()`, except `quak collections` and
+`quak files` with a collection ID.
 
-`await lib.freshCollections()` is the fresh read of the collection list alone:
+`await lib.freshCollections()` is the fresh read of the collection list alone.
-one collections request, answered with every live collection that decrypts, as
+It sends one collections request. It answers with every live collection that
-the `Collection` objects `listCollections()` returns, in the server's order. It
+decrypts, in the server's order, as the `Collection` objects that
-fetches no collection's files and stores nothing, not even the list. A refresh
+`listCollections()` returns. It fetches no collection's files and stores
-fetches a collection's files only when the collection is new or has changed
+nothing, not even the list. A refresh fetches a collection's files only when the
-since the stored copy. A collection stored without its files would never get
+collection is new or has changed since the stored copy. So a collection stored
-them. `quak collections` uses it (issue
+without its files would never get them. `quak collections` uses
-https://git.eeqj.de/sneak/quak/issues/210).
+`freshCollections()`.
 
 `await lib.refreshCollection(collection)` brings one collection up to date in
 the cache, given its `Collection` from `freshCollections()`. It does for that
 collection what a refresh does for a changed one, and fetches no other
-collection's files. It sends no request when the cache's copy has the same
+collection's files. It fetches the collection's files changed since the cache's
-`updationTime` or a later one. It also sends none when a refresh has already
+copy, or all of them when the cache has no copy. It then stores and saves them.
-dropped the collection from the cache, because the server deleted it or it did
+It sends no request when the cache's copy has the same `updationTime` or a later
-not decrypt. Otherwise it fetches the collection's files changed since the
+one. It also sends none when a refresh has already dropped the collection from
-cache's copy, or all of them when there is none, then stores and saves them. The
+the cache, because the server deleted it or it did not decrypt.
-refresh cursor stays where it was, so the next refresh still fetches the files
-of every other collection that changed. It never runs at the same time as a
-refresh. It first waits for a running refresh to end. A `fresh()` or
-`lib.backup()` called after that waits for it. `quak files` with a collection ID
-uses it.
 
-An album or file in the server's answer that quak cannot decrypt, or whose
+`lib.refreshCollection()` leaves the refresh cursor where it was, so the next
-metadata is not JSON, does not fail the refresh. The refresh leaves it out, an
+refresh still fetches the files of every other collection that changed. It never
-album with all its files, and drops any older copy the cache held, so reads
+runs at the same time as a refresh. It first waits for a running refresh or
-answer from what did decrypt. The library keeps a list of what it left out, with
+`lib.refreshCollection()` to end. While it runs, a `fresh()` or `lib.backup()`
-the reason, in `metadata.json`. The next refresh asks the server only for what
+waits for it to end. `quak files` with a collection ID uses it.
-changed, so it would not see the record again. A record stays on that list until
+
-a later refresh or `lib.refreshCollection()` stores a copy that decrypts, or
+A refresh does not fail on an album or file that quak cannot decrypt, or whose
-learns that the server deleted it. `lib.backup()` reports each such album, and
+metadata is not JSON. The refresh leaves out such a file, and such an album
-each such file in its scope that has no copy that decrypted (see "Backup
+together with all its files. It also drops any older copy that the cache held,
+so reads answer from what did decrypt. The library keeps a list of what it left
+out, with the reason, in `metadata.json`. The next refresh asks the server only
+for what changed, so it would not see them again. An entry stays on that list
+until a later refresh or `lib.refreshCollection()` stores a copy that decrypts,
+or learns that the server deleted it. `lib.backup()` reports each such album,
+and each such file in its scope that has no copy that decrypted (see "Backup
 layout").
 
 ### Read surface
@@ -1312,10 +1345,16 @@ The GUI-facing records hold no key material and no binary, so they survive
 
 `lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first,
 photos newest first). `lib.subscribe({ onChange })` returns `{ unsubscribe }`.
-It delivers a `LibraryChange` (`albumsChanged`, `photosChanged`,
+It calls `onChange` with a `LibraryChange` whenever a refresh or
-`fileIDsRemoved`, `albumIDsRemoved`, `refreshedAt`) whenever a refresh or
+`lib.refreshCollection()` changes these records. A refresh or
-`lib.refreshCollection()` alters the projection. One that changes nothing
+`lib.refreshCollection()` that changes nothing does not call it. A
-delivers nothing.
+`LibraryChange` holds:
+
+- `albumsChanged`
+- `photosChanged`
+- `fileIDsRemoved`
+- `albumIDsRemoved`
+- `refreshedAt`
 
 ### Thumbnails, ML search, and backup
 
@@ -1331,30 +1370,44 @@ delivers nothing.
   `SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
   default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
   a query vector the caller produced elsewhere.
-- `await lib.backup(opts?)` → `BackupResult`. It takes the lock in the download
+- `await lib.backup(opts?)` → `BackupResult`. It takes the lock, `backup.lock`
-  directory, and fails at once with an error whose `code` is `ELOCKED` while
+  in the download directory. It fails at once with an error whose `code` is
-  another backup of it runs. A caller can instead take the lock itself, before
+  `ELOCKED` while another backup of that directory runs. A caller can instead
-  it opens its library, as `quak backup` does. `await lockBackupDirectory(dir)`
+  take the lock itself before it opens its library, as `quak backup` does.
-  takes it, failing the same way, and returns a `BackupLock`. Its `release()`
+  `await lockBackupDirectory(dir)` takes it, failing the same way, and returns a
-  removes the lock unless it was lost, and its `lost` holds why it was lost. The
+  `BackupLock`. Its `release()` removes the lock unless it was lost, and its
-  caller passes it to the backup as `lock`, and releases it once the backup
+  `lost` holds why it was lost. The caller passes it to the backup as `lock`,
-  ends. The backup waits for a refresh as `fresh()` does. It then puts every
+  and releases it once the backup ends.
-  in-scope original not already at its save path there, as `photo.download()`
+
-  does, through the content cache. With `includeThumbnails` it also fetches
+    The backup waits for a refresh as `fresh()` does. It then fetches each
-  thumbnails. It waits for an ML data fetch, and rebuilds the on-disk backup
+    in-scope original that is not yet at its save path, as `photo.download()`
-  tree with a durable failure ledger. Each file's JSON holds its ML data and
+    does, through the content cache. With `includeThumbnails` it also fetches
-  keywords. An image's JSON also holds the EXIF and XMP of its original, and its
+    thumbnails. It waits for an ML data fetch, then rebuilds the backup tree. It
-  dimensions if it is a JPEG. A live photo's JSON holds these for its image. A
+    records each file that failed in `failures.json`.
-  video's JSON holds no EXIF, XMP or dimensions. A fetched original is written
+
-  straight to its save path and not into the cache, which then counts it as
+    Each file's JSON holds its ML data and keywords. An image's JSON also holds
-  present; one the cache already held is copied from there. `BackupOptions`:
+    the EXIF and XMP of its original, and its dimensions if it is a JPEG. A live
-  `downloadDirectory` (falls back to the library's), `includeOriginals` (default
+    photo's JSON holds these for its image. A video's JSON holds no EXIF, XMP or
-  `true`), `includeThumbnails` (default `false`), `onlyAlbumNames`, `verify`
+    dimensions.
-  (default `false`), `onProgress`, and `lock`. See Backup layout above for the
+
-  tree it writes, and for the albums `BackupResult` lists in
+    A fetched original is written straight to its save path, not into the cache.
-  `undecryptableAlbums` and `failedAlbums`. Whether the backup took the lock or
+    The cache then counts it as present. An original that the cache already held
-  its caller did, once the lock is lost the backup rejects with
+    is copied from there.
-  `lost the lock on <dir>: <reason>` before it starts another file.
+
+    `BackupOptions` holds:
+    - `downloadDirectory`, which falls back to the library's
+    - `includeOriginals`, `true` by default
+    - `includeThumbnails`, `false` by default
+    - `onlyAlbumNames`
+    - `verify`, `false` by default
+    - `onProgress`
+    - `lock`
+
+    "Backup layout" above describes the tree that the backup writes. It also
+    describes the albums that `BackupResult` lists in `undecryptableAlbums` and
+    `failedAlbums`. Once the lock is lost, the backup rejects with
+    `lost the lock on <dir>: <reason>` before it starts another file. This holds
+    whether the backup or its caller took the lock.
 
 ### Request pools