quak files COLLECTION fetches that collection's files alone (closes #217)

`quak files COLLECTION` used to wait for a full refresh, which on an
empty cache fetched the files of every collection before it listed one.
It now reads the collection list, then calls the new
`Library.refreshCollection()`. That brings one collection up to date in
the store as a refresh would, and saves it. It runs in the same slot as
a refresh, so the two never run at the same time, and `close()` waits
for it. It never stores a collection that a refresh has already dropped.
It leaves the refresh cursor alone. `open()` therefore awaits the first
refresh on a cache whose cursor is still 0. `quak files` with no
argument still waits for a refresh.

Model: opus-5-5

README.md

@@ -2,31 +2,21 @@
 
 quak is a WTFPL-licensed TypeScript client library and CLI by
 [@sneak](https://sneak.berlin) for the [Ente](https://ente.io) end-to-end
-encrypted photo hosting service. It logs in, lists collections and their files,
+encrypted photo hosting service. It logs in, enumerates collections and files,
-and downloads and decrypts single files.
+and downloads individual images while decrypting them on the way to disk.
 
-`quak backup` downloads every file in the account into a local directory tree.
+quak also includes a resilient backup command that downloads every file in the
-It downloads and stores each file once, however many collections hold it. It
+account into a deduplicated local directory tree, skipping files that already
-skips files already on disk. It logs a failed download and moves on to the next
+exist on disk and continuing past individual download failures instead of
-file. Beside each file it writes a JSON file that holds:
+crashing. For each file it persists the basic metadata fields quak keeps (title,
-
+file type, creation and modification time, latitude, longitude, content hash),
-- the file's title and file type
+its update time, and the private and public magic metadata in full. It keeps any
-- its creation and modification times
+ML data Ente has for the file, except the CLIP embedding, and writes the file's
-- its latitude and longitude
+keywords in place of the embedding. For an image (for a live photo, its image),
-- its content hash
+it keeps its original's EXIF and XMP, with its dimensions for a JPEG only. For a
-- its update time
+video it keeps none of these three. It runs unattended from cron (see "Running
-- its private and public magic metadata, in full
+the backup from cron"). A helper subcommand can detect and regenerate missing
-- the ML data that Ente has for it, without the CLIP embedding
+thumbnails, encrypting and uploading them back to the server.
-- its keywords in place of that embedding (see "Backup layout")
-- for an image, the EXIF and XMP of its original, and its dimensions if it is a
-  JPEG
-- for a live photo, the same for its image
-
-The backup runs unattended from cron (see "Running the backup from cron").
-
-`quak helper list-missing-thumbnails` finds files whose thumbnail is missing.
-`quak helper fix-missing-thumbnails` makes a new thumbnail for such a file, then
-encrypts it and uploads it to the server.
 
 ## Getting Started
 
@@ -70,11 +60,10 @@ const client = await Client.login({
     password: "your-password",
 });
 
-// Open a cache-backed library. On an empty cache, or one that only
+// Open a cache-backed library. On an empty cache this awaits one server
-// `lib.refreshCollection()` wrote, this awaits one server refresh. Otherwise
+// refresh; on an existing cache it returns immediately and refreshes in the
-// it returns immediately and refreshes in the background. A background timer
+// background. Later refreshes start `refreshIntervalSeconds` (default 3) after
-// then refreshes again `refreshIntervalSeconds` (default 3) after its previous
+// the previous one ends.
-// refresh ends.
 const lib = await Library.open({ client });
 
 // Default reads answer synchronously from the local cache — no network.
@@ -691,38 +680,26 @@ quak helper fix-missing-thumbnails [--file ids] [--json]  generate + upload miss
 ```
 
 Every command except `login`, `whoami` and `logout` runs on the cache-backed
-library. Each read command makes a fresh server round-trip before it answers. It
+library. The read commands — `collections`, `files`, `get`, `get-thumb`,
-then reports the account's current state, not what the cache last held. The read
+`backup-metadata`, `helper list-missing-thumbnails` and
-commands are:
+`helper fix-missing-thumbnails` — force a fresh server round-trip before they
-
+answer, so they report current account state rather than whatever the cache last
-- `collections`
+held. `collections` asks the server for the collection list alone, in one
-- `files`
+request, and neither fetches any collection's files nor changes the cache.
-- `get`
+`files` with a collection ID asks for the collection list the same way, then for
-- `get-thumb`
+the files of that collection alone, and stores them in the cache. It asks for
-- `backup-metadata`
+none of them when the cache's copy of that collection is current. `files`
-- `helper list-missing-thumbnails`
+without a collection ID and the other read commands wait for a refresh. With an
-- `helper fix-missing-thumbnails`
+empty cache, as on a first run, that refresh fetches the files of every
-
+collection, one collection after another, and stores nothing until it has them
-`collections` asks the server for the collection list alone, in one request. It
+all, which on a large account takes a while. Meanwhile the command logs a line
-fetches no collection's files and leaves the cache as it is. `files` with a
+on stderr for each collection whose files it has fetched, such as
-collection ID asks for the collection list the same way. It then asks for the
+`Fetched the files of 3 of 120 collections`, with `--json` too; `backup` does
-files of that collection alone, and stores them in the cache. It asks for no
+the same while the refresh it starts with runs. stdout holds only the command's
-files when the cache's copy of that collection is current. The other read
+output. If that round-trip fails, the command prints the error on one line and
-commands, and `files` without a collection ID, wait for a refresh.
-
-On an empty cache, as on a first run, that refresh fetches the files of every
-collection, one collection after another. It stores nothing until it has them
-all, and on a large account that takes a while. Meanwhile the command logs a
-line on stderr for each collection whose files it has fetched, such as
-`Fetched the files of 3 of 120 collections`. It logs these lines with `--json`
-too. `backup` logs them during the refresh it starts with. stdout holds only the
-command's output.
-
-If the server round-trip fails, the command prints the error on one line. It
 exits 1, or 3 when the server no longer accepts the saved session (see "Session
-handling"). An album or file that quak cannot decrypt does not fail the command.
+handling"). An album or file quak cannot decrypt does not fail it; the command
-The command answers from what did decrypt (see "Default reads vs. fresh reads").
+answers from what did decrypt (see "Default reads vs. fresh reads").
-
 `--cache-dir` overrides where the cache lives; without it each account gets its
 own directory under the per-user cache path.
 
@@ -1119,17 +1096,16 @@ against a fake of the Ente HTTP API, and `yarn test` verifies them.
 ### Opening a library
 
 `Library.open(options)` loads the on-disk cache, starts the background refresh
-loop, and resolves to a `Library`. On an empty cache, or one that only
+loop, and resolves to a `Library`. On an empty cache it awaits the first
-`lib.refreshCollection()` wrote, it awaits one refresh. It then opens onto the
+refresh, so it opens onto the account's data whenever the server is reachable.
-account's data whenever the server is reachable. If that refresh fails, it opens
+It also awaits it on a cache that holds only what `lib.refreshCollection()`
-onto what the cache holds, and records the error in `lib.status()`. On any other
+stored. If that refresh fails, it opens onto what the cache holds and records
-cache it returns immediately and refreshes in the background. An unreachable
+the error in `lib.status()`. On any other cache it returns immediately and
-server then does not block opening.
+refreshes in the background, so an unreachable server does not block opening.
-
+With `refreshOnOpen: false` it starts no refresh, and returns at once onto what
-With `refreshOnOpen: false`, `open()` starts no refresh. It returns at once onto
+the cache holds, empty or not. A refresh then starts only when `fresh()` or
-what the cache holds, even when the cache is empty. A refresh then starts only
+`lib.backup()` asks, or the background timer fires. The CLI's read commands open
-when `fresh()` or `lib.backup()` asks, or the background timer fires. The CLI's
+it so (see "CLI surface").
-read commands open the library this way (see "CLI surface").
 
 `LibraryOptions`:
 
@@ -1156,35 +1132,28 @@ when the library opens; `lib.downloadDirectory` holds the result.
 `onProgress` receives a `RefreshEvent` with status `"started"` as a refresh, an
 ML data fetch or a precache sweep starts, and `"done"` or `"failed"` as it ends.
 In between, a refresh sends `"progress"` after each collection whose files it
-fetched. That event's `fetched` is the number of collections whose files the
+fetched. That event's `fetched` is how many collections it has fetched the files
-refresh has fetched so far. Its `total` is the number of collections whose files
+of so far, and its `total` how many it will. A refresh fetches the files of each
-it will fetch in all. A refresh fetches the files of each collection that is new
+collection that is new or changed since the cache's copy, so on an empty cache
-or changed since the cache's copy. On an empty cache, that is every collection.
+it fetches those of every collection.
 
 Concurrency is set through `pools`: construct
 `new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
 and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).
 
-`lib.status()` returns a `LibraryStatus`, which holds:
+`lib.status()` returns a `LibraryStatus` (collection/file counts, last
-
+refresh/ML times and errors, originals usage and effective limit, precache
-- the collection and file counts
+progress, and `closed`).
-- the time of the last refresh that succeeded, and the error of the latest
-  refresh if it failed
-- the same two for the ML data fetch
-- the originals cache's usage and its effective limit
-- the precache progress
-- `closed`
 
 `lib.close()` stops the background timer, and is idempotent. A refresh or
-`lib.refreshCollection()` in flight is left to finish. `close()` returns a
+`lib.refreshCollection()` in flight is left to finish. The promise `close()`
-promise that resolves once all of these have finished:
+returns resolves once all of these have finished, so the cache directory can
+then be removed:
 
 - that refresh or `lib.refreshCollection()`, with its cache write
 - the ML data fetch
 - the precache fetches already running
 
-The cache directory can then be removed.
-
 `lib.status().mlDataErrors` lists, as `{ fileID, error }`, each file whose ML
 data the last ML data fetch that succeeded got but could not decrypt, gunzip or
 parse; the `fetchMLData` event with status `"done"` carries the same list. A
@@ -1200,56 +1169,54 @@ synchronously from the copy held in RAM and never touch the network. A refresh
 changes that copy only once all its server requests have succeeded.
 `lib.refreshCollection()` changes it the same way for one collection. So a
 default read is immediate, but only as current as the last refresh or
-`lib.refreshCollection()` whose requests all succeeded. Right after `open()`
+`lib.refreshCollection()` whose requests all succeeded. Right after opening an
-returns without awaiting a refresh, that copy is the one loaded from disk.
+existing cache, it is the copy on disk.
 
 The background timer takes a turn `refreshIntervalSeconds` after its last turn
 ends. Each turn starts a refresh, unless a refresh or `lib.refreshCollection()`
 is already running.
 
-`await lib.fresh()` waits for a refresh to complete and persist. It then returns
+`await lib.fresh()` waits for a refresh to complete and persist, and returns the
-the same `{ albums, photos, timeline }` namespaces, which reflect a completed
+same `{ albums, photos, timeline }` namespaces, which then reflect a completed
-server round-trip. If a `lib.refreshCollection()` is running, `fresh()` first
+server round-trip. It first waits for a running `lib.refreshCollection()` to
-waits for it to end. When a refresh is already running, background or not,
+end. When a refresh is already running, background or not, `fresh()` waits for
-`fresh()` waits for that one. Its answer can then come from requests made before
+that one. Its answer can then come from requests made before the call. Only when
-the call. Only when none is running does it start one. A refresh that fails
+none is running does it start one. A refresh that fails rejects the caller
-rejects the caller (default reads stay silent and keep serving the last good
+(default reads stay silent and keep serving the last good copy). The CLI's read
-copy). Every CLI read command calls `fresh()`, except `quak collections` and
+commands use fresh reads (issue https://git.eeqj.de/sneak/quak/issues/75).
-`quak files` with a collection ID.
 
-`await lib.freshCollections()` is the fresh read of the collection list alone.
+`await lib.freshCollections()` is the fresh read of the collection list alone:
-It sends one collections request. It answers with every live collection that
+one collections request, answered with every live collection that decrypts, as
-decrypts, in the server's order, as the `Collection` objects that
+the `Collection` objects `listCollections()` returns, in the server's order. It
-`listCollections()` returns. It fetches no collection's files and stores
+fetches no collection's files and stores nothing, not even the list. A refresh
-nothing, not even the list. A refresh fetches a collection's files only when the
+fetches a collection's files only when the collection is new or has changed
-collection is new or has changed since the stored copy. So a collection stored
+since the stored copy. A collection stored without its files would never get
-without its files would never get them. `quak collections` uses
+them. `quak collections` uses it (issue
-`freshCollections()`.
+https://git.eeqj.de/sneak/quak/issues/210).
 
 `await lib.refreshCollection(collection)` brings one collection up to date in
 the cache, given its `Collection` from `freshCollections()`. It does for that
 collection what a refresh does for a changed one, and fetches no other
-collection's files. It fetches the collection's files changed since the cache's
+collection's files. It sends no request when the cache's copy has the same
-copy, or all of them when the cache has no copy. It then stores and saves them.
+`updationTime` or a later one. It also sends none when a refresh has already
-It sends no request when the cache's copy has the same `updationTime` or a later
+dropped the collection from the cache, because the server deleted it or it did
-one. It also sends none when a refresh has already dropped the collection from
+not decrypt. Otherwise it fetches the collection's files changed since the
-the cache, because the server deleted it or it did not decrypt.
+cache's copy, or all of them when there is none, then stores and saves them. The
+refresh cursor stays where it was, so the next refresh still fetches the files
+of every other collection that changed. It never runs at the same time as a
+refresh. It first waits for a running refresh to end. A `fresh()` or
+`lib.backup()` called after that waits for it. `quak files` with a collection ID
+uses it.
 
-`lib.refreshCollection()` leaves the refresh cursor where it was, so the next
+An album or file in the server's answer that quak cannot decrypt, or whose
-refresh still fetches the files of every other collection that changed. It never
+metadata is not JSON, does not fail the refresh. The refresh leaves it out, an
-runs at the same time as a refresh. It first waits for a running refresh or
+album with all its files, and drops any older copy the cache held, so reads
-`lib.refreshCollection()` to end. While it runs, a `fresh()` or `lib.backup()`
+answer from what did decrypt. The library keeps a list of what it left out, with
-waits for it to end. `quak files` with a collection ID uses it.
+the reason, in `metadata.json`. The next refresh asks the server only for what
-
+changed, so it would not see the record again. A record stays on that list until
-A refresh does not fail on an album or file that quak cannot decrypt, or whose
+a later refresh or `lib.refreshCollection()` stores a copy that decrypts, or
-metadata is not JSON. The refresh leaves out such a file, and such an album
+learns that the server deleted it. `lib.backup()` reports each such album, and
-together with all its files. It also drops any older copy that the cache held,
+each such file in its scope that has no copy that decrypted (see "Backup
-so reads answer from what did decrypt. The library keeps a list of what it left
-out, with the reason, in `metadata.json`. The next refresh asks the server only
-for what changed, so it would not see them again. An entry stays on that list
-until a later refresh or `lib.refreshCollection()` stores a copy that decrypts,
-or learns that the server deleted it. `lib.backup()` reports each such album,
-and each such file in its scope that has no copy that decrypted (see "Backup
 layout").
 
 ### Read surface
@@ -1345,16 +1312,10 @@ The GUI-facing records hold no key material and no binary, so they survive
 
 `lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first,
 photos newest first). `lib.subscribe({ onChange })` returns `{ unsubscribe }`.
-It calls `onChange` with a `LibraryChange` whenever a refresh or
+It delivers a `LibraryChange` (`albumsChanged`, `photosChanged`,
-`lib.refreshCollection()` changes these records. A refresh or
+`fileIDsRemoved`, `albumIDsRemoved`, `refreshedAt`) whenever a refresh or
-`lib.refreshCollection()` that changes nothing does not call it. A
+`lib.refreshCollection()` alters the projection. One that changes nothing
-`LibraryChange` holds:
+delivers nothing.
-
-- `albumsChanged`
-- `photosChanged`
-- `fileIDsRemoved`
-- `albumIDsRemoved`
-- `refreshedAt`
 
 ### Thumbnails, ML search, and backup
 
@@ -1370,44 +1331,30 @@ It calls `onChange` with a `LibraryChange` whenever a refresh or
   `SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
   default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
   a query vector the caller produced elsewhere.
-- `await lib.backup(opts?)` → `BackupResult`. It takes the lock, `backup.lock`
+- `await lib.backup(opts?)` → `BackupResult`. It takes the lock in the download
-  in the download directory. It fails at once with an error whose `code` is
+  directory, and fails at once with an error whose `code` is `ELOCKED` while
-  `ELOCKED` while another backup of that directory runs. A caller can instead
+  another backup of it runs. A caller can instead take the lock itself, before
-  take the lock itself before it opens its library, as `quak backup` does.
+  it opens its library, as `quak backup` does. `await lockBackupDirectory(dir)`
-  `await lockBackupDirectory(dir)` takes it, failing the same way, and returns a
+  takes it, failing the same way, and returns a `BackupLock`. Its `release()`
-  `BackupLock`. Its `release()` removes the lock unless it was lost, and its
+  removes the lock unless it was lost, and its `lost` holds why it was lost. The
-  `lost` holds why it was lost. The caller passes it to the backup as `lock`,
+  caller passes it to the backup as `lock`, and releases it once the backup
-  and releases it once the backup ends.
+  ends. The backup waits for a refresh as `fresh()` does. It then puts every
-
+  in-scope original not already at its save path there, as `photo.download()`
-    The backup waits for a refresh as `fresh()` does. It then fetches each
+  does, through the content cache. With `includeThumbnails` it also fetches
-    in-scope original that is not yet at its save path, as `photo.download()`
+  thumbnails. It waits for an ML data fetch, and rebuilds the on-disk backup
-    does, through the content cache. With `includeThumbnails` it also fetches
+  tree with a durable failure ledger. Each file's JSON holds its ML data and
-    thumbnails. It waits for an ML data fetch, then rebuilds the backup tree. It
+  keywords. An image's JSON also holds the EXIF and XMP of its original, and its
-    records each file that failed in `failures.json`.
+  dimensions if it is a JPEG. A live photo's JSON holds these for its image. A
-
+  video's JSON holds no EXIF, XMP or dimensions. A fetched original is written
-    Each file's JSON holds its ML data and keywords. An image's JSON also holds
+  straight to its save path and not into the cache, which then counts it as
-    the EXIF and XMP of its original, and its dimensions if it is a JPEG. A live
+  present; one the cache already held is copied from there. `BackupOptions`:
-    photo's JSON holds these for its image. A video's JSON holds no EXIF, XMP or
+  `downloadDirectory` (falls back to the library's), `includeOriginals` (default
-    dimensions.
+  `true`), `includeThumbnails` (default `false`), `onlyAlbumNames`, `verify`
-
+  (default `false`), `onProgress`, and `lock`. See Backup layout above for the
-    A fetched original is written straight to its save path, not into the cache.
+  tree it writes, and for the albums `BackupResult` lists in
-    The cache then counts it as present. An original that the cache already held
+  `undecryptableAlbums` and `failedAlbums`. Whether the backup took the lock or
-    is copied from there.
+  its caller did, once the lock is lost the backup rejects with
-
+  `lost the lock on <dir>: <reason>` before it starts another file.
-    `BackupOptions` holds:
-    - `downloadDirectory`, which falls back to the library's
-    - `includeOriginals`, `true` by default
-    - `includeThumbnails`, `false` by default
-    - `onlyAlbumNames`
-    - `verify`, `false` by default
-    - `onProgress`
-    - `lock`
-
-    "Backup layout" above describes the tree that the backup writes. It also
-    describes the albums that `BackupResult` lists in `undecryptableAlbums` and
-    `failedAlbums`. Once the lock is lost, the backup rejects with
-    `lost the lock on <dir>: <reason>` before it starts another file. This holds
-    whether the backup or its caller took the lock.
 
 ### Request pools