quak files COLLECTION fetches that collection's files alone (closes #217)

`quak files COLLECTION` used to wait for a full refresh, which on an
empty cache fetched the files of every collection first. It now reads
the collection list, then calls the new `Library.refreshCollection()`.
That brings one collection up to date in the store as a refresh would.
Once the stored cursor is past 0, it only updates a collection the store
holds. For a new one, `quak files COLLECTION` waits for a refresh. A
refresh from a cursor of 0 replaces all the store holds, so collections
then sit in the server's order. `open()` awaits the first refresh while
the cursor is 0. The store lists what did not decrypt in a fixed order.

Model: opus-5-5

Makefile

@@ -26,8 +26,10 @@ check:
 build:
 	@script/build
 
-build-bin:
-	@script/build-bin
+# Bundles the built dist/, so the binary reports the version script/build
+# stamped.
+build-bin: build
+	nix-shell -p bun --run "bun build dist/bin/quak.js --compile --outfile bin/quak"
 
 install: build-bin
 	mkdir -p ~/bin

TODO.md

@@ -2,7 +2,7 @@
 
 - branch from `next`
 - do the work in Next Step
-- delete Next Step from `TODO.md`
+- move Next Step to the top of Completed Steps
 - move the top item of Future Steps into Next Step
 - commit (`TODO.md` changes in the same commit as the work)
 - push
@@ -24,6 +24,723 @@ only sneak has, so it waits on him.
 Tagging and releases are decided by sneak alone, and happen only when he
 declares one.
 
+# Completed Steps
+
+- 2026-10-25: `quak files COLLECTION` fetches the files of that collection alone
+  (https://git.eeqj.de/sneak/quak/issues/217). It used to wait for a refresh,
+  which on an empty cache fetched the files of every collection first. It now
+  reads the collection list through `lib.freshCollections()`, then brings that
+  one collection up to date through the new `lib.refreshCollection()`. That
+  stores and saves its files as a refresh would, and leaves the refresh cursor
+  where it was. When the stored copy is current, it asks for none of the files.
+  It never runs at the same time as a refresh. Once a refresh has received the
+  account's collections, it stores no collection the cache does not hold, and
+  `quak files COLLECTION` waits for a refresh instead. A refresh from a cursor
+  of 0 replaces all the cache holds, so collections keep the server's order. A
+  library opened on a cache that only `lib.refreshCollection()` wrote awaits its
+  first refresh. `quak files` with no argument still waits for a refresh. The
+  store lists what did not decrypt by collection ID and file ID.
+
+- 2026-10-22: The JSON `quak backup` and `quak backup-metadata` write for a file
+  holds `keywords` in place of the CLIP embedding in its ML data
+  (https://git.eeqj.de/sneak/quak/issues/218). The keywords are the titles of
+  the Discover categories of Ente's photos app that the file falls in, by the
+  app's rule: the cosine similarity of the file's CLIP embedding and the text
+  embedding of a category's query is at least the category's minimum score, and
+  videos stay out of the categories that leave them out. `src/clip-keywords.ts`
+  applies the rule. `src/clip-keyword-vectors.ts` holds the 16 categories with
+  their text embeddings. `bin/clip-keyword-vectors` writes it with Ente's CLIP
+  text model, and writes the CLIP embedding of a public-domain photo of a sunset
+  that the tests use.
+
+- 2026-10-18: `-v`/`--verbose` raises the CLI's log level once for each repeat,
+  and `-q`/`--quiet` prints only fatal errors
+  (https://git.eeqj.de/sneak/quak/issues/212). The levels are fatal, error,
+  warning, info, debug and trace; the default is info, everything quak printed
+  on stderr before, `-v` gives debug and `-vv` trace. `src/cli-log.ts` holds the
+  level and the log the commands write stderr lines through. Debug adds each
+  retry, which `withRetry` reports through the new `onRetry`, and the start and
+  end of each refresh and ML data fetch; trace adds each request, which
+  `ApiClient` reports through the new `onRequest`. `--json` no longer keeps the
+  progress lines off stderr, so `backup` and `--json` runs log a refresh's
+  progress too. Every line that reports a failed file or album is at error,
+  which the progress callbacks of `backup`, `backup-metadata` and the thumbnail
+  helpers now say, and a command that exits 1 because files or albums failed
+  ends with one fatal line saying how many and where to look.
+
+- 2026-10-18: `quak files` takes the collection ID as an optional argument
+  instead of the `--collection` option, which is gone
+  (https://git.eeqj.de/sneak/quak/issues/211). With no argument it lists every
+  file in the account once, collection by collection; a file in several
+  collections is listed with the first of them, which is its `collectionID` in
+  `--json`. The fresh read in `src/cli-read.ts` does this.
+
+- 2026-10-18: Names quak builds from an album name or a title fit in a file name
+  (https://git.eeqj.de/sneak/quak/issues/205). `shortenKeepingExtension` in
+  `src/filename.ts` cuts a name short to a number of bytes, keeping its
+  extension when it fits and always its first character, and can put text right
+  before the extension; the backup's `withID` uses it for an ID.
+  `defaultFileName` there gives the name a file is saved under when the caller
+  gives no path: a prefix such as `thumb_` and the title with unsafe characters
+  replaced, cut short with it to 255 bytes, and, for a live photo's image or
+  video, with that part's own extension before the cut. `quak get`,
+  `quak get-thumb`, `downloadFile` and `downloadThumbnail` use it without
+  `--out` or `outPath`. `backup-metadata` cuts an album's name short so that its
+  folder, `<id>-<name>`, fits in 255 bytes. `safeExtension` gives `.bin` for an
+  extension of more than 16 letters and digits, so a save path's name always
+  fits.
+
+- 2026-10-18: `quak collections` no longer waits for the files of every
+  collection (issue 210). On an empty cache it opened its library with a refresh
+  that fetched and decrypted the files of each collection in turn, printing
+  nothing and storing nothing until all were done, so on a large account it
+  seemed to hang. It now reads the collection list alone through the new
+  `lib.freshCollections()`, one request that stores nothing. The read commands
+  open their library with the new `refreshOnOpen: false`, so each makes only the
+  refresh it asks for. A refresh sends a `"progress"` event after each
+  collection whose files it fetched, and the commands that wait on one print it
+  on stderr, except with `--json`.
+
+- 2026-10-16: ML data the server sends that does not decrypt, gunzip or parse is
+  no longer taken for no ML data (issue 198). `fetchMLDataBatch` returns why for
+  each such file beside the payloads; ML data that parses to something other
+  than a JSON object, such as `null`, is one of them. The library keeps the
+  reasons from its last ML data fetch that succeeded in memory, not in the
+  cache, and gives them in `status().mlDataErrors` and in the `fetchMLData`
+  event with status `"done"`; a fetch still running or one that fails does not
+  change them. `quak backup` writes the reason as `mlDataError` in the file's
+  JSON when no ML data is cached for it, and records the file in `failures.json`
+  with an error starting `ML data: `; `backup-metadata` writes `mlDataError`
+  too. Both exit 1. Tests serve ML data encrypted under a key that is not the
+  file's.
+
+- 2026-10-16: With `--verify`, an original whose hash does not match is no
+  longer removed before its replacement is downloaded (issue 199).
+  `replaceOriginal` in `src/library/content.ts` downloads the replacement to a
+  temporary file beside it, a live photo's image and video each to its own,
+  hashes it and renames it over the original only when it matches. When the
+  download fails or does not match either, the original keeps its bytes and
+  links, and the file goes into `failures.json`. The replacement is always
+  downloaded, through the new `fetchOriginal` of the content cache, never copied
+  from the cache.
+
+- 2026-10-16: The `~/bin/quak` that `make install` builds retries a refused
+  connection and a name that does not resolve, and the backup records either as
+  transient (issue 206). Bun's `fetch` reports them with codes of its own,
+  `ConnectionRefused` and `FailedToOpenSocket`, which `src/retry.ts` now counts
+  as transport failures. A `POST` or `PUT` is not replayed on them, because bun
+  also gives them when it resends a request on a new connection after a
+  connection it kept open closed, and the server may already have the request.
+  Bun already gave `ECONNRESET` for a reset and a `TimeoutError` for a deadline.
+  `classify` in `src/backup.ts` now records any `TimeoutError` as transient, so
+  a stalled download, whose error says only that it stalled, is no longer
+  `unknown`. The shapes were found by running the binary against local
+  listeners.
+
+- 2026-10-15: An album whose folder or JSON `quak backup` cannot write no longer
+  ends the run before the other albums and `failures.json` are written (issue
+  197). The run prints `FAILED album <name>: <reason>`, lists the album's ID in
+  `failedAlbums` in the result and in `--json`, goes on, and exits 1. An album
+  whose folder would have the name of another album's JSON, as `x.json` beside
+  `x` would, gets its ID added. A folder or link name over 255 bytes is cut
+  short between two characters, by `shortenToBytes` in `src/filename.ts`, and
+  gets the ID.
+
+- 2026-10-15: A network failure is printed and recorded with its reason, not as
+  just `fetch failed` (issue 196). `describeError` in `src/errors.ts` gives an
+  error's message followed by the message of each error on its `cause` chain,
+  such as `fetch failed: connect ECONNREFUSED 1.2.3.4:443`, and each address's
+  reason when a host has several and none could be reached. `run` prints that,
+  and the backup logs it and records it in `failures.json`, where `classify` now
+  calls the failure transient when the chain holds an errno that
+  `hasTransportCode` in `src/retry.ts` recognises, for a failed ML data fetch
+  too. The walk along the chain moved from `src/retry.ts` to `src/errors.ts`,
+  which both use. Tests refuse every connection of a backup, then only one
+  file's download, and fail the ML data fetch.
+
+- 2026-10-15: The README says that `make install` needs nix (issue 200), in
+  Getting Started and in "Running the backup from cron": `make build-bin` runs
+  bun through `nix-shell -p bun`. The `Makefile` is unchanged.
+
+- 2026-10-14: `lib.backup()`, and so `quak backup`, works on up to the content
+  pool's limit of originals at once, 5 by default, instead of one at a time
+  (issue 182). That many workers take files from one shared list, each doing all
+  the work for one original, its `--verify` hashing included, so no more
+  originals than that are open at once. `BackupLibrary` says the limit as
+  `contentConcurrency`. A test holds each download at the fake Ente server to
+  count those in flight. Once a worker has thrown, as one does when `onProgress`
+  throws, or the lock is lost, no worker takes another file; the backup fails
+  once the originals under way are in place, and a lost lock stops it before it
+  fetches ML data.
+
+- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's
+  review, though that issue closed on 2026-10-03 (issue 194). It now names the
+  issues open on the tracker.
+
+- 2026-10-14: A backup whose lock is lost no longer dies with a stack trace
+  (issue 184). `lockBackupDirectory` gives proper-lockfile an `onCompromised`
+  that records why the lock was lost instead of throwing from its timer, and
+  returns the lock, which `quak backup` passes to `lib.backup()` as `lock`. The
+  backup then stops before it starts another file with
+  `lost the lock on DIR: REASON`, which `quak backup` prints as one line before
+  it exits 1. Releasing a lost lock leaves `backup.lock` alone. Tests in
+  `test/cli/backup-end-to-end.test.ts` remove the lock, or put another backup's
+  in its place, while a download is held open.
+
+- 2026-10-14: An album or file quak cannot decrypt, or whose metadata is not
+  JSON, no longer fails the refresh, so one bad record no longer stops
+  `quak backup` from storing anything (issue 181). The refresh leaves it out, an
+  album with its files, and the library keeps it with the reason in
+  `metadata.json` until the server sends a copy that decrypts or deletes it.
+  `quak backup` stores every other file, records each such file in
+  `failures.json` with an error starting `could not be decrypted:`, names each
+  such album's ID in a printed line, or with `--json` in `undecryptableAlbums`,
+  and exits 1 while any remain. A file whose copy in another album decrypts is
+  not recorded: its original is stored from that copy, and the album where it
+  does not decrypt neither links nor lists it. The read commands answer from
+  what decrypted.
+
+- 2026-10-14: `quak backup` no longer asks the server for changes every 3
+  seconds while it runs (issue 183). It opens its library with
+  `refreshIntervalSeconds: Infinity`, which the library now takes as no
+  background timer, so the backup refreshes only as it starts and no timed
+  refresh starts another ML data pass during the run. A test holds a backup's
+  download open while two hours pass and counts its `/collections/v2` requests.
+
+- 2026-10-14: `test/cli/backup-end-to-end.test.ts` runs `quak backup` as cron
+  does (issue 186): `backupCommand` and `run`, the real `loadSession` reading a
+  session saved after logging in, and every request answered by a fake of the
+  Ente HTTP API serving data encrypted with libsodium. It covers the backup tree
+  of an account with two albums of its own and one shared with it, a second run
+  that downloads nothing, a download that succeeds after two 503s, a 404 that
+  lands in `failures.json` with exit 1, a refresh answered 401 with exit 3, and
+  `--verify` putting back an original changed on disk. The fake server and the
+  account it logs in moved from the backup-metadata test into
+  `test/ente-server.ts`, unchanged.
+
+- 2026-10-14: `files --collection`, `get`, `get-thumb` and
+  `helper fix-missing-thumbnails --file` take an ID only when it is written in
+  decimal digits and is a safe integer (issue 185). Before, `Number()` read
+  `0x10` as 16, `1e3` as 1000 and an empty string as 0, and `--file` dropped an
+  ID it could not read without a word. Anything else now prints
+  `Invalid file ID: <arg>` or `Invalid collection ID: <arg>` and exits 1 before
+  any request; one bad `--file` ID fails the whole command. `parseID` in
+  `src/cli-commands.ts` does the check for all four.
+
+- 2026-10-06: The README says how to run `quak backup` from cron (issue 170):
+  log in once as the job's user, a crontab with a backup every night and
+  `--verify` on Sundays appending to a log file, and the `HOME` and
+  `XDG_DATA_HOME` the job needs to find the saved session. A table gives each
+  exit code: 0, the backup is complete; 1, files are in `failures.json` or
+  another error stopped the run; 2, another backup of the directory is running;
+  3, there is no usable session. The introduction lists everything the backup
+  keeps for each file.
+
+- 2026-10-06: `quak backup --verify` and `lib.backup({ verify: true })` hash
+  each original already at its save path as the download check does, streamed, a
+  live photo as `<imageHash>:<videoHash>` (issue 168). One that does not match
+  the content hash its metadata records is logged, removed (both files of a live
+  photo) and put back in the same run, downloaded or copied from the cache, and
+  what is put back is hashed too. One that still does not match, or whose
+  download fails, goes into `failures.json`. One with no recorded hash is left
+  alone. The result, `--json` and the summary gain `verified`, `mismatched` and
+  `unchecked`. Without `--verify` nothing is hashed.
+
+- 2026-10-06: Two backups of the same directory never run at once (issue 169).
+  `lib.backup()` takes a lock, `backup.lock` in its download directory, made
+  with `proper-lockfile`, before its refresh, and removes it when it ends,
+  whether it succeeds or fails. A second backup of the directory, from another
+  process or the same one, fails at once with an error naming the directory.
+  `quak backup` takes the lock before it opens its library, so a refused run
+  sends no request; it prints the error as one line and exits 2. A lock that has
+  gone 10 seconds untouched, left by a run that could not remove it, is taken
+  over by the next run.
+
+- 2026-10-06: `quak backup` writes each original's EXIF, XMP and dimensions into
+  the file's JSON as `imageMetadata`, what `backup-metadata --exif` records
+  (issue 167): for a live photo from its image, for a video nothing, and `{}`
+  for an original with none of them. A failed read puts the reason in
+  `imageMetadataError` and fails neither the file nor the run. An original is
+  read when the run stores it or when its JSON has neither field; otherwise the
+  field is taken from that JSON. The hand-built JPEGs moved to
+  `test/exif-jpeg.ts`, beside the HEIC.
+
+- 2026-10-06: When the server answers HTTP 401 and that ends a command that
+  loads the saved session, because the server no longer accepts its token, the
+  command prints one line,
+  `quak: the saved session is no longer valid; run "quak login"`, and exits 3
+  (issue 164). `quak backup` meets that 401 on the refresh it starts with,
+  before it touches any file. For the same commands, a missing or corrupt
+  session file keeps its message and also exits 3, so a cron job can tell that
+  the user must log in again. `quak logout` is unchanged. `run` in
+  `src/cli-run.ts` recognises the 401, which reaches it unchanged.
+
+- 2026-10-06: `quak backup` retries a failed request for longer than the other
+  commands do (issue 165). `src/retry.ts` exports `UNATTENDED_RETRY_OPTIONS`
+  beside the unchanged default: 10 attempts, a 1 s base delay and a 60 s cap, so
+  a request that keeps failing waits at most 243 s before it gives up.
+  `bin/quak.ts` loads the backup's session with them, so its refresh, ML data
+  and downloads all use them. What is retried and the backoff formula are
+  unchanged.
+
+- 2026-10-06: The files this repository copies from `sneak/prompts` are copied
+  again from its commit `dd4027b` (issue 171). `.gitignore` and `.dockerignore`
+  keep out more secret files, and this repository's build artifacts follow the
+  copied content. `script/check` runs `script/fmt-check` again, `script/cibuild`
+  runs `script/bootstrap` and `script/check` before the image build,
+  `script/fmt` and `script/fmt-check` find the pinned yarn under nvm, and the
+  image's last stage marks `/app` safe for git.
+
+- 2026-10-06: `quak backup` writes the account and album records
+  `backup-metadata` writes (issue 166): `account.json` with the account's
+  `email` and `userID`, and in each album's JSON its `ownerID`, `isShared`,
+  `updationTime` and, when present, its three layers of magic metadata. Each
+  file's JSON gains its `updationTime`.
+
+- 2026-10-05: `quak backup` writes each file's ML data (its faces and CLIP
+  embedding) into the file's JSON as `mlData`, the payload
+  `lib.mldata.forFile()` returns (issue 163). `lib.backup()` waits for an ML
+  data fetch before it writes the JSON files. When that fetch fails, each file
+  whose ML data is not cached gets the reason in `mlDataError` and counts as
+  failed, and the next run fetches it again.
+
+- 2026-10-03: The download-albums example test no longer fails on a disk with
+  under 50 GiB free (issue 160). It opens its libraries with
+  `freeBelowBytes: 0`, so the free space of the disk it runs on cannot shrink
+  the cache, evict the original it cached and add a fetch to the count. No other
+  test's result depends on that free space. The 50 GiB default is unchanged.
+
+- 2026-10-02: `docker build .` stamps the commit's tag or short commit, not
+  `dev` (issue 154). `script/build` writes the version `script/version` prints
+  into `dist/package.json`: the `VERSION` environment variable or build arg when
+  one is given, otherwise `git describe --tags --always`, otherwise the version
+  `package.json` declares. `.dockerignore` sends `.git`, and a checkout with
+  `.git` whose version comes out empty, `dev` or `unknown` fails the build.
+  `make build-bin` bundles the built `dist/`, so the single binary reports the
+  same version.
+
+- 2026-10-02: `photo.exif()` returns every EXIF tag in the file as `ExifTags`,
+  keyed by tag name, each as exifreader decodes it, not only the thirteen common
+  fields (issue 156). The embedded thumbnail's tags are under `Thumbnail`,
+  without the thumbnail image. The thirteen typed methods stay, each picking its
+  field from the tags `exif()` returns, typed as in `PhotoExif`. The example
+  script's JSON files now carry every tag.
+
+- 2026-10-01: The content cache no longer looks for a live photo that an earlier
+  version cached as one ZIP (issue 151). When the cache opens, a live photo's
+  file that no JSON file names is now always left alone.
+
+- 2026-10-01: `examples/download-albums.ts` logs in, opens the library, and for
+  every album downloads each photo to its save path, writes the photo's record
+  and EXIF fields to a JSON file beside it, and writes the album's photos to
+  `albums/<collectionID>.json` (issue 144). The build compiles it to
+  `dist/examples/`; the README's "Examples" section says how to run it.
+
+- 2026-10-01: A `Photo` has one async method for each field of `exif()`, named
+  and typed as in `PhotoExif`: `make()`, `model()`, `lensModel()`,
+  `dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`,
+  `iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()`
+  and `gpsAltitude()` (issue 148). Each calls `exif()` and returns its one
+  field, or undefined when the file lacks it. `Photo` implements a type with one
+  method per `PhotoExif` field, so the build's type check fails when a field has
+  no method. A test checks, on the JPEG and the HEIC, that each method gives the
+  same value as `exif()`.
+
+- 2026-10-01: Each original's save path is
+  `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
+  download directory, which defaults to `photos` in the working directory (issue
+  143). The date is the photo's `takenAt` in the machine's time zone.
+  `photo.savePath` is always a string, with or without a content cache, and
+  `isLocal` is true only when the original is at its save path.
+  `photo.download()` puts the original there, copied from the cache when the
+  cache holds it and fetched otherwise. `lib.backup()` does the same for each
+  file, writes each file's JSON beside its original, and links `collections/` to
+  the save paths.
+
+- 2026-10-01: `photo.exif()` and `backup-metadata --exif` read EXIF from
+  HEIC/HEIF originals, a live photo's HEIC image included, as well as JPEG and
+  the other image formats `exifreader` reads (issue 145). `exifreader` replaces
+  `exif-reader` and the JPEG segment scan; `PhotoExif` is unchanged. The `exif`
+  field of `backup-metadata --exif` is now exifreader's tag output, and
+  `exifRaw` holds the whole EXIF block it could not read. The tests use a real
+  HEIC, `test/exif.heic`.
+
+- 2026-10-01: A `Photo` has `savePath`, `isLocal`, `content()`, `exif()`,
+  `modifiedAt`, `hash` and `year` (issue 141). `savePath` is where
+  `lib.backup()` writes the original under the library's download directory; for
+  a live photo not yet stored, it carries the title's extension, and the backup
+  may store the image under a different one. `isLocal` says whether the whole
+  original is there. Both look only at the disk. `content()` returns the
+  original's bytes and `exif()` the common EXIF fields of a JPEG; both may
+  download the original, and `exif()` downloads no video. `PhotoRecord` gains
+  `modifiedAt` and `hash`, and the JPEG EXIF scan moved to `src/exif.ts`.
+
+- 2026-09-29: The "Workflow" list at the top of this file now says to branch
+  from `next` and open a pull request that targets `next`, that the repository
+  manager squash-merges a reviewed pull request into `next`, and that only sneak
+  merges `next` into `main`, as the README does (issue 137).
+
+- 2026-09-29: The README's "Development workflow" and "For LLMs" sections now
+  say work branches from `next`, every pull request targets `next`, the
+  repository manager squash-merges reviewed pull requests into `next`, and only
+  sneak merges `next` into `main` (issue 135).
+
+- 2026-09-29: Brought the README and this file in line with `next` after the
+  milestone merge (issue 132). The Next Step says no implementation work is open
+  and the cache design (issue 36) waits on sneak's review. README corrections:
+  the Getting Started comments say when the library refreshes; most, not all,
+  Makefile targets call a script; `script/lint` and `script/test` have no host
+  path, though `yarn test` does; the SRP handshake uses `fast-srp-hap`, outside
+  `crypto/`, and a thumbnail upload's MD5 uses `node:crypto`; the SRP password
+  is the first 16 bytes of a 32-byte subkey; the key attributes and token come
+  after SRP, after the TOTP code SRP may ask for, or after the email OTP that
+  replaces SRP when the account has email MFA on, and quak cannot answer a
+  passkey; the auth token is sent as URL-safe base64 with padding; every
+  `TypeError` is retried; each download attempt writes its own temporary files,
+  two for a live photo, and only the attempt that completes renames them into
+  place; `runMetadataBackup` records a failed download in the file's JSON; a
+  second `client.logout()` does not throw; `quak logout` with no session exits
+  0, and names the cache directory only when it knows it; `login`, `whoami` and
+  `logout` open no library; `--exif` records XMP and, for a JPEG, EXIF, and no
+  IPTC; which commands take `--json`; a failed ML data request may not have been
+  retried; the thumbnail fixer is not limited to baseline JPEG; `quak backup`
+  still fetches ML data; a backup's JSON holds the basic metadata fields quak
+  keeps and the private and public magic metadata, not every decrypted field,
+  and the README no longer lists what the magic metadata holds; the default
+  cache directory is the per-user one, not an XDG path on macOS; pinned
+  originals can exceed `cacheOriginalsMaxBytes`; which tests cover which
+  operations; a default read is only as current as the last refresh whose
+  requests all succeeded; `fresh()` and `lib.backup()` join a refresh already
+  running; a `Photo` has no `thumbnailPath` or `originalPath`; and `408` and
+  `429` are retried.
+
+- 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks
+  that the image's and the video's temp files are fsynced before either is
+  renamed into place, and the directory after both renames, as the `writeAtomic`
+  test does for one file.
+
+- 2026-09-23: Live photos are stored as their image and their video (issue 107).
+  A live photo, which Ente stores as one ZIP, is unpacked as it downloads into
+  `<fileID>.<ext>` for the image and for the video, each with its extension from
+  the ZIP, beside `<fileID>.livephoto.json`, which names the two. Both are
+  checked against the recorded hash and renamed into place only when both are
+  complete. A ZIP holding a second image or video, or whose two files come to
+  more than 20 times its size plus 16 MiB, is refused. The backup and the
+  content cache count the live photo as stored only with both files, the album
+  folder links both, `quak get` writes both, and the content result gives the
+  video as `videoPath`. A ZIP an earlier version stored is replaced on the next
+  backup run, and removed from the cache when the library opens it.
+
+- 2026-09-23: Settled the package metadata (issue 6). quak is not published, so
+  `package.json` is marked `"private": true` and the `files` field is gone.
+  `engines.node` is `>=22`, the major version `script/bootstrap` and the
+  `Dockerfile` use. An `exports` map makes `.` and `./package.json` the only
+  importable paths; `runMetadataBackup`, the thumbnail helpers and their types
+  stay internal to the CLI.
+
+- 2026-09-23: Brought the README and this file in line with the tree (issue
+  111). The layout lists `src/library/` and the other source files, the backup
+  layout names `failures.json` and the optional `thumbnails/`, "Opening a
+  library" says what happens when the first refresh fails, the Testing section
+  gives the 60-second hard cap and 20-second target for `make test` and names
+  the 90-second `timeout` in the `Dockerfile` as the backstop for a hung test,
+  and "Tag v1.0.0" is no longer listed as the next step.
+
+- 2026-09-23: Tested `quak login` and `backup-metadata --exif` (issue 110).
+  `loginCommand` takes its login function and its prompts from `CliContext`, and
+  `bin/quak.ts` passes `Client.login` and the terminal prompts. Tests cover a
+  login from `QUAK_EMAIL` and `QUAK_PASSWORD` with no prompt, the TOTP prompt, a
+  failed login, and the saved session's modes, and show that `--exif` and
+  `--all` each turn on EXIF extraction and that it is off without them.
+
+- 2026-09-23: `quak backup` writes each original once and no longer fills the
+  cache (issue 106). An original fetched for a backup is written by the download
+  writer straight into the backup's `originals/`, and the content cache records
+  it there instead of keeping its own copy; one the cache already held is still
+  copied. `quak backup` opens its library with the thumbnail and originals
+  precache off.
+
+- 2026-09-23: `backup-metadata`, `helper list-missing-thumbnails` and
+  `helper fix-missing-thumbnails` refresh before they answer (issue 100). Each
+  awaits `lib.fresh()` before reading, so a file added since the cache was
+  written is included, and a failed refresh prints one line and exits 1 instead
+  of answering from a stale or empty cache. The README lists them among the
+  commands that refresh first.
+
+- 2026-09-23: `quak backup` waits for the server refresh and fails when it fails
+  (issue 99). `lib.backup()` joins a refresh already running or starts one, as
+  `fresh()` does, and rejects before touching any file when it fails, leaving
+  `failures.json` as it was, so `quak backup` prints the error as one line and
+  exits 1 instead of backing up the previous run's file list, or nothing, and
+  exiting 0.
+
+- 2026-09-23: CLI errors print a message instead of a stack trace (issue 102).
+  An error a command throws is printed as one `quak: MESSAGE` line on stderr and
+  the CLI exits 1 once output has drained. The wrapper that does this moved from
+  `bin/quak.ts` to `src/cli-run.ts`, and `bin/quak.ts` now awaits
+  `program.parseAsync()`.
+
+- 2026-09-23: Opening a library no longer deletes another process's download in
+  progress (issue 105). The download writer's temp files are named
+  `.quak-<pid>-<random>.tmp`, and `removeLeftoverTempFiles`, moved from the
+  backup into the download module, deletes a `.quak-*.tmp` file only when the
+  process ID in its name is no longer running. The content cache calls it at
+  `open()` for `originals/` and `thumbnails/`, the backup as before.
+
+- 2026-09-23: Re-vendored the lint and test setup from the template (issue 96).
+  Linting and testing are the `lint` and `test` phases of the `Dockerfile`;
+  `script/lint` and `script/test` each build one with `--no-cache`, and the last
+  stage compiles and depends on both, so `script/cibuild` is one build.
+  `Dockerfile.lint`, `CHECK_EPOCH`, `LINT_EPOCH` and the tests that checked them
+  are gone; `REPO_POLICIES.md` is re-copied.
+
+- 2026-09-23: Stopped `helper fix-missing-thumbnails` retrying files the server
+  always refuses (issue 109). Both thumbnail helpers skip a file another account
+  owns without fetching it. The fixer skips a file whose recorded thumbnail size
+  is 0 or unknown before downloading it, and otherwise tries smaller encodings
+  (720 px quality 50 down to 160 px quality 20) until the encrypted thumbnail is
+  no larger than that size, skipping the file if none fits.
+
+- 2026-09-23: Tested the live-photo hash check's error paths (issue 117). Tests
+  download a live photo whose ZIP names an unknown compression method, one whose
+  ZIP has no image entry and one with no video entry, and check that nothing is
+  stored and the error names the file ID; the unreadable one is not retried.
+
+- 2026-09-23: `quak logout` ends the session on the server (issue 108). It calls
+  `POST /users/logout` through the new `Client.logoutOnServer()`, then deletes
+  `session.json` even when that call fails, says so and exits 1. It prints the
+  account's cache directory and says it still holds decrypted data. The default
+  cache path is now `defaultCacheDirectory()` in the library, shared with
+  `Library.open`.
+
+- 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files
+  in one collection with the same title, and two collections with the same name,
+  each get their ID added to the name (`IMG_0001 (12345).JPG`, `Trip (10)/`), so
+  none replaces another's symlink or JSON. Each run removes symlinks into
+  `originals/` for files no longer in the collection, and the folders of deleted
+  or renamed collections, leaving anything else in `collections/` alone. The
+  README backup layout states the naming rule.
+
+- 2026-09-23: Checked downloaded originals against their recorded content hash
+  (issue 68). `downloadFile`, which `quak get`, the content cache and backup all
+  use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and
+  stores nothing on a mismatch, failing with an error naming the file ID. A live
+  photo ZIP is unpacked as it streams with `fflate` and its image and video
+  hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older
+  clients' `imageHash` and `videoHash` fields for live photos. A file with no
+  recorded hash is stored unchecked.
+
+- 2026-09-23: Kept one account's cache from mixing with another's (issue 104).
+  When `metadata.json` in the cache directory was written for a different,
+  non-zero user ID than the client's, `Library.open` deletes it and `mldata/`
+  and starts empty, so the first refresh enumerates from 0. This only happens
+  with `--cache-dir` or an explicit `cacheDirectory`; the default path already
+  includes the user ID. A test opens one account's cache as another account.
+
+- 2026-09-23: `backup-metadata` no longer stops on one failed ML data request
+  (issue 101). Each request of up to 200 files is tried on its own; a failed one
+  is logged, its files are written with the reason in `mlDataError`, and the
+  command exits 1 once the dump is complete. `fetchMLData`, which only this
+  command used, is gone; the command calls `fetchMLDataBatch` per batch.
+- 2026-09-23: Single-sourced the version string (issue 5). `package.json` is the
+  only place it is written: `src/index.ts` imports it for `VERSION` and
+  `bin/quak.ts` passes `VERSION` to commander. tsc copies `package.json` to
+  `dist/package.json`, so the import resolves from the built output too, and
+  `script/build` runs the built CLI with `--version` to prove it. A test checks
+  that `VERSION` and `quak --version` both equal the `package.json` version.
+- 2026-09-23: Tested that `Library.close()` waits for the originals precache
+  (issue 93). The test that holds a precache fetch open while `close()` runs now
+  runs once with only the thumbnail fill and once with only the originals fill,
+  so dropping either wait from `Precache.close()` fails a test.
+- 2026-09-23: Fixed two intermittently failing library tests (issue 90).
+  `Library.close()` now returns a promise that resolves once an in-flight
+  refresh (including its cache write), the ML data fetch and running precache
+  sweeps have finished; the library tests await it, so `afterEach` no longer
+  removes the cache directory while something is still writing into it. The
+  precache test waits for both fills to report "done" instead of for its stub
+  source to be called, which happened before the cache recorded the file.
+- 2026-09-23: Pinned three guards reviewers found untested (issue 89). The
+  download idle deadline's timer is unref'd, so it can never keep the process
+  alive, and a test checks no timer is left after a download completes or fails.
+  A test covers the rejection of `#` in a request path. The EXIF scan compares
+  the `Exif` header only in an APP1 segment of length 8 or more, so it never
+  reads the next segment's bytes, with a test for a short one.
+- 2026-09-23: Made the download deadline an idle deadline (issue 24).
+  `downloadTimeoutMs` now aborts a file or thumbnail download only after no
+  bytes have arrived for that long, default 60 seconds, instead of bounding the
+  whole transfer at 10 minutes, so a slow download that keeps making progress
+  completes. A download that fails before reading the whole body cancels it, so
+  a failed file no longer holds its connection.
+- 2026-09-23: Every `ApiClient` request URL is now built by one function next to
+  the class (issue 18), so a self-hosted `apiOrigin` with a base path keeps it
+  on every request, a path works with or without a leading slash, and query
+  parameters are percent-encoded. A path containing `?` or `#` is rejected with
+  an error instead of being silently cut.
+- 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies
+  moved from `bin/quak.ts` into `src/cli-commands.ts` as functions that take
+  their options and a context (output streams, session directory, cache
+  directory, session loader) and return an exit code; `bin/quak.ts` only wires
+  them to commander and exits with the code once stdout and stderr have drained,
+  so nothing below it calls `process.exit`. `test/cli/commands.test.ts` drives
+  them with a fake client: session file modes, logout, the missing and corrupt
+  session paths, and the output and exit code of `whoami`, `collections`,
+  `files`, `get`, `get-thumb`, `backup` and `helper list-missing-thumbnails`.
+- 2026-09-23: Hardened the backup tree's atomic copy (issue 22). `copyAtomic`
+  fsyncs its temp file before the rename and the directory after it, through the
+  download writer's `fsyncPath`; each backup run deletes `.quak-backup-*.tmp`
+  files whose process is no longer running. The README backup layout names the
+  temp files and states that the rename replaces a symlink and takes the temp
+  file's permissions. Added tests for a missing and an unwritable destination
+  directory for `downloadFile` and `downloadThumbnail`.
+- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue
+  11). Every segment length is checked against the remaining bytes and lengths
+  under 2 stop the scan, so a truncated or corrupt original can neither throw
+  nor loop. A malformed or unparseable EXIF segment is recorded as
+  `imageMetadata.exifError`, and a failure to read the original as
+  `imageMetadataError` in the per-file JSON, instead of the field being left
+  out.
+- 2026-09-22: Hardened the retry classifier (issue 80). A `POST` or `PUT` is
+  replayed only when every errno in the cause chain is a connect errno, and it
+  no longer follows redirects. `getRetryOptions()` returns a copy. Tests pin
+  every errno the classifier names, the cause-chain depth limit, cycle
+  termination, and a fresh deadline per attempt for every retrying entry point.
+  The README's endpoint list is the one place that names the requests the replay
+  rule covers.
+- 2026-09-22: Stopped `make test` collecting tests from checkouts nested under
+  `.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a
+  nested checkout ran the whole suite again; `vitest.config.ts` now adds
+  `.claude/**` to vitest's default excludes, and
+  `test/packaging/nested-checkout.test.ts` plants a nested checkout in a temp
+  directory and fails if vitest would collect it.
+- 2026-09-22: Dropped the deprecated `@types/libsodium-wrappers-sumo` stub from
+  `devDependencies` (issue 27). It shipped no declarations; the types come from
+  `libsodium-wrappers-sumo` itself. `yarn.lock` regenerated by `yarn remove`.
+- 2026-09-22: Hardened the client session lifecycle (issue 10).
+  `Client.fromJSON` checks every snapshot field and each key's decoded length
+  and names the bad field; `toJSON` reads the token through
+  `ApiClient.getAuthToken` and throws when there is none; `logout` zeroes the
+  key buffers, and `collectionsSince` re-checks for logout after its request so
+  it never decrypts with zeroed keys. The CLI reports a corrupt session file
+  separately from a missing one (`src/cli-session.ts`).
+- 2026-09-22: Sanitized file names taken from server metadata (issue 9). A new
+  `src/filename.ts` holds the one sanitizer, used by `quak get`/`get-thumb`
+  without `--out`, `downloadFile`/`downloadThumbnail` without `outPath`, and the
+  backup and metadata backup trees; it removes separators, control characters,
+  leading dots and Windows device names, and falls back to a name built from the
+  ID for an empty title. Originals-cache extensions are letters and digits only,
+  else `.bin`. A user-supplied path is used as is. `decryptFile` reads a missing
+  or non-string title as "" and rejects metadata that is not a JSON object.
+- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
+  snippets) to match the shipped cache/API library on `next` (issue 53, issue
+  13). Documented `Library.open` and its options, the default-read vs `fresh()`
+  distinction (and that the CLI's read commands are fresh), the record types and
+  `snapshot()`/`subscribe()`, the `albums`/`photos`/`timeline` read surface,
+  `Photo` content methods, `thumbnails.ensure`, the `mldata` search surface,
+  `backup()`, the three request pools (10/5/25), and the on-disk cache layout;
+  noted the deferred content-hash integrity check (issue 68). Docs-only; no code
+  changed.
+- 2026-09-22: Added resumable, deletion-aware enumeration to `Client` (issue 38,
+  closes issue 7). `collectionsSince`/`filesSince` take a starting cursor,
+  decrypt live records, surface tombstoned ids in a separate `deleted` list (a
+  tombstone has nothing to decrypt, so it is a bare id, not a hollow record),
+  and return the max `updationTime` seen as the cursor to resume from.
+  `filesSince` refuses to loop when the diff reports `hasMore` without advancing
+  the cursor (issue 7). `listCollections`/`listFiles` are now thin wrappers that
+  enumerate from `sinceTime: 0` and drop deletions, so existing callers are
+  unaffected.
+- 2026-09-22: Carried file size, thumbnail size, and the deletion flag through
+  `decryptFile` (issue 37, foundation for the cache/API design). Live files now
+  populate `file.size`/`thumbnail.size` from the server's `info` (left
+  `undefined` when the server omits it), and `isDeleted` is carried from the
+  diff row onto `EnteFile`. No caller change: `listFiles` still filters deleted
+  rows before decrypting. Surfacing a tombstone through decryption belongs to
+  the enumeration unit (issue 38).
+- 2026-08-10: Made `lint-once.test.ts` enforce what its header claims. It walked
+  `make check` only, so it never read `Dockerfile` — the image CI builds through
+  `script/cibuild` — and a second `prettier --check .` could be added there with
+  the suite staying green. The walk now also starts at
+  `.gitea/workflows/check.yml` and follows its `run:` steps, so the graph under
+  test is the one CI executes rather than the one someone assumed it executes.
+  The lockfile assertion was a substring check against the whole of
+  `script/bootstrap`, which has two install sites and so reported the branch the
+  containers never take; the two branches are now resolved separately and every
+  `yarn install` in each is required to be `--frozen-lockfile`. Prettier is
+  counted per occurrence instead of per line, so two invocations chained with
+  `&&` no longer read as one, and edges are followed on counted lines instead of
+  being skipped. Every way for the walk to reach nothing — an unknown target, an
+  unknown script, a missing file, a node with no commands, an unknown node kind
+  — is a thrown error rather than a quiet zero. Every assertion in the file was
+  mutation-tested individually.
+- 2026-08-10: Stopped `make check` running `prettier --check .` twice. Since
+  linting moved into Docker, the duplicate was one container pass and one host
+  pass of the same check: `script/lint` builds `Dockerfile.lint`, which runs
+  prettier as a build step, and `script/check` then called `script/fmt-check` as
+  well. The host call is gone from `script/check` and from `script/precommit`;
+  the container keeps checking formatting, because a successful
+  `Dockerfile.lint` build is what CI treats as proof of a clean tree, and it is
+  also what still fails the pre-commit hook on a badly formatted tree.
+  `script/fmt-check` survives as a standalone entrypoint, whose verdict cannot
+  drift from the container's. A test walks the invocation graph from each
+  entrypoint — through the Makefile shims, the `script/` calls and the
+  `docker build` — and asserts the prettier count, so the duplication cannot
+  come back unnoticed.
+- 2026-08-10: Moved all linting into Docker. `script/lint` builds a new root
+  `Dockerfile.lint`, which copies the repo into the digest-pinned node image and
+  runs eslint and prettier as build steps, so a successful build is a clean
+  lint; no host lint path remains and `yarn lint` is gone from `package.json`. A
+  fail-closed `LINT_EPOCH` guard stops Docker serving the linter layers from
+  cache, which is how a lint build returns success in under a second having
+  linted nothing. The lint stage inside `Dockerfile` and its `COPY --from=lint`
+  ordering hack are gone: that image now runs `make test` and `make build` only,
+  because `script/check` calls `script/lint` and running it in a container would
+  mean docker inside docker. `script/cibuild` builds the lint image first, then
+  the test and build image.
+- 2026-08-09: Made `make docker` green and policy-conformant. Multi-stage
+  Dockerfile: a lint stage runs `make fmt-check` and `make lint`, and the check
+  stage takes a `COPY --from=lint` dependency on it before running `make check`
+  and `make build`. `CHECK_EPOCH` and a fail-closed guard stop Docker serving
+  those two layers from cache, which is what let a build report success without
+  running the suite. `script/projectname` says `quak`, so the image is tagged
+  `quak`; `script/bootstrap` updates apt lists before installing, so a Debian
+  base works; `.dockerignore` no longer ships the compiled binary, the caches or
+  agent worktrees into the build context, and keeps `.gitignore` in it for
+  prettier.
+- 2026-08-09: Fixed the TypeScript build. `rootDir` is the repo root, so `bin/`
+  compiles alongside `src/` instead of failing with TS6059; output is
+  `dist/src/` and `dist/bin/`, which is where `main`, `types` and `bin.quak` now
+  point. `script/build` verifies the declared entrypoints exist after the
+  compiler runs and makes the CLI executable, the Dockerfile runs `make build`
+  as well as `make check`, and a `quak` script makes the README's
+  `yarn quak <command>` examples work.
+- 2026-08-09: Retry policy: no retry on 4xx (except `408` and `429`),
+  exponential backoff with full jitter on 5xx, transport failures and truncated
+  transfers, under per-attempt deadlines that cover the response body as well as
+  the request. Downloads retry request, stream consumption and decryption as one
+  unit; `postJSON` and `putJSON` are replayed only when the connection was never
+  established.
+- 2026-08-09: Downloads verify the secretstream terminated on `TAG_FINAL` and
+  write output atomically: a truncated body is rejected instead of landing on
+  disk as a short file, and plaintext is staged in a sibling temp file and
+  renamed into place, so a failed download leaves the destination untouched.
+- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
+  shims, README Entrypoints section
+- 2026-06-10: Decrypted collections shared by other users (sealed-box keys);
+  listCollections drops deleted-collection tombstones.
+- 2026-06-10: Login hardening: dual-2FA empty-string fields handled, TOTP
+  preferred when a passkey is also enrolled, interactive input via
+  @inquirer/prompts.
+- 2026-06-10: Replaced sharp with pure JS (jpeg-js + exif-reader); added
+  single-binary bun build and make install.
+- 2026-06-09: Added backup-metadata command (ML data always included, --exif
+  opt-in); rewrote README to match the implementation; added thumbnail helper
+  tests.
+- 2026-05-13: Full CLI surface: login, backup with dedup symlink layout,
+  collections, files, get, get-thumb, thumbnail repair helpers.
+- 2026-05-13: Client OO API with literate usage tests; file download and
+  decryption; all three metadata layers decrypted and persisted; renamed quack
+  to quak.
+- 2026-05-11: SRP login flow (email OTP + TOTP) and ApiClient.
+
 # Future Steps
 
 - Future desktop client, separate repo:

bin/quak.ts

@@ -74,11 +74,7 @@ const context = (): CliContext => ({
     sessionDir: paths.data,
     cacheDir: program.opts<ProgramOptions>().cacheDir,
     loadSession: (path) => loadSession(path, logRequests(log())),
-    login: (opts) =>
-        Client.login({
-            ...opts,
-            apiOptions: { ...opts.apiOptions, ...logRequests(log()) },
-        }),
+    login: (opts) => Client.login({ ...opts, apiOptions: logRequests(log()) }),
     prompt: (message) => input({ message }),
     promptSecret: (message) => password({ message, mask: true }),
 });
@@ -89,12 +85,7 @@ const run = (command: Promise<number>): Promise<void> =>
 program
     .command("login")
     .description("Log in to an Ente account and save the session")
-    .option(
-        "--server <url>",
-        "Address of a self-hosted Ente server's API, saved with the session " +
-            "(default: QUAK_SERVER, then Ente's hosted service)",
-    )
-    .action((opts: { server?: string }) => run(loginCommand(context(), opts)));
+    .action(() => run(loginCommand(context())));
 
 program
     .command("whoami")
@@ -132,6 +123,7 @@ program
     .description("Download and decrypt a single file")
     .argument("<fileID>", "File ID (from `quak files`)")
     .option("--out <path>", "Output file path")
+    .option("--collection <id>", "Accepted for compatibility; ignored")
     .action((fileID: string, opts: { out?: string }) =>
         run(getCommand(context(), fileID, opts)),
     );
@@ -141,6 +133,7 @@ program
     .description("Download and decrypt a thumbnail")
     .argument("<fileID>", "File ID (from `quak files`)")
     .option("--out <path>", "Output file path")
+    .option("--collection <id>", "Accepted for compatibility; ignored")
     .action((fileID: string, opts: { out?: string }) =>
         run(getThumbCommand(context(), fileID, opts)),
     );

package.json

@@ -12,7 +12,7 @@
     "private": true,
     "type": "module",
     "engines": {
-        "node": "^22.13.0 || >=23.5.0"
+        "node": ">=22"
     },
     "main": "./dist/src/index.js",
     "types": "./dist/src/index.d.ts",

script/build-bin

@@ -1,85 +0,0 @@
-#!/bin/sh
-# script/build-bin: run script/build, then bundle the built dist/ into one
-# executable, bin/quak, which reports the version script/build stamped. Our own
-# extension to scripts-to-rule-them-all.
-#
-# bun compiles the binary from a copy of its own executable, so the binary
-# loads the same libraries as the bun that built it. nix's bun loads libraries
-# from the nix store: on Linux its C library, on macOS its ICU. A binary
-# compiled from it loads them too. nix-collect-garbage can delete them, and the
-# binary then no longer starts. This script therefore compiles with bun's own
-# release build, which loads only the system's libraries. It downloads that
-# build and checks it against the hash below before running it.
-set -eu
-
-ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
-
-# bun 1.3.13, 2026-11-11, and the SHA-256 of each release archive below. The
-# x86-64 archives are the baseline builds, which also run on CPUs without AVX2.
-# The musl archives are for Linux systems that use the musl C library, such as
-# Alpine.
-BUN_VERSION="1.3.13"
-BUN_LINUX_X64_BASELINE_SHA256="9d8a24292a7068090205daac0a5a223f5f69736f5287e37bf88d3b4031edc750"
-BUN_LINUX_X64_MUSL_BASELINE_SHA256="88ca7c7ad235b498f549eea2f770f434e9f0f5e9ba95168a2d3a1f235184c394"
-BUN_LINUX_AARCH64_SHA256="70bae41b3908b0a120e1e58c5c8af30e74afae3b8d11b0d3fdd8e787ddfb4b22"
-BUN_LINUX_AARCH64_MUSL_SHA256="5385e978107ce4934298d8d6afe9bfbb898683f6cc23e6753a0da60bc60c5b81"
-BUN_DARWIN_X64_BASELINE_SHA256="a98ba6a480f22fda9b343626b906a4e26aa53618bf85d2bc5928ecf2ba45f0ed"
-BUN_DARWIN_AARCH64_SHA256="5467e3f65dba526b9fea98f0cce04efafc0c63e169733ec27b876a3ad32da190"
-
-# compile_with_downloaded_bun <archive-name> <sha256>
-compile_with_downloaded_bun() {
-    tmp="$(mktemp -d)"
-    trap 'rm -rf "$tmp"' EXIT
-    curl -fsSL -o "$tmp/$1.zip" \
-        "https://github.com/oven-sh/bun/releases/download/bun-v$BUN_VERSION/$1.zip"
-    # macOS always has shasum, but not every macOS has sha256sum.
-    if command -v sha256sum >/dev/null 2>&1; then
-        echo "$2  $tmp/$1.zip" | sha256sum -c -
-    else
-        echo "$2  $tmp/$1.zip" | shasum -a 256 -c -
-    fi
-    unzip -q "$tmp/$1.zip" -d "$tmp"
-    "$tmp/$1/bun" build dist/bin/quak.js --compile --outfile bin/quak
-}
-
-main() {
-    cd "$ROOT"
-    "$ROOT/script/build"
-    platform="$(uname -s)-$(uname -m)"
-    # musl's ldd begins its version output with "musl libc"; glibc's does not.
-    if ldd --version 2>&1 | grep -q '^musl libc'; then
-        platform="$platform-musl"
-    fi
-    case "$platform" in
-        Linux-x86_64)
-            compile_with_downloaded_bun bun-linux-x64-baseline \
-                "$BUN_LINUX_X64_BASELINE_SHA256"
-            ;;
-        Linux-x86_64-musl)
-            compile_with_downloaded_bun bun-linux-x64-musl-baseline \
-                "$BUN_LINUX_X64_MUSL_BASELINE_SHA256"
-            ;;
-        Linux-aarch64)
-            compile_with_downloaded_bun bun-linux-aarch64 \
-                "$BUN_LINUX_AARCH64_SHA256"
-            ;;
-        Linux-aarch64-musl)
-            compile_with_downloaded_bun bun-linux-aarch64-musl \
-                "$BUN_LINUX_AARCH64_MUSL_SHA256"
-            ;;
-        Darwin-x86_64)
-            compile_with_downloaded_bun bun-darwin-x64-baseline \
-                "$BUN_DARWIN_X64_BASELINE_SHA256"
-            ;;
-        Darwin-arm64)
-            compile_with_downloaded_bun bun-darwin-aarch64 \
-                "$BUN_DARWIN_AARCH64_SHA256"
-            ;;
-        *)
-            echo "build-bin: bun has no release build for $(uname -s) on $(uname -m)" >&2
-            exit 1
-            ;;
-    esac
-}
-
-main "$@"

src/api/client.ts

@@ -354,13 +354,7 @@ export class ApiClient {
         });
     }
 
-    // `contentMD5` is the value given to `getUploadURL`. The server signs it
-    // into the URL, and storage refuses a PUT without that `Content-MD5`.
-    async putFile(
-        presignedURL: string,
-        data: Uint8Array,
-        contentMD5: string,
-    ): Promise<void> {
+    async putFile(presignedURL: string, data: Uint8Array): Promise<void> {
         // Idempotent despite being a write: a presigned PUT stores one whole
         // object at one key in one request, so replaying it either overwrites
         // the same bytes or lands them for the first time. There is no partial
@@ -376,7 +370,6 @@ export class ApiClient {
                     headers: {
                         "Content-Type": "application/octet-stream",
                         "Content-Length": String(data.length),
-                        "Content-MD5": contentMD5,
                     },
                     body: data,
                     signal: AbortSignal.timeout(this.requestTimeoutMs),

src/cli-commands.ts

@@ -37,16 +37,9 @@ import {
     thumbnailName,
 } from "./cli-output.js";
 import { type Log, logRefresh } from "./cli-log.js";
-import {
-    cachedFilesByID,
-    freshCollections,
-    freshFiles,
-    freshFile,
-    freshFilesByID,
-} from "./cli-read.js";
+import { freshCollections, freshFiles, freshFile } from "./cli-read.js";
 import { withExtension } from "./filename.js";
 import { runMetadataBackup } from "./metadata-backup.js";
-import type { Collection, EnteFile } from "./model/types.js";
 import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js";
 
 export interface CliContext {
@@ -137,13 +130,12 @@ const readLibraryClient = (client: Client): LibraryClient => ({
 
 // Open a library for a single point command. The aggressive background
 // precache (issue #48) is off, because a one-shot `collections` or `get` must
-// not start downloading the whole account. No refresh starts as the library
-// opens or on a timer. Each command asks for the refresh it needs.
-// `collections` and `files` with a collection ID need none (issues #210 and
-// #217). Nor do `get` and `get-thumb` on an empty cache (issue #222). Nor does
-// `helper fix-missing-thumbnails --file` on an empty cache (issue #223). While
-// a refresh runs, each collection whose files it has fetched is logged, so a
-// large account shows progress rather than nothing.
+// not start downloading the whole account. No refresh starts as it opens or on
+// a timer. Each command asks for the refresh it needs. `collections` needs none
+// (issue #210). `files COLLECTION` needs one only when `refreshCollection()`
+// resolves to `false` (issue #217). While a refresh runs,
+// each collection whose files it has fetched is logged, so a large account
+// shows progress rather than nothing.
 const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
     Library.open({
         client: readLibraryClient(client),
@@ -155,31 +147,8 @@ const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
         onProgress: logRefresh(ctx.log),
     });
 
-// Logs in to the server `--server` or `QUAK_SERVER` names, the flag first, or
-// to Ente's hosted service when neither is set. The saved session keeps the
-// server, so every later command uses it.
-// Whether `text` can be an Ente server's address: an http or https URL, which
-// may have a path, and no query or fragment, which request URLs could not
-// carry.
-const isServerAddress = (text: string): boolean =>
-    URL.canParse(text) &&
-    ["http:", "https:"].includes(new URL(text).protocol) &&
-    !/[?#]/.test(text);
-
-export const loginCommand = async (
-    ctx: CliContext,
-    opts: { server?: string },
-): Promise<number> => {
+export const loginCommand = async (ctx: CliContext): Promise<number> => {
     await init();
-    const server = opts.server ?? process.env.QUAK_SERVER;
-    if (server !== undefined && !isServerAddress(server)) {
-        ctx.log(
-            "fatal",
-            `Invalid server address "${server}": it must be an http or ` +
-                `https URL with no query or fragment`,
-        );
-        return 1;
-    }
     const email = process.env.QUAK_EMAIL ?? (await ctx.prompt("Email"));
     const password =
         process.env.QUAK_PASSWORD ?? (await ctx.promptSecret("Password"));
@@ -191,8 +160,6 @@ export const loginCommand = async (
             password,
             totp: async () => ctx.prompt("TOTP code: "),
             emailOTP: async () => ctx.prompt("Email verification code: "),
-            apiOptions:
-                server === undefined ? undefined : { apiOrigin: server },
         });
 
         saveSession(ctx.sessionDir, client.toJSON());
@@ -317,11 +284,11 @@ export const filesCommand = async (
 
     const lib = await openReadLibrary(ctx, client);
     try {
-        // Ask the server for current state (issue #36 amendment, issue #52).
-        // Each file prints from its own decrypted metadata (raw title,
-        // microsecond creationTime) via cli-output. The files print in the
-        // order `freshFiles` returns them, not the projection's newest-first
-        // order.
+        // Force a server round-trip and list in enumeration order (issue #36
+        // amendment, issue #52). Each file prints from its own decrypted
+        // metadata (raw title, microsecond creationTime) via cli-output, and in
+        // the pre-library CLI's enumeration order, not the projection's
+        // newest-first order.
         const files = await freshFiles(lib, collectionID);
         if (!files) {
             ctx.log("fatal", `Collection ${collectionID} not found`);
@@ -358,13 +325,14 @@ export const getCommand = async (
     try {
         // Force a server round-trip so the file resolves against current state
         // (issue #36 amendment, issue #52).
-        const file = await freshFile(lib, fileID);
-        if (!file) {
+        const resolved = await freshFile(lib, fileID);
+        if (!resolved) {
             ctx.log("fatal", `File ${fileID} not found`);
             return 1;
         }
+        const { photo, file } = resolved;
 
-        const result = await lib.originalOf(file);
+        const result = await photo.original();
         // Default name is the file's own title, as the pre-library CLI used
         // (not the editedName-preferring projection title) (issue #52).
         if (result.videoPath === undefined) {
@@ -415,13 +383,14 @@ export const getThumbCommand = async (
     try {
         // Force a server round-trip so the file resolves against current state
         // (issue #36 amendment, issue #52).
-        const file = await freshFile(lib, fileID);
-        if (!file) {
+        const resolved = await freshFile(lib, fileID);
+        if (!resolved) {
             ctx.log("fatal", `File ${fileID} not found`);
             return 1;
         }
+        const { photo, file } = resolved;
 
-        const result = await lib.thumbnailOf(file);
+        const result = await photo.thumbnail();
         // Default name is thumb_<file's own title>, as the pre-library CLI
         // used (not the projection title) (issue #52).
         const outPath = opts.out ?? thumbnailName(file);
@@ -598,13 +567,10 @@ export const fixMissingThumbnailsCommand = async (
     }
     const lib = await openReadLibrary(ctx, client);
     try {
-        let found: Map<number, { file: EnteFile; collection: Collection }>;
-        if (fileIDs.length > 0) {
-            found = await freshFilesByID(lib, fileIDs);
-        } else {
-            // Refresh first so files added since the cache was written are
-            // checked; a failed refresh throws.
-            await lib.fresh();
+        // Refresh first so files added since the cache was written are found;
+        // a failed refresh throws.
+        await lib.fresh();
+        if (fileIDs.length === 0) {
             ctx.log("info", "Scanning for missing thumbnails...");
             const missing = await listMissingThumbnails(
                 lib,
@@ -617,14 +583,12 @@ export const fixMissingThumbnailsCommand = async (
                 return 0;
             }
             ctx.log("info", `Found ${fileIDs.length} file(s) to fix.`);
-            found = cachedFilesByID(lib, fileIDs);
         }
 
         const results = await fixMissingThumbnails(
             lib,
             client,
             fileIDs,
-            found,
             (msg, level) => ctx.log(level ?? "info", msg),
         );
 

src/cli-log.ts

@@ -1,18 +1,15 @@
 // The CLI's log level, and the one function the commands write stderr lines
 // through.
 //
-// Every line quak logs goes to stderr, at one of six levels, from the
-// quietest: fatal, error, warning, info, debug, trace. A run prints the lines
-// at its own level and at every quieter one. Its level is info unless `-v` or
-// `-q` asks for another (see "What quak prints" in the README). A command that
-// runs and exits non-zero logs a fatal line saying why, so even `-q`, which
-// prints only fatal lines, never leaves such an exit unexplained. An error
+// Every line quak prints on stderr is a log line at one of six levels, from
+// the quietest: fatal, error, warning, info, debug, trace. A run prints the
+// lines at its own level and at every quieter one. Its level is info unless
+// `-v` or `-q` asks for another (see "CLI surface" in the README). Every
+// command that exits non-zero logs a fatal line saying why, so even `-q`, which
+// prints only fatal lines, never leaves a non-zero exit unexplained. An error
 // line reports something that failed, such as a file or an album, and a
 // warning something that did not fail but needs the user's attention. stdout
-// carries a command's output and the prompts of `login`, and is not a log.
-// Two things on stderr are not log lines: the `error:` message for a
-// command line that quak cannot parse, and the usage printed when no command
-// is given.
+// carries only a command's output and is not a log.
 //
 // The library itself writes no lines: its layers report through callbacks, and
 // the CLI logs what they report. `logRequests` gives the API client options

src/cli-read.ts

@@ -2,40 +2,47 @@
 //
 // `collections`, `files`, `get`, and `get-thumb` must answer for
 // the account's state at the moment the command runs, not for whatever the
-// local cache last happened to hold (owner amendment, issue #36). `get`,
-// `get-thumb` and `files` without a collection ID therefore force a server
-// round-trip through `Library.fresh()` and only then read, so a change made
-// elsewhere is visible. `collections` asks the library for the collection list
-// alone through `Library.freshCollections()` (issue #210): a refresh would also
-// fetch the files of every collection, which on a large account takes long and
-// which `collections` does not print. `files` with a collection ID reads that
-// list too, then that one collection's files through `Library.freshFiles()`
-// (issue #217). On an empty cache, `get` and `get-thumb` read that list, then
-// one collection's files at a time, until they find the file (issue #222).
-// `helper fix-missing-thumbnails --file` does the same until it finds every
-// file it names (issue #223).
+// local cache last happened to hold (owner amendment, issue #36). `files`
+// without a collection ID, `get` and `get-thumb` therefore force a server
+// round-trip through `Library.fresh()`, and only then read. A collection, file,
+// or metadata change made elsewhere is then visible. `collections` asks the
+// library for the collection list alone through `Library.freshCollections()`
+// (issue #210). A refresh would also fetch the files of every collection, which
+// `collections` does not print, and on a large account that takes long.
+// `files COLLECTION` reads the collection list the same way, for the same
+// reason. It then brings that one collection up to date through
+// `Library.refreshCollection()` (issue #217), and fetches no other collection's
+// files. Once a refresh has received the account's collections,
+// `refreshCollection()` stores no collection the library does not hold.
+// `files COLLECTION` then forces a refresh instead.
 //
-// `collections` and `files` with a collection ID list in the server's order.
-// `files` without one lists in the library's own enumeration order
-// (`listFiles()`). Neither command uses the `albums`/`photos` projection's
-// newest-first order, which re-sorts the rows. The field values still come
-// from each record's raw metadata via `cli-output.ts`. `files` without a
-// collection ID lists every file in the account once (issue #211): the library
-// keeps a record of a file for each collection that holds it, and only the
-// first collection's record is listed.
+// `collections` lists in the server's order. `files` lists in the library's
+// own enumeration order: `listCollections()`, then `listFiles()` for each.
+// Those are the orders the pre-library CLI printed. The `albums`/`photos`
+// projection would re-sort the rows newest first. After a refresh, the
+// library holds collections in the order the server first sent them. The
+// field values still come from each record's raw metadata via
+// `cli-output.ts`. `files` without a collection ID lists every file in the
+// account once (issue #211).
+// The library keeps a record of a file for each collection that holds it, and
+// only the first collection's record is listed.
 
 import type { Collection, EnteFile } from "./model/types.js";
+import type { Photo, PhotosAPI } from "./library/index.js";
 
 // The slice of `Library` these helpers read. `Library` satisfies it
-// structurally; a test can drive them with a stand-in that records each fresh
-// read and serves records in a known enumeration order.
+// structurally. A test can drive them with a stand-in that records the
+// `fresh()`, `freshCollections()` and `refreshCollection()` calls. The
+// stand-in serves records in a known enumeration order.
 export interface FreshReadLibrary {
     fresh(): Promise<unknown>;
     freshCollections(): Promise<Collection[]>;
-    freshFiles(collection: Collection): Promise<EnteFile[]>;
+    refreshCollection(collection: Collection): Promise<boolean>;
+    getCollection(id: number): Collection | undefined;
     listCollections(): Collection[];
     listFiles(collectionID: number): EnteFile[];
     getFileByID(fileID: number): EnteFile | undefined;
+    photos: Pick<PhotosAPI, "byID">;
 }
 
 // Every live collection, current as of one collections request, in the
@@ -44,11 +51,14 @@ export const freshCollections = (
     lib: FreshReadLibrary,
 ): Promise<Collection[]> => lib.freshCollections();
 
-// The files of one collection, as the server holds them now, in the server's
-// order. `undefined` (not an empty list) when the collection does not exist, so
-// the caller can tell "no such collection" from "an empty collection". Without
-// a collection ID, every file in the account as of a forced refresh, each once,
-// as held by the first collection that holds it.
+// The files of one collection, in enumeration order. `undefined` (not an empty
+// list) when the collection does not exist, so the caller can tell "no such
+// collection" from "an empty collection". They are current as of one
+// collections request and the requests that bring that collection alone up to
+// date. When `refreshCollection()` resolves to `false`, they are current as of
+// a forced refresh instead. Without a collection ID, every file in the account,
+// current as of a forced refresh. Each file is listed once, as held by the
+// first collection that holds it.
 export const freshFiles = async (
     lib: FreshReadLibrary,
     collectionID?: number,
@@ -56,7 +66,12 @@ export const freshFiles = async (
     if (collectionID !== undefined) {
         const collections = await lib.freshCollections();
         const collection = collections.find((c) => c.id === collectionID);
-        return collection ? lib.freshFiles(collection) : undefined;
+        if (!collection) return undefined;
+        if (!(await lib.refreshCollection(collection))) {
+            await lib.fresh();
+            if (!lib.getCollection(collectionID)) return undefined;
+        }
+        return lib.listFiles(collectionID);
     }
     await lib.fresh();
     const listed = new Set<number>();
@@ -71,80 +86,16 @@ export const freshFiles = async (
     return files;
 };
 
-// One file as the server holds it now, or `undefined` when no collection holds
-// it. When the cache holds the account's collections, it refreshes and looks
-// the file up in the cache. On an empty cache, it reads the collection list,
-// then one collection's files at a time, and stops at the first collection
-// that holds the file. That search stores nothing.
+// One file, current as of a forced refresh, resolved to both its content
+// handle (`Photo`, for fetching bytes) and its raw record (`EnteFile`, for the
+// default output name and field values). `undefined` when the file is unknown.
 export const freshFile = async (
     lib: FreshReadLibrary,
     fileID: number,
-): Promise<EnteFile | undefined> => {
-    // Only a refresh stores collections, so a cache holding any holds them all.
-    if (lib.listCollections().length > 0) {
-        await lib.fresh();
-        return lib.getFileByID(fileID);
-    }
-    for (const collection of await lib.freshCollections()) {
-        const files = await lib.freshFiles(collection);
-        const file = files.find((f) => f.id === fileID);
-        if (file) return file;
-    }
-    return undefined;
-};
-
-// The most recently updated collection first, and the higher ID first on a
-// tie. `Library.albums.list()` lists albums in this order.
-const byNewestCollection = (a: Collection, b: Collection): number =>
-    b.updationTime - a.updationTime || b.id - a.id;
-
-// Each file named by `fileIDs` that the cache holds, mapped to its record and
-// the collection it was found in. Collections are searched in
-// `byNewestCollection` order, and a file is found in the first that holds it.
-// So `helper fix-missing-thumbnails` names a file after the same album as
-// `helper list-missing-thumbnails`. It reads the cache alone, so the caller
-// refreshes first.
-export const cachedFilesByID = (
-    lib: FreshReadLibrary,
-    fileIDs: number[],
-): Map<number, { file: EnteFile; collection: Collection }> => {
-    const wanted = new Set(fileIDs);
-    const found = new Map<number, { file: EnteFile; collection: Collection }>();
-    const collections = [...lib.listCollections()].sort(byNewestCollection);
-    for (const collection of collections) {
-        for (const file of lib.listFiles(collection.id)) {
-            if (wanted.has(file.id) && !found.has(file.id)) {
-                found.set(file.id, { file, collection });
-            }
-        }
-    }
-    return found;
-};
-
-// Each file named by `fileIDs` as the server holds it now, mapped to its record
-// and collection as `cachedFilesByID` maps it. When the cache holds the
-// account's collections, it refreshes and looks the files up in the cache. On
-// an empty cache, it reads the collection list, then one collection's files at
-// a time, and stops once it has found every file. That search stores nothing.
-export const freshFilesByID = async (
-    lib: FreshReadLibrary,
-    fileIDs: number[],
-): Promise<Map<number, { file: EnteFile; collection: Collection }>> => {
-    // Only a refresh stores collections, so a cache holding any holds them all.
-    if (lib.listCollections().length > 0) {
-        await lib.fresh();
-        return cachedFilesByID(lib, fileIDs);
-    }
-    const wanted = new Set(fileIDs);
-    const found = new Map<number, { file: EnteFile; collection: Collection }>();
-    const collections = await lib.freshCollections();
-    for (const collection of [...collections].sort(byNewestCollection)) {
-        if (found.size === wanted.size) break;
-        for (const file of await lib.freshFiles(collection)) {
-            if (wanted.has(file.id) && !found.has(file.id)) {
-                found.set(file.id, { file, collection });
-            }
-        }
-    }
-    return found;
+): Promise<{ photo: Photo; file: EnteFile } | undefined> => {
+    await lib.fresh();
+    const photo = lib.photos.byID({ fileID });
+    const file = lib.getFileByID(fileID);
+    if (!photo || !file) return undefined;
+    return { photo, file };
 };

src/client.ts

@@ -40,9 +40,6 @@ export interface ClientSnapshot {
     masterKey: string;
     secretKey: string;
     publicKey: string;
-    // The `apiOrigin` the client was given, so a restored client talks to
-    // the server that issued its token. Absent means the default server.
-    apiOrigin?: string;
 }
 
 // The result of a resumable enumeration. Live decrypted records and deleted
@@ -76,7 +73,6 @@ export class Client {
     private readonly masterKey: Uint8Array;
     private readonly secretKey: Uint8Array;
     private readonly publicKey: Uint8Array;
-    private readonly apiOrigin: string | undefined;
     private loggedOut = false;
 
     private constructor(
@@ -86,7 +82,6 @@ export class Client {
         masterKey: Uint8Array,
         secretKey: Uint8Array,
         publicKey: Uint8Array,
-        apiOrigin: string | undefined,
     ) {
         this.api = api;
         this.email = email;
@@ -94,7 +89,6 @@ export class Client {
         this.masterKey = masterKey;
         this.secretKey = secretKey;
         this.publicKey = publicKey;
-        this.apiOrigin = apiOrigin;
     }
 
     static async login(opts: LoginOptions): Promise<Client> {
@@ -136,15 +130,12 @@ export class Client {
             unwrapped.masterKey,
             unwrapped.secretKey,
             unwrapped.publicKey,
-            opts.apiOptions?.apiOrigin,
         );
     }
 
     // Restore a client from a `toJSON()` snapshot. The snapshot usually comes
     // straight from `JSON.parse` of a file on disk, so every field is checked
-    // before use; a bad one throws an error naming it. The snapshot's
-    // `apiOrigin`, when it has one, is used instead of `apiOptions.apiOrigin`.
-    // Needs `init()` first.
+    // before use; a bad one throws an error naming it. Needs `init()` first.
     static fromJSON(snapshot: unknown, apiOptions?: ApiClientOptions): Client {
         const invalid = (field: string, problem: string): Error =>
             new Error(`Invalid session data: ${field} ${problem}`);
@@ -161,14 +152,6 @@ export class Client {
         if (!Number.isInteger(s.userID)) {
             throw invalid("userID", "must be an integer");
         }
-        if (
-            s.apiOrigin !== undefined &&
-            (typeof s.apiOrigin !== "string" || s.apiOrigin === "")
-        ) {
-            throw invalid("apiOrigin", "must be a non-empty string");
-        }
-        const apiOrigin =
-            (s.apiOrigin as string | undefined) ?? apiOptions?.apiOrigin;
         const key = (field: string): Uint8Array => {
             const value = s[field];
             if (typeof value !== "string") {
@@ -192,7 +175,6 @@ export class Client {
 
         const api = new ApiClient({
             ...apiOptions,
-            apiOrigin,
             authToken: s.token as string,
         });
         return new Client(
@@ -202,7 +184,6 @@ export class Client {
             key("masterKey"),
             key("secretKey"),
             key("publicKey"),
-            apiOrigin,
         );
     }
 
@@ -241,9 +222,6 @@ export class Client {
             masterKey: toBase64(this.masterKey),
             secretKey: toBase64(this.secretKey),
             publicKey: toBase64(this.publicKey),
-            ...(this.apiOrigin === undefined
-                ? {}
-                : { apiOrigin: this.apiOrigin }),
         };
     }
 

src/library/content.ts

@@ -581,34 +581,14 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         fileID: number,
         opts?: ContentOptions,
     ): Promise<ContentResult> {
-        const file = this.fileOrThrow(fileID);
-        return this.get(file, "original", "on-demand", opts?.onProgress);
+        return this.get(fileID, "original", "on-demand", opts?.onProgress);
     }
 
     async thumbnail(
         fileID: number,
         opts?: ContentOptions,
     ): Promise<ContentResult> {
-        const file = this.fileOrThrow(fileID);
-        return this.get(file, "thumbnail", "on-demand", opts?.onProgress);
-    }
-
-    // The original of `file`, fetched and cached as `original` does. `file` is
-    // used as given, so `getFile` need not know it.
-    async originalOf(
-        file: EnteFile,
-        opts?: ContentOptions,
-    ): Promise<ContentResult> {
-        return this.get(file, "original", "on-demand", opts?.onProgress);
-    }
-
-    // The thumbnail of `file`, fetched and cached as `thumbnail` does. `file`
-    // is used as given, so `getFile` need not know it.
-    async thumbnailOf(
-        file: EnteFile,
-        opts?: ContentOptions,
-    ): Promise<ContentResult> {
-        return this.get(file, "thumbnail", "on-demand", opts?.onProgress);
+        return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
     }
 
     // Put the original at the save path of `file` under the download directory
@@ -635,7 +615,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         destination: string,
     ): Promise<ContentResult> {
         const result = await this.acquire(
-            this.fileOrThrow(fileID),
+            fileID,
             "original",
             "on-demand",
             undefined,
@@ -658,7 +638,8 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         fileID: number,
         destination: string,
     ): Promise<{ path: string; videoPath?: string }> {
-        const file = this.fileOrThrow(fileID);
+        const file = this.getFile(fileID);
+        if (!file) throw new Error(`content cache: unknown file ${fileID}`);
         return this.pools.content.run(
             async () => {
                 const fetched = await this.fetchInto(
@@ -740,8 +721,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         onProgress: ((event: EnsureEvent) => void) | undefined,
     ): Promise<EnsureResult> {
         try {
-            const file = this.fileOrThrow(fileID);
-            const result = await this.acquire(file, kind, priority, signal);
+            const result = await this.acquire(fileID, kind, priority, signal);
             const status = result.cached ? "skipped" : "done";
             onProgress?.({ fileID, status, path: result.path });
             return { fileID, path: result.path };
@@ -756,16 +736,8 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         }
     }
 
-    // The record `getFile` gives for `fileID`. It throws for a file `getFile`
-    // does not know.
-    private fileOrThrow(fileID: number): EnteFile {
-        const file = this.getFile(fileID);
-        if (!file) throw new Error(`content cache: unknown file ${fileID}`);
-        return file;
-    }
-
     private async get(
-        file: EnteFile,
+        fileID: number,
         kind: Kind,
         priority: Priority,
         onProgress: ((event: ContentEvent) => void) | undefined,
@@ -773,7 +745,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         const onByte: ProgressCallback | undefined = onProgress
             ? (bytesDone) => onProgress({ status: "downloading", bytesDone })
             : undefined;
-        const result = await this.acquire(file, kind, priority, undefined, {
+        const result = await this.acquire(fileID, kind, priority, undefined, {
             onByte,
         });
         onProgress?.(
@@ -795,7 +767,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
     // photo's original is present only with its video, and is returned with
     // it.
     private async acquire(
-        file: EnteFile,
+        fileID: number,
         kind: Kind,
         priority: Priority,
         signal: AbortSignal | undefined,
@@ -806,7 +778,8 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         videoPath?: string;
         cached: boolean;
     }> {
-        const fileID = file.id;
+        const file = this.getFile(fileID);
+        if (!file) throw new Error(`content cache: unknown file ${fileID}`);
         const isLivePhoto =
             kind === "original" && file.metadata.fileType === "livePhoto";
 

src/library/index.ts

@@ -1,29 +1,37 @@
 // The library surface over the local cache.
 //
 // `Library.open()` loads the on-disk metadata store (issue #41), then starts
-// the refresh loop. When the cache loaded empty it awaits the first refresh,
-// so the library never opens onto an empty store it could have filled; when an
-// existing copy loaded, that first refresh runs in the background and `open()`
-// returns as soon as the cached data is ready to serve — a slow or unreachable
-// server no longer stalls opening. A background timer then refreshes every
-// `refreshIntervalSeconds`. Every default read is answered from RAM — no
-// default read touches the network. There is deliberately no `sync()`, no
-// `refresh()`, no `serverReachable` flag, and no "before each read" mode
-// (design #36).
+// the refresh loop. When the stored cursor is 0, `open()` awaits the first
+// refresh. No refresh has then stored any collection: the cache is empty, or
+// holds only what `refreshCollection()` stored. The library must not serve
+// that as the whole account. Otherwise that first refresh runs in the
+// background. `open()` returns as soon as the cached data is ready to serve,
+// so a slow or unreachable server does not stall opening. A background timer
+// then takes a turn `refreshIntervalSeconds` after its last turn ends. Each
+// turn starts a refresh, unless a refresh or `refreshCollection()` is already
+// running. Every default read is answered from RAM — no default read touches
+// the network. There is deliberately no `sync()`, no `refresh()`, no
+// `serverReachable` flag, and no "before each read" mode (design #36).
 //
-// `fresh()` is the one exception (issue #75, an owner amendment to #36): it
-// forces a refresh, awaits it, and only then hands back the read namespaces, so
-// a caller that needs server-current data can ask for it. Concurrent `fresh()`
-// calls coalesce onto one in-flight refresh, and a refresh that fails rejects
-// the caller (the default reads stay silent and serve the last good copy). The
-// default methods and the background loop are unchanged.
+// `fresh()` is an exception (issue #75, an owner amendment to #36). It awaits
+// a refresh, and only then hands back the read namespaces. A caller that needs
+// server-current data uses it. Concurrent `fresh()` calls coalesce onto one
+// in-flight refresh. A refresh that fails rejects the caller. The default
+// reads stay silent and serve the last good copy. The default methods and the
+// background loop are unchanged.
 //
 // `freshCollections()` is the fresh read of the collection list alone (issue
-// #210): one collections request, no collection's files, and nothing stored,
-// for a caller such as `quak collections` that needs no more. `freshFiles()`
-// is the fresh read of one collection's files, and stores nothing either
-// (issue #217). A caller that asks for each refresh it needs can also open the
-// library with `refreshOnOpen: false`, so that `open()` starts none.
+// #210). It sends one collections request and stores nothing.
+// `quak collections` uses it. `refreshCollection()` brings one collection
+// from that list up to date in the store (issue #217). It fetches that
+// collection's files alone, as a refresh would. `quak files COLLECTION` uses
+// it. It never runs at the same time as a refresh. Once the stored cursor is
+// past 0, it only updates a collection the store holds, and only a refresh
+// adds one. A refresh from a stored cursor of 0 replaces all the store holds.
+// So after a refresh, the store holds collections in the order the server
+// first sent them. A caller that asks for
+// each refresh it needs can also open the library with `refreshOnOpen: false`,
+// so that `open()` starts none.
 //
 // A refresh reports through `onProgress` each collection whose files it has
 // fetched, so a large account, whose first refresh fetches the files of every
@@ -32,10 +40,12 @@
 // A refresh stages all of its network work first and only mutates the store
 // once every fetch has succeeded. A refresh that fails partway therefore never
 // becomes visible to reads: the last good snapshot stays in place, and the
-// failure surfaces through `onProgress` and `status()` instead. A commit that
-// mutates RAM but then fails to persist keeps `status().lastError` set and the
-// store marked unsaved until a later save actually lands, so a stuck disk is
-// never masked by a subsequent empty refresh.
+// failure surfaces through `onProgress` and `status()` instead. A save can
+// still fail after the store has changed in RAM. The store then stays marked
+// unsaved until a later save lands. Until then, every `commit()` retries the
+// save. A refresh whose save fails sets `status().lastError`, which stays set
+// until a later refresh succeeds. `refreshCollection()` never sets or clears
+// it. So a stuck disk is never hidden by a later refresh that changes nothing.
 
 import { rm } from "node:fs/promises";
 import { join, resolve } from "node:path";
@@ -66,8 +76,6 @@ import {
     ContentCache,
     savePath,
     storedAtSavePath,
-    type ContentOptions,
-    type ContentResult,
     type ContentSource,
     type ThumbnailsAPI,
     type EnsureOptions,
@@ -162,21 +170,23 @@ export interface LibraryClient {
         payloads: Map<number, MLData>;
         errors: Map<number, unknown>;
     }>;
-    // The byte source for the on-disk content cache. Optional so a mock client
-    // that only serves metadata still satisfies the interface; when absent (and
-    // no explicit `contentSource` is passed to `open`) the content cache is
-    // disabled and `Photo.original`/`thumbnail` and `thumbnails.ensure` throw.
+    // The byte source for the on-disk content cache. Optional, so a mock
+    // client that only serves metadata still satisfies the interface. Without
+    // it, and with no `contentSource` passed to `open`, the content cache is
+    // disabled. `Photo.original`/`thumbnail` and `thumbnails.ensure` then
+    // throw.
     contentSource?(): ContentSource;
 }
 
-// A progress event for one unit of background work. A metadata "refresh" or an
-// ML "fetchMLData" pass each fire "started" before their network work and then
-// exactly one of "done" or "failed"; "failed" carries the error message and
-// an ML "done" reports how many payloads it stored and, in `mlDataErrors`,
-// each file whose ML data did not decrypt, gunzip or parse, with the reason.
-// In between, a refresh fires "progress" after each collection whose files it
-// fetched: `fetched` is how many collections it has fetched the files of so
-// far, `total` how many it will. The precache fills
+// A progress event for one unit of background work. A metadata "refresh" and
+// an ML "fetchMLData" pass each fire "started" before their network work. Each
+// then fires exactly one of "done" or "failed". A "failed" event carries the
+// error message. An ML "done" reports in `fetched` how many payloads it
+// stored. Its `mlDataErrors` lists each file whose ML data did not decrypt,
+// gunzip or parse, with the reason. In between, a refresh fires "progress"
+// after each collection whose files it fetched. Its `fetched` is how many
+// collections it has fetched the files of so far, and `total` how many it
+// will. The precache fills
 // ("precacheThumbnails"/"precacheOriginals", #48) fire "started"/"done" around
 // each sweep that has work, "done" reporting the count newly cached.
 export interface RefreshEvent {
@@ -204,9 +214,10 @@ export interface LibraryOptions {
     // directory at open. The content cache treats an original already stored
     // at its save path as present.
     downloadDirectory?: string;
-    // How long after one refresh ends the background timer starts the next.
-    // `Infinity` starts none: the library then refreshes only as it opens and
-    // when `fresh()` or `backup()` asks.
+    // How long after its last turn ends the background timer takes the next.
+    // A turn starts a refresh unless a refresh or `refreshCollection()` is
+    // running. With `Infinity` the timer takes no turn: the library then
+    // refreshes only as it opens and when `fresh()` or `backup()` asks.
     refreshIntervalSeconds?: number;
     // Whether `open()` starts a refresh. Defaults to true. With false, `open()`
     // returns at once onto what the cache holds, empty or not, and a refresh
@@ -245,8 +256,9 @@ export interface LibraryStatus {
     // Wall-clock ms of the last refresh that succeeded, or undefined if none
     // has yet.
     lastRefreshAt?: number;
-    // The message from the most recent refresh, set only while that refresh
-    // failed; cleared by the next success.
+    // The error of the most recent refresh, set only when it failed. The next
+    // refresh that succeeds clears it. `refreshCollection()` never sets or
+    // clears it.
     lastError?: string;
     // Wall-clock ms of the last ML fetch pass that succeeded, or undefined if
     // none has yet (or ML fetching is disabled).
@@ -309,12 +321,17 @@ export class Library {
     private readonly precache?: Precache;
 
     private timer?: ReturnType<typeof setTimeout>;
-    // The in-flight refresh cycle, or undefined when none runs. One slot serves
-    // both paths: the background loop skips when it is set, and a fresh read
-    // (issue #75) coalesces onto it or starts one. The promise carries the
-    // cycle's real outcome (it rejects on failure); the background loop ignores
-    // that, a fresh read propagates it.
+    // The in-flight refresh cycle or `refreshCollection()`, or undefined when
+    // neither runs. One slot holds both, so they never run at the same time.
+    // The background loop skips while it is set. A fresh read (issue #75)
+    // coalesces onto a running refresh. It waits for a running
+    // `refreshCollection()` to end, and then joins or starts a refresh. The
+    // promise carries the real outcome: it rejects on failure. The background
+    // loop ignores a failure. A fresh read passes on the failure of the
+    // refresh it joins or starts, and `refreshCollection()` passes on its own.
     private cycle?: Promise<void>;
+    // Whether `cycle` is a refresh cycle rather than a `refreshCollection()`.
+    private cycleIsRefresh = false;
     // Guards the ML fetch pass so a slow backfill never runs twice at once; a
     // refresh whose pass is still running kicks nothing new. Holds the running
     // pass, so `close()` and `backup()` can wait for it. It rejects when the
@@ -327,19 +344,20 @@ export class Library {
     private lastMLError?: string;
     // For each file whose ML data the last ML fetch pass that succeeded got
     // but could not decrypt, gunzip or parse, why not, by fileID. A pass
-    // gathers its own and puts them here when it ends without failing, so
-    // `status()` and `backup()` never see the list of a pass that is still
-    // running or that failed. Kept in RAM only, never in the cache.
+    // gathers its own and puts them here only when it ends without failing.
+    // So `status()` and `backup()` never see the list of a pass still running
+    // or one that failed. Kept in RAM only, never in the cache.
     private mlDataErrors = new Map<number, unknown>();
-    // The plain-record projection as of the last refresh, and the GUI change
-    // subscribers. A refresh that alters the projection notifies each with the
-    // delta; `lastRecords` is kept current every refresh so a subscriber that
-    // joins later diffs against the state its own `snapshot()` already returned.
+    // The plain-record projection as of the last store change, and the GUI
+    // change subscribers. A refresh or `refreshCollection()` that alters the
+    // projection notifies each subscriber with the delta. `lastRecords`
+    // advances on each `commit()` that changes the store. A subscriber that
+    // joins later then diffs against the state its own `snapshot()` returned.
     private readonly subscribers = new Set<(change: LibraryChange) => void>();
     private lastRecords: DerivedRecords;
     // RAM holds changes disk has not yet accepted (an earlier save failed).
-    // Cleared only when a save actually succeeds; keeps the store trying to
-    // persist and the failure visible in `status()` until then.
+    // Cleared only when a save succeeds. Until then, every `commit()` retries
+    // the save.
     private unsaved = false;
 
     private constructor(args: {
@@ -369,7 +387,7 @@ export class Library {
         this.lastRecords = this.deriveNow();
 
         // The read namespaces derive fresh from the store on each call, so they
-        // always reflect the latest refresh.
+        // always reflect what it holds now.
         const derive = (): DerivedRecords => this.deriveNow();
         const root = this.downloadDirectory;
         const saves: SavePathLookup = {
@@ -396,14 +414,16 @@ export class Library {
         this.mldata = makeMLDataAPI(() => this.mlStore);
     }
 
-    // Load the cache and start the refresh loop. With an empty cache the first
-    // refresh is awaited, so `open()` resolves onto populated data whenever the
-    // server is reachable; that awaited refresh may still fail, and the library
-    // then opens empty with the failure recorded in `status()`. With an
-    // existing cache the first refresh runs in the background and `open()`
-    // returns as soon as the cached data is ready — an unreachable server does
-    // not block opening. With `refreshOnOpen: false` there is no first
-    // refresh: `open()` returns onto the cache as it is.
+    // Load the cache and start the refresh loop. When the stored cursor is 0,
+    // the first refresh is awaited. Such a cache is empty, or holds only what
+    // `refreshCollection()` stored. `open()` then resolves onto the whole
+    // account whenever the server is reachable. If that refresh fails, the
+    // library opens onto what the cache holds, with the failure recorded in
+    // `status()`. With any other cache the first refresh runs in the
+    // background, and `open()` returns as soon as the cached data is ready.
+    // An unreachable server then does not block opening. With
+    // `refreshOnOpen: false` there is no first refresh: `open()` returns onto
+    // the cache as it is.
     static async open(opts: LibraryOptions): Promise<Library> {
         const { userID } = opts.client.whoami();
         const cacheDirectory =
@@ -419,8 +439,8 @@ export class Library {
         let store = await MetadataStore.load(metadataPath);
         // A cache directory given explicitly can hold another account's cache.
         // Its records and cursor are not this account's, so delete it and the
-        // ML data beside it and start empty. A user ID of 0 means the cache
-        // was never refreshed and so holds nothing to discard.
+        // ML data beside it and start empty. A user ID of 0 means nothing was
+        // ever stored there, so there is nothing to discard.
         if (store.userID !== 0 && store.userID !== userID) {
             await rm(metadataPath, { force: true });
             await rm(join(cacheDirectory, "mldata"), {
@@ -443,10 +463,11 @@ export class Library {
             ? await MLDataStore.open(join(cacheDirectory, "mldata"))
             : undefined;
 
-        // Build the content cache from an explicit source or the client's own,
-        // and take its record of what is already cached (and reap orphan temp
-        // files) before the first projection, so cached paths are present from
-        // the start and the first refresh raises no spurious path-change diff.
+        // Build the content cache from an explicit source or the client's own.
+        // Take its record of what is already cached, and reap orphan temp
+        // files, before the first projection. Cached paths are then present
+        // from the start, and the first `commit()` raises no spurious
+        // path-change diff.
         const source = opts.contentSource ?? opts.client.contentSource?.();
         let cache: ContentCache | undefined;
         let precache: Precache | undefined;
@@ -500,22 +521,26 @@ export class Library {
             precache,
         });
 
-        // Start filling from whatever the loaded store already holds; each
-        // refresh below re-kicks with the new files (and retries any that
-        // failed). An empty store starts empty here and fills after its first
-        // refresh.
+        // Start filling from whatever the loaded store already holds. Each
+        // later `commit()` re-kicks the fills with its new files, and retries
+        // any that failed. An empty store starts empty here, and fills once a
+        // `commit()` stores files.
         precache?.start();
 
         if (opts.refreshOnOpen === false) {
             // The caller asks for each refresh it needs.
             lib.scheduleNext();
-        } else if (store.loadedFromDisk) {
-            // An existing copy already answers reads; refresh in the background
-            // and start the interval once that first cycle settles.
+        } else if (store.collectionsSinceTime > 0) {
+            // The stored cursor has moved, so a refresh has stored the
+            // account's collections and the cache's copy answers reads.
+            // Refresh in the background, and start the interval once that
+            // first cycle settles.
             void lib.runRefresh().then(() => lib.scheduleNext());
         } else {
-            // Nothing was cached: wait for the first refresh to fill the store
-            // (or fail) rather than resolve onto an empty library.
+            // The stored cursor is still 0, so no refresh has stored any
+            // collection: the cache is empty, or holds only what
+            // `refreshCollection()` stored. Wait for the first refresh to fill
+            // the store (or fail) rather than resolve onto part of the account.
             await lib.runRefresh();
             lib.scheduleNext();
         }
@@ -553,9 +578,9 @@ export class Library {
         return snapshotFrom(this.deriveNow(), Date.now());
     }
 
-    // Deliver a `LibraryChange` whenever a refresh alters the projection. A
-    // refresh that changes nothing delivers nothing. The returned handle's
-    // `unsubscribe` stops delivery.
+    // Deliver a `LibraryChange` whenever a refresh or `refreshCollection()`
+    // alters the projection. One that changes nothing delivers nothing. The
+    // returned handle's `unsubscribe` stops delivery.
     subscribe(args: { onChange: (change: LibraryChange) => void }): {
         unsubscribe: () => void;
     } {
@@ -598,12 +623,13 @@ export class Library {
         };
     }
 
-    // Fresh reads (issue #75, owner amendment to design #36). Force a refresh,
-    // wait for it to complete and persist, then hand back the same
-    // `albums`/`photos`/`timeline` namespaces — now guaranteed to reflect a
-    // completed server round-trip. Concurrent calls coalesce onto one refresh;
-    // a refresh that fails rejects here, where the default namespaces would
-    // instead stay silent and serve the last good copy.
+    // Fresh reads (issue #75, owner amendment to design #36). Wait for a
+    // refresh to complete and persist, then hand back the same
+    // `albums`/`photos`/`timeline` namespaces. They then reflect a completed
+    // server round-trip. A running `refreshCollection()` is waited for first.
+    // Concurrent calls coalesce onto one refresh. A refresh that fails rejects
+    // here, where the default namespaces would stay silent and serve the last
+    // good copy.
     async fresh(): Promise<FreshReads> {
         await this.refreshNow();
         return {
@@ -616,10 +642,10 @@ export class Library {
     // The fresh read of the collection list alone (issue #210): every live
     // collection that decrypts, in the server's order, from one collections
     // request. Unlike a refresh, it fetches no collection's files, which on a
-    // large account take long. Nothing is stored: a refresh fetches a
-    // collection's files only when the collection is new or has changed since
-    // the stored copy, so a collection stored here without its files would
-    // never have them fetched.
+    // large account take long. It stores nothing. A refresh from a stored
+    // cursor past 0 fetches a collection's files only when the collection is
+    // new or has changed since the stored copy. So does `refreshCollection()`.
+    // Neither would fetch the files of a collection stored here without them.
     async freshCollections(): Promise<Collection[]> {
         if (this.closed) throw new Error("the library is closed");
         const { collections } = await this.client.collectionsSince({
@@ -628,67 +654,44 @@ export class Library {
         return collections;
     }
 
-    // The fresh read of one collection's files (issue #217): every live file
-    // in it that decrypts, in the server's order. `collection` is a record
-    // `freshCollections()` returned, whose key decrypts the files. It fetches
-    // no other collection's files. Like `freshCollections()`, it stores
-    // nothing. It returns each file once, as the latest copy the server sent,
-    // where the first copy was.
-    async freshFiles(collection: Collection): Promise<EnteFile[]> {
+    // Bring one collection up to date in the store (issue #217), given its
+    // record from `freshCollections()`. It does for that collection what a
+    // refresh does for a changed one. It fetches no other collection's files.
+    // It resolves to whether the store then holds the collection.
+    //
+    // It adds a collection the store does not hold only while the stored
+    // cursor is 0, before any refresh has received a collection. After that,
+    // it stores nothing for such a collection and resolves to false. Only a
+    // refresh may then add it, after the collections the store holds.
+    //
+    // It runs in the same slot as a refresh, so the two never run at the same
+    // time. It first waits for a running refresh or `refreshCollection()` to
+    // end. While its own work runs, a fresh read or `backup()` waits for it,
+    // and the background loop skips its turn.
+    async refreshCollection(collection: Collection): Promise<boolean> {
+        while (this.cycle) await this.cycle.catch(() => {});
         if (this.closed) throw new Error("the library is closed");
-        const { files } = await this.client.filesSince({
-            collectionID: collection.id,
-            collectionKey: collection.key,
-            sinceTime: 0,
+        let held = false;
+        await this.startCycle(false, async () => {
+            held = await this.refreshCollectionOnce(collection);
         });
-        const byID = new Map<number, EnteFile>();
-        for (const file of files) byID.set(file.id, file);
-        return [...byID.values()];
-    }
-
-    // The original of `file`, fetched and cached as `Photo.original()` does.
-    // `file` is used as given, so the store need not hold it. On an empty
-    // cache, `quak get` passes a file that `freshFiles()` returned.
-    async originalOf(
-        file: EnteFile,
-        opts?: ContentOptions,
-    ): Promise<ContentResult> {
-        if (!this.cache) {
-            throw new Error(
-                "originalOf requires a library opened with a content cache",
-            );
-        }
-        return this.cache.originalOf(file, opts);
-    }
-
-    // The thumbnail of `file`, fetched and cached as `Photo.thumbnail()` does.
-    // `file` is used as given, so the store need not hold it. On an empty
-    // cache, `quak get-thumb` passes a file that `freshFiles()` returned.
-    async thumbnailOf(
-        file: EnteFile,
-        opts?: ContentOptions,
-    ): Promise<ContentResult> {
-        if (!this.cache) {
-            throw new Error(
-                "thumbnailOf requires a library opened with a content cache",
-            );
-        }
-        return this.cache.thumbnailOf(file, opts);
+        return held;
     }
 
     // Back up every in-scope file to `opts.downloadDirectory`, or else the
     // library's, each original at its save path, with a durable failure
-    // ledger (issue #51). Takes the lock in that directory first, unless the
-    // caller passes the lock it took as `opts.lock`, failing at once while
-    // another backup of it runs, and before it starts another file once the
-    // lock is lost (see `lockBackupDirectory`). Waits for a
-    // completed refresh, as `fresh()` does, joining one already running, and
-    // rejects before touching any file but the lock when it fails. Then puts
-    // pending originals at their save paths as `Photo.download()` does (and
-    // optional thumbnails) through the content cache and pools, waits for an
-    // ML data fetch, and rebuilds the derived symlink/JSON views from the
-    // model, each file's JSON with its ML data, beside an `account.json` with
-    // the account's email and user ID.
+    // ledger (issue #51). It first takes the lock in that directory, unless
+    // the caller passes the lock it took as `opts.lock`. Taking it fails at
+    // once while another backup of that directory runs. Once the lock is lost,
+    // the backup fails before it starts another file (see
+    // `lockBackupDirectory`). It waits for a completed refresh as `fresh()`
+    // does, joining one already running. When that refresh fails, it rejects
+    // before touching any file but the lock. It then puts pending originals
+    // at their save paths as `Photo.download()` does, through the content
+    // cache and pools. With `includeThumbnails` it also fetches thumbnails. It
+    // waits for an ML data fetch. It rebuilds the derived symlink/JSON views
+    // from the model, each file's JSON with its ML data. An `account.json`
+    // beside them holds the account's email and user ID.
     // Throws before any network work when no content cache backs the
     // originals it must fetch.
     backup(opts?: BackupOptions): Promise<BackupResult> {
@@ -725,12 +728,15 @@ export class Library {
         );
     }
 
-    // Stop the background timer. Idempotent. An in-flight refresh is left to
-    // finish; it will not schedule another cycle once closed. The returned
-    // promise resolves once that refresh (including its cache write), the ML
-    // fetch pass and the precache fetches already running have all finished,
-    // so a caller can then remove the cache directory. A refresh failure is
-    // reported through `status()`, not thrown here.
+    // Stop the background timer. Idempotent. A refresh or `refreshCollection()`
+    // in flight is left to finish, and nothing starts after it. The returned
+    // promise resolves once all of these have finished:
+    //  - that refresh or `refreshCollection()`, with its cache write;
+    //  - the ML fetch pass;
+    //  - the precache fetches already running.
+    // A caller can then remove the cache directory. A refresh failure is
+    // reported through `status()`, and a `refreshCollection()` failure to its
+    // own caller. Neither is thrown here.
     async close(): Promise<void> {
         this.closed = true;
         const precacheClosed = this.precache?.close();
@@ -753,33 +759,41 @@ export class Library {
         this.timer.unref?.();
     }
 
-    // The background loop's refresh: run a cycle unless one is already in flight
-    // (or the library is closed), and never let a failure escape — the
-    // background path reports errors through `status()`/`onProgress`, it does
-    // not throw. Resolves once the cycle it started (or skipped past) settles.
+    // The background loop's refresh: run a cycle unless a refresh or
+    // `refreshCollection()` is in flight (or the library is closed), and never
+    // let a failure escape. The background path reports errors through
+    // `status()`/`onProgress`; it does not throw. It resolves once the cycle
+    // it started settles, or at once when it starts none.
     private runRefresh(): Promise<void> {
         if (this.closed || this.cycle) return Promise.resolve();
-        return this.startCycle().catch(() => {});
+        return this.startCycle(true, () => this.refreshCycle()).catch(() => {});
     }
 
-    // A fresh read's refresh (issue #75): force a cycle and await it, rejecting
-    // if it fails. Concurrent fresh reads coalesce onto the one in-flight cycle
-    // — the background loop's included — so they never fan out into redundant
-    // server round-trips.
-    private refreshNow(): Promise<void> {
-        if (this.closed) {
-            return Promise.reject(new Error("the library is closed"));
+    // A fresh read's refresh (issue #75): join the running refresh or start
+    // one, and await it, rejecting if it fails. Concurrent fresh reads
+    // coalesce onto the one in-flight cycle — the background loop's included —
+    // so they never fan out into redundant server round-trips. A running
+    // `refreshCollection()` is waited for first.
+    private async refreshNow(): Promise<void> {
+        while (this.cycle && !this.cycleIsRefresh) {
+            await this.cycle.catch(() => {});
         }
-        return this.cycle ?? this.startCycle();
+        if (this.closed) throw new Error("the library is closed");
+        return this.cycle ?? this.startCycle(true, () => this.refreshCycle());
     }
 
-    // Start one refresh cycle and record it as the in-flight cycle so every
-    // caller coalesces onto it. The returned promise carries the cycle's real
-    // outcome; each caller attaches the handling its own path needs, and the
-    // slot is cleared once the cycle settles.
-    private startCycle(): Promise<void> {
-        const cycle = this.refreshCycle();
+    // Start `work` and record it in the in-flight slot, so every other caller
+    // sees it. `isRefresh` says whether it is a refresh cycle, which a fresh
+    // read can coalesce onto. The returned promise carries the real outcome.
+    // Each caller attaches the handling its own path needs, and the slot is
+    // cleared once the work settles.
+    private startCycle(
+        isRefresh: boolean,
+        work: () => Promise<void>,
+    ): Promise<void> {
+        const cycle = work();
         this.cycle = cycle;
+        this.cycleIsRefresh = isRefresh;
         void cycle.then(
             () => {
                 if (this.cycle === cycle) this.cycle = undefined;
@@ -802,9 +816,9 @@ export class Library {
             this.lastError = undefined;
             this.emit({ operation: "refresh", status: "done" });
             // Backfill ML data for the files this refresh knows about. It runs
-            // outside the refresh's success/failure so a fetch or disk problem
-            // there never marks the metadata refresh failed, and it is not
-            // awaited so it never stalls the refresh interval. Its failure is
+            // outside the refresh's success or failure, so a fetch or disk
+            // problem there never marks the metadata refresh failed. It is not
+            // awaited, so it never stalls the refresh interval. Its failure is
             // reported through `status()` and `onProgress`.
             void this.fetchMLDataNow().catch(() => {});
         } catch (err) {
@@ -817,11 +831,14 @@ export class Library {
 
     // Fetch every change since the stored cursor, then commit. All network
     // reads happen before any store mutation, so a fetch that throws leaves the
-    // store untouched and the previous snapshot intact. A collection or file
-    // that does not decrypt is left out instead, as if deleted, with the reason
-    // kept in the store: the cursor moves past it, so no later refresh sees it
-    // again until the server's copy changes.
+    // store untouched and the previous snapshot intact.
+    //
+    // From a stored cursor of 0, the server lists every live collection. The
+    // refresh then replaces all the store holds, and counts every collection
+    // as new. Only `refreshCollection()` can have stored anything before it.
+    // So the store ends as it would had `refreshCollection()` never run.
     private async refreshOnce(): Promise<void> {
+        const replace = this.store.collectionsSinceTime === 0;
         const page = await this.client.collectionsSince({
             sinceTime: this.store.collectionsSinceTime,
         });
@@ -829,16 +846,19 @@ export class Library {
         // Stage per-collection file diffs. A collection's files are
         // re-enumerated only when its updationTime has advanced past the cached
         // copy; an unchanged album's file list cannot have changed. New
-        // collections enumerate from the beginning of time. Each collection
+        // collections enumerate from the beginning of time. When the refresh
+        // replaces the store, every collection counts as new. Each collection
         // whose files are fetched is reported, with how many there are to
         // fetch.
+        const storedCopy = (id: number): Collection | undefined =>
+            replace ? undefined : this.store.getCollection(id);
         const changedCollections = page.collections.filter((collection) => {
-            const known = this.store.getCollection(collection.id);
+            const known = storedCopy(collection.id);
             return !known || collection.updationTime > known.updationTime;
         });
         const filePages: { collectionID: number; page: FilesPage }[] = [];
         for (const collection of changedCollections) {
-            const known = this.store.getCollection(collection.id);
+            const known = storedCopy(collection.id);
             const filePage = await this.client.filesSince({
                 collectionID: collection.id,
                 collectionKey: collection.key,
@@ -853,8 +873,56 @@ export class Library {
             });
         }
 
-        // Network work done; commit to the store and persist only if something
-        // actually changed.
+        await this.commit(page, filePages, replace);
+    }
+
+    // `refreshCollection()`'s work, run in the slot: fetch the files of one
+    // collection changed since the stored copy, then commit. It resolves to
+    // whether the store then holds the collection.
+    private async refreshCollectionOnce(
+        collection: Collection,
+    ): Promise<boolean> {
+        const known = this.store.getCollection(collection.id);
+        // A refresh has received the account's collections, and this one is
+        // not stored. It is new since then, or a refresh dropped it. Leave it
+        // to the next refresh, which adds new collections after the stored
+        // ones, in the server's order.
+        if (!known && this.store.collectionsSinceTime > 0) return false;
+        // The stored copy is as new or newer, so there is nothing to fetch.
+        if (known && collection.updationTime <= known.updationTime) return true;
+        const filePage = await this.client.filesSince({
+            collectionID: collection.id,
+            collectionKey: collection.key,
+            sinceTime: known ? known.updationTime : 0,
+        });
+        // The stored cursor stays where it is, so the next refresh still
+        // fetches the files of every other collection that changed.
+        await this.commit(
+            {
+                collections: [collection],
+                deleted: [],
+                cursor: this.store.collectionsSinceTime,
+            },
+            [{ collectionID: collection.id, page: filePage }],
+            false,
+        );
+        return true;
+    }
+
+    // Commit a collections page and the file pages fetched for its collections
+    // to the store. With `replace`, the page lists every live collection, so
+    // the commit first deletes every stored collection. A collection or file
+    // that does not decrypt is left out, as if deleted. The reason is kept in
+    // the store. Until the server's copy changes, only a refresh from a stored
+    // cursor of 0 fetches it again. Last, save the store when it holds changes
+    // not yet saved.
+    private async commit(
+        page: CollectionsPage,
+        filePages: { collectionID: number; page: FilesPage }[],
+        replace: boolean,
+    ): Promise<void> {
+        // The network work is done, so change the store now. `changed` says
+        // whether this commit changed anything.
         let changed = false;
 
         if (this.store.userID !== this.userID) {
@@ -862,6 +930,13 @@ export class Library {
             changed = true;
         }
 
+        if (replace) {
+            for (const { id } of this.store.listCollections()) {
+                this.store.deleteCollection(id);
+                changed = true;
+            }
+        }
+
         for (const id of page.deleted) {
             if (this.store.getCollection(id)) {
                 this.store.deleteCollection(id);
@@ -915,10 +990,10 @@ export class Library {
         if (changed) this.unsaved = true;
 
         // Reproject and notify subscribers of the delta. This tracks RAM (what
-        // reads see), so it fires whether or not the save below succeeds; a
-        // save failure surfaces separately through `status().lastError`.
-        // `lastRecords` advances every changed refresh so the next diff is
-        // against current state.
+        // reads see), so it fires whether or not the save below succeeds. A
+        // failed save rejects the caller, and a refresh also records it in
+        // `status().lastError`. `lastRecords` advances on every commit that
+        // changes something, so the next diff is against current state.
         if (changed) {
             const next = this.deriveNow();
             if (this.subscribers.size > 0) {
@@ -929,16 +1004,16 @@ export class Library {
             // Recompute the fill orders and pinned set against the new library.
             this.precache?.update(next);
         }
-        // Re-kick the fills every cycle: a finished sweep starts afresh to pick
-        // up new files and retry any that failed, and a running one is left be.
+        // Re-kick the fills on every commit. A finished sweep starts afresh to
+        // pick up new files and retry any that failed. A running one is left
+        // be.
         this.precache?.start();
 
         // Persist whenever RAM holds changes disk has not accepted — including
         // changes an earlier cycle staged whose save failed. `unsaved` clears
-        // only once a save lands, so a save failure both stays visible through
-        // `status().lastError` (the throw below records it) and keeps being
-        // retried, instead of a later empty refresh silently clearing it while
-        // the on-disk cache is still behind RAM.
+        // only once a save lands, so a later commit retries a failed save even
+        // when it changes nothing. A failed save rejects the caller, and a
+        // refresh also records it in `status().lastError`.
         if (this.unsaved) {
             await this.store.save();
             this.unsaved = false;
@@ -954,13 +1029,14 @@ export class Library {
         return this.mlFetch;
     }
 
-    // One ML fetch pass: fetch, decrypt and store the ML data for every file
-    // the store knows about that is not cached (or whose `updationTime` has
-    // advanced), through the metadata pool, and update the CLIP index. For a
-    // file whose ML data does not decrypt, gunzip or parse, it gathers the
-    // reason instead, and puts the reasons it gathered in `mlDataErrors` when
-    // it ends without failing. A failure leaves `mlDataErrors` as it was and
-    // is reported through `status()` and `onProgress`, then thrown.
+    // One ML fetch pass. It fetches, decrypts and stores the ML data of every
+    // file the store knows about that is not cached, or whose `updationTime`
+    // has advanced. It fetches through the metadata pool, and updates the CLIP
+    // index. For a file whose ML data does not decrypt, gunzip or parse, it
+    // gathers the reason instead. It puts the reasons it gathered in
+    // `mlDataErrors` when it ends without failing. A failure leaves
+    // `mlDataErrors` as it was and is reported through `status()` and
+    // `onProgress`, then thrown.
     private async runMLFetch(): Promise<void> {
         const mldata = this.mlStore;
         // Bind so the call keeps the client as its receiver when invoked
@@ -1028,9 +1104,9 @@ export class Library {
         }));
     }
 
-    // The distinct files the store holds, one entry per fileID (a file in
-    // several collections shares its ML data), each carrying the key and the
-    // newest `updationTime` seen across its memberships.
+    // The distinct files the store holds, one entry per fileID, since a file
+    // in several collections shares its ML data. Each entry carries the key
+    // and the newest `updationTime` seen across the file's memberships.
     private uniqueFiles(): {
         id: number;
         key: Uint8Array;

src/library/precache.ts

@@ -122,7 +122,7 @@ export class Precache {
     }
 
     // Recompute the fill orders and the pinned set from the current projection.
-    // Called at open and after every refresh that changes the library.
+    // Called at open and after every change to the store.
     update(records: DerivedRecords): void {
         const photos = [...records.photos.values()].sort(
             (a, b) => b.takenAt - a.takenAt || b.fileID - a.fileID,

src/library/records.ts

@@ -71,13 +71,14 @@ export interface LibrarySnapshot {
     takenAt: number;
 }
 
+// What one refresh or `refreshCollection()` changed in the records.
 export interface LibraryChange {
-    // Full records for albums/photos added or changed by the refresh.
+    // Full records for albums/photos added or changed.
     albumsChanged: AlbumRecord[];
     photosChanged: PhotoRecord[];
     fileIDsRemoved: number[];
     albumIDsRemoved: number[];
-    // Wall-clock milliseconds of the refresh that produced this change.
+    // Wall-clock milliseconds when the store changed.
     refreshedAt: number;
 }
 

src/library/store.ts

@@ -3,7 +3,7 @@
 // The store keeps one `metadata.json` file holding the account's server
 // state: the user id, a schema version, the cursor for the incremental
 // collections listing, the decrypted collection and file records, and the
-// collections and files the refresh could not decrypt. The whole file is read
+// collections and files that did not decrypt. The whole file is read
 // into RAM on load and rewritten as a whole on save; there is no partial
 // update and no lock file. A separate refresh unit populates the store from
 // the server — this module only stores what it is given.
@@ -34,9 +34,11 @@ const FILE_MODE = 0o600;
 type StoredCollection = Omit<Collection, "key"> & { key: string };
 type StoredFile = Omit<EnteFile, "key"> & { key: string };
 
-// A collection, or with `fileID` a file in it, that the last refresh to fetch
-// it could not decrypt or parse, and why. It is kept until a later refresh
-// fetches it again, because a refresh fetches only what changed.
+// A collection, or with `fileID` a file in it, that did not decrypt or parse,
+// and why. A refresh or `refreshCollection()` records it on fetching such a
+// copy. A later fetch drops it on getting a copy that decrypts, or on learning
+// that the server deleted it. Each fetch asks only for what changed, so the
+// store keeps the record between fetches.
 export interface Undecryptable {
     collectionID: number;
     fileID?: number;
@@ -70,12 +72,6 @@ export class MetadataStore {
     userID = 0;
     collectionsSinceTime: Microseconds = 0;
 
-    // True when `load` populated this store from a valid existing file; false
-    // on a first run or a missing/corrupt/wrong-version file that loaded empty.
-    // `Library.open` reads it to decide whether the first refresh may run in
-    // the background (an existing copy already serves reads) or must be awaited.
-    loadedFromDisk = false;
-
     private readonly collections = new Map<number, Collection>();
     private readonly files = new Map<string, EnteFile>();
     private undecryptable: Undecryptable[] = [];
@@ -102,7 +98,6 @@ export class MetadataStore {
             if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) {
                 return store;
             }
-            store.loadedFromDisk = true;
             store.userID = parsed.userID ?? 0;
             store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0;
             for (const stored of parsed.collections ?? []) {
@@ -229,8 +224,14 @@ export class MetadataStore {
         this.files.delete(fileKey(collectionID, fileID));
     }
 
+    // Sorted by collection ID, then file ID, so the order does not depend on
+    // when each record was fetched.
     listUndecryptable(): Undecryptable[] {
-        return [...this.undecryptable];
+        return [...this.undecryptable].sort(
+            (a, b) =>
+                a.collectionID - b.collectionID ||
+                (a.fileID ?? 0) - (b.fileID ?? 0),
+        );
     }
 
     // Replaces any earlier record of the same collection or file.

src/thumbnails.ts

@@ -5,7 +5,7 @@ import type { Client } from "./client.js";
 import type { Library } from "./library/index.js";
 import { ApiError } from "./api/client.js";
 import { encryptBlob, toBase64 } from "./crypto/index.js";
-import type { Collection, EnteFile } from "./model/types.js";
+import type { EnteFile } from "./model/types.js";
 
 // The server refuses a thumbnail larger than the one it already records for the
 // file (`thumbnail.size`, the encrypted size), so these encodings are tried
@@ -226,18 +226,16 @@ const encryptThumbnailWithin = (
     return undefined;
 };
 
-// Regenerate and upload a thumbnail for each requested file. `found` holds the
-// record of each requested file that a collection holds, and the collection it
-// was found in. Each original is read through the library's content cache, by
-// `lib.originalOf(file)`. The new thumbnail is a JPEG, encrypted under the
-// file's own key. It is uploaded, then registered with the server. Each file is
+// Regenerate and upload a thumbnail for each requested file. Originals are read
+// through the library's content cache (`photo.original()`); the generated
+// thumbnail is JPEG-encoded, encrypted under the file's own key, and registered
+// with the server — the encrypt-and-upload path is unchanged. Each file is
 // resolved to one outcome (fixed / skipped / failed) and a failure on one file
 // never stops the others.
 export const fixMissingThumbnails = async (
     lib: Library,
     client: Client,
     fileIDs: number[],
-    found: Map<number, { file: EnteFile; collection: Collection }>,
     onProgress?: ProgressCallback,
 ): Promise<ThumbnailFixResult[]> => {
     const log = onProgress ?? (() => {});
@@ -245,8 +243,31 @@ export const fixMissingThumbnails = async (
     const api = client.getApiClient();
     const { userID } = client.whoami();
 
+    // Resolve each requested fileID to its file record and owning album by
+    // enumerating the library, each file taken from the first album that holds
+    // it. The raw `EnteFile` carries the per-file key the thumbnail is
+    // encrypted under, which the projected records deliberately do not.
+    const wanted = new Set(fileIDs);
+    const fileMap = new Map<
+        number,
+        { file: EnteFile; collectionName: string }
+    >();
+    for (const album of lib.albums.list()) {
+        for (const photo of album.photos.list()) {
+            if (!wanted.has(photo.fileID) || fileMap.has(photo.fileID))
+                continue;
+            const file = lib.getFile(album.collectionID, photo.fileID);
+            if (file) {
+                fileMap.set(photo.fileID, {
+                    file,
+                    collectionName: album.name,
+                });
+            }
+        }
+    }
+
     for (const fileID of fileIDs) {
-        const entry = found.get(fileID);
+        const entry = fileMap.get(fileID);
         if (!entry) {
             results.push({
                 fileID,
@@ -258,8 +279,7 @@ export const fixMissingThumbnails = async (
             continue;
         }
 
-        const { file } = entry;
-        const collectionName = entry.collection.name;
+        const { file, collectionName } = entry;
         const title = file.metadata.title;
 
         const skipReason = reasonToSkip(file, userID);
@@ -277,10 +297,15 @@ export const fixMissingThumbnails = async (
         const maxSize = file.thumbnail.size!;
 
         try {
+            const photo = lib.photos.byID({ fileID });
+            if (!photo) {
+                throw new Error("file not present in the library cache");
+            }
+
             log(
                 `[${collectionName}] Downloading ${title} for thumbnail generation...`,
             );
-            const { path } = await lib.originalOf(file);
+            const { path } = await photo.original();
             const fileBytes = new Uint8Array(readFileSync(path));
 
             if (!isJpeg(fileBytes)) {
@@ -325,7 +350,7 @@ export const fixMissingThumbnails = async (
                 ciphertext.length,
                 md5,
             );
-            await api.putFile(url, ciphertext, md5);
+            await api.putFile(url, ciphertext);
             await api.updateThumbnail(file.id, objectKey, toBase64(header));
 
             log(`[${collectionName}] Thumbnail uploaded for ${title}`);

test/api/client.test.ts

@@ -780,7 +780,7 @@ describe("ApiClient timeouts", () => {
 
         await client.getJSON("/a");
         await client.postJSON("/b", {});
-        await client.putFile("https://s3.example/x", new Uint8Array([1]), "m");
+        await client.putFile("https://s3.example/x", new Uint8Array([1]));
         await client.putJSON("/c", {});
 
         for (const call of calls) {
@@ -825,8 +825,7 @@ describe("ApiClient timeouts", () => {
             [
                 "putFile",
                 () => new Response(null, { status: 200 }),
-                (c) =>
-                    c.putFile("https://s3.example/x", new Uint8Array([1]), "m"),
+                (c) => c.putFile("https://s3.example/x", new Uint8Array([1])),
             ],
             [
                 "getFileStream",
@@ -1050,7 +1049,7 @@ describe("ApiClient error typing", () => {
         const client = new ApiClient({ fetch, retry: noWait });
 
         const err: unknown = await client
-            .putFile("https://s3.example/obj", new Uint8Array([1, 2]), "m")
+            .putFile("https://s3.example/obj", new Uint8Array([1, 2]))
             .catch((e: unknown) => e);
 
         expect(err).toBeInstanceOf(ApiError);
@@ -1068,11 +1067,7 @@ describe("ApiClient error typing", () => {
         );
         const client = new ApiClient({ fetch, retry: noWait });
 
-        await client.putFile(
-            "https://s3.example/obj",
-            new Uint8Array([1, 2]),
-            "m",
-        );
+        await client.putFile("https://s3.example/obj", new Uint8Array([1, 2]));
         expect(calls).toHaveLength(2);
     });
 
@@ -1282,7 +1277,6 @@ describe("ApiClient onRequest", () => {
         await client.putFile(
             "https://s3.example/obj?X-Amz-Signature=abc",
             new Uint8Array([1, 2]),
-            "m",
         );
 
         expect(requests).toEqual([

test/api/upload.test.ts

@@ -80,20 +80,13 @@ describe("ApiClient.putFile", () => {
         const client = new ApiClient({ fetch, authToken: "secret-tok" });
         const data = new Uint8Array([1, 2, 3, 4, 5]);
 
-        await client.putFile(
-            "https://s3.example.com/presigned?sig=abc",
-            data,
-            "md5base64==",
-        );
+        await client.putFile("https://s3.example.com/presigned?sig=abc", data);
 
         expect(calls[0]!.url).toBe("https://s3.example.com/presigned?sig=abc");
         expect(calls[0]!.init?.method).toBe("PUT");
         const headers = new Headers(calls[0]!.init?.headers as HeadersInit);
         expect(headers.get("Content-Type")).toBe("application/octet-stream");
         expect(headers.get("Content-Length")).toBe("5");
-        // The server signs the MD5 into the URL; storage refuses the PUT
-        // without it.
-        expect(headers.get("Content-MD5")).toBe("md5base64==");
         // Must NOT leak auth headers to S3
         expect(headers.has("X-Auth-Token")).toBe(false);
         expect(headers.has("X-Client-Package")).toBe(false);
@@ -106,11 +99,7 @@ describe("ApiClient.putFile", () => {
         const client = new ApiClient({ fetch });
 
         await expect(
-            client.putFile(
-                "https://s3.example.com/bad",
-                new Uint8Array(10),
-                "md5base64==",
-            ),
+            client.putFile("https://s3.example.com/bad", new Uint8Array(10)),
         ).rejects.toThrow(/403/);
     });
 });

test/cli/backup-end-to-end.test.ts

@@ -418,7 +418,7 @@ describe("the first backup of the account", () => {
         expect(downloads.sort()).toEqual([100, 200, 300]);
     });
 
-    // The tree under "The backup directory" in the README. Each original is at
+    // The tree "Backup layout" in the README describes. Each original is at
     // its save path, `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, with
     // the extension of its title; the live photo is its image and its video,
     // each with the extension it had inside the ZIP, and a `.livephoto.json`

test/cli/bin.test.ts

@@ -1,19 +1,12 @@
 /**
  * Tests for the retry options `bin/quak.ts` loads the saved session with
  * (`quak backup` gets the unattended ones, every other command the default),
- * each with the callbacks that log its requests and retries, for `quak files`
- * taking its collection ID as an optional argument, for `quak get` and
- * `quak get-thumb` refusing `--collection`, and for `quak login` taking
- * `--server`.
+ * each with the callbacks that log its requests and retries, and for
+ * `quak files` taking its collection ID as an optional argument.
  *
  * Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
  * replaced by one that records the options it is given and reports no session,
  * so the command stops before it makes any request.
- *
- * The first load of the sources `bin/quak.ts` imports compiles them, and the
- * first `init()` starts libsodium. On a loaded host that takes longer than a
- * test's timeout, so both happen here, before the tests run. Each test's
- * `vi.resetModules()` then only runs the compiled sources again.
  */
 
 import { mkdtempSync, rmSync } from "node:fs";
@@ -21,13 +14,8 @@ import { tmpdir } from "node:os";
 import { join } from "node:path";
 import { afterAll, afterEach, describe, expect, it, vi } from "vitest";
 import type { ApiClientOptions } from "../../src/api/client.js";
-import "../../src/cli-commands.js";
-import { init } from "../../src/crypto/index.js";
-import "../../src/index.js";
 import { UNATTENDED_RETRY_OPTIONS } from "../../src/retry.js";
 
-await init();
-
 const loaded = vi.hoisted(() => [] as (ApiClientOptions | undefined)[]);
 
 vi.mock("../../src/cli-session.js", () => ({
@@ -107,24 +95,3 @@ describe("bin/quak.ts files", () => {
         expect(await quak("files", "7")).toMatch(/^Not logged in/);
     });
 });
-
-describe("bin/quak.ts get and get-thumb", () => {
-    it.each(["get", "get-thumb"])(
-        "%s refuses --collection as an unknown option",
-        async (command) => {
-            expect(await quak(command, "1", "--collection", "2")).toMatch(
-                /^error: unknown option '--collection'/,
-            );
-        },
-    );
-});
-
-// The command refuses an address that is not a URL before it asks for anything
-// or sends any request.
-describe("bin/quak.ts login", () => {
-    it("hands --server to the command", async () => {
-        expect(await quak("login", "--server", "ente.example.com")).toMatch(
-            /^Invalid server address "ente\.example\.com"/,
-        );
-    });
-});

test/cli/commands.test.ts

@@ -124,20 +124,35 @@ const FILES: Record<number, EnteFile[]> = {
 // thumbnail as empty. `withNewFile` adds new.jpg (102) to Vacation, advancing
 // the collection's updationTime as the server does. `beachInWork` adds
 // beach.jpg (100) to Work as well, so that one file is in both albums.
-// `refreshError` makes listing collections fail with that message.
+// `newAlbums` adds the albums A (3) and B (4), in that order, which both hold
+// shared.jpg (300). `refreshError` makes listing collections fail with that
+// message.
 const fakeClient = (
     opts: {
         failID?: number;
         emptyThumbID?: number;
         withNewFile?: boolean;
         beachInWork?: boolean;
+        newAlbums?: boolean;
         refreshError?: string;
     } = {},
 ) => {
-    const collections = opts.withNewFile
+    let collections = opts.withNewFile
         ? [{ ...COLLECTIONS[0], updationTime: 2 }, COLLECTIONS[1]]
         : COLLECTIONS;
     let files = FILES;
+    if (opts.newAlbums) {
+        collections = [
+            ...collections,
+            { ...collection(3, "A"), updationTime: 2 },
+            { ...collection(4, "B"), updationTime: 2 },
+        ];
+        files = {
+            ...files,
+            3: [file(300, 3, "shared.jpg")],
+            4: [file(300, 4, "shared.jpg")],
+        };
+    }
     if (opts.withNewFile) {
         files = {
             ...files,
@@ -194,6 +209,9 @@ const FETCHED_BOTH_ALBUMS =
     "Fetched the files of 1 of 2 collections\n" +
     "Fetched the files of 2 of 2 collections\n";
 
+// The same for an account with one album.
+const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n";
+
 // Collects everything written to it.
 class Output {
     text = "";
@@ -328,7 +346,6 @@ describe("login", () => {
         prompts = [];
         vi.stubEnv("QUAK_EMAIL", "cli@example.com");
         vi.stubEnv("QUAK_PASSWORD", "hunter2");
-        vi.stubEnv("QUAK_SERVER", undefined);
     });
 
     afterEach(() => {
@@ -341,14 +358,12 @@ describe("login", () => {
             calls.push(opts);
             return loggedIn;
         });
-        expect(await loginCommand(ctx, {})).toBe(0);
+        expect(await loginCommand(ctx)).toBe(0);
 
         expect(prompts).toEqual([]);
         expect(calls).toHaveLength(1);
         expect(calls[0]!.email).toBe("cli@example.com");
         expect(calls[0]!.password).toBe("hunter2");
-        // No server named: the client uses Ente's hosted service.
-        expect(calls[0]!.apiOptions).toBeUndefined();
         const path = join(ctx.sessionDir, "session.json");
         expect(stderr.text).toBe(
             "Authenticating...\n" +
@@ -359,7 +374,7 @@ describe("login", () => {
 
     it("saves the session with mode 0600 in a directory with mode 0700", async () => {
         const ctx = loginContext(async () => loggedIn);
-        expect(await loginCommand(ctx, {})).toBe(0);
+        expect(await loginCommand(ctx)).toBe(0);
 
         expect(statSync(ctx.sessionDir).mode & 0o777).toBe(0o700);
         const path = join(ctx.sessionDir, "session.json");
@@ -373,7 +388,7 @@ describe("login", () => {
             code = await opts.totp!();
             return loggedIn;
         });
-        expect(await loginCommand(ctx, {})).toBe(0);
+        expect(await loginCommand(ctx)).toBe(0);
 
         expect(prompts).toEqual(["TOTP code: "]);
         expect(code).toBe("123456");
@@ -383,68 +398,13 @@ describe("login", () => {
         const ctx = loginContext(async () => {
             throw new Error("HTTP 401 from server");
         });
-        expect(await loginCommand(ctx, {})).toBe(1);
+        expect(await loginCommand(ctx)).toBe(1);
 
         expect(stderr.text).toBe(
             "Authenticating...\nLogin failed: HTTP 401 from server\n",
         );
         expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
     });
-
-    it("logs in to the server --server names, ahead of QUAK_SERVER", async () => {
-        vi.stubEnv("QUAK_SERVER", "https://other.example.com");
-        const calls: LoginOptions[] = [];
-        const ctx = loginContext(async (opts) => {
-            calls.push(opts);
-            return loggedIn;
-        });
-        expect(
-            await loginCommand(ctx, { server: "https://ente.example.com" }),
-        ).toBe(0);
-
-        expect(calls[0]!.apiOptions).toEqual({
-            apiOrigin: "https://ente.example.com",
-        });
-    });
-
-    it("logs in to the server QUAK_SERVER names without --server", async () => {
-        vi.stubEnv("QUAK_SERVER", "http://192.0.2.1:8080/ente");
-        const calls: LoginOptions[] = [];
-        const ctx = loginContext(async (opts) => {
-            calls.push(opts);
-            return loggedIn;
-        });
-        expect(await loginCommand(ctx, {})).toBe(0);
-
-        expect(calls[0]!.apiOptions).toEqual({
-            apiOrigin: "http://192.0.2.1:8080/ente",
-        });
-    });
-
-    it.each([
-        "",
-        "ente.example.com",
-        "ftp://ente.example.com",
-        "https://ente.example.com/?region=eu",
-        "https://ente.example.com/#top",
-    ])(
-        "refuses the server address %j before asking for anything",
-        async (server) => {
-            const ctx = loginContext(async () => {
-                throw new Error("login must not be called");
-            });
-            expect(await loginCommand(ctx, { server })).toBe(1);
-
-            expect(prompts).toEqual([]);
-            expect(stderr.text).toBe(
-                `Invalid server address "${server}": it must be an http or ` +
-                    `https URL with no query or fragment\n`,
-            );
-            expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(
-                false,
-            );
-        },
-    );
 });
 
 // These use a real client read from the session file, over a fake API that
@@ -624,34 +584,85 @@ describe("files", () => {
         ]);
         expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS);
     });
+
+    // `files 2` on an empty cache stores Work alone. The refresh that `files`
+    // then waits for must still store Vacation first, as the server lists it.
+    it("without a collection ID lists in the server's order after files 2 on an empty cache", async () => {
+        const ctx = context(fakeClient({ beachInWork: true }));
+        expect(await filesCommand(ctx, "2", {})).toBe(0);
+        stdout.text = "";
+        stderr.text = "";
+
+        expect(await filesCommand(ctx, undefined, { json: true })).toBe(0);
+        expect(
+            JSON.parse(stdout.text).map(
+                (row: { id: number; collectionID: number }) => [
+                    row.id,
+                    row.collectionID,
+                ],
+            ),
+        ).toEqual([
+            [100, 1],
+            [101, 1],
+            [200, 2],
+        ]);
+        expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS);
+    });
+
+    // After a refresh, the server adds A and then B. `files 4` must not store B
+    // ahead of A, so it waits for a refresh, which stores both in that order.
+    it("without a collection ID lists new albums in the server's order after files with the second one's ID", async () => {
+        expect(await filesCommand(context(), undefined, {})).toBe(0);
+        const ctx = context(fakeClient({ newAlbums: true }));
+        stdout.text = "";
+
+        expect(await filesCommand(ctx, "4", {})).toBe(0);
+        expect(stdout.text).toBe("300\timage\tshared.jpg\n");
+        stdout.text = "";
+
+        expect(await filesCommand(ctx, undefined, { json: true })).toBe(0);
+        expect(
+            JSON.parse(stdout.text).map(
+                (row: { id: number; collectionID: number }) => [
+                    row.id,
+                    row.collectionID,
+                ],
+            ),
+        ).toEqual([
+            [100, 1],
+            [101, 1],
+            [200, 2],
+            [300, 3],
+        ]);
+    });
 });
 
 describe("get and get-thumb", () => {
-    it("get finds a file in any album", async () => {
+    it("get finds a file in any album without --collection", async () => {
         const out = join(root, "diagram.png");
         expect(await getCommand(context(), "200", { out })).toBe(0);
         expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
-        expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
+        expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + `7 bytes -> ${out}\n`);
     });
 
-    it("get-thumb finds a file in any album", async () => {
+    it("get-thumb finds a file in any album without --collection", async () => {
         const out = join(root, "thumb.jpg");
         expect(await getThumbCommand(context(), "200", { out })).toBe(0);
         expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
-        expect(stderr.text).toBe(`3 bytes -> ${out}\n`);
+        expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + `3 bytes -> ${out}\n`);
     });
 
     it("get exits 1 when no album has the file", async () => {
         const out = join(root, "x");
         expect(await getCommand(context(), "999", { out })).toBe(1);
-        expect(stderr.text).toBe("File 999 not found\n");
+        expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + "File 999 not found\n");
         expect(existsSync(out)).toBe(false);
     });
 
     it("get-thumb exits 1 when no album has the file", async () => {
         const out = join(root, "x");
         expect(await getThumbCommand(context(), "999", { out })).toBe(1);
-        expect(stderr.text).toBe("File 999 not found\n");
+        expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + "File 999 not found\n");
         expect(existsSync(out)).toBe(false);
     });
 });
@@ -699,7 +710,7 @@ describe("names from an album name or a title over 255 bytes", () => {
         expect(Buffer.byteLength(name)).toBe(255);
         expect(code).toBe(0);
         expect(readdirSync(dir)).toEqual([name]);
-        expect(stderr.text).toBe(`7 bytes -> ${name}\n`);
+        expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `7 bytes -> ${name}\n`);
     });
 
     it("get-thumb cuts the title short so that thumb_ and it fit in 255 bytes, and exits 0", async () => {
@@ -711,7 +722,7 @@ describe("names from an album name or a title over 255 bytes", () => {
         expect(Buffer.byteLength(name)).toBe(255);
         expect(code).toBe(0);
         expect(readdirSync(dir)).toEqual([name]);
-        expect(stderr.text).toBe(`3 bytes -> ${name}\n`);
+        expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `3 bytes -> ${name}\n`);
     });
 
     it("backup-metadata cuts the album name short so that its folder's name fits in 255 bytes, and exits 0", async () => {
@@ -800,7 +811,8 @@ describe("a live photo", () => {
             Buffer.from(VIDEO),
         );
         expect(stderr.text).toBe(
-            `${IMAGE.length} bytes -> IMG_0300.heic\n` +
+            FETCHED_ONE_ALBUM +
+                `${IMAGE.length} bytes -> IMG_0300.heic\n` +
                 `${VIDEO.length} bytes -> IMG_0300.mov\n`,
         );
     });
@@ -854,7 +866,8 @@ describe("a live photo", () => {
         expect(readFileSync(out)).toEqual(Buffer.from(IMAGE));
         expect(readFileSync(video)).toEqual(Buffer.from(VIDEO));
         expect(stderr.text).toBe(
-            `${IMAGE.length} bytes -> ${out}\n${VIDEO.length} bytes -> ${video}\n`,
+            FETCHED_ONE_ALBUM +
+                `${IMAGE.length} bytes -> ${out}\n${VIDEO.length} bytes -> ${video}\n`,
         );
     });
 
@@ -868,7 +881,8 @@ describe("a live photo", () => {
         expect(existsSync(out)).toBe(false);
         expect(existsSync(join(root, "photo.mov"))).toBe(false);
         expect(stderr.text).toBe(
-            `File 300 is a live photo, and its video would also be written to ${out}\n`,
+            FETCHED_ONE_ALBUM +
+                `File 300 is a live photo, and its video would also be written to ${out}\n`,
         );
     });
 
@@ -1411,23 +1425,6 @@ describe("helper fix-missing-thumbnails", () => {
         ).toBe(1);
         expect(stderr.text).toBe(last);
     });
-
-    // beach.jpg (100) is in Vacation (1) and Work (2), both updated at the same
-    // time. The fixer names a file after the first album holding it in the
-    // order of `Library.albums.list()`. On a tie, the album with the higher ID
-    // comes first, so beach.jpg is named after Work.
-    it("names a photo in two albums after the same album on an empty cache as on a cache that holds the account", async () => {
-        const ctx = context(fakeClient({ beachInWork: true }));
-        const opts = { file: ["100"], json: true };
-        const inWork = [{ fileID: 100, collection: "Work", status: "skipped" }];
-        expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
-        expect(JSON.parse(stdout.text)).toMatchObject(inWork);
-
-        expect(await filesCommand(ctx, undefined, {})).toBe(0);
-        stdout.text = "";
-        expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
-        expect(JSON.parse(stdout.text)).toMatchObject(inWork);
-    });
 });
 
 describe("backup-metadata --exif", () => {
@@ -1457,8 +1454,8 @@ describe("backup-metadata --exif", () => {
 });
 
 // Each test first runs `files` so the cache holds the account as it was, then
-// changes the server under it. (`collections` and `files` with a collection ID
-// would not do: they store nothing.)
+// changes the server under it. (`collections` would not do: it reads the
+// collection list alone and stores nothing.)
 describe("backup-metadata and the thumbnail helpers refresh first", () => {
     beforeEach(async () => {
         expect(await filesCommand(context(), undefined, {})).toBe(0);
@@ -1644,83 +1641,22 @@ describe("an ID given on the command line", () => {
     );
 });
 
-// An account with many albums (issues #210, #217 and #222). On an empty cache
-// `collections`, `files` with a collection ID and `get` used to wait for a
-// refresh that fetched and decrypted the files of every album, one album after
-// another. So did `helper fix-missing-thumbnails --file` (issue #223). On such
-// an account they seemed to hang. The account here has 40 albums of one photo
-// each, encrypted as Ente's clients encrypt them, on the fake Ente server of
-// `test/ente-server.ts`. The tests record the path and query of each request
-// quak sends. The session is the one logging in to that server saves.
+// An account with many albums (issue #210). On an empty cache `collections`
+// used to wait for a refresh that fetched and decrypted the files of every
+// album, one album after another. It printed nothing until then, so on such an
+// account it seemed to hang. `files` with an album's ID did the same before it
+// listed that album (issue #217). The account here has 40 albums of one photo
+// each, encrypted as Ente's clients encrypt them. The fake Ente server of
+// `test/ente-server.ts` serves them. A wrapper around it records the path and
+// query of each request quak sends. The session is the one logging in to that
+// server saves.
 describe("an account with many albums", () => {
     const ALBUMS = 40;
     const PASSWORD = "many-albums";
-    const requests: string[] = [];
+    const paths: string[] = [];
     let server: typeof globalThis.fetch;
     let snapshot: ClientSnapshot;
 
-    // What `files` without a collection ID prints on an empty cache: every
-    // album's photo on stdout, and on stderr each album whose files its
-    // refresh fetched.
-    let everyPhoto = "";
-    let progress = "";
-    for (let n = 1; n <= ALBUMS; n++) {
-        everyPhoto += `${n * 100}\timage\tphoto-${n}.jpg\n`;
-        progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`;
-    }
-
-    // The key of each album, by album ID.
-    const albumKeys = new Map<number, Uint8Array>();
-
-    // A photo in album `collectionID` as the server sends it, and its original
-    // encrypted as the server stores it. The original's bytes are the title.
-    // These tests fetch no thumbnail. The server records a thumbnail size, so
-    // `helper fix-missing-thumbnails` downloads the original before it skips
-    // the photo as not a JPEG.
-    const photoRow = (photo: {
-        id: number;
-        collectionID: number;
-        title: string;
-        updationTime: number;
-    }): { row: Record<string, unknown>; original: Uint8Array } => {
-        const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
-        const photoKeyBox = encryptSecretbox(
-            photoKey,
-            albumKeys.get(photo.collectionID)!,
-        );
-        const metadata = encryptStreamBlob(
-            new TextEncoder().encode(
-                JSON.stringify({
-                    title: photo.title,
-                    fileType: 0,
-                    creationTime: 1000,
-                    modificationTime: 1000,
-                }),
-            ),
-            photoKey,
-        );
-        const original = encryptStreamBlob(
-            new TextEncoder().encode(photo.title),
-            photoKey,
-        );
-        const row = {
-            id: photo.id,
-            collectionID: photo.collectionID,
-            ownerID: USER_ID,
-            encryptedKey: toBase64(photoKeyBox.ciphertext),
-            keyDecryptionNonce: toBase64(photoKeyBox.nonce),
-            metadata: {
-                encryptedData: toBase64(metadata.ciphertext),
-                decryptionHeader: toBase64(metadata.header),
-            },
-            file: { decryptionHeader: toBase64(original.header) },
-            thumbnail: { decryptionHeader: "" },
-            info: { thumbSize: 1000 },
-            updationTime: photo.updationTime,
-        };
-        return { row, original: original.ciphertext };
-    };
-
     // Album n is named `Album n` and holds one photo, n * 100, titled
     // `photo-n.jpg`.
     beforeAll(async () => {
@@ -1728,10 +1664,8 @@ describe("an account with many albums", () => {
         const account = await buildAccount(PASSWORD);
         const collections: Record<string, unknown>[] = [];
         const filesByCollection: Record<number, Record<string, unknown>[]> = {};
-        const fileCiphertexts: Record<number, Uint8Array> = {};
         for (let id = 1; id <= ALBUMS; id++) {
             const albumKey = sodium.crypto_secretbox_keygen();
-            albumKeys.set(id, albumKey);
             const albumKeyBox = encryptSecretbox(albumKey, account.masterKey);
             const name = encryptSecretbox(
                 new TextEncoder().encode(`Album ${id}`),
@@ -1747,14 +1681,37 @@ describe("an account with many albums", () => {
                 type: "album",
                 updationTime: 1,
             });
-            const { row, original } = photoRow({
-                id: id * 100,
-                collectionID: id,
-                title: `photo-${id}.jpg`,
-                updationTime: 1,
-            });
-            filesByCollection[id] = [row];
-            fileCiphertexts[id * 100] = original;
+            const photoKey =
+                sodium.crypto_secretstream_xchacha20poly1305_keygen();
+            const photoKeyBox = encryptSecretbox(photoKey, albumKey);
+            const metadata = encryptStreamBlob(
+                new TextEncoder().encode(
+                    JSON.stringify({
+                        title: `photo-${id}.jpg`,
+                        fileType: 0,
+                        creationTime: 1000,
+                        modificationTime: 1000,
+                    }),
+                ),
+                photoKey,
+            );
+            filesByCollection[id] = [
+                {
+                    id: id * 100,
+                    collectionID: id,
+                    ownerID: USER_ID,
+                    encryptedKey: toBase64(photoKeyBox.ciphertext),
+                    keyDecryptionNonce: toBase64(photoKeyBox.nonce),
+                    metadata: {
+                        encryptedData: toBase64(metadata.ciphertext),
+                        decryptionHeader: toBase64(metadata.header),
+                    },
+                    // These tests fetch no original and no thumbnail.
+                    file: { decryptionHeader: "" },
+                    thumbnail: { decryptionHeader: "" },
+                    updationTime: 1,
+                },
+            ];
         }
         const ente = buildMetaFetch({
             verifier: account.verifier,
@@ -1764,7 +1721,7 @@ describe("an account with many albums", () => {
             collections,
             filesByCollection,
             encryptedMLData: {},
-            fileCiphertexts,
+            fileCiphertexts: {},
         });
         server = (async (input: RequestInfo | URL, init?: RequestInit) => {
             const url =
@@ -1774,7 +1731,7 @@ describe("an account with many albums", () => {
                       ? input.href
                       : input.url;
             const { pathname, search } = new URL(url);
-            requests.push(pathname + search);
+            paths.push(pathname + search);
             return ente(input, init);
         }) as typeof globalThis.fetch;
         const client = await Client.login({
@@ -1786,17 +1743,16 @@ describe("an account with many albums", () => {
     });
 
     beforeEach(() => {
-        requests.length = 0;
+        paths.length = 0;
     });
 
-    // A context whose saved session sends its requests to `fetch`, by default
-    // the fake server.
-    const accountContext = (fetch = server): CliContext => {
+    // A context whose saved session sends its requests to the fake server.
+    const accountContext = (): CliContext => {
         const ctx = context();
         saveSession(ctx.sessionDir, snapshot);
         return {
             ...ctx,
-            loadSession: (path) => loadSession(path, { fetch }),
+            loadSession: (path) => loadSession(path, { fetch: server }),
         };
     };
 
@@ -1809,260 +1765,72 @@ describe("an account with many albums", () => {
             albums += `${id}\talbum\tAlbum ${id}\n`;
         expect(stdout.text).toBe(albums);
         expect(stderr.text).toBe("");
-        expect(requests).toEqual(["/collections/v2?sinceTime=0"]);
+        expect(paths).toEqual(["/collections/v2?sinceTime=0"]);
 
-        // It stored nothing that would make a refresh skip an album's files:
-        // `files` after it still lists every album's photo.
+        // It stored nothing that would make `refreshCollection()` skip an
+        // album's files: `files` after it still finds the album's photo.
         stdout.text = "";
-        expect(await filesCommand(ctx, undefined, {})).toBe(0);
-        expect(stdout.text).toBe(everyPhoto);
+        expect(await filesCommand(ctx, "7", {})).toBe(0);
+        expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
     });
 
-    it("files with a collection ID asks for the album list and that album's files alone", async () => {
+    it("files fetches the files of the album it lists and of no other", async () => {
         expect(await filesCommand(accountContext(), "7", {})).toBe(0);
 
         expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
         expect(stderr.text).toBe("");
-        expect(requests).toEqual([
+        expect(paths).toEqual([
             "/collections/v2?sinceTime=0",
             "/collections/v2/diff?collectionID=7&sinceTime=0",
         ]);
     });
 
-    // Here the server sends album 7's files in two pages. Photo 700 changed
-    // between them, so both pages hold a copy of it.
-    it("files with a collection ID lists only the later copy of a photo sent in two pages", async () => {
-        const firstPage = [
-            photoRow({
-                id: 700,
-                collectionID: 7,
-                title: "photo-7.jpg",
-                updationTime: 1,
-            }).row,
-            photoRow({
-                id: 701,
-                collectionID: 7,
-                title: "photo-701.jpg",
-                updationTime: 1,
-            }).row,
-        ];
-        const secondPage = [
-            photoRow({
-                id: 700,
-                collectionID: 7,
-                title: "photo-7-edited.jpg",
-                updationTime: 2,
-            }).row,
-        ];
-        const paged = (async (input: RequestInfo | URL, init?: RequestInit) => {
-            const url = new URL(input instanceof Request ? input.url : input);
-            if (
-                url.pathname !== "/collections/v2/diff" ||
-                url.searchParams.get("collectionID") !== "7"
-            ) {
-                return server(input, init);
-            }
-            const first = url.searchParams.get("sinceTime") === "0";
-            return new Response(
-                JSON.stringify({
-                    diff: first ? firstPage : secondPage,
-                    hasMore: first,
-                }),
-            );
-        }) as typeof globalThis.fetch;
-
-        expect(await filesCommand(accountContext(paged), "7", {})).toBe(0);
-
-        // Photo 700 is listed once, with the later title, before photo 701.
-        expect(stdout.text).toBe(
-            "700\timage\tphoto-7-edited.jpg\n701\timage\tphoto-701.jpg\n",
-        );
-    });
-
-    // The cache stays empty, so a later `files` without a collection ID still
-    // fetches the files of all 40 albums.
-    it("files with a collection ID stores nothing in the cache", async () => {
+    // The first listing stored the album, so the second finds its stored copy
+    // current and asks for no album's files.
+    it("files run again for the same album sends no request for its files", async () => {
         const ctx = accountContext();
         expect(await filesCommand(ctx, "7", {})).toBe(0);
-        expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
-
         stdout.text = "";
-        expect(await filesCommand(ctx, undefined, {})).toBe(0);
-        expect(stdout.text).toBe(everyPhoto);
-        expect(stderr.text).toBe(progress);
+        paths.length = 0;
+
+        expect(await filesCommand(ctx, "7", {})).toBe(0);
+        expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
+        expect(paths).toEqual(["/collections/v2?sinceTime=0"]);
     });
 
-    // Photo 200 is in album 2, so `get` asks for no file of album 3 or later.
-    // The last request downloads the photo's original.
-    it("get on an empty cache asks for the albums' files only until one holds the photo", async () => {
-        const out = join(root, "photo.jpg");
-        expect(await getCommand(accountContext(), "200", { out })).toBe(0);
-
-        expect(readFileSync(out, "utf-8")).toBe("photo-2.jpg");
-        expect(stderr.text).toBe(`11 bytes -> ${out}\n`);
-        expect(requests).toEqual([
-            "/collections/v2?sinceTime=0",
-            "/collections/v2/diff?collectionID=1&sinceTime=0",
-            "/collections/v2/diff?collectionID=2&sinceTime=0",
-            "/?fileID=200",
-        ]);
-        expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
-    });
-
-    // A self-hosted server serves downloads under its own address, at
-    // `/files/download/<id>`, so no request goes to Ente's hosted service.
-    it("login --server saves the server, and a later get sends every request to it", async () => {
-        vi.stubEnv("QUAK_EMAIL", "many@example.com");
-        vi.stubEnv("QUAK_PASSWORD", PASSWORD);
-        vi.stubEnv("QUAK_SERVER", undefined);
-        const hosts = new Set<string>();
-        const fetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
-            hosts.add(
-                new URL(input instanceof Request ? input.url : input).host,
-            );
-            return server(input, init);
-        }) as typeof globalThis.fetch;
-        const ctx: CliContext = {
-            ...context(),
-            login: (opts) =>
-                Client.login({
-                    ...opts,
-                    apiOptions: { ...opts.apiOptions, fetch },
-                }),
-            loadSession: (path) => loadSession(path, { fetch }),
-        };
-        const out = join(root, "photo.jpg");
-        try {
-            const opts = { server: "https://ente.example.com" };
-            expect(await loginCommand(ctx, opts)).toBe(0);
-            requests.length = 0;
-            expect(await getCommand(ctx, "200", { out })).toBe(0);
-        } finally {
-            vi.unstubAllEnvs();
-        }
-
-        const path = join(ctx.sessionDir, "session.json");
-        expect(JSON.parse(readFileSync(path, "utf-8")).apiOrigin).toBe(
-            "https://ente.example.com",
-        );
-        expect(readFileSync(out, "utf-8")).toBe("photo-2.jpg");
-        expect(requests.at(-1)).toBe("/files/download/200");
-        expect([...hosts]).toEqual(["ente.example.com"]);
-    });
-
-    // `files` first fills the cache, so `get` waits for a refresh. No album has
-    // changed since, so the refresh asks for no album's files.
-    it("get on a cache that holds the account asks for no unchanged album's files, also for a photo in no album", async () => {
-        const ctx = accountContext();
-        expect(await filesCommand(ctx, undefined, {})).toBe(0);
-        requests.length = 0;
-        stderr.text = "";
-
-        const out = join(root, "photo.jpg");
-        expect(await getCommand(ctx, "4000", { out })).toBe(0);
-        expect(readFileSync(out, "utf-8")).toBe("photo-40.jpg");
-        expect(stderr.text).toBe(`12 bytes -> ${out}\n`);
-        expect(requests).toEqual([
-            "/collections/v2?sinceTime=1",
-            "/?fileID=4000",
-        ]);
-
-        requests.length = 0;
-        stderr.text = "";
-        expect(await getCommand(ctx, "999", { out })).toBe(1);
-        expect(stderr.text).toBe("File 999 not found\n");
-        expect(requests).toEqual(["/collections/v2?sinceTime=1"]);
-    });
-
-    // The helper searches the most recently updated album first. All 40 were
-    // updated at the same time, and on a tie the higher ID comes first. Photo
-    // 3900 is in album 39, so the helper asks for no file of album 38 or
-    // earlier. The last request downloads the photo's original.
-    it("fix-missing-thumbnails --file on an empty cache asks for the albums' files only until one holds the photo", async () => {
-        const opts = { file: ["3900"], json: true };
-        expect(await fixMissingThumbnailsCommand(accountContext(), opts)).toBe(
-            0,
-        );
-
-        const reason =
-            "unsupported image format (only baseline JPEG can be regenerated)";
-        expect(JSON.parse(stdout.text)).toEqual([
-            {
-                fileID: 3900,
-                title: "photo-39.jpg",
-                collection: "Album 39",
-                status: "skipped",
-                reason,
-            },
-        ]);
-        expect(stderr.text).toBe(
-            "[Album 39] Downloading photo-39.jpg for thumbnail generation...\n" +
-                `[Album 39] Skipping photo-39.jpg: ${reason}\n`,
-        );
-        expect(requests).toEqual([
-            "/collections/v2?sinceTime=0",
-            "/collections/v2/diff?collectionID=40&sinceTime=0",
-            "/collections/v2/diff?collectionID=39&sinceTime=0",
-            "/?fileID=3900",
-        ]);
-        expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
-    });
-
-    // `files` first fills the cache, so the helper waits for a refresh. No
-    // album has changed since, so the refresh asks for no album's files.
-    it("fix-missing-thumbnails --file on a cache that holds the account asks for no unchanged album's files, also for a photo in no album", async () => {
-        const ctx = accountContext();
-        expect(await filesCommand(ctx, undefined, {})).toBe(0);
-        requests.length = 0;
-        stdout.text = "";
-        stderr.text = "";
-
-        const opts = { file: ["3900"], json: true };
-        expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
-        const reason =
-            "unsupported image format (only baseline JPEG can be regenerated)";
-        expect(JSON.parse(stdout.text)).toEqual([
-            {
-                fileID: 3900,
-                title: "photo-39.jpg",
-                collection: "Album 39",
-                status: "skipped",
-                reason,
-            },
-        ]);
-        expect(stderr.text).toBe(
-            "[Album 39] Downloading photo-39.jpg for thumbnail generation...\n" +
-                `[Album 39] Skipping photo-39.jpg: ${reason}\n`,
-        );
-        expect(requests).toEqual([
-            "/collections/v2?sinceTime=1",
-            "/?fileID=3900",
-        ]);
-
-        requests.length = 0;
-        stdout.text = "";
-        const notFound = { file: ["999"], json: true };
-        expect(await fixMissingThumbnailsCommand(ctx, notFound)).toBe(1);
-        expect(JSON.parse(stdout.text)).toMatchObject([
-            { fileID: 999, reason: "file not found in any collection" },
-        ]);
-        expect(requests).toEqual(["/collections/v2?sinceTime=1"]);
-    });
-
-    it("files counts on stderr each album whose files its refresh fetched", async () => {
+    it("files with no album's ID counts on stderr each album whose files its refresh fetched", async () => {
         expect(await filesCommand(accountContext(), undefined, {})).toBe(0);
 
-        expect(stdout.text).toBe(everyPhoto);
+        let photos = "";
+        let progress = "";
+        for (let n = 1; n <= ALBUMS; n++) {
+            photos += `${n * 100}\timage\tphoto-${n}.jpg\n`;
+            progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`;
+        }
+        expect(stdout.text).toBe(photos);
         expect(stderr.text).toBe(progress);
     });
 
     // `--json` shapes stdout only, so the refresh's progress is still counted
     // on stderr.
-    it("files --json counts on stderr each album whose files its refresh fetched", async () => {
-        const opts = { json: true };
-        expect(await filesCommand(accountContext(), undefined, opts)).toBe(0);
+    it("files --json with no album's ID counts on stderr each album whose files its refresh fetched", async () => {
+        expect(
+            await filesCommand(accountContext(), undefined, { json: true }),
+        ).toBe(0);
 
-        expect(JSON.parse(stdout.text)).toHaveLength(ALBUMS);
+        const rows = JSON.parse(stdout.text);
+        expect(rows).toHaveLength(ALBUMS);
+        expect(rows[6]).toEqual({
+            id: 700,
+            title: "photo-7.jpg",
+            fileType: "image",
+            creationTime: 1000,
+            collectionID: 7,
+        });
+        let progress = "";
+        for (let n = 1; n <= ALBUMS; n++) {
+            progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`;
+        }
         expect(stderr.text).toBe(progress);
     });
 });

test/cli/log.test.ts

@@ -234,15 +234,13 @@ describe("-q", () => {
         });
     });
 
-    // No album holds file 999, so fixing it fails. The summary from
-    // `--- Done ---` to the end of the `Failed files:` list is left out. The
-    // line saying why the command failed is the fatal one, and it is printed.
+    // The line saying why the command failed is the fatal one, and it is
+    // printed.
     it("prints only the line a failing command exits with", async () => {
-        const fix = ["helper", "fix-missing-thumbnails", "--file", "999"];
-        expect(await quak("-q", ...fix)).toEqual({
+        expect(await quak("-q", "files", "9")).toEqual({
             code: 1,
             stdout: "",
-            stderr: `quak: 1 file(s) failed; see the "Failed files:" list, or the JSON with --json\n`,
+            stderr: "Collection 9 not found\n",
         });
     });
 
@@ -328,7 +326,7 @@ describe("what the library reports", () => {
         ]);
     });
 
-    // Each operation is named in plain words.
+    // Each operation is named in plain words, as the README names it.
     it("logs a refresh's progress at info, and its start and end at debug", () => {
         const { log, lines } = collect();
         const onProgress = logRefresh(log);

test/cli/read.test.ts

@@ -3,25 +3,25 @@
  * issue #36, issue #52).
  *
  * The `collections`, `files`, `get`, and `get-thumb` commands must answer for
- * current server state, not the local cache. `get`, `get-thumb` and `files`
- * without a collection ID force a `Library.fresh()` round-trip before they
- * read. The stand-in library below serves no cached record until `fresh()` has
- * been awaited, so a helper that read without refreshing would come back empty
- * and fail here. `collections` asks the library for the collection list alone
- * (`freshCollections()`, issue #210). `files` with a collection ID asks for
- * that list, then for that one collection's files (`freshFiles()`, issue
- * #217). Neither may wait for a refresh, which would fetch the files of every
- * collection. On an empty cache, `get` and `get-thumb` ask for that list, then
- * for one collection's files at a time, until they find the file (issue #222).
- * `helper fix-missing-thumbnails --file` does the same until it finds every
- * file it names, reading the most recently updated collection first (issue
- * #223).
+ * current server state, not the local cache. `files` without a collection ID,
+ * `get` and `get-thumb` force a `Library.fresh()` round-trip before they read.
+ * The stand-in library below serves nothing until `fresh()` has been awaited.
+ * A helper that read without refreshing would come back empty and fail here.
+ * `collections` asks the library for the collection list alone
+ * (`freshCollections()`, issue #210). It must not wait for a refresh, which
+ * would fetch the files of every collection. `files COLLECTION` reads that list
+ * too, then brings that one collection up to date (`refreshCollection()`,
+ * issue #217). The stand-in serves a collection's files once `fresh()` has run,
+ * or once `refreshCollection()` has resolved to `true` for that collection.
+ * When `refreshCollection()` resolves to `false`, `files COLLECTION` forces a
+ * refresh instead.
  *
- * `collections` and `files` also list in the order the library gives them, not
+ * `collections` and `files` also list in the library's enumeration order
+ * (`listCollections`/`listFiles`) — the order the pre-library CLI printed — not
  * the albums/photos projection's newest-first order. The fixtures are seeded in
- * an order that a newest-first sort would rearrange, so a regression to the
- * projection order would fail here too. Field values still come from the raw
- * metadata via `cli-output.ts`.
+ * an enumeration order that a newest-first sort would rearrange, so a
+ * regression to the projection order would fail here too. Field values still
+ * come from the raw metadata via `cli-output.ts`.
  */
 
 import { describe, it, expect } from "vitest";
@@ -30,10 +30,10 @@ import {
     freshCollections,
     freshFiles,
     freshFile,
-    freshFilesByID,
     type FreshReadLibrary,
 } from "../../src/cli-read.js";
 import { fileListRow } from "../../src/cli-output.js";
+import type { Photo } from "../../src/library/index.js";
 import type { Collection, EnteFile } from "../../src/model/types.js";
 
 const collection = (id: number, updationTime: number): Collection => ({
@@ -67,15 +67,20 @@ const file = (
     updationTime: creationTime,
 });
 
-// A library that reveals its cached records only after `fresh()` has been
-// awaited, and serves them in the enumeration order it was given.
-// `freshCollections()` and `freshFiles()` answer in that order without a
-// refresh.
+// A library that reveals its records only after `fresh()` has been awaited, and
+// serves them in the enumeration order it was given. `freshCollections()`
+// answers the collection list in that order without a refresh. When
+// `refreshCollection()` resolves to true, it reveals the files of that one
+// collection.
+// `photos.byID` returns a stand-in `Photo` carrying just the fileID the helper
+// passes through.
 class FakeLibrary implements FreshReadLibrary {
     freshCalls = 0;
     freshCollectionsCalls = 0;
-    // The ID of each collection `freshFiles()` was asked for.
-    freshFilesCalls: number[] = [];
+    refreshCollectionIDs: number[] = [];
+    // What `refreshCollection()` resolves to. With false it reveals nothing,
+    // as a library that has had a refresh does for a collection it lacks.
+    refreshCollectionHolds = true;
     private refreshed = false;
 
     constructor(
@@ -94,9 +99,14 @@ class FakeLibrary implements FreshReadLibrary {
         return this.collections;
     }
 
-    async freshFiles(collection: Collection): Promise<EnteFile[]> {
-        this.freshFilesCalls.push(collection.id);
-        return this.files.filter((f) => f.collectionID === collection.id);
+    async refreshCollection(collection: Collection): Promise<boolean> {
+        this.refreshCollectionIDs.push(collection.id);
+        return this.refreshCollectionHolds;
+    }
+
+    getCollection(id: number): Collection | undefined {
+        if (!this.refreshed) return undefined;
+        return this.collections.find((c) => c.id === id);
     }
 
     listCollections(): Collection[] {
@@ -104,7 +114,11 @@ class FakeLibrary implements FreshReadLibrary {
     }
 
     listFiles(collectionID: number): EnteFile[] {
-        return this.refreshed
+        const current =
+            this.refreshed ||
+            (this.refreshCollectionHolds &&
+                this.refreshCollectionIDs.includes(collectionID));
+        return current
             ? this.files.filter((f) => f.collectionID === collectionID)
             : [];
     }
@@ -113,6 +127,14 @@ class FakeLibrary implements FreshReadLibrary {
         if (!this.refreshed) return undefined;
         return this.files.find((f) => f.id === fileID);
     }
+
+    photos = {
+        byID: ({ fileID }: { fileID: number }): Photo | undefined => {
+            if (!this.refreshed) return undefined;
+            if (!this.files.some((f) => f.id === fileID)) return undefined;
+            return { fileID } as unknown as Photo;
+        },
+    };
 }
 
 describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
@@ -136,9 +158,10 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
         expect(rows.map((c) => c.id)).not.toEqual(newestFirst);
     });
 
-    it("freshFiles with a collection ID reads that collection's files alone and keeps their order and raw fields", async () => {
-        // Given in the order 10, 11, 12; creationTimes ascending, so a
-        // newest-first sort would reverse them.
+    it("freshFiles brings the collection alone up to date, lists in enumeration order, keeps raw fields", async () => {
+        // Enumeration order by id 10, 11, 12; creationTimes ascending, so a
+        // newest-first sort would reverse them. Collection 2's file is not
+        // listed.
         const files = [
             file(10, 1, 1_700_000_000_000_000),
             file(11, 1, 1_700_000_000_000_001),
@@ -146,14 +169,14 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
             file(20, 2, 1_700_000_000_000_003),
         ];
         const lib = new FakeLibrary(
-            [collection(1, 100), collection(2, 200)],
+            [collection(1, 100), collection(2, 100)],
             files,
         );
 
         const rows = await freshFiles(lib, 1);
-        expect(lib.freshCalls).toBe(0);
         expect(lib.freshCollectionsCalls).toBe(1);
-        expect(lib.freshFilesCalls).toEqual([1]);
+        expect(lib.refreshCollectionIDs).toEqual([1]);
+        expect(lib.freshCalls).toBe(0);
         expect(rows?.map((f) => f.id)).toEqual([10, 11, 12]);
 
         // Field values come from raw metadata: microsecond creationTime and the
@@ -207,120 +230,45 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
         ]);
     });
 
+    it("freshFiles forces a refresh when refreshCollection resolves to false", async () => {
+        const lib = new FakeLibrary(
+            [collection(1, 100)],
+            [file(10, 1, 1_700_000_000_000_000)],
+        );
+        lib.refreshCollectionHolds = false;
+
+        const rows = await freshFiles(lib, 1);
+        expect(lib.refreshCollectionIDs).toEqual([1]);
+        expect(lib.freshCalls).toBe(1);
+        expect(rows?.map((f) => f.id)).toEqual([10]);
+    });
+
     it("freshFiles returns undefined for an unknown collection", async () => {
         const lib = new FakeLibrary([collection(1, 100)], []);
         const rows = await freshFiles(lib, 999);
+        expect(lib.freshCollectionsCalls).toBe(1);
+        expect(lib.refreshCollectionIDs).toEqual([]);
         expect(lib.freshCalls).toBe(0);
-        expect(lib.freshFilesCalls).toEqual([]);
         expect(rows).toBeUndefined();
     });
 
-    // Three collections, each holding one file.
-    const threeCollections = (): FakeLibrary =>
-        new FakeLibrary(
-            [collection(1, 100), collection(2, 200), collection(3, 300)],
-            [
-                file(10, 1, 1_700_000_000_000_000),
-                file(20, 2, 1_700_000_000_000_001),
-                file(30, 3, 1_700_000_000_000_002),
-            ],
+    it("freshFile refreshes first, then resolves the photo and its raw record", async () => {
+        const f = file(10, 1, 1_700_000_000_000_000);
+        const lib = new FakeLibrary([collection(1, 100)], [f]);
+
+        const resolved = await freshFile(lib, 10);
+        expect(lib.freshCalls).toBe(1);
+        expect(resolved?.photo.fileID).toBe(10);
+        expect(resolved?.file.metadata.title).toBe("file-10.jpg");
+        expect(resolved?.file.metadata.creationTime).toBe(
+            1_700_000_000_000_000,
         );
-
-    it("freshFile on an empty cache reads each collection's files in the server's order until one holds the file", async () => {
-        const lib = threeCollections();
-
-        const found = await freshFile(lib, 20);
-        expect(lib.freshCalls).toBe(0);
-        expect(lib.freshCollectionsCalls).toBe(1);
-        expect(lib.freshFilesCalls).toEqual([1, 2]);
-        expect(found?.metadata.title).toBe("file-20.jpg");
-        expect(found?.metadata.creationTime).toBe(1_700_000_000_000_001);
     });
 
-    // The stand-in's cache holds what the server holds once `fresh()` has run.
-    it("freshFile on a cache that holds the collections refreshes, then looks the file up in the cache", async () => {
-        const lib = threeCollections();
-        await lib.fresh();
-
-        const found = await freshFile(lib, 30);
-        expect(lib.freshCalls).toBe(2);
-        expect(lib.freshCollectionsCalls).toBe(0);
-        expect(lib.freshFilesCalls).toEqual([]);
-        expect(found?.collectionID).toBe(3);
-    });
-
-    it("freshFile on an empty cache returns undefined for an unknown file, after reading every collection's files", async () => {
-        const lib = threeCollections();
-
-        const found = await freshFile(lib, 404);
-        expect(lib.freshCalls).toBe(0);
-        expect(lib.freshFilesCalls).toEqual([1, 2, 3]);
-        expect(found).toBeUndefined();
-    });
-
-    // The server lists collections 1, 2 and 3, and collection 3 was updated
-    // most recently. File 11 is in collections 1 and 2.
-    const fileInTwoCollections = (): FakeLibrary =>
-        new FakeLibrary(
-            [collection(1, 100), collection(2, 200), collection(3, 300)],
-            [
-                file(10, 1, 1_700_000_000_000_000),
-                file(11, 1, 1_700_000_000_000_001),
-                file(11, 2, 1_700_000_000_000_001),
-                file(20, 2, 1_700_000_000_000_002),
-                file(30, 3, 1_700_000_000_000_003),
-            ],
-        );
-
-    it("freshFilesByID on an empty cache reads the most recently updated collection's files first, until it has found every file", async () => {
-        const lib = fileInTwoCollections();
-
-        const found = await freshFilesByID(lib, [11, 20]);
-        expect(lib.freshCalls).toBe(0);
-        expect(lib.freshCollectionsCalls).toBe(1);
-        expect(lib.freshFilesCalls).toEqual([3, 2]);
-        expect(found.get(11)?.collection.id).toBe(2);
-        expect(found.get(11)?.file.collectionID).toBe(2);
-        expect(found.get(20)?.collection.id).toBe(2);
-    });
-
-    // Collection 1 was updated last, though collections 2 and 3 have higher
-    // IDs. File 11 is in collections 1 and 2.
-    it("freshFilesByID on an empty cache reads the most recently updated collection first, though its ID is the lowest", async () => {
-        const lib = new FakeLibrary(
-            [collection(1, 400), collection(2, 200), collection(3, 300)],
-            [
-                file(11, 1, 1_700_000_000_000_001),
-                file(11, 2, 1_700_000_000_000_001),
-            ],
-        );
-
-        const found = await freshFilesByID(lib, [11]);
-        expect(lib.freshFilesCalls).toEqual([1]);
-        expect(found.get(11)?.collection.id).toBe(1);
-        expect(found.get(11)?.file.collectionID).toBe(1);
-    });
-
-    // The stand-in's cache holds what the server holds once `fresh()` has run.
-    it("freshFilesByID on a cache that holds the collections refreshes, then finds each file in the most recently updated collection that holds it", async () => {
-        const lib = fileInTwoCollections();
-        await lib.fresh();
-
-        const found = await freshFilesByID(lib, [11, 404]);
-        expect(lib.freshCalls).toBe(2);
-        expect(lib.freshCollectionsCalls).toBe(0);
-        expect(lib.freshFilesCalls).toEqual([]);
-        expect(found.get(11)?.collection.id).toBe(2);
-        expect(found.get(11)?.file.collectionID).toBe(2);
-        expect(found.has(404)).toBe(false);
-    });
-
-    it("freshFilesByID on an empty cache reads every collection's files when a file is in none of them", async () => {
-        const lib = fileInTwoCollections();
-
-        const found = await freshFilesByID(lib, [10, 404]);
-        expect(lib.freshFilesCalls).toEqual([3, 2, 1]);
-        expect(found.get(10)?.collection.id).toBe(1);
-        expect(found.has(404)).toBe(false);
+    it("freshFile returns undefined for an unknown file", async () => {
+        const lib = new FakeLibrary([collection(1, 100)], []);
+        const resolved = await freshFile(lib, 404);
+        expect(lib.freshCalls).toBe(1);
+        expect(resolved).toBeUndefined();
     });
 });

test/client/session.test.ts

@@ -47,20 +47,6 @@ describe("Client.toJSON", () => {
         expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
     });
 
-    it("round-trips a snapshot that names a server", () => {
-        const snapshot = {
-            ...validSnapshot(),
-            apiOrigin: "https://ente.example.com",
-        };
-        expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
-    });
-
-    it("names the server apiOptions gave fromJSON", () => {
-        const apiOrigin = "https://ente.example.com";
-        const client = Client.fromJSON(validSnapshot(), { apiOrigin });
-        expect(client.toJSON().apiOrigin).toBe(apiOrigin);
-    });
-
     it("throws instead of emitting a snapshot without a token", () => {
         const client = Client.fromJSON(validSnapshot());
         client.getApiClient().clearAuthToken();
@@ -91,8 +77,6 @@ describe("Client.fromJSON", () => {
         ["publicKey", undefined],
         ["publicKey", "not base64!"],
         ["publicKey", shortKey],
-        ["apiOrigin", 7],
-        ["apiOrigin", ""],
     ])("rejects %s = %j, naming the field", (field, value) => {
         const snapshot: Record<string, unknown> = { ...validSnapshot() };
         snapshot[field] = value;
@@ -101,30 +85,6 @@ describe("Client.fromJSON", () => {
         );
     });
 
-    // The token belongs to the snapshot's server, so that server wins over
-    // the one in `apiOptions`.
-    it("sends requests to the snapshot's server", async () => {
-        const urls: string[] = [];
-        const client = Client.fromJSON(
-            { ...validSnapshot(), apiOrigin: "https://ente.example.com/" },
-            {
-                apiOrigin: "https://other.example.com",
-                fetch: async (input) => {
-                    urls.push(String(input));
-                    return new Response(JSON.stringify({ collections: [] }), {
-                        status: 200,
-                        headers: { "content-type": "application/json" },
-                    });
-                },
-            },
-        );
-
-        await client.listCollections();
-        expect(urls).toEqual([
-            "https://ente.example.com/collections/v2?sinceTime=0",
-        ]);
-    });
-
     it.each([null, "a string", 42])("rejects a non-object %j", (value) => {
         expect(() => Client.fromJSON(value)).toThrow(
             /^Invalid session data: not a JSON object/,

test/ente-server.ts

@@ -19,11 +19,8 @@ import { SRP, SrpServer } from "fast-srp-hap";
 import { toBase64, deriveKEK, deriveLoginSubkey } from "../src/crypto/index.js";
 import type { KeyAttributes } from "../src/auth/types.js";
 
-// The account's password is stretched at libsodium's lowest Argon2id cost.
-// Every login stretches it again, and at a realistic cost that takes seconds on
-// a loaded host.
-const TEST_OPS = 1;
-const TEST_MEM = 8192;
+const TEST_OPS = 2;
+const TEST_MEM = 64 * 1024 * 1024;
 
 export interface MetaMockState {
     verifier: Buffer;

test/library/content-library.test.ts

@@ -253,12 +253,6 @@ describe("Library content wiring", () => {
         await expect(
             lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
         ).rejects.toThrow(/content cache/i);
-        await expect(lib.originalOf(file(1, 1))).rejects.toThrow(
-            /content cache/i,
-        );
-        await expect(lib.thumbnailOf(file(1, 1))).rejects.toThrow(
-            /content cache/i,
-        );
         await lib.close();
     });
 });

test/library/content.test.ts

@@ -375,25 +375,6 @@ describe("ContentCache.original / thumbnail", () => {
         await cache.open();
         await expect(cache.original(999)).rejects.toThrow(/unknown file/i);
     });
-
-    // `getFile` knows no file here, so the cache can fetch only the file it is
-    // given.
-    it("originalOf and thumbnailOf fetch and cache the file they are given", async () => {
-        const { cache, source } = buildCache({ files: [] });
-        await cache.open();
-
-        const original = await cache.originalOf(file(5));
-        const thumbnail = await cache.thumbnailOf(file(5));
-
-        expect(source.originalCalls).toEqual([5]);
-        expect(source.thumbnailCalls).toEqual([5]);
-        expect(original.path).toBe(join(cacheDir, "originals", "5.jpg"));
-        expect(thumbnail.path).toBe(join(cacheDir, "thumbnails", "5.jpg"));
-        expect(cache.pathsFor(5)).toEqual({
-            originalPath: original.path,
-            thumbnailPath: thumbnail.path,
-        });
-    });
 });
 
 describe("ContentCache.ensureThumbnails", () => {

test/library/fresh.test.ts

@@ -17,6 +17,14 @@
  *  4. `freshCollections()` reads the collection list alone, with one request
  *     and no collection's files, and stores nothing, so the next refresh still
  *     fetches the files of every collection (issue #210).
+ *  5. `refreshCollection()` brings one collection up to date in the store
+ *     (issue #217). It leaves the stored cursor where it was, so the next
+ *     refresh still fetches the files of every other collection that changed.
+ *     It never runs at the same time as a refresh. After a refresh, it stores
+ *     no collection the store does not hold: not a new one, and not one a
+ *     refresh deleted. `close()` waits for it. A library opened on a cache that
+ *     only it wrote awaits its first refresh. That refresh replaces what it
+ *     stored, in the server's order.
  *
  * The client is the same metadata-only mock the background-refresh tests use:
  * no crypto, no network, scripted pages, and a record of each call. A long
@@ -30,6 +38,7 @@ import { tmpdir } from "node:os";
 import { join } from "node:path";
 
 import { Library } from "../../src/library/index.js";
+import { MetadataStore } from "../../src/library/store.js";
 import type { CollectionsPage, FilesPage } from "../../src/client.js";
 import type { Collection, EnteFile } from "../../src/model/types.js";
 
@@ -144,6 +153,26 @@ const openSeeded = async (
     return { client, lib };
 };
 
+// Hold the client's next files request until `release()` is called. `started`
+// resolves once that request has been sent.
+const holdNextFilesRequest = (
+    client: MockClient,
+): { started: Promise<void>; release: () => void } => {
+    let release!: () => void;
+    const held = new Promise<void>((r) => (release = r));
+    let sent!: () => void;
+    const started = new Promise<void>((r) => (sent = r));
+    const realFilesSince = client.filesSince.bind(client);
+    client.filesSince = async (args) => {
+        client.filesSince = realFilesSince;
+        const page = await realFilesSince(args);
+        sent();
+        await held;
+        return page;
+    };
+    return { started, release };
+};
+
 describe("Library.fresh", () => {
     let dir: string;
     let cacheDirectory: string;
@@ -287,10 +316,10 @@ describe("Library.freshCollections", () => {
         rmSync(dir, { recursive: true, force: true });
     });
 
-    // A refresh fetches a collection's files only when the collection is new
-    // or its updationTime has advanced past the stored copy's. Had the
-    // collection list been stored, the next refresh would take every
-    // collection for one it already holds the files of, and fetch none.
+    // A refresh from a stored cursor past 0 fetches a collection's files only
+    // when the collection is new or its updationTime has advanced past the
+    // stored copy's. So does `refreshCollection()`. Neither would fetch the
+    // files of a collection stored here without them.
     it("asks for the collection list alone and stores nothing", async () => {
         const client = new MockClient();
         const page = {
@@ -340,3 +369,243 @@ describe("Library.freshCollections", () => {
         }
     });
 });
+
+describe("Library.refreshCollection", () => {
+    let dir: string;
+    let cacheDirectory: string;
+
+    beforeEach(() => {
+        dir = mkdtempSync(join(tmpdir(), "quak-refresh-collection-"));
+        cacheDirectory = join(dir, "cache");
+    });
+
+    afterEach(() => {
+        rmSync(dir, { recursive: true, force: true });
+    });
+
+    // Collections 1 and 2 are stored at updationTime 100, with a stored cursor
+    // of 100, and then both change on the server.
+    it("fetches the files of one changed collection and leaves the others to the next refresh", async () => {
+        const client = new MockClient();
+        client.collectionsQueue.push({
+            collections: [collection(1, 100), collection(2, 100)],
+            deleted: [],
+            cursor: 100,
+        });
+        client.filesFor(1, {
+            files: [file(1001, 1, 90)],
+            deleted: [],
+            cursor: 90,
+        });
+        client.filesFor(2, {
+            files: [file(2001, 2, 90)],
+            deleted: [],
+            cursor: 90,
+        });
+        const lib = await Library.open({
+            client,
+            cacheDirectory,
+            refreshIntervalSeconds: SLOW_INTERVAL,
+        });
+        try {
+            client.filesCalls.length = 0;
+            client.filesFor(1, {
+                files: [file(1002, 1, 110)],
+                deleted: [],
+                cursor: 110,
+            });
+            client.filesFor(2, {
+                files: [file(2002, 2, 120)],
+                deleted: [],
+                cursor: 120,
+            });
+
+            // Collection 1's files alone, since the stored copy.
+            expect(await lib.refreshCollection(collection(1, 110))).toBe(true);
+            expect(client.filesCalls).toEqual([
+                { collectionID: 1, sinceTime: 100 },
+            ]);
+            expect(lib.listFiles(1).map((f) => f.id)).toEqual([1001, 1002]);
+            expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001]);
+
+            // The stored copy is now current, so no request.
+            await lib.refreshCollection(collection(1, 110));
+            expect(client.filesCalls).toHaveLength(1);
+
+            // The next refresh asks from the stored cursor, 100, so it still
+            // sees collection 2 changed and fetches its files.
+            client.collectionsQueue.push({
+                collections: [collection(1, 110), collection(2, 120)],
+                deleted: [],
+                cursor: 120,
+            });
+            await lib.fresh();
+            expect(client.collectionsSinceTimes).toEqual([0, 100]);
+            expect(client.filesCalls).toEqual([
+                { collectionID: 1, sinceTime: 100 },
+                { collectionID: 2, sinceTime: 100 },
+            ]);
+            expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001, 2002]);
+        } finally {
+            await lib.close();
+        }
+    });
+
+    // Collection 1 is stored at 100. `refreshCollection()` asks for it at 110,
+    // which adds file 1002. While that files request waits, a fresh read
+    // starts a refresh. The server tells that refresh that collection 1 was
+    // deleted at 130.
+    it("holds off a refresh that starts during the files request, which then deletes the collection", async () => {
+        const { client, lib } = await openSeeded(cacheDirectory);
+        const files = holdNextFilesRequest(client);
+        try {
+            client.filesFor(1, {
+                files: [file(1002, 1, 110)],
+                deleted: [],
+                cursor: 110,
+            });
+            client.collectionsQueue.push({
+                collections: [],
+                deleted: [1],
+                cursor: 130,
+            });
+
+            const refreshing = lib.refreshCollection(collection(1, 110));
+            await files.started;
+            const reading = lib.fresh();
+            // The refresh has not sent its collections request yet.
+            await new Promise((r) => setImmediate(r));
+            expect(client.collectionsSinceTimes).toEqual([0]);
+
+            files.release();
+            await refreshing;
+            await reading;
+            expect(client.collectionsSinceTimes).toEqual([0, 100]);
+            expect(lib.getCollection(1)).toBeUndefined();
+            expect(lib.listFiles(1)).toEqual([]);
+        } finally {
+            files.release();
+            await lib.close();
+        }
+    });
+
+    // `freshCollections()` lists collection 1 at 110. Before
+    // `refreshCollection()` is called, a refresh learns that it was deleted at
+    // 130.
+    it("stores nothing for a collection that a refresh deleted before the call", async () => {
+        const { client, lib } = await openSeeded(cacheDirectory);
+        try {
+            client.collectionsQueue.push(
+                { collections: [collection(1, 110)], deleted: [], cursor: 110 },
+                { collections: [], deleted: [1], cursor: 130 },
+            );
+            const [listed] = await lib.freshCollections();
+            await lib.fresh();
+            client.filesCalls.length = 0;
+
+            expect(await lib.refreshCollection(listed)).toBe(false);
+            expect(client.filesCalls).toEqual([]);
+            expect(lib.getCollection(1)).toBeUndefined();
+        } finally {
+            await lib.close();
+        }
+    });
+
+    // A refresh has stored collection 1. Collection 2 is new on the server
+    // since then. Only the next refresh may add it, after collection 1.
+    it("stores nothing for a collection that is new since the last refresh", async () => {
+        const { client, lib } = await openSeeded(cacheDirectory);
+        try {
+            client.filesCalls.length = 0;
+
+            expect(await lib.refreshCollection(collection(2, 110))).toBe(false);
+            expect(client.filesCalls).toEqual([]);
+            expect(lib.getCollection(2)).toBeUndefined();
+        } finally {
+            await lib.close();
+        }
+    });
+
+    // The caller may remove the cache directory once `close()` resolves, so
+    // the cache write must have landed by then.
+    it("close() waits for a refreshCollection() in flight", async () => {
+        const { client, lib } = await openSeeded(cacheDirectory);
+        const files = holdNextFilesRequest(client);
+        client.filesFor(1, {
+            files: [file(1002, 1, 110)],
+            deleted: [],
+            cursor: 110,
+        });
+
+        const refreshing = lib.refreshCollection(collection(1, 110));
+        await files.started;
+        let closed = false;
+        const closing = lib.close().then(() => {
+            closed = true;
+        });
+        await new Promise((r) => setImmediate(r));
+        expect(closed).toBe(false);
+
+        files.release();
+        await closing;
+        const saved = await MetadataStore.load(
+            join(cacheDirectory, "metadata.json"),
+        );
+        expect(saved.listFiles(1).map((f) => f.id)).toEqual([1001, 1002]);
+        await refreshing;
+    });
+
+    // `quak files 2` on an empty cache stores collection 2 alone. A library
+    // opened later on that cache must not serve it as the whole account. Its
+    // first refresh replaces what the cache holds, in the server's order.
+    it("open() awaits the first refresh on a cache that only refreshCollection() wrote", async () => {
+        const client = new MockClient();
+        client.filesFor(2, {
+            files: [file(2001, 2, 90)],
+            deleted: [],
+            cursor: 90,
+        });
+        const first = await Library.open({
+            client,
+            cacheDirectory,
+            refreshOnOpen: false,
+            refreshIntervalSeconds: SLOW_INTERVAL,
+        });
+        expect(await first.refreshCollection(collection(2, 100))).toBe(true);
+        await first.close();
+
+        client.collectionsQueue.push({
+            collections: [collection(1, 100), collection(2, 100)],
+            deleted: [],
+            cursor: 100,
+        });
+        client.filesFor(1, {
+            files: [file(1001, 1, 90)],
+            deleted: [],
+            cursor: 90,
+        });
+        client.filesFor(2, {
+            files: [file(2001, 2, 90)],
+            deleted: [],
+            cursor: 90,
+        });
+        client.filesCalls.length = 0;
+        const lib = await Library.open({
+            client,
+            cacheDirectory,
+            refreshIntervalSeconds: SLOW_INTERVAL,
+        });
+        try {
+            expect(lib.status().lastRefreshAt).toBeDefined();
+            expect(lib.listCollections().map((c) => c.id)).toEqual([1, 2]);
+            // It fetches collection 2's files again, from the beginning.
+            expect(client.filesCalls).toEqual([
+                { collectionID: 1, sinceTime: 0 },
+                { collectionID: 2, sinceTime: 0 },
+            ]);
+            expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001]);
+        } finally {
+            await lib.close();
+        }
+    });
+});

test/library/library.test.ts

@@ -20,9 +20,9 @@
  * 5. `close()` stops the timer and is idempotent, and its promise resolves
  *    only once an in-flight refresh has written the cache file.
  * 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id.
- * 7. `open()` branches on the cache: an empty cache awaits the first refresh
- *    (it has nothing to serve yet); an existing cache serves its copy at once
- *    and refreshes in the background, so a slow or dead server never stalls
+ * 7. `open()` branches on the stored cursor. At 0, as on an empty cache, it
+ *    awaits the first refresh. Past 0 it serves the cache's copy at once and
+ *    refreshes in the background, so a slow or dead server never stalls
  *    opening.
  * 8. A save failure that leaves RAM ahead of disk keeps `status().lastError`
  *    set and keeps retrying the write; a later empty refresh does not clear it.

test/library/snapshot.test.ts

@@ -4,7 +4,8 @@
  * These are the surface the GUI consumes across Electron IPC. `snapshot()` is
  * synchronous — it reads the in-RAM store and projects it into plain records
  * (no keys) — and `subscribe({ onChange })` delivers a `LibraryChange` whenever
- * a background refresh actually changes the derived records. The contracts:
+ * a refresh or `refreshCollection()` changes the derived records. The
+ * contracts:
  *
  *  1. `snapshot()` deduplicates a file across memberships into one record with
  *     every album id, orders photos newest first, and carries no key material.

test/library/store.test.ts

@@ -172,7 +172,7 @@ describe("MetadataStore", () => {
         expect(store.listCollections()).toEqual([]);
     });
 
-    it("loads an empty store when the list of what the refresh could not decrypt is damaged", async () => {
+    it("loads an empty store when the list of what did not decrypt is damaged", async () => {
         // A backup reads that list on every run, so a damaged one that loaded
         // would fail every backup until the cache was deleted. A string is
         // damaged too, though JavaScript can loop over its characters.
@@ -266,4 +266,23 @@ describe("MetadataStore", () => {
         expect(store.getFile(12345, 67890)).toBeUndefined();
         expect(store.getFile(55555, 67890)?.key).toEqual(inB.key);
     });
+
+    // A backup reports these in this order. It must not depend on whether a
+    // refresh or `Library.refreshCollection()` fetched each one first.
+    it("lists what did not decrypt by collection ID, then file ID", async () => {
+        const store = await MetadataStore.load(path);
+        store.putUndecryptable({ collectionID: 2, fileID: 7, error: "a" });
+        store.putUndecryptable({ collectionID: 3, error: "b" });
+        store.putUndecryptable({ collectionID: 1, fileID: 9, error: "c" });
+        store.putUndecryptable({ collectionID: 2, fileID: 5, error: "d" });
+
+        expect(
+            store.listUndecryptable().map((r) => [r.collectionID, r.fileID]),
+        ).toEqual([
+            [1, 9],
+            [2, 5],
+            [2, 7],
+            [3, undefined],
+        ]);
+    });
 });

test/packaging/build-bin.test.ts

@@ -1,241 +0,0 @@
-// `script/build-bin` compiles `bin/quak`, the single binary that the README's
-// cron setup runs. bun compiles it from a copy of its own executable, so the
-// binary loads the same libraries as the bun that built it. nix's bun loads
-// libraries from the nix store, which garbage collection can delete: on Linux
-// its C library, on macOS its ICU. The script therefore downloads bun's own
-// release build, which loads only the system's libraries.
-//
-// What the script promises, and why each test matters:
-//
-// - Each system, machine and C library that bun publishes a build for gets its
-//   own archive: Linux on x86-64 or arm64, with glibc or with musl, as on
-//   Alpine, and macOS on Intel or Apple silicon. A bun built for another
-//   system, machine or C library does not start, and `make install` fails.
-// - The archive's SHA-256 must match the one in the script before anything in
-//   the archive runs. Without that check, a tampered download would build the
-//   binary that cron runs.
-// - Any other system or machine stops with an error, and never falls back to
-//   nix's bun. Building with nix's bun would bring back the dependency on the
-//   nix store.
-//
-// Each test copies the script into a fresh directory, which the script then
-// treats as the checkout, and runs it there. Stubs first on PATH stand in for
-// `uname`, `ldd`, `curl`, `unzip`, `nix-shell` and `shasum`, so no test
-// downloads anything or runs a real bun.
-import { afterEach, beforeEach, describe, expect, it } from "vitest";
-import { execFileSync, spawnSync } from "node:child_process";
-import {
-    chmodSync,
-    copyFileSync,
-    existsSync,
-    mkdirSync,
-    mkdtempSync,
-    readFileSync,
-    rmSync,
-    symlinkSync,
-    writeFileSync,
-} from "node:fs";
-import { tmpdir } from "node:os";
-import { join } from "node:path";
-import { fileURLToPath } from "node:url";
-
-const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
-
-// The stubs, keyed by the command each replaces. `uname` and `ldd` describe the
-// system that runBuildBin passes in the environment. `curl`, `unzip`,
-// `nix-shell` and `shasum` write the arguments they were given, one per line,
-// to a file named after the command in $STUB_LOG_DIR, which also shows that
-// they ran.
-const stubs: Record<string, string> = {
-    uname: `
-        case "$1" in
-            -s) echo "$STUB_UNAME_S" ;;
-            -m) echo "$STUB_UNAME_M" ;;
-        esac`,
-    // The script looks only at the start of the line: "musl libc" or not.
-    ldd: `
-        if [ "$STUB_LIBC" = musl ]; then
-            echo "musl libc (x86_64)"
-        else
-            echo "ldd (GNU libc) 2.36"
-        fi`,
-    // Writes a file that is not the archive to the path after -o.
-    curl: `
-        printf '%s\\n' "$@" > "$STUB_LOG_DIR/curl"
-        while [ $# -gt 1 ]; do
-            if [ "$1" = -o ]; then echo "not the archive" > "$2"; fi
-            shift
-        done`,
-    // Unpacks a stub bun where the real archive puts bun: in a directory named
-    // after the archive, inside the directory after -d. That bun writes
-    // bin/quak, as a real compile would.
-    unzip: `
-        printf '%s\\n' "$@" > "$STUB_LOG_DIR/unzip"
-        for arg in "$@"; do
-            case "$arg" in *.zip) archive="$arg" ;; esac
-            if [ "$previous" = -d ]; then directory="$arg"; fi
-            previous="$arg"
-        done
-        bun="$directory/$(basename "$archive" .zip)/bun"
-        mkdir -p "$(dirname "$bun")"
-        echo '#!/bin/sh' > "$bun"
-        echo 'mkdir -p bin && echo binary > bin/quak' >> "$bun"
-        chmod +x "$bun"`,
-    "nix-shell": `
-        printf '%s\\n' "$@" > "$STUB_LOG_DIR/nix-shell"`,
-    // The script checks the hash with `shasum` where there is no `sha256sum`.
-    // The stub fails, as `shasum` does when the hash does not match.
-    shasum: `
-        printf '%s\\n' "$@" > "$STUB_LOG_DIR/shasum"
-        exit 1`,
-};
-
-let checkout = "";
-
-const writeScript = (path: string, body: string): void => {
-    writeFileSync(path, `#!/bin/sh\n${body}\n`);
-    chmodSync(path, 0o755);
-};
-
-// A checkout holding the script, a stub script/build that does nothing, the
-// stubs, and the directory they log to.
-beforeEach(() => {
-    checkout = mkdtempSync(join(tmpdir(), "quak-build-bin-"));
-    mkdirSync(join(checkout, "script"));
-    copyFileSync(
-        join(repoRoot, "script/build-bin"),
-        join(checkout, "script/build-bin"),
-    );
-    chmodSync(join(checkout, "script/build-bin"), 0o755);
-    writeScript(join(checkout, "script/build"), "exit 0");
-    mkdirSync(join(checkout, "stubs"));
-    for (const [command, body] of Object.entries(stubs)) {
-        writeScript(join(checkout, "stubs", command), body);
-    }
-    mkdirSync(join(checkout, "log"));
-});
-
-afterEach(() => {
-    rmSync(checkout, { recursive: true, force: true });
-});
-
-// Runs the script with nothing in its environment but the stubs' settings and a
-// PATH. Unless the caller passes one, the PATH finds the stubs first and then
-// the host's commands.
-const runBuildBin = (
-    system: string,
-    machine: string,
-    {
-        libc = "glibc",
-        path = `${join(checkout, "stubs")}:${process.env.PATH}`,
-    } = {},
-) =>
-    spawnSync(join(checkout, "script/build-bin"), {
-        cwd: checkout,
-        encoding: "utf-8",
-        env: {
-            PATH: path,
-            STUB_UNAME_S: system,
-            STUB_UNAME_M: machine,
-            STUB_LIBC: libc,
-            STUB_LOG_DIR: join(checkout, "log"),
-        },
-    });
-
-// The real commands that the script and the stubs run.
-const hostCommands = [
-    "basename",
-    "chmod",
-    "dirname",
-    "grep",
-    "mkdir",
-    "mktemp",
-    "rm",
-];
-
-// A Mac has no `ldd`, and not every Mac has `sha256sum`. This makes a PATH that
-// finds neither: the stubs other than `ldd`, then links to hostCommands.
-const makeMacPath = (): string => {
-    rmSync(join(checkout, "stubs", "ldd"));
-    mkdirSync(join(checkout, "host-commands"));
-    for (const command of hostCommands) {
-        const target = execFileSync("sh", ["-c", `command -v ${command}`], {
-            encoding: "utf-8",
-        }).trim();
-        symlinkSync(target, join(checkout, "host-commands", command));
-    }
-    return `${join(checkout, "stubs")}:${join(checkout, "host-commands")}`;
-};
-
-const ran = (command: string): boolean =>
-    existsSync(join(checkout, "log", command));
-
-const argumentsOf = (command: string): string[] =>
-    readFileSync(join(checkout, "log", command), "utf-8")
-        .trimEnd()
-        .split("\n");
-
-describe("script/build-bin", () => {
-    // The stub curl writes a file that is not the archive, so the hash check
-    // must stop the script before unzip runs. A script without the check would
-    // run unzip, and the stub bun it unpacks would write bin/quak. On Linux the
-    // check must use `sha256sum`, since a stock Alpine has no `shasum`.
-    it.each([
-        ["Linux", "x86_64", "glibc", "bun-linux-x64-baseline.zip"],
-        ["Linux", "x86_64", "musl", "bun-linux-x64-musl-baseline.zip"],
-        ["Linux", "aarch64", "glibc", "bun-linux-aarch64.zip"],
-        ["Linux", "aarch64", "musl", "bun-linux-aarch64-musl.zip"],
-    ])(
-        "on %s %s with %s, downloads %s and stops when its hash does not match",
-        (system, machine, libc, archive) => {
-            const result = runBuildBin(system, machine, { libc });
-            expect(argumentsOf("curl")).toContain(
-                "https://github.com/oven-sh/bun/releases/download/bun-v1.3.13/" +
-                    archive,
-            );
-            expect(ran("shasum")).toBe(false);
-            expect(result.status).not.toBe(0);
-            expect(ran("unzip")).toBe(false);
-            expect(ran("nix-shell")).toBe(false);
-            expect(existsSync(join(checkout, "bin/quak"))).toBe(false);
-        },
-    );
-
-    // On the PATH from makeMacPath the script finds no `sha256sum` and checks
-    // the hash with the stub `shasum`. That stub fails, so the script must stop
-    // before unzip runs.
-    // On Apple silicon `uname -m` says arm64, where bun's archive says aarch64.
-    it.each([
-        ["Darwin", "x86_64", "bun-darwin-x64-baseline.zip"],
-        ["Darwin", "arm64", "bun-darwin-aarch64.zip"],
-    ])(
-        "on %s %s, downloads %s and stops when shasum rejects its hash",
-        (system, machine, archive) => {
-            const result = runBuildBin(system, machine, {
-                path: makeMacPath(),
-            });
-            expect(argumentsOf("curl")).toContain(
-                "https://github.com/oven-sh/bun/releases/download/bun-v1.3.13/" +
-                    archive,
-            );
-            expect(argumentsOf("shasum")).toEqual(["-a", "256", "-c", "-"]);
-            expect(result.status).not.toBe(0);
-            expect(ran("unzip")).toBe(false);
-            expect(ran("nix-shell")).toBe(false);
-            expect(existsSync(join(checkout, "bin/quak"))).toBe(false);
-        },
-    );
-
-    it.each([
-        ["Linux", "riscv64"],
-        ["FreeBSD", "amd64"],
-    ])(
-        "stops on %s %s, which bun publishes no build for",
-        (system, machine) => {
-            const result = runBuildBin(system, machine);
-            expect(result.status).not.toBe(0);
-            expect(ran("curl")).toBe(false);
-            expect(ran("nix-shell")).toBe(false);
-        },
-    );
-});

test/thumbnails/thumbnails.test.ts

@@ -6,12 +6,10 @@
  * working thumbnails, others return 404 or empty bodies. The tests
  * verify that the detection and repair logic handles each case correctly.
  *
- * `listMissingThumbnails` takes an open `Library` to enumerate the files. Both
- * helpers take the `Client` for the API operations: the thumbnail existence
- * check, and the encrypt-and-upload path. `fixMissingThumbnails` takes the
- * record of each file, which these tests look up with `cachedFilesByID`. It
- * reads each original through the library's content cache, by
- * `lib.originalOf(file)`.
+ * As of issue #52 both helpers take an open `Library` for enumeration and the
+ * `Client` for the API operations that stay unchanged (the thumbnail existence
+ * check, and the encrypt-and-upload path). `fixMissingThumbnails` reads each
+ * original through the library's content cache (`photo.original()`).
  *
  * `fixMissingThumbnails` is the most complex function in quak: it
  * downloads the original file, generates a JPEG thumbnail with jpeg-js,
@@ -43,7 +41,6 @@ import {
 import { SRP, SrpServer } from "fast-srp-hap";
 import { Client } from "../../src/client.js";
 import { Library } from "../../src/library/index.js";
-import { cachedFilesByID } from "../../src/cli-read.js";
 import {
     listMissingThumbnails,
     fixMissingThumbnails,
@@ -57,11 +54,8 @@ import type { RetryOptions } from "../../src/retry.js";
 
 const TEST_EMAIL = "thumb@example.com";
 const TEST_PASSWORD = "thumbpass";
-// The password is stretched at libsodium's lowest Argon2id cost. Each test
-// builds an account and logs in, and at a realistic cost that takes seconds on
-// a loaded host.
-const TEST_OPS = 1;
-const TEST_MEM = 8192;
+const TEST_OPS = 2;
+const TEST_MEM = 64 * 1024 * 1024;
 const TEST_TIME = 1700000000000000;
 
 interface ThumbMockState {
@@ -273,7 +267,6 @@ const buildThumbMock = async (opts?: {
 
 const buildThumbFetch = (m: ThumbMockState) => {
     let srpServer: SrpServer;
-    let signedMD5: string | undefined;
 
     return (async (
         input: RequestInfo | URL,
@@ -366,9 +359,8 @@ const buildThumbFetch = (m: ThumbMockState) => {
             return new Response("not found", { status: 404 });
         }
 
-        // Upload URL minting. The real server signs the MD5 into the URL.
+        // Upload URL minting
         if (path === "/files/upload-url") {
-            signedMD5 = JSON.parse(init?.body as string).contentMD5;
             return json({
                 objectKey: `42/thumb-${Date.now()}`,
                 url: "https://s3.mock.test/presigned-put",
@@ -377,11 +369,6 @@ const buildThumbFetch = (m: ThumbMockState) => {
 
         // Presigned PUT (S3 upload)
         if (url.startsWith("https://s3.mock.test/")) {
-            // Storage refuses a PUT that lacks the signed `Content-MD5`.
-            const headers = new Headers(init?.headers);
-            if (headers.get("Content-MD5") !== signedMD5) {
-                return new Response("AccessDenied", { status: 400 });
-            }
             const body = init?.body;
             // Store the uploaded bytes for later inspection
             if (body instanceof Uint8Array) {
@@ -669,12 +656,7 @@ describe("fixMissingThumbnails", () => {
         const client = await login(buildThumbFetch(fixMock));
         const lib = await openLib(client);
 
-        const results = await fixMissingThumbnails(
-            lib,
-            client,
-            [101],
-            cachedFilesByID(lib, [101]),
-        );
+        const results = await fixMissingThumbnails(lib, client, [101]);
         lib.close();
 
         expect(results.length).toBe(1);
@@ -711,12 +693,7 @@ describe("fixMissingThumbnails", () => {
         const client = await login(buildThumbFetch(fixMock));
         const lib = await openLib(client);
 
-        const results = await fixMissingThumbnails(
-            lib,
-            client,
-            [999],
-            cachedFilesByID(lib, [999]),
-        );
+        const results = await fixMissingThumbnails(lib, client, [999]);
         lib.close();
 
         expect(results.length).toBe(1);
@@ -738,7 +715,6 @@ describe("fixMissingThumbnails", () => {
             lib,
             client,
             [101, 102],
-            cachedFilesByID(lib, [101, 102]),
             (msg, level) => {
                 if (level === "error") errors.push(msg);
             },
@@ -769,7 +745,6 @@ describe("fixMissingThumbnails", () => {
             lib,
             client,
             [101, 103, 104],
-            cachedFilesByID(lib, [101, 103, 104]),
         );
         lib.close();
 
@@ -804,12 +779,7 @@ describe("fixMissingThumbnails", () => {
         const client = await login(counted.fetch);
         const lib = await openLib(client);
 
-        const results = await fixMissingThumbnails(
-            lib,
-            client,
-            [101],
-            cachedFilesByID(lib, [101]),
-        );
+        const results = await fixMissingThumbnails(lib, client, [101]);
         lib.close();
 
         expect(results[0]!.status).toBe("skipped");
@@ -830,12 +800,7 @@ describe("fixMissingThumbnails", () => {
         const client = await login(counted.fetch);
         const lib = await openLib(client);
 
-        const results = await fixMissingThumbnails(
-            lib,
-            client,
-            [101],
-            cachedFilesByID(lib, [101]),
-        );
+        const results = await fixMissingThumbnails(lib, client, [101]);
         lib.close();
 
         expect(results[0]!.status).toBe("skipped");
@@ -876,12 +841,7 @@ describe("fixMissingThumbnails", () => {
         const client = await login(buildThumbFetch(fixMock));
         const lib = await openLib(client);
 
-        const results = await fixMissingThumbnails(
-            lib,
-            client,
-            [101],
-            cachedFilesByID(lib, [101]),
-        );
+        const results = await fixMissingThumbnails(lib, client, [101]);
         lib.close();
 
         expect(results[0]!.status).toBe("fixed");

vitest.config.ts

@@ -3,15 +3,8 @@ import { configDefaults, defineConfig } from "vitest/config";
 // vitest does not read .gitignore when looking for tests. A checkout nested
 // under .claude/ has its own test/ tree, and without this exclude the suite
 // runs once per nested checkout and still reports success.
-//
-// vitest starts one worker per CPU core by default, and a shared host has
-// dozens. Four keep the suite well inside the Dockerfile's time limit. vitest
-// 2 rejects a maxWorkers below its default minWorkers, so minWorkers is set
-// too.
 export default defineConfig({
     test: {
         exclude: [...configDefaults.exclude, ".claude/**"],
-        minWorkers: 1,
-        maxWorkers: 4,
     },
 });