make build-bin: on macOS, build with bun's pinned release build (closes #251)

nixpkgs points its macOS bun at an ICU in the nix store, and bun build
--compile copied that reference into bin/quak, so nix-collect-garbage could
break the cron backup. script/build-bin now downloads bun 1.3.13's release
archive on macOS too, Intel (baseline) or Apple silicon, checks its SHA-256,
and compiles with it. Those builds load only /usr/lib libraries. nix-shell is
no longer used; any system without a bun release build stops with an error.
The hash check falls back to shasum where sha256sum is missing.

Model: opus-5-5

Makefile

@@ -26,10 +26,8 @@ check:
 build:
 	@script/build
 
-# Bundles the built dist/, so the binary reports the version script/build
+build-bin:
-# stamped.
+	@script/build-bin
-build-bin: build
-	nix-shell -p bun --run "bun build dist/bin/quak.js --compile --outfile bin/quak"
 
 install: build-bin
 	mkdir -p ~/bin

TODO.md

@@ -2,7 +2,7 @@
 
 - branch from `next`
 - do the work in Next Step
-- move Next Step to the top of Completed Steps
+- delete Next Step from `TODO.md`
 - move the top item of Future Steps into Next Step
 - commit (`TODO.md` changes in the same commit as the work)
 - push
@@ -17,594 +17,13 @@ pre-1.0
 
 # Next Step
 
-None: no implementation work is open. The check of quak against a live Ente
+No bug is open, and no change sneak asked for is open. The check of quak against
-account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
+a live Ente account, https://git.eeqj.de/sneak/quak/issues/187, needs an account
-has, so it waits on him.
+only sneak has, so it waits on him.
 
 Tagging and releases are decided by sneak alone, and happen only when he
 declares one.
 
-# Completed Steps
-
-- 2026-10-14: `lib.backup()`, and so `quak backup`, works on up to the content
-  pool's limit of originals at once, 5 by default, instead of one at a time
-  (issue 182). That many workers take files from one shared list, each doing all
-  the work for one original, its `--verify` hashing included, so no more
-  originals than that are open at once. `BackupLibrary` says the limit as
-  `contentConcurrency`. A test holds each download at the fake Ente server to
-  count those in flight. Once a worker has thrown, as one does when `onProgress`
-  throws, or the lock is lost, no worker takes another file; the backup fails
-  once the originals under way are in place, and a lost lock stops it before it
-  fetches ML data.
-
-- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's
-  review, though that issue closed on 2026-10-03 (issue 194). It now names the
-  issues open on the tracker.
-
-- 2026-10-14: A backup whose lock is lost no longer dies with a stack trace
-  (issue 184). `lockBackupDirectory` gives proper-lockfile an `onCompromised`
-  that records why the lock was lost instead of throwing from its timer, and
-  returns the lock, which `quak backup` passes to `lib.backup()` as `lock`. The
-  backup then stops before it starts another file with
-  `lost the lock on DIR: REASON`, which `quak backup` prints as one line before
-  it exits 1. Releasing a lost lock leaves `backup.lock` alone. Tests in
-  `test/cli/backup-end-to-end.test.ts` remove the lock, or put another backup's
-  in its place, while a download is held open.
-
-- 2026-10-14: An album or file quak cannot decrypt, or whose metadata is not
-  JSON, no longer fails the refresh, so one bad record no longer stops
-  `quak backup` from storing anything (issue 181). The refresh leaves it out, an
-  album with its files, and the library keeps it with the reason in
-  `metadata.json` until the server sends a copy that decrypts or deletes it.
-  `quak backup` stores every other file, records each such file in
-  `failures.json` with an error starting `could not be decrypted:`, names each
-  such album's ID in a printed line, or with `--json` in `undecryptableAlbums`,
-  and exits 1 while any remain. A file whose copy in another album decrypts is
-  not recorded: its original is stored from that copy, and the album where it
-  does not decrypt neither links nor lists it. The read commands answer from
-  what decrypted.
-
-- 2026-10-14: `quak backup` no longer asks the server for changes every 3
-  seconds while it runs (issue 183). It opens its library with
-  `refreshIntervalSeconds: Infinity`, which the library now takes as no
-  background timer, so the backup refreshes only as it starts and no timed
-  refresh starts another ML data pass during the run. A test holds a backup's
-  download open while two hours pass and counts its `/collections/v2` requests.
-
-- 2026-10-14: `test/cli/backup-end-to-end.test.ts` runs `quak backup` as cron
-  does (issue 186): `backupCommand` and `run`, the real `loadSession` reading a
-  session saved after logging in, and every request answered by a fake of the
-  Ente HTTP API serving data encrypted with libsodium. It covers the backup tree
-  of an account with two albums of its own and one shared with it, a second run
-  that downloads nothing, a download that succeeds after two 503s, a 404 that
-  lands in `failures.json` with exit 1, a refresh answered 401 with exit 3, and
-  `--verify` putting back an original changed on disk. The fake server and the
-  account it logs in moved from the backup-metadata test into
-  `test/ente-server.ts`, unchanged.
-
-- 2026-10-14: `files --collection`, `get`, `get-thumb` and
-  `helper fix-missing-thumbnails --file` take an ID only when it is written in
-  decimal digits and is a safe integer (issue 185). Before, `Number()` read
-  `0x10` as 16, `1e3` as 1000 and an empty string as 0, and `--file` dropped an
-  ID it could not read without a word. Anything else now prints
-  `Invalid file ID: <arg>` or `Invalid collection ID: <arg>` and exits 1 before
-  any request; one bad `--file` ID fails the whole command. `parseID` in
-  `src/cli-commands.ts` does the check for all four.
-
-- 2026-10-06: The README says how to run `quak backup` from cron (issue 170):
-  log in once as the job's user, a crontab with a backup every night and
-  `--verify` on Sundays appending to a log file, and the `HOME` and
-  `XDG_DATA_HOME` the job needs to find the saved session. A table gives each
-  exit code: 0, the backup is complete; 1, files are in `failures.json` or
-  another error stopped the run; 2, another backup of the directory is running;
-  3, there is no usable session. The introduction lists everything the backup
-  keeps for each file.
-
-- 2026-10-06: `quak backup --verify` and `lib.backup({ verify: true })` hash
-  each original already at its save path as the download check does, streamed, a
-  live photo as `<imageHash>:<videoHash>` (issue 168). One that does not match
-  the content hash its metadata records is logged, removed (both files of a live
-  photo) and put back in the same run, downloaded or copied from the cache, and
-  what is put back is hashed too. One that still does not match, or whose
-  download fails, goes into `failures.json`. One with no recorded hash is left
-  alone. The result, `--json` and the summary gain `verified`, `mismatched` and
-  `unchecked`. Without `--verify` nothing is hashed.
-
-- 2026-10-06: Two backups of the same directory never run at once (issue 169).
-  `lib.backup()` takes a lock, `backup.lock` in its download directory, made
-  with `proper-lockfile`, before its refresh, and removes it when it ends,
-  whether it succeeds or fails. A second backup of the directory, from another
-  process or the same one, fails at once with an error naming the directory.
-  `quak backup` takes the lock before it opens its library, so a refused run
-  sends no request; it prints the error as one line and exits 2. A lock that has
-  gone 10 seconds untouched, left by a run that could not remove it, is taken
-  over by the next run.
-
-- 2026-10-06: `quak backup` writes each original's EXIF, XMP and dimensions into
-  the file's JSON as `imageMetadata`, what `backup-metadata --exif` records
-  (issue 167): for a live photo from its image, for a video nothing, and `{}`
-  for an original with none of them. A failed read puts the reason in
-  `imageMetadataError` and fails neither the file nor the run. An original is
-  read when the run stores it or when its JSON has neither field; otherwise the
-  field is taken from that JSON. The hand-built JPEGs moved to
-  `test/exif-jpeg.ts`, beside the HEIC.
-
-- 2026-10-06: When the server answers HTTP 401 and that ends a command that
-  loads the saved session, because the server no longer accepts its token, the
-  command prints one line,
-  `quak: the saved session is no longer valid; run "quak login"`, and exits 3
-  (issue 164). `quak backup` meets that 401 on the refresh it starts with,
-  before it touches any file. For the same commands, a missing or corrupt
-  session file keeps its message and also exits 3, so a cron job can tell that
-  the user must log in again. `quak logout` is unchanged. `run` in
-  `src/cli-run.ts` recognises the 401, which reaches it unchanged.
-
-- 2026-10-06: `quak backup` retries a failed request for longer than the other
-  commands do (issue 165). `src/retry.ts` exports `UNATTENDED_RETRY_OPTIONS`
-  beside the unchanged default: 10 attempts, a 1 s base delay and a 60 s cap, so
-  a request that keeps failing waits at most 243 s before it gives up.
-  `bin/quak.ts` loads the backup's session with them, so its refresh, ML data
-  and downloads all use them. What is retried and the backoff formula are
-  unchanged.
-
-- 2026-10-06: The files this repository copies from `sneak/prompts` are copied
-  again from its commit `dd4027b` (issue 171). `.gitignore` and `.dockerignore`
-  keep out more secret files, and this repository's build artifacts follow the
-  copied content. `script/check` runs `script/fmt-check` again, `script/cibuild`
-  runs `script/bootstrap` and `script/check` before the image build,
-  `script/fmt` and `script/fmt-check` find the pinned yarn under nvm, and the
-  image's last stage marks `/app` safe for git.
-
-- 2026-10-06: `quak backup` writes the account and album records
-  `backup-metadata` writes (issue 166): `account.json` with the account's
-  `email` and `userID`, and in each album's JSON its `ownerID`, `isShared`,
-  `updationTime` and, when present, its three layers of magic metadata. Each
-  file's JSON gains its `updationTime`.
-
-- 2026-10-05: `quak backup` writes each file's ML data (its faces and CLIP
-  embedding) into the file's JSON as `mlData`, the payload
-  `lib.mldata.forFile()` returns (issue 163). `lib.backup()` waits for an ML
-  data fetch before it writes the JSON files. When that fetch fails, each file
-  whose ML data is not cached gets the reason in `mlDataError` and counts as
-  failed, and the next run fetches it again.
-
-- 2026-10-03: The download-albums example test no longer fails on a disk with
-  under 50 GiB free (issue 160). It opens its libraries with
-  `freeBelowBytes: 0`, so the free space of the disk it runs on cannot shrink
-  the cache, evict the original it cached and add a fetch to the count. No other
-  test's result depends on that free space. The 50 GiB default is unchanged.
-
-- 2026-10-02: `docker build .` stamps the commit's tag or short commit, not
-  `dev` (issue 154). `script/build` writes the version `script/version` prints
-  into `dist/package.json`: the `VERSION` environment variable or build arg when
-  one is given, otherwise `git describe --tags --always`, otherwise the version
-  `package.json` declares. `.dockerignore` sends `.git`, and a checkout with
-  `.git` whose version comes out empty, `dev` or `unknown` fails the build.
-  `make build-bin` bundles the built `dist/`, so the single binary reports the
-  same version.
-
-- 2026-10-02: `photo.exif()` returns every EXIF tag in the file as `ExifTags`,
-  keyed by tag name, each as exifreader decodes it, not only the thirteen common
-  fields (issue 156). The embedded thumbnail's tags are under `Thumbnail`,
-  without the thumbnail image. The thirteen typed methods stay, each picking its
-  field from the tags `exif()` returns, typed as in `PhotoExif`. The example
-  script's JSON files now carry every tag.
-
-- 2026-10-01: The content cache no longer looks for a live photo that an earlier
-  version cached as one ZIP (issue 151). When the cache opens, a live photo's
-  file that no JSON file names is now always left alone.
-
-- 2026-10-01: `examples/download-albums.ts` logs in, opens the library, and for
-  every album downloads each photo to its save path, writes the photo's record
-  and EXIF fields to a JSON file beside it, and writes the album's photos to
-  `albums/<collectionID>.json` (issue 144). The build compiles it to
-  `dist/examples/`; the README's "Examples" section says how to run it.
-
-- 2026-10-01: A `Photo` has one async method for each field of `exif()`, named
-  and typed as in `PhotoExif`: `make()`, `model()`, `lensModel()`,
-  `dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`,
-  `iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()`
-  and `gpsAltitude()` (issue 148). Each calls `exif()` and returns its one
-  field, or undefined when the file lacks it. `Photo` implements a type with one
-  method per `PhotoExif` field, so the build's type check fails when a field has
-  no method. A test checks, on the JPEG and the HEIC, that each method gives the
-  same value as `exif()`.
-
-- 2026-10-01: Each original's save path is
-  `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
-  download directory, which defaults to `photos` in the working directory (issue
-  143). The date is the photo's `takenAt` in the machine's time zone.
-  `photo.savePath` is always a string, with or without a content cache, and
-  `isLocal` is true only when the original is at its save path.
-  `photo.download()` puts the original there, copied from the cache when the
-  cache holds it and fetched otherwise. `lib.backup()` does the same for each
-  file, writes each file's JSON beside its original, and links `collections/` to
-  the save paths.
-
-- 2026-10-01: `photo.exif()` and `backup-metadata --exif` read EXIF from
-  HEIC/HEIF originals, a live photo's HEIC image included, as well as JPEG and
-  the other image formats `exifreader` reads (issue 145). `exifreader` replaces
-  `exif-reader` and the JPEG segment scan; `PhotoExif` is unchanged. The `exif`
-  field of `backup-metadata --exif` is now exifreader's tag output, and
-  `exifRaw` holds the whole EXIF block it could not read. The tests use a real
-  HEIC, `test/exif.heic`.
-
-- 2026-10-01: A `Photo` has `savePath`, `isLocal`, `content()`, `exif()`,
-  `modifiedAt`, `hash` and `year` (issue 141). `savePath` is where
-  `lib.backup()` writes the original under the library's download directory; for
-  a live photo not yet stored, it carries the title's extension, and the backup
-  may store the image under a different one. `isLocal` says whether the whole
-  original is there. Both look only at the disk. `content()` returns the
-  original's bytes and `exif()` the common EXIF fields of a JPEG; both may
-  download the original, and `exif()` downloads no video. `PhotoRecord` gains
-  `modifiedAt` and `hash`, and the JPEG EXIF scan moved to `src/exif.ts`.
-
-- 2026-09-29: The "Workflow" list at the top of this file now says to branch
-  from `next` and open a pull request that targets `next`, that the repository
-  manager squash-merges a reviewed pull request into `next`, and that only sneak
-  merges `next` into `main`, as the README does (issue 137).
-
-- 2026-09-29: The README's "Development workflow" and "For LLMs" sections now
-  say work branches from `next`, every pull request targets `next`, the
-  repository manager squash-merges reviewed pull requests into `next`, and only
-  sneak merges `next` into `main` (issue 135).
-
-- 2026-09-29: Brought the README and this file in line with `next` after the
-  milestone merge (issue 132). The Next Step says no implementation work is open
-  and the cache design (issue 36) waits on sneak's review. README corrections:
-  the Getting Started comments say when the library refreshes; most, not all,
-  Makefile targets call a script; `script/lint` and `script/test` have no host
-  path, though `yarn test` does; the SRP handshake uses `fast-srp-hap`, outside
-  `crypto/`, and a thumbnail upload's MD5 uses `node:crypto`; the SRP password
-  is the first 16 bytes of a 32-byte subkey; the key attributes and token come
-  after SRP, after the TOTP code SRP may ask for, or after the email OTP that
-  replaces SRP when the account has email MFA on, and quak cannot answer a
-  passkey; the auth token is sent as URL-safe base64 with padding; every
-  `TypeError` is retried; each download attempt writes its own temporary files,
-  two for a live photo, and only the attempt that completes renames them into
-  place; `runMetadataBackup` records a failed download in the file's JSON; a
-  second `client.logout()` does not throw; `quak logout` with no session exits
-  0, and names the cache directory only when it knows it; `login`, `whoami` and
-  `logout` open no library; `--exif` records XMP and, for a JPEG, EXIF, and no
-  IPTC; which commands take `--json`; a failed ML data request may not have been
-  retried; the thumbnail fixer is not limited to baseline JPEG; `quak backup`
-  still fetches ML data; a backup's JSON holds the basic metadata fields quak
-  keeps and the private and public magic metadata, not every decrypted field,
-  and the README no longer lists what the magic metadata holds; the default
-  cache directory is the per-user one, not an XDG path on macOS; pinned
-  originals can exceed `cacheOriginalsMaxBytes`; which tests cover which
-  operations; a default read is only as current as the last refresh whose
-  requests all succeeded; `fresh()` and `lib.backup()` join a refresh already
-  running; a `Photo` has no `thumbnailPath` or `originalPath`; and `408` and
-  `429` are retried.
-
-- 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks
-  that the image's and the video's temp files are fsynced before either is
-  renamed into place, and the directory after both renames, as the `writeAtomic`
-  test does for one file.
-
-- 2026-09-23: Live photos are stored as their image and their video (issue 107).
-  A live photo, which Ente stores as one ZIP, is unpacked as it downloads into
-  `<fileID>.<ext>` for the image and for the video, each with its extension from
-  the ZIP, beside `<fileID>.livephoto.json`, which names the two. Both are
-  checked against the recorded hash and renamed into place only when both are
-  complete. A ZIP holding a second image or video, or whose two files come to
-  more than 20 times its size plus 16 MiB, is refused. The backup and the
-  content cache count the live photo as stored only with both files, the album
-  folder links both, `quak get` writes both, and the content result gives the
-  video as `videoPath`. A ZIP an earlier version stored is replaced on the next
-  backup run, and removed from the cache when the library opens it.
-
-- 2026-09-23: Settled the package metadata (issue 6). quak is not published, so
-  `package.json` is marked `"private": true` and the `files` field is gone.
-  `engines.node` is `>=22`, the major version `script/bootstrap` and the
-  `Dockerfile` use. An `exports` map makes `.` and `./package.json` the only
-  importable paths; `runMetadataBackup`, the thumbnail helpers and their types
-  stay internal to the CLI.
-
-- 2026-09-23: Brought the README and this file in line with the tree (issue
-  111). The layout lists `src/library/` and the other source files, the backup
-  layout names `failures.json` and the optional `thumbnails/`, "Opening a
-  library" says what happens when the first refresh fails, the Testing section
-  gives the 60-second hard cap and 20-second target for `make test` and names
-  the 90-second `timeout` in the `Dockerfile` as the backstop for a hung test,
-  and "Tag v1.0.0" is no longer listed as the next step.
-
-- 2026-09-23: Tested `quak login` and `backup-metadata --exif` (issue 110).
-  `loginCommand` takes its login function and its prompts from `CliContext`, and
-  `bin/quak.ts` passes `Client.login` and the terminal prompts. Tests cover a
-  login from `QUAK_EMAIL` and `QUAK_PASSWORD` with no prompt, the TOTP prompt, a
-  failed login, and the saved session's modes, and show that `--exif` and
-  `--all` each turn on EXIF extraction and that it is off without them.
-
-- 2026-09-23: `quak backup` writes each original once and no longer fills the
-  cache (issue 106). An original fetched for a backup is written by the download
-  writer straight into the backup's `originals/`, and the content cache records
-  it there instead of keeping its own copy; one the cache already held is still
-  copied. `quak backup` opens its library with the thumbnail and originals
-  precache off.
-
-- 2026-09-23: `backup-metadata`, `helper list-missing-thumbnails` and
-  `helper fix-missing-thumbnails` refresh before they answer (issue 100). Each
-  awaits `lib.fresh()` before reading, so a file added since the cache was
-  written is included, and a failed refresh prints one line and exits 1 instead
-  of answering from a stale or empty cache. The README lists them among the
-  commands that refresh first.
-
-- 2026-09-23: `quak backup` waits for the server refresh and fails when it fails
-  (issue 99). `lib.backup()` joins a refresh already running or starts one, as
-  `fresh()` does, and rejects before touching any file when it fails, leaving
-  `failures.json` as it was, so `quak backup` prints the error as one line and
-  exits 1 instead of backing up the previous run's file list, or nothing, and
-  exiting 0.
-
-- 2026-09-23: CLI errors print a message instead of a stack trace (issue 102).
-  An error a command throws is printed as one `quak: MESSAGE` line on stderr and
-  the CLI exits 1 once output has drained. The wrapper that does this moved from
-  `bin/quak.ts` to `src/cli-run.ts`, and `bin/quak.ts` now awaits
-  `program.parseAsync()`.
-
-- 2026-09-23: Opening a library no longer deletes another process's download in
-  progress (issue 105). The download writer's temp files are named
-  `.quak-<pid>-<random>.tmp`, and `removeLeftoverTempFiles`, moved from the
-  backup into the download module, deletes a `.quak-*.tmp` file only when the
-  process ID in its name is no longer running. The content cache calls it at
-  `open()` for `originals/` and `thumbnails/`, the backup as before.
-
-- 2026-09-23: Re-vendored the lint and test setup from the template (issue 96).
-  Linting and testing are the `lint` and `test` phases of the `Dockerfile`;
-  `script/lint` and `script/test` each build one with `--no-cache`, and the last
-  stage compiles and depends on both, so `script/cibuild` is one build.
-  `Dockerfile.lint`, `CHECK_EPOCH`, `LINT_EPOCH` and the tests that checked them
-  are gone; `REPO_POLICIES.md` is re-copied.
-
-- 2026-09-23: Stopped `helper fix-missing-thumbnails` retrying files the server
-  always refuses (issue 109). Both thumbnail helpers skip a file another account
-  owns without fetching it. The fixer skips a file whose recorded thumbnail size
-  is 0 or unknown before downloading it, and otherwise tries smaller encodings
-  (720 px quality 50 down to 160 px quality 20) until the encrypted thumbnail is
-  no larger than that size, skipping the file if none fits.
-
-- 2026-09-23: Tested the live-photo hash check's error paths (issue 117). Tests
-  download a live photo whose ZIP names an unknown compression method, one whose
-  ZIP has no image entry and one with no video entry, and check that nothing is
-  stored and the error names the file ID; the unreadable one is not retried.
-
-- 2026-09-23: `quak logout` ends the session on the server (issue 108). It calls
-  `POST /users/logout` through the new `Client.logoutOnServer()`, then deletes
-  `session.json` even when that call fails, says so and exits 1. It prints the
-  account's cache directory and says it still holds decrypted data. The default
-  cache path is now `defaultCacheDirectory()` in the library, shared with
-  `Library.open`.
-
-- 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files
-  in one collection with the same title, and two collections with the same name,
-  each get their ID added to the name (`IMG_0001 (12345).JPG`, `Trip (10)/`), so
-  none replaces another's symlink or JSON. Each run removes symlinks into
-  `originals/` for files no longer in the collection, and the folders of deleted
-  or renamed collections, leaving anything else in `collections/` alone. The
-  README backup layout states the naming rule.
-
-- 2026-09-23: Checked downloaded originals against their recorded content hash
-  (issue 68). `downloadFile`, which `quak get`, the content cache and backup all
-  use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and
-  stores nothing on a mismatch, failing with an error naming the file ID. A live
-  photo ZIP is unpacked as it streams with `fflate` and its image and video
-  hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older
-  clients' `imageHash` and `videoHash` fields for live photos. A file with no
-  recorded hash is stored unchecked.
-
-- 2026-09-23: Kept one account's cache from mixing with another's (issue 104).
-  When `metadata.json` in the cache directory was written for a different,
-  non-zero user ID than the client's, `Library.open` deletes it and `mldata/`
-  and starts empty, so the first refresh enumerates from 0. This only happens
-  with `--cache-dir` or an explicit `cacheDirectory`; the default path already
-  includes the user ID. A test opens one account's cache as another account.
-
-- 2026-09-23: `backup-metadata` no longer stops on one failed ML data request
-  (issue 101). Each request of up to 200 files is tried on its own; a failed one
-  is logged, its files are written with the reason in `mlDataError`, and the
-  command exits 1 once the dump is complete. `fetchMLData`, which only this
-  command used, is gone; the command calls `fetchMLDataBatch` per batch.
-- 2026-09-23: Single-sourced the version string (issue 5). `package.json` is the
-  only place it is written: `src/index.ts` imports it for `VERSION` and
-  `bin/quak.ts` passes `VERSION` to commander. tsc copies `package.json` to
-  `dist/package.json`, so the import resolves from the built output too, and
-  `script/build` runs the built CLI with `--version` to prove it. A test checks
-  that `VERSION` and `quak --version` both equal the `package.json` version.
-- 2026-09-23: Tested that `Library.close()` waits for the originals precache
-  (issue 93). The test that holds a precache fetch open while `close()` runs now
-  runs once with only the thumbnail fill and once with only the originals fill,
-  so dropping either wait from `Precache.close()` fails a test.
-- 2026-09-23: Fixed two intermittently failing library tests (issue 90).
-  `Library.close()` now returns a promise that resolves once an in-flight
-  refresh (including its cache write), the ML data fetch and running precache
-  sweeps have finished; the library tests await it, so `afterEach` no longer
-  removes the cache directory while something is still writing into it. The
-  precache test waits for both fills to report "done" instead of for its stub
-  source to be called, which happened before the cache recorded the file.
-- 2026-09-23: Pinned three guards reviewers found untested (issue 89). The
-  download idle deadline's timer is unref'd, so it can never keep the process
-  alive, and a test checks no timer is left after a download completes or fails.
-  A test covers the rejection of `#` in a request path. The EXIF scan compares
-  the `Exif` header only in an APP1 segment of length 8 or more, so it never
-  reads the next segment's bytes, with a test for a short one.
-- 2026-09-23: Made the download deadline an idle deadline (issue 24).
-  `downloadTimeoutMs` now aborts a file or thumbnail download only after no
-  bytes have arrived for that long, default 60 seconds, instead of bounding the
-  whole transfer at 10 minutes, so a slow download that keeps making progress
-  completes. A download that fails before reading the whole body cancels it, so
-  a failed file no longer holds its connection.
-- 2026-09-23: Every `ApiClient` request URL is now built by one function next to
-  the class (issue 18), so a self-hosted `apiOrigin` with a base path keeps it
-  on every request, a path works with or without a leading slash, and query
-  parameters are percent-encoded. A path containing `?` or `#` is rejected with
-  an error instead of being silently cut.
-- 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies
-  moved from `bin/quak.ts` into `src/cli-commands.ts` as functions that take
-  their options and a context (output streams, session directory, cache
-  directory, session loader) and return an exit code; `bin/quak.ts` only wires
-  them to commander and exits with the code once stdout and stderr have drained,
-  so nothing below it calls `process.exit`. `test/cli/commands.test.ts` drives
-  them with a fake client: session file modes, logout, the missing and corrupt
-  session paths, and the output and exit code of `whoami`, `collections`,
-  `files`, `get`, `get-thumb`, `backup` and `helper list-missing-thumbnails`.
-- 2026-09-23: Hardened the backup tree's atomic copy (issue 22). `copyAtomic`
-  fsyncs its temp file before the rename and the directory after it, through the
-  download writer's `fsyncPath`; each backup run deletes `.quak-backup-*.tmp`
-  files whose process is no longer running. The README backup layout names the
-  temp files and states that the rename replaces a symlink and takes the temp
-  file's permissions. Added tests for a missing and an unwritable destination
-  directory for `downloadFile` and `downloadThumbnail`.
-- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue
-  11). Every segment length is checked against the remaining bytes and lengths
-  under 2 stop the scan, so a truncated or corrupt original can neither throw
-  nor loop. A malformed or unparseable EXIF segment is recorded as
-  `imageMetadata.exifError`, and a failure to read the original as
-  `imageMetadataError` in the per-file JSON, instead of the field being left
-  out.
-- 2026-09-22: Hardened the retry classifier (issue 80). A `POST` or `PUT` is
-  replayed only when every errno in the cause chain is a connect errno, and it
-  no longer follows redirects. `getRetryOptions()` returns a copy. Tests pin
-  every errno the classifier names, the cause-chain depth limit, cycle
-  termination, and a fresh deadline per attempt for every retrying entry point.
-  The README's endpoint list is the one place that names the requests the replay
-  rule covers.
-- 2026-09-22: Stopped `make test` collecting tests from checkouts nested under
-  `.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a
-  nested checkout ran the whole suite again; `vitest.config.ts` now adds
-  `.claude/**` to vitest's default excludes, and
-  `test/packaging/nested-checkout.test.ts` plants a nested checkout in a temp
-  directory and fails if vitest would collect it.
-- 2026-09-22: Dropped the deprecated `@types/libsodium-wrappers-sumo` stub from
-  `devDependencies` (issue 27). It shipped no declarations; the types come from
-  `libsodium-wrappers-sumo` itself. `yarn.lock` regenerated by `yarn remove`.
-- 2026-09-22: Hardened the client session lifecycle (issue 10).
-  `Client.fromJSON` checks every snapshot field and each key's decoded length
-  and names the bad field; `toJSON` reads the token through
-  `ApiClient.getAuthToken` and throws when there is none; `logout` zeroes the
-  key buffers, and `collectionsSince` re-checks for logout after its request so
-  it never decrypts with zeroed keys. The CLI reports a corrupt session file
-  separately from a missing one (`src/cli-session.ts`).
-- 2026-09-22: Sanitized file names taken from server metadata (issue 9). A new
-  `src/filename.ts` holds the one sanitizer, used by `quak get`/`get-thumb`
-  without `--out`, `downloadFile`/`downloadThumbnail` without `outPath`, and the
-  backup and metadata backup trees; it removes separators, control characters,
-  leading dots and Windows device names, and falls back to a name built from the
-  ID for an empty title. Originals-cache extensions are letters and digits only,
-  else `.bin`. A user-supplied path is used as is. `decryptFile` reads a missing
-  or non-string title as "" and rejects metadata that is not a JSON object.
-- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
-  snippets) to match the shipped cache/API library on `next` (issue 53, issue
-  13). Documented `Library.open` and its options, the default-read vs `fresh()`
-  distinction (and that the CLI's read commands are fresh), the record types and
-  `snapshot()`/`subscribe()`, the `albums`/`photos`/`timeline` read surface,
-  `Photo` content methods, `thumbnails.ensure`, the `mldata` search surface,
-  `backup()`, the three request pools (10/5/25), and the on-disk cache layout;
-  noted the deferred content-hash integrity check (issue 68). Docs-only; no code
-  changed.
-- 2026-09-22: Added resumable, deletion-aware enumeration to `Client` (issue 38,
-  closes issue 7). `collectionsSince`/`filesSince` take a starting cursor,
-  decrypt live records, surface tombstoned ids in a separate `deleted` list (a
-  tombstone has nothing to decrypt, so it is a bare id, not a hollow record),
-  and return the max `updationTime` seen as the cursor to resume from.
-  `filesSince` refuses to loop when the diff reports `hasMore` without advancing
-  the cursor (issue 7). `listCollections`/`listFiles` are now thin wrappers that
-  enumerate from `sinceTime: 0` and drop deletions, so existing callers are
-  unaffected.
-- 2026-09-22: Carried file size, thumbnail size, and the deletion flag through
-  `decryptFile` (issue 37, foundation for the cache/API design). Live files now
-  populate `file.size`/`thumbnail.size` from the server's `info` (left
-  `undefined` when the server omits it), and `isDeleted` is carried from the
-  diff row onto `EnteFile`. No caller change: `listFiles` still filters deleted
-  rows before decrypting. Surfacing a tombstone through decryption belongs to
-  the enumeration unit (issue 38).
-- 2026-08-10: Made `lint-once.test.ts` enforce what its header claims. It walked
-  `make check` only, so it never read `Dockerfile` — the image CI builds through
-  `script/cibuild` — and a second `prettier --check .` could be added there with
-  the suite staying green. The walk now also starts at
-  `.gitea/workflows/check.yml` and follows its `run:` steps, so the graph under
-  test is the one CI executes rather than the one someone assumed it executes.
-  The lockfile assertion was a substring check against the whole of
-  `script/bootstrap`, which has two install sites and so reported the branch the
-  containers never take; the two branches are now resolved separately and every
-  `yarn install` in each is required to be `--frozen-lockfile`. Prettier is
-  counted per occurrence instead of per line, so two invocations chained with
-  `&&` no longer read as one, and edges are followed on counted lines instead of
-  being skipped. Every way for the walk to reach nothing — an unknown target, an
-  unknown script, a missing file, a node with no commands, an unknown node kind
-  — is a thrown error rather than a quiet zero. Every assertion in the file was
-  mutation-tested individually.
-- 2026-08-10: Stopped `make check` running `prettier --check .` twice. Since
-  linting moved into Docker, the duplicate was one container pass and one host
-  pass of the same check: `script/lint` builds `Dockerfile.lint`, which runs
-  prettier as a build step, and `script/check` then called `script/fmt-check` as
-  well. The host call is gone from `script/check` and from `script/precommit`;
-  the container keeps checking formatting, because a successful
-  `Dockerfile.lint` build is what CI treats as proof of a clean tree, and it is
-  also what still fails the pre-commit hook on a badly formatted tree.
-  `script/fmt-check` survives as a standalone entrypoint, whose verdict cannot
-  drift from the container's. A test walks the invocation graph from each
-  entrypoint — through the Makefile shims, the `script/` calls and the
-  `docker build` — and asserts the prettier count, so the duplication cannot
-  come back unnoticed.
-- 2026-08-10: Moved all linting into Docker. `script/lint` builds a new root
-  `Dockerfile.lint`, which copies the repo into the digest-pinned node image and
-  runs eslint and prettier as build steps, so a successful build is a clean
-  lint; no host lint path remains and `yarn lint` is gone from `package.json`. A
-  fail-closed `LINT_EPOCH` guard stops Docker serving the linter layers from
-  cache, which is how a lint build returns success in under a second having
-  linted nothing. The lint stage inside `Dockerfile` and its `COPY --from=lint`
-  ordering hack are gone: that image now runs `make test` and `make build` only,
-  because `script/check` calls `script/lint` and running it in a container would
-  mean docker inside docker. `script/cibuild` builds the lint image first, then
-  the test and build image.
-- 2026-08-09: Made `make docker` green and policy-conformant. Multi-stage
-  Dockerfile: a lint stage runs `make fmt-check` and `make lint`, and the check
-  stage takes a `COPY --from=lint` dependency on it before running `make check`
-  and `make build`. `CHECK_EPOCH` and a fail-closed guard stop Docker serving
-  those two layers from cache, which is what let a build report success without
-  running the suite. `script/projectname` says `quak`, so the image is tagged
-  `quak`; `script/bootstrap` updates apt lists before installing, so a Debian
-  base works; `.dockerignore` no longer ships the compiled binary, the caches or
-  agent worktrees into the build context, and keeps `.gitignore` in it for
-  prettier.
-- 2026-08-09: Fixed the TypeScript build. `rootDir` is the repo root, so `bin/`
-  compiles alongside `src/` instead of failing with TS6059; output is
-  `dist/src/` and `dist/bin/`, which is where `main`, `types` and `bin.quak` now
-  point. `script/build` verifies the declared entrypoints exist after the
-  compiler runs and makes the CLI executable, the Dockerfile runs `make build`
-  as well as `make check`, and a `quak` script makes the README's
-  `yarn quak <command>` examples work.
-- 2026-08-09: Retry policy: no retry on 4xx (except `408` and `429`),
-  exponential backoff with full jitter on 5xx, transport failures and truncated
-  transfers, under per-attempt deadlines that cover the response body as well as
-  the request. Downloads retry request, stream consumption and decryption as one
-  unit; `postJSON` and `putJSON` are replayed only when the connection was never
-  established.
-- 2026-08-09: Downloads verify the secretstream terminated on `TAG_FINAL` and
-  write output atomically: a truncated body is rejected instead of landing on
-  disk as a short file, and plaintext is staged in a sibling temp file and
-  renamed into place, so a failed download leaves the destination untouched.
-- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
-  shims, README Entrypoints section
-- 2026-06-10: Decrypted collections shared by other users (sealed-box keys);
-  listCollections drops deleted-collection tombstones.
-- 2026-06-10: Login hardening: dual-2FA empty-string fields handled, TOTP
-  preferred when a passkey is also enrolled, interactive input via
-  @inquirer/prompts.
-- 2026-06-10: Replaced sharp with pure JS (jpeg-js + exif-reader); added
-  single-binary bun build and make install.
-- 2026-06-09: Added backup-metadata command (ML data always included, --exif
-  opt-in); rewrote README to match the implementation; added thumbnail helper
-  tests.
-- 2026-05-13: Full CLI surface: login, backup with dedup symlink layout,
-  collections, files, get, get-thumb, thumbnail repair helpers.
-- 2026-05-13: Client OO API with literate usage tests; file download and
-  decryption; all three metadata layers decrypted and persisted; renamed quack
-  to quak.
-- 2026-05-11: SRP login flow (email OTP + TOTP) and ApiClient.
-
 # Future Steps
 
 - Future desktop client, separate repo:

bin/clip-keyword-vectors

@@ -0,0 +1,31 @@
+#!/bin/sh
+# Writes src/clip-keyword-vectors.ts and test/clip-keywords-sunset.json by
+# running bin/clip-keyword-vectors.py in a Python image pinned by digest, with
+# its packages pinned by hash. Run it by hand from a checkout, then `make fmt`;
+# no gate runs it. Besides the Python image, it downloads about 265 MB, most of
+# it Ente's two CLIP models.
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+# python:3.13-slim (Python 3.13.17 on Debian 13.7), 2026-10-21. The wheels
+# below are the ones for this image on linux/amd64.
+IMAGE=python@sha256:57b29a7c050c4e679e4bea572e14b3965cf4b5839de2da49d55e49ffc4024f39
+
+main() {
+    cd "$ROOT"
+    docker run --rm -i --platform linux/amd64 \
+        --user "$(id -u):$(id -g)" -e HOME=/tmp \
+        -v "$ROOT:/repo" -w /repo "$IMAGE" sh -c '
+            python -m venv /tmp/venv &&
+            /tmp/venv/bin/pip install --quiet --disable-pip-version-check \
+                --no-cache-dir --no-deps --require-hashes -r /dev/stdin &&
+            /tmp/venv/bin/python bin/clip-keyword-vectors.py' <<'EOF'
+numpy==2.5.3 --hash=sha256:a5fa86b80fd24bcd1aff83ad23be44ea323de3f787be8f8b15d4a65621e25321
+onnxruntime==1.30.0 --hash=sha256:86f940afc801ea9681a4da8af84fbe95e1d9ea7d80903952cc1bfad54faad38f
+pillow==12.3.0 --hash=sha256:0847a763afefb695bc912d7c131e7e0632d4edc1d8698f58ddabec8e46b8b6d3
+tokenizers==0.23.2 --hash=sha256:41c2f84d172449b4dadb9cdc508e3e364076613c35b16e76ecfe47a60d1e3305
+EOF
+}
+
+main "$@"

bin/clip-keyword-vectors.py

@@ -0,0 +1,158 @@
+"""Write src/clip-keyword-vectors.ts and test/clip-keywords-sunset.json.
+
+bin/clip-keyword-vectors runs this in a pinned Python image. It downloads
+Ente's Discover list, Ente's CLIP text and image models, the CLIP tokenizer
+and a public-domain photo of a sunset, checks each against its sha256, and
+writes:
+
+- src/clip-keyword-vectors.ts: each Discover category of Ente's photos app,
+  with the text embedding Ente's CLIP text model gives its query;
+- test/clip-keywords-sunset.json: the CLIP embedding Ente's image model gives
+  the photo, prepared as Ente's apps prepare a photo.
+
+Both embeddings are scaled to length 1, as Ente's apps scale them.
+"""
+
+import hashlib
+import io
+import json
+import urllib.request
+
+import numpy as np
+import onnxruntime
+from PIL import Image, ImageOps
+from tokenizers import Tokenizer
+
+ENTE_COMMIT = "f5ad70a860ed25d0c8057b0efb4bd6eaefe4b0c5"
+DISCOVER_PATH = "mobile/apps/photos/assets/discover.json"
+DISCOVER_URL = (
+    f"https://raw.githubusercontent.com/ente-io/ente/{ENTE_COMMIT}/{DISCOVER_PATH}"
+)
+DISCOVER_SHA256 = "cf70ac7116b592f7921ca4cf67cdbb421a8e6b287fbe44b2be678b8d198e5c24"
+
+TEXT_MODEL = "mobileclip_s2_text_opset18_quant.onnx"
+TEXT_MODEL_SHA256 = "d92f33dfcff83077fc2e0d3414250710efbb51795dfd89767bdbefb5fdc47322"
+IMAGE_MODEL = "mobileclip_s2_image_gelu_opset20.onnx"
+IMAGE_MODEL_SHA256 = "205a430af825e501c5138e5bb9abea942482a7a4fd4a680e98e47cf0830dce7e"
+
+# The CLIP tokenizer Hugging Face publishes for OpenAI's CLIP. Its vocabulary
+# and merges are those of bpe_simple_vocab_16e6.txt, the vocabulary Ente's
+# apps tokenize with.
+TOKENIZER_URL = (
+    "https://huggingface.co/openai/clip-vit-base-patch32/resolve/"
+    "3d74acf9a28c67741b2f4f2ea7635f0aaf6f0268/tokenizer.json"
+)
+TOKENIZER_SHA256 = "b556ac8c99757ffb677208af34bc8c6721572114111a6e0aaf5fa69ff0b8d842"
+
+# "Sunset off the Massachusetts coast", a NOAA photo in the public domain:
+# https://commons.wikimedia.org/wiki/File:Sunset_off_the_Massachusetts_coast_-_NOAA.jpg
+PHOTO_URL = (
+    "https://upload.wikimedia.org/wikipedia/commons/d/db/"
+    "Sunset_off_the_Massachusetts_coast_-_NOAA.jpg"
+)
+PHOTO_SHA256 = "be62f3289c9bf7b75af68cd6978021d54e8a531d5d4cc493f6e9fbf5e1ab74d3"
+
+# Ente's text model takes 77 token ids, padded with 0; its image model takes
+# a 256x256 RGB image.
+TOKEN_COUNT = 77
+IMAGE_SIZE = 256
+
+
+def download(url, sha256):
+    # Wikimedia refuses requests without a User-Agent.
+    request = urllib.request.Request(url, headers={"User-Agent": "quak"})
+    with urllib.request.urlopen(request) as response:
+        data = response.read()
+    digest = hashlib.sha256(data).hexdigest()
+    if digest != sha256:
+        raise SystemExit(f"{url} has sha256 {digest}, not {sha256}")
+    return data
+
+
+def ente_model(name, sha256):
+    return onnxruntime.InferenceSession(
+        download(f"https://models.ente.com/{name}", sha256),
+        providers=["CPUExecutionProvider"],
+    )
+
+
+def run(session, tensor):
+    (output,) = session.run(None, {session.get_inputs()[0].name: tensor})
+    embedding = output[0]
+    return embedding / np.linalg.norm(embedding)
+
+
+def text_embedding(session, tokenizer, query):
+    ids = tokenizer.encode(query).ids
+    ids += [0] * (TOKEN_COUNT - len(ids))
+    return run(session, np.array([ids], dtype=np.int32))
+
+
+def image_embedding(session, photo):
+    image = ImageOps.exif_transpose(Image.open(io.BytesIO(photo))).convert("RGB")
+    # Scale to cover IMAGE_SIZE x IMAGE_SIZE and crop the centre.
+    square = ImageOps.fit(image, (IMAGE_SIZE, IMAGE_SIZE), Image.Resampling.BILINEAR)
+    pixels = np.asarray(square, dtype=np.float32) / 255
+    return run(session, pixels.transpose(2, 0, 1)[np.newaxis])
+
+
+def numbers(embedding):
+    return ", ".join(str(value) for value in embedding)
+
+
+def write_categories(prompts, embeddings):
+    lines = [
+        "// Written by bin/clip-keyword-vectors; do not edit by hand.",
+        "//",
+        "// The Discover categories of Ente's photos app, in their order, from",
+        f"// {DISCOVER_PATH} in https://github.com/ente-io/ente",
+        f"// at commit {ENTE_COMMIT}. Each has the text",
+        f"// embedding Ente's CLIP text model ({TEXT_MODEL})",
+        "// gives its query, scaled to length 1. `showVideo` is false for a category",
+        "// that leaves out videos.",
+        "",
+        "export const DISCOVER_CATEGORIES = [",
+    ]
+    for prompt, embedding in zip(prompts, embeddings):
+        lines += [
+            "    {",
+            f"        title: {json.dumps(prompt['title'])},",
+            f"        query: {json.dumps(prompt['query'])},",
+            f"        minScore: {json.dumps(prompt['minScore'])},",
+            f"        showVideo: {json.dumps(prompt.get('showVideo', True))},",
+            f"        textEmbedding: [{numbers(embedding)}],",
+            "    },",
+        ]
+    lines.append("];")
+    with open("src/clip-keyword-vectors.ts", "w", encoding="utf-8") as file:
+        file.write("\n".join(lines) + "\n")
+
+
+def write_fixture(embedding):
+    with open("test/clip-keywords-sunset.json", "w", encoding="utf-8") as file:
+        file.write(
+            "{\n"
+            f'    "photo": {json.dumps(PHOTO_URL)},\n'
+            f'    "sha256": {json.dumps(PHOTO_SHA256)},\n'
+            f'    "embedding": [{numbers(embedding)}]\n'
+            "}\n"
+        )
+
+
+def main():
+    prompts = json.loads(download(DISCOVER_URL, DISCOVER_SHA256))["prompts"]
+    tokenizer = Tokenizer.from_str(
+        download(TOKENIZER_URL, TOKENIZER_SHA256).decode("utf-8")
+    )
+    text_model = ente_model(TEXT_MODEL, TEXT_MODEL_SHA256)
+    embeddings = [
+        text_embedding(text_model, tokenizer, prompt["query"]) for prompt in prompts
+    ]
+    write_categories(prompts, embeddings)
+
+    image_model = ente_model(IMAGE_MODEL, IMAGE_MODEL_SHA256)
+    write_fixture(image_embedding(image_model, download(PHOTO_URL, PHOTO_SHA256)))
+
+
+if __name__ == "__main__":
+    main()

bin/quak.ts

@@ -2,8 +2,9 @@
 
 import { stdout, stderr } from "node:process";
 import { input, password } from "@inquirer/prompts";
-import { Command } from "commander";
+import { Command, Option } from "commander";
 import envPaths from "env-paths";
+import { logLevel, logRequests, stderrLog, type Log } from "../src/cli-log.js";
 import { init } from "../src/crypto/index.js";
 import {
     type CliContext,
@@ -29,6 +30,9 @@ const paths = envPaths("quak", { suffix: "" });
 
 const program = new Command();
 
+// Program options, so they are taken before or after any command's name.
+// `-v` counts its repeats, the way commander's documentation counts a repeated
+// option.
 program
     .name("quak")
     .description("CLI for the Ente end-to-end encrypted photo service")
@@ -37,21 +41,46 @@ program
         "--cache-dir <path>",
         "Directory for the local metadata/content cache " +
             "(default: the per-user cache directory)",
-    );
+    )
+    .option(
+        "-v, --verbose",
+        "Log more on stderr: debug lines with -v, trace lines too with -vv",
+        (_value: string, previous: number) => previous + 1,
+        0,
+    )
+    .addOption(
+        new Option(
+            "-q, --quiet",
+            "Print nothing on stderr but the error a failing command exits with",
+        ).conflicts("verbose"),
+    )
+    .configureHelp({ showGlobalOptions: true });
+
+interface ProgramOptions {
+    cacheDir?: string;
+    verbose: number;
+    quiet?: boolean;
+}
+
+// The log for this run, at the level `-v` and `-q` ask for.
+const log = (): Log => {
+    const { verbose, quiet } = program.opts<ProgramOptions>();
+    return stderrLog(stderr, logLevel(verbose, quiet ?? false));
+};
 
 const context = (): CliContext => ({
     stdout,
-    stderr,
+    log: log(),
     sessionDir: paths.data,
-    cacheDir: program.opts<{ cacheDir?: string }>().cacheDir,
+    cacheDir: program.opts<ProgramOptions>().cacheDir,
-    loadSession,
+    loadSession: (path) => loadSession(path, logRequests(log())),
-    login: (opts) => Client.login(opts),
+    login: (opts) => Client.login({ ...opts, apiOptions: logRequests(log()) }),
     prompt: (message) => input({ message }),
     promptSecret: (message) => password({ message, mask: true }),
 });
 
 const run = (command: Promise<number>): Promise<void> =>
-    runCommand(command, stdout, stderr, (code) => process.exit(code));
+    runCommand(command, stdout, stderr, log(), (code) => process.exit(code));
 
 program
     .command("login")
@@ -78,14 +107,15 @@ program
 
 program
     .command("files")
-    .description("List files in a collection")
+    .description("List the files in a collection, or every file in the account")
-    .requiredOption(
+    .argument(
-        "--collection <id>",
+        "[collectionID]",
-        "Collection ID (from `quak collections`)",
+        "Collection ID (from `quak collections`); without it, every file " +
+            "is listed once",
     )
     .option("--json", "Output as JSON array")
-    .action((opts: { collection: string; json?: boolean }) =>
+    .action((collectionID: string | undefined, opts: { json?: boolean }) =>
-        run(filesCommand(context(), opts)),
+        run(filesCommand(context(), collectionID, opts)),
     );
 
 program
@@ -93,7 +123,6 @@ program
     .description("Download and decrypt a single file")
     .argument("<fileID>", "File ID (from `quak files`)")
     .option("--out <path>", "Output file path")
-    .option("--collection <id>", "Accepted for compatibility; ignored")
     .action((fileID: string, opts: { out?: string }) =>
         run(getCommand(context(), fileID, opts)),
     );
@@ -103,7 +132,6 @@ program
     .description("Download and decrypt a thumbnail")
     .argument("<fileID>", "File ID (from `quak files`)")
     .option("--out <path>", "Output file path")
-    .option("--collection <id>", "Accepted for compatibility; ignored")
     .action((fileID: string, opts: { out?: string }) =>
         run(getThumbCommand(context(), fileID, opts)),
     );
@@ -142,7 +170,10 @@ program
                 {
                     ...context(),
                     loadSession: (path) =>
-                        loadSession(path, { retry: UNATTENDED_RETRY_OPTIONS }),
+                        loadSession(
+                            path,
+                            logRequests(log(), UNATTENDED_RETRY_OPTIONS),
+                        ),
                 },
                 dir,
                 opts,

package.json

@@ -12,7 +12,7 @@
     "private": true,
     "type": "module",
     "engines": {
-        "node": ">=22"
+        "node": "^22.13.0 || >=23.5.0"
     },
     "main": "./dist/src/index.js",
     "types": "./dist/src/index.d.ts",

script/build-bin

@@ -0,0 +1,85 @@
+#!/bin/sh
+# script/build-bin: run script/build, then bundle the built dist/ into one
+# executable, bin/quak, which reports the version script/build stamped. Our own
+# extension to scripts-to-rule-them-all.
+#
+# bun compiles the binary from a copy of its own executable, so the binary
+# loads the same libraries as the bun that built it. nix's bun loads libraries
+# from the nix store: on Linux its C library, on macOS its ICU. A binary
+# compiled from it loads them too. nix-collect-garbage can delete them, and the
+# binary then no longer starts. This script therefore compiles with bun's own
+# release build, which loads only the system's libraries. It downloads that
+# build and checks it against the hash below before running it.
+set -eu
+
+ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
+
+# bun 1.3.13, 2026-11-11, and the SHA-256 of each release archive below. The
+# x86-64 archives are the baseline builds, which also run on CPUs without AVX2.
+# The musl archives are for Linux systems that use the musl C library, such as
+# Alpine.
+BUN_VERSION="1.3.13"
+BUN_LINUX_X64_BASELINE_SHA256="9d8a24292a7068090205daac0a5a223f5f69736f5287e37bf88d3b4031edc750"
+BUN_LINUX_X64_MUSL_BASELINE_SHA256="88ca7c7ad235b498f549eea2f770f434e9f0f5e9ba95168a2d3a1f235184c394"
+BUN_LINUX_AARCH64_SHA256="70bae41b3908b0a120e1e58c5c8af30e74afae3b8d11b0d3fdd8e787ddfb4b22"
+BUN_LINUX_AARCH64_MUSL_SHA256="5385e978107ce4934298d8d6afe9bfbb898683f6cc23e6753a0da60bc60c5b81"
+BUN_DARWIN_X64_BASELINE_SHA256="a98ba6a480f22fda9b343626b906a4e26aa53618bf85d2bc5928ecf2ba45f0ed"
+BUN_DARWIN_AARCH64_SHA256="5467e3f65dba526b9fea98f0cce04efafc0c63e169733ec27b876a3ad32da190"
+
+# compile_with_downloaded_bun <archive-name> <sha256>
+compile_with_downloaded_bun() {
+    tmp="$(mktemp -d)"
+    trap 'rm -rf "$tmp"' EXIT
+    curl -fsSL -o "$tmp/$1.zip" \
+        "https://github.com/oven-sh/bun/releases/download/bun-v$BUN_VERSION/$1.zip"
+    # macOS always has shasum, but not every macOS has sha256sum.
+    if command -v sha256sum >/dev/null 2>&1; then
+        echo "$2  $tmp/$1.zip" | sha256sum -c -
+    else
+        echo "$2  $tmp/$1.zip" | shasum -a 256 -c -
+    fi
+    unzip -q "$tmp/$1.zip" -d "$tmp"
+    "$tmp/$1/bun" build dist/bin/quak.js --compile --outfile bin/quak
+}
+
+main() {
+    cd "$ROOT"
+    "$ROOT/script/build"
+    platform="$(uname -s)-$(uname -m)"
+    # musl's ldd begins its version output with "musl libc"; glibc's does not.
+    if ldd --version 2>&1 | grep -q '^musl libc'; then
+        platform="$platform-musl"
+    fi
+    case "$platform" in
+        Linux-x86_64)
+            compile_with_downloaded_bun bun-linux-x64-baseline \
+                "$BUN_LINUX_X64_BASELINE_SHA256"
+            ;;
+        Linux-x86_64-musl)
+            compile_with_downloaded_bun bun-linux-x64-musl-baseline \
+                "$BUN_LINUX_X64_MUSL_BASELINE_SHA256"
+            ;;
+        Linux-aarch64)
+            compile_with_downloaded_bun bun-linux-aarch64 \
+                "$BUN_LINUX_AARCH64_SHA256"
+            ;;
+        Linux-aarch64-musl)
+            compile_with_downloaded_bun bun-linux-aarch64-musl \
+                "$BUN_LINUX_AARCH64_MUSL_SHA256"
+            ;;
+        Darwin-x86_64)
+            compile_with_downloaded_bun bun-darwin-x64-baseline \
+                "$BUN_DARWIN_X64_BASELINE_SHA256"
+            ;;
+        Darwin-arm64)
+            compile_with_downloaded_bun bun-darwin-aarch64 \
+                "$BUN_DARWIN_AARCH64_SHA256"
+            ;;
+        *)
+            echo "build-bin: bun has no release build for $(uname -s) on $(uname -m)" >&2
+            exit 1
+            ;;
+    esac
+}
+
+main "$@"

src/api/client.ts

@@ -36,6 +36,15 @@ export interface ApiClientOptions {
     retry?: RetryOptions;
     requestTimeoutMs?: number;
     downloadTimeoutMs?: number;
+    // Called once for each request sent, with the status of its response or
+    // the error it failed with, for a caller that logs requests. A request
+    // that is retried is reported once for each attempt.
+    onRequest?: (request: {
+        method: string;
+        url: string;
+        status?: number;
+        error?: unknown;
+    }) => void;
 }
 
 export interface StreamOptions {
@@ -159,6 +168,7 @@ export class ApiClient {
     private readonly retry: ResolvedRetryOptions;
     private readonly requestTimeoutMs: number;
     private readonly downloadTimeoutMs: number;
+    private readonly onRequest: NonNullable<ApiClientOptions["onRequest"]>;
     private token: string | undefined;
 
     constructor(opts?: ApiClientOptions) {
@@ -180,6 +190,7 @@ export class ApiClient {
             opts?.requestTimeoutMs ?? DEFAULT_REQUEST_TIMEOUT_MS;
         this.downloadTimeoutMs =
             opts?.downloadTimeoutMs ?? DEFAULT_DOWNLOAD_TIMEOUT_MS;
+        this.onRequest = opts?.onRequest ?? (() => {});
         this.token = opts?.authToken;
     }
 
@@ -214,6 +225,33 @@ export class ApiClient {
         return h;
     }
 
+    // Send one request and report it to `onRequest`, with the status of its
+    // response or the error it failed with. `reportedURL` is the URL given to
+    // `onRequest`, when that must not be the one sent.
+    private async send(
+        url: string,
+        init: RequestInit & { method: string },
+        reportedURL = url,
+    ): Promise<Response> {
+        let resp: Response;
+        try {
+            resp = await this._fetch(url, init);
+        } catch (err) {
+            this.onRequest({
+                method: init.method,
+                url: reportedURL,
+                error: err,
+            });
+            throw err;
+        }
+        this.onRequest({
+            method: init.method,
+            url: reportedURL,
+            status: resp.status,
+        });
+        return resp;
+    }
+
     private async throwIfError(resp: Response): Promise<void> {
         if (resp.ok) return;
         const requestID = resp.headers.get("x-request-id") ?? undefined;
@@ -257,7 +295,7 @@ export class ApiClient {
         const url = buildURL(this.apiOrigin, path, query);
         // A GET changes nothing, so it is retried under the full policy.
         return withRetry(async () => {
-            const resp = await this._fetch(url, {
+            const resp = await this.send(url, {
                 method: "GET",
                 headers: this.headers(),
                 signal: AbortSignal.timeout(this.requestTimeoutMs),
@@ -280,7 +318,7 @@ export class ApiClient {
         // 3xx status, which is not retried.
         return withRetry(
             async () => {
-                const resp = await this._fetch(url, {
+                const resp = await this.send(url, {
                     method: "POST",
                     headers: this.headers({
                         "Content-Type": "application/json",
@@ -316,21 +354,35 @@ export class ApiClient {
         });
     }
 
-    async putFile(presignedURL: string, data: Uint8Array): Promise<void> {
+    // `contentMD5` is the value given to `getUploadURL`. The server signs it
+    // into the URL, and storage refuses a PUT without that `Content-MD5`.
+    async putFile(
+        presignedURL: string,
+        data: Uint8Array,
+        contentMD5: string,
+    ): Promise<void> {
         // Idempotent despite being a write: a presigned PUT stores one whole
         // object at one key in one request, so replaying it either overwrites
         // the same bytes or lands them for the first time. There is no partial
         // state to protect, hence the full policy rather than the POST rule.
+        // It is reported without its query, which holds the upload's
+        // signature: a log keeps what it is given.
+        const { origin, pathname } = new URL(presignedURL);
         await withRetry(async () => {
-            const resp = await this._fetch(presignedURL, {
+            const resp = await this.send(
+                presignedURL,
+                {
                     method: "PUT",
                     headers: {
                         "Content-Type": "application/octet-stream",
                         "Content-Length": String(data.length),
+                        "Content-MD5": contentMD5,
                     },
                     body: data,
                     signal: AbortSignal.timeout(this.requestTimeoutMs),
-            });
+                },
+                origin + pathname,
+            );
             if (!resp.ok) {
                 // An ApiError, not a bare Error: without the status on the
                 // error the upload path cannot be classified at all, and a
@@ -348,7 +400,7 @@ export class ApiClient {
         // Same replay and redirect rules as `postJSON`, for the same reasons.
         return withRetry(
             async () => {
-                const resp = await this._fetch(url, {
+                const resp = await this.send(url, {
                     method: "PUT",
                     headers: this.headers({
                         "Content-Type": "application/json",
@@ -394,7 +446,7 @@ export class ApiClient {
             // headers, when no bytes have arrived either.
             const deadline = idleDeadline(this.downloadTimeoutMs);
             try {
-                const resp = await this._fetch(url, {
+                const resp = await this.send(url, {
                     method: "GET",
                     headers: this.headers(),
                     signal: deadline.signal,

src/backup.ts

@@ -14,9 +14,9 @@
 //         YYYY/YYYY-MM/YYYY-MM-DD/
 //             YYYY-MM-DD.<fileID>.<ext>    the decrypted bytes (the save path)
 //             YYYY-MM-DD.<fileID>.json     per-file metadata sidecar, with
-//                                          the file's ML data and its
+//                                          the file's ML data, its
-//                                          original's EXIF, XMP and
+//                                          keywords, and its original's
-//                                          dimensions
+//                                          EXIF, XMP and dimensions
 //         collections/<name>/<title>       symlink to the original
 //         collections/<name>.json          per-collection metadata
 //         account.json                     the account's email and user ID
@@ -41,8 +41,9 @@
 //
 // With `verify`, each original already at its save path is hashed as the
 // download check hashes it, and one that does not match the content hash its
-// metadata records is removed and fetched again in the same run. What is put
+// metadata records is replaced once a matching copy is downloaded, in the same
-// back is hashed too, and recorded as failed if it still does not match.
+// run. Until then it stays as it is: a download that fails, or that does not
+// match either, leaves it in place and is recorded as failed.
 //
 // Resilience (issue #8): no per-file condition aborts the run. A failed
 // download, a failed symlink, or ML data missing because the ML data fetch
@@ -50,7 +51,9 @@
 // running attempt count, and the last-tried time, and the run continues.
 // A file the library could not decrypt is recorded there too, unless a copy of
 // it in another album decrypted; an album it could not decrypt is logged by ID
-// and listed in `result.undecryptableAlbums`, as its name is not known.
+// and listed in `result.undecryptableAlbums`, as its name is not known. An
+// album whose folder or JSON cannot be written is logged by name and
+// listed in `result.failedAlbums`, and the run goes on to the next album.
 // `result.failed` — and thus the CLI's exit code — stays non-zero while any
 // failure remains unresolved and clears once every one succeeds. Each run
 // reconciles the ledger against the files it attempted, so an entry for a file
@@ -59,7 +62,6 @@
 // code.
 
 import {
-    createReadStream,
     lstatSync,
     mkdirSync,
     readdirSync,
@@ -75,27 +77,35 @@ import { readFile } from "node:fs/promises";
 import { dirname, extname, join, relative, resolve } from "node:path";
 import lockfile from "proper-lockfile";
 
-import {
+import { clipKeywords } from "./clip-keywords.js";
-    chunkHashFinal,
-    chunkHashInit,
-    chunkHashUpdate,
-    init,
-} from "./crypto/index.js";
 import { removeLeftoverTempFiles } from "./download/index.js";
-import { sanitizeFileName, withExtension } from "./filename.js";
+import { describeError } from "./errors.js";
+import {
+    MAX_NAME_BYTES,
+    sanitizeFileName,
+    shortenKeepingExtension,
+    shortenToBytes,
+    withExtension,
+} from "./filename.js";
 import {
     copyAtomic,
     placeOriginal,
+    replaceOriginal,
     savePath,
     storedAtSavePath,
+    storedHash,
 } from "./library/content.js";
 import { representative } from "./library/records.js";
 import type { Undecryptable } from "./library/store.js";
 import { extractImageMetadata } from "./metadata-backup.js";
 import type { MLData } from "./mldata-fetch.js";
 import type { Collection, EnteFile, FileMetadata } from "./model/types.js";
+import { hasTransportCode } from "./retry.js";
 
-export type ProgressCallback = (message: string) => void;
+// Reports one line of the backup's progress. `level` is "error" when the line
+// reports something that failed, such as a file or an album, and is left out
+// for every other line.
+export type ProgressCallback = (message: string, level?: "error") => void;
 
 // The lock in a backup directory, as `lockBackupDirectory` returns it.
 export interface BackupLock {
@@ -143,9 +153,9 @@ export interface BackupResult {
     // Originals already at their save path and left untouched.
     skipped: number;
     // With `verify`, the originals already at their save path whose hash
-    // matched, those whose hash did not (each removed and fetched again), and
+    // matched, those whose hash did not (each replaced once a matching copy is
-    // those whose metadata records no hash (left as they are). All three are
+    // downloaded), and those whose metadata records no hash (left as they
-    // zero without `verify`.
+    // are). All three are zero without `verify`.
     verified: number;
     mismatched: number;
     unchecked: number;
@@ -158,6 +168,9 @@ export interface BackupResult {
     // The IDs of the albums the library could not decrypt, whose files it
     // could not list; the CLI exits non-zero while there are any.
     undecryptableAlbums: number[];
+    // The IDs of the albums whose folder or JSON this run could not write;
+    // the CLI exits non-zero while there are any.
+    failedAlbums: number[];
 }
 
 // The slice of the library that backup drives. `Library` implements it; a test
@@ -180,6 +193,15 @@ export interface BackupLibrary {
         fileID: number,
         destination: string,
     ): Promise<{ path: string; videoPath?: string }>;
+    // Fetch an original's bytes to `destination` through the content pool,
+    // a live photo as its image and its video beside it, even when the cache
+    // or the library's save path already holds it, and return where they
+    // landed. With `verify`, an original whose hash does not match is replaced
+    // with what this fetches.
+    fetchOriginal(
+        fileID: number,
+        destination: string,
+    ): Promise<{ path: string; videoPath?: string }>;
     thumbnail(fileID: number): Promise<{ path: string }>;
     // How many originals the backup works on at once: the limit of the
     // library's content pool.
@@ -191,6 +213,11 @@ export interface BackupLibrary {
     // A file's cached ML data, as `lib.mldata.forFile()` returns it, or
     // undefined when none is cached.
     mlData(fileID: number): Promise<MLData | undefined>;
+    // Why the ML data the last ML data fetch that succeeded got for a file did
+    // not decrypt, gunzip or parse, or undefined when it did or the fetch got
+    // none. A fetch still running does not change it. A fetch that fails
+    // leaves it as it was.
+    mlDataError(fileID: number): unknown;
 }
 
 type FailureClass = "transient" | "permanent" | "unknown";
@@ -219,11 +246,16 @@ const isPresent = (path: string): boolean => {
 
 // Best-effort classification for the ledger. Retryable server/network problems
 // are transient; refusals and local filesystem/decrypt errors are permanent;
-// anything else is unknown. Both the error code and message are inspected.
+// anything else is unknown. The error code and the error as `describeError`
+// gives it are inspected, and so is each errno on the error's cause chain,
+// where Node's `fetch` puts it. An error named `TimeoutError` is transient
+// whatever its message says, as `isRetryable` retries it: each of quak's
+// deadlines aborts with one, and the download's says only that it stalled.
 const classify = (err: unknown): FailureClass => {
+    if (hasTransportCode(err)) return "transient";
     const e = err as NodeJS.ErrnoException;
-    const text =
+    if (e?.name === "TimeoutError") return "transient";
-        `${e?.code ?? ""} ${err instanceof Error ? err.message : String(err)}`.toLowerCase();
+    const text = `${e?.code ?? ""} ${describeError(err)}`.toLowerCase();
     if (
         /timeout|timed out|econnreset|econnrefused|econnaborted|network|socket|eai_again|throttl|temporarily|429|500|502|503|504/.test(
             text,
@@ -241,9 +273,6 @@ const classify = (err: unknown): FailureClass => {
     return "unknown";
 };
 
-const errorMessage = (err: unknown): string =>
-    err instanceof Error ? err.message : String(err);
-
 // Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or
 // non-symlink entry. Throws on failure (a directory in the way, no permission)
 // so the caller records it and moves on rather than aborting the run.
@@ -263,31 +292,46 @@ const rebuildSymlink = (linkPath: string, target: string): void => {
 // The on-disk names for the entries of one directory, in entry order. Each
 // name is used as is unless another entry would get the same name, ignoring
 // case (two names that differ only in case are one entry on a case-insensitive
-// file system); then every entry sharing it gets ` (<id>)`, before the
+// file system), or it is too long for a file name; then it gets ` (<id>)`,
-// extension when `beforeExtension` is set. A name with an ID added can match
+// right before the extension when `beforeExtension` is set. To fit, the part
-// another entry's own name (`IMG (6).JPG`), so this repeats until no name is
+// before the ID is cut short as far as it must be. When the extension leaves
-// shared. IDs are stable, so the names are too.
+// no room for the ID and the first character of the name, the extension is
+// cut short, and the part before the ID keeps that character and any bytes
+// the cut leaves over (see `shortenKeepingExtension`). With `withJSON`, each
+// entry also has the JSON file `<name>.json` beside it, as an album does: a
+// name then leaves room for `.json`, and one that is another entry's JSON name
+// gets its ID added too.
+// Two names with different IDs added never match, but one can match another
+// entry's own name (`IMG (6).JPG`), so this repeats until no name is shared.
+// IDs are stable, so the names are too.
 const uniqueNames = (
     entries: { id: number; name: string }[],
-    beforeExtension: boolean,
+    opts: { beforeExtension?: boolean; withJSON?: boolean },
 ): string[] => {
+    const maxBytes = MAX_NAME_BYTES - (opts.withJSON ? ".json".length : 0);
     const withID = (id: number, name: string): string => {
-        const ext = beforeExtension ? extname(name) : "";
+        const tag = ` (${id})`;
-        const stem = name.slice(0, name.length - ext.length);
+        return opts.beforeExtension
-        return `${stem} (${id})${ext}`;
+            ? shortenKeepingExtension(name, maxBytes, tag)
+            : shortenToBytes(name, maxBytes - Buffer.byteLength(tag)) + tag;
     };
     const names = entries.map((e) => e.name);
     const suffixed = new Set<number>();
     for (;;) {
         const counts = new Map<string, number>();
-        for (const name of names) {
+        const count = (name: string): void => {
             const key = name.toLowerCase();
             counts.set(key, (counts.get(key) ?? 0) + 1);
+        };
+        for (const name of names) {
+            count(name);
+            if (opts.withJSON) count(`${name}.json`);
         }
         let changed = false;
         for (const [i, { id, name }] of entries.entries()) {
             if (suffixed.has(i)) continue;
-            if (counts.get(name.toLowerCase()) === 1) continue;
+            const fits = Buffer.byteLength(name) <= maxBytes;
+            if (fits && counts.get(name.toLowerCase()) === 1) continue;
             names[i] = withID(id, name);
             suffixed.add(i);
             changed = true;
@@ -414,26 +458,6 @@ const saveLedger = (path: string, ledger: Map<number, FailureEntry>): void => {
     );
 };
 
-// The content hash of the original stored at `stored`, computed as the download
-// check computes it: over each file's bytes, read in chunks, and for a live
-// photo `<imageHash>:<videoHash>`.
-const storedHash = async (stored: {
-    path: string;
-    videoPath?: string;
-}): Promise<string> => {
-    await init();
-    const hashFile = async (path: string): Promise<string> => {
-        const state = chunkHashInit();
-        for await (const chunk of createReadStream(path)) {
-            chunkHashUpdate(state, chunk as Buffer);
-        }
-        return chunkHashFinal(state);
-    };
-    const hash = await hashFile(stored.path);
-    if (stored.videoPath === undefined) return hash;
-    return `${hash}:${await hashFile(stored.videoPath)}`;
-};
-
 // A file's EXIF, XMP and dimensions as its JSON holds them: what
 // `extractImageMetadata` found in its original, or why the original could not
 // be read.
@@ -470,12 +494,13 @@ const imageMetadataFor = async (
         const bytes = await readFile(originalPath);
         return { imageMetadata: extractImageMetadata(bytes) ?? {} };
     } catch (err) {
-        return { imageMetadataError: errorMessage(err) };
+        return { imageMetadataError: describeError(err) };
     }
 };
 
-// The file's JSON: its basic fields, its magic metadata, its ML data or the
+// The file's JSON: its basic fields, its magic metadata, its ML data without
-// reason the ML data is missing, and its image metadata.
+// the CLIP embedding and the keywords that embedding gives, or the reason the
+// ML data is missing, and its image metadata.
 const writeSidecar = (
     path: string,
     file: EnteFile,
@@ -491,7 +516,13 @@ const writeSidecar = (
     };
     if (file.magicMetadata) meta.magicMetadata = file.magicMetadata;
     if (file.pubMagicMetadata) meta.pubMagicMetadata = file.pubMagicMetadata;
-    if (ml.mlData) meta.mlData = ml.mlData;
+    if (ml.mlData) {
+        const { clip: _clip, ...mlDataWithoutClip } = ml.mlData;
+        meta.mlData = mlDataWithoutClip;
+        const isVideo = file.metadata.fileType === "video";
+        const keywords = clipKeywords(ml.mlData, isVideo);
+        if (keywords) meta.keywords = keywords;
+    }
     if (ml.mlDataError) meta.mlDataError = ml.mlDataError;
     if (image.imageMetadata) meta.imageMetadata = image.imageMetadata;
     if (image.imageMetadataError) {
@@ -610,7 +641,7 @@ const runLockedBackup = async (
         // and thumbnail both fail this run must not double its attempt count
         // or appear twice in errors.
         if (failedThisRun.has(fileID)) return;
-        const error = errorMessage(err);
+        const error = describeError(err);
         errors.push({ fileID, title, collection, error });
         const prior = ledger.get(fileID);
         ledger.set(fileID, {
@@ -634,7 +665,7 @@ const runLockedBackup = async (
     for (const record of lib.undecryptable()) {
         const error = `could not be decrypted: ${record.error}`;
         if (record.fileID === undefined) {
-            log(`FAILED album ${record.collectionID}: ${error}`);
+            log(`FAILED album ${record.collectionID}: ${error}`, "error");
             undecryptableAlbums.push(record.collectionID);
         } else if (scope.has(record.collectionID)) {
             const copies = memberships.get(record.fileID);
@@ -655,11 +686,13 @@ const runLockedBackup = async (
     // through the content cache/pools, as `Photo.download()` does, and fetch
     // the optional thumbnails; a present file is left as is. With `verify`, a
     // present original is hashed first, and one that does not match the hash
-    // its metadata records is removed and fetched like a missing one. One
+    // its metadata records is downloaded again and replaced by
-    // that cannot be read is recorded as failed and left where it is.
+    // `replaceOriginal`, which leaves it as it was when that download fails
-    // `storeOriginal` does this for one original, and records its failure.
+    // or does not match either. One that cannot be read is recorded as failed
+    // and left where it is. `storeOriginal` does this for one original, and
+    // records its failure.
     const storeOriginal = async (file: EnteFile): Promise<void> => {
-        let stored = storedAtSavePath(downloadDirectory, file);
+        const stored = storedAtSavePath(downloadDirectory, file);
         let mismatch = false;
         if (stored !== undefined && verify) {
             try {
@@ -673,15 +706,11 @@ const runLockedBackup = async (
                     );
                     mismatched++;
                     mismatch = true;
-                    rmSync(stored.path);
-                    if (stored.videoPath !== undefined) {
-                        rmSync(stored.videoPath);
-                    }
-                    stored = undefined;
                 }
             } catch (err) {
                 log(
-                    `FAILED verifying original ${file.metadata.title}: ${errorMessage(err)}`,
+                    `FAILED verifying original ${file.metadata.title}: ${describeError(err)}`,
+                    "error",
                 );
                 recordFailure(
                     file.id,
@@ -692,32 +721,31 @@ const runLockedBackup = async (
                 return;
             }
         }
-        if (stored !== undefined) {
+        if (stored !== undefined && !mismatch) {
             skipped++;
             return;
         }
         try {
             log(`Fetching original ${file.metadata.title} (${file.id})...`);
+            if (mismatch) {
+                await replaceOriginal(downloadDirectory, file, (dest) =>
+                    lib.fetchOriginal(file.id, dest),
+                );
+            } else {
                 // A fetched original is written straight to its save path (a
                 // live photo beside it); only one that was already cached
                 // elsewhere is copied.
-            const placed = await placeOriginal(
+                await placeOriginal(downloadDirectory, file, (dest) =>
-                downloadDirectory,
+                    lib.original(file.id, dest),
-                file,
-                (dest) => lib.original(file.id, dest),
-            );
-            storedThisRun.add(file.id);
-            // A copy from the cache is not checked as a download is and can
-            // hold the same bad bytes, so what is put back after a mismatch
-            // is hashed too. A bad copy stays where it is and the file fails.
-            if (mismatch && (await storedHash(placed)) !== file.metadata.hash) {
-                throw new Error(
-                    "the original put back does not match its content hash either",
                 );
             }
+            storedThisRun.add(file.id);
             downloaded++;
         } catch (err) {
-            log(`FAILED original ${file.metadata.title}: ${errorMessage(err)}`);
+            log(
+                `FAILED original ${file.metadata.title}: ${describeError(err)}`,
+                "error",
+            );
             recordFailure(
                 file.id,
                 file.metadata.title,
@@ -781,17 +809,18 @@ const runLockedBackup = async (
     // Phase 2: rebuild the derived views from the model. Sidecars first, for
     // every present original (this repairs stale ones), each with the file's
     // ML data once an ML data fetch has completed, and its image metadata.
-    // When the fetch fails, a file with no cached ML data gets the reason
+    // When the fetch got a file's ML data but could not decrypt, gunzip or
-    // instead and is recorded as failed. The next run fetches its ML data
+    // parse it, or when the fetch failed, a file with no cached ML data gets
-    // again because none is cached.
+    // the reason instead and is recorded as failed. The next run fetches its
+    // ML data again because none is cached.
     if (includeOriginals) {
-        let mlDataError: string | undefined;
+        let mlDataFailure: unknown;
         try {
             log("Fetching ML data...");
             await lib.fetchMLData();
         } catch (err) {
-            mlDataError = errorMessage(err);
+            mlDataFailure = err;
-            log(`FAILED ML data: ${mlDataError}`);
+            log(`FAILED ML data: ${describeError(err)}`, "error");
         }
         for (const file of distinct.values()) {
             if (lock.lost) throw lock.lost;
@@ -808,13 +837,19 @@ const runLockedBackup = async (
                 storedThisRun.has(file.id),
             );
             const mlData = await lib.mlData(file.id);
-            if (mlData === undefined && mlDataError !== undefined) {
+            const failure = lib.mlDataError(file.id) ?? mlDataFailure;
-                writeSidecar(path, file, { mlDataError }, image);
+            if (mlData === undefined && failure !== undefined) {
+                writeSidecar(
+                    path,
+                    file,
+                    { mlDataError: describeError(failure) },
+                    image,
+                );
                 recordFailure(
                     file.id,
                     file.metadata.title,
                     collectionName.get(file.collectionID) ?? "",
-                    new Error(`ML data: ${mlDataError}`),
+                    new Error("ML data", { cause: failure }),
                 );
             } else {
                 writeSidecar(path, file, { mlData }, image);
@@ -833,7 +868,7 @@ const runLockedBackup = async (
             id: c.id,
             name: sanitizeFileName(c.name, `collection-${c.id}`),
         })),
-        false,
+        { withJSON: true },
     );
     const albumDirNames = new Map(
         allCollections.map((c, i) => [c.id, dirNames[i]!]),
@@ -846,17 +881,25 @@ const runLockedBackup = async (
             downloadDirectory,
         );
     } catch (err) {
-        log(`FAILED removing old album directories: ${errorMessage(err)}`);
+        log(
+            `FAILED removing old album directories: ${describeError(err)}`,
+            "error",
+        );
     }
 
+    // An album whose folder or JSON cannot be written, such as one whose
+    // folder name is taken by a file the user put there, is logged and listed
+    // in the result, and the run goes on to the next album.
+    const failedAlbums: number[] = [];
     for (const c of collections) {
         if (lock.lost) throw lock.lost;
+        try {
             const colDirName = albumDirNames.get(c.id)!;
             const colDir = join(collectionsDir, colDirName);
             mkdirSync(colDir, { recursive: true });
 
-        // Every album links the one original, saved from the file's entry in
+            // Every album links the one original, saved from the file's entry
-        // `distinct`.
+            // in `distinct`.
             const files = filesByCollection.get(c.id) ?? [];
             const links = files.flatMap((f) =>
                 linksFor(
@@ -864,11 +907,14 @@ const runLockedBackup = async (
                     storedAtSavePath(downloadDirectory, distinct.get(f.id)!),
                 ),
             );
-        const linkNames = uniqueNames(links, true);
+            const linkNames = uniqueNames(links, { beforeExtension: true });
             try {
                 removeStaleLinks(colDir, new Set(linkNames), downloadDirectory);
             } catch (err) {
-            log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`);
+                log(
+                    `FAILED removing old links in ${c.name}: ${describeError(err)}`,
+                    "error",
+                );
             }
 
             const metaFiles = files.map((f) => ({
@@ -885,7 +931,8 @@ const runLockedBackup = async (
                     );
                 } catch (err) {
                     log(
-                    `FAILED symlink ${c.name}/${linkName}: ${errorMessage(err)}`,
+                        `FAILED symlink ${c.name}/${linkName}: ${describeError(err)}`,
+                        "error",
                     );
                     recordFailure(
                         link.file.id,
@@ -901,6 +948,10 @@ const runLockedBackup = async (
                 c,
                 metaFiles,
             );
+        } catch (err) {
+            log(`FAILED album ${c.name}: ${describeError(err)}`, "error");
+            failedAlbums.push(c.id);
+        }
     }
 
     // Reconcile the ledger against what this run actually attempted: an entry
@@ -926,6 +977,7 @@ const runLockedBackup = async (
         failed: ledger.size,
         errors,
         undecryptableAlbums,
+        failedAlbums,
     };
 };
 

src/cli-commands.ts

@@ -4,7 +4,10 @@
 // code; a thrown error is left to the caller. Nothing here calls
 // `process.exit`: `bin/quak.ts` wires these to the command line, and `run` in
 // `cli-run.ts` prints a thrown error as one line and exits once output has
-// drained. Output must stay byte-identical (see `cli-output.ts`).
+// drained. Output must stay byte-identical (see `cli-output.ts`). Each line on
+// stderr goes through `ctx.log` at its level (see `cli-log.ts`). Every command
+// that exits non-zero logs a fatal line saying why; one that exits 1 because
+// files or albums failed logs it last, saying how many and where to look.
 
 import {
     copyFileSync,
@@ -33,14 +36,23 @@ import {
     originalName,
     thumbnailName,
 } from "./cli-output.js";
-import { freshCollections, freshFiles, freshFile } from "./cli-read.js";
+import { type Log, logRefresh } from "./cli-log.js";
+import {
+    cachedFilesByID,
+    freshCollections,
+    freshFiles,
+    freshFile,
+    freshFilesByID,
+} from "./cli-read.js";
 import { withExtension } from "./filename.js";
 import { runMetadataBackup } from "./metadata-backup.js";
+import type { Collection, EnteFile } from "./model/types.js";
 import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js";
 
 export interface CliContext {
     stdout: { write(text: string): unknown };
-    stderr: { write(text: string): unknown };
+    // Writes a line on stderr at a log level, if the run's level prints it.
+    log: Log;
     // Directory holding `session.json`.
     sessionDir: string;
     // The `--cache-dir` global, or undefined to let the library pick its
@@ -82,16 +94,16 @@ const requireSession = (ctx: CliContext): Client | undefined => {
     try {
         client = ctx.loadSession(sessionPath(ctx));
     } catch (err) {
-        ctx.stderr.write(
+        ctx.log("fatal", err instanceof Error ? err.message : String(err));
-            `${err instanceof Error ? err.message : err}\n` +
+        ctx.log(
-                `Run "quak logout" and then "quak login" to replace it.\n`,
+            "fatal",
+            `Run "quak logout" and then "quak login" to replace it.`,
         );
         return undefined;
     }
     if (!client) {
-        ctx.stderr.write(
+        ctx.log("fatal", `Not logged in. Run "quak login" first.`);
-            `Not logged in. Run "quak login" first.\nSession file: ${sessionPath(ctx)}\n`,
+        ctx.log("fatal", `Session file: ${sessionPath(ctx)}`);
-        );
         return undefined;
     }
     return client;
@@ -107,7 +119,7 @@ const parseID = (
 ): number | undefined => {
     const id = Number(text);
     if (/^[0-9]+$/.test(text) && Number.isSafeInteger(id)) return id;
-    ctx.stderr.write(`Invalid ${kind} ID: ${text}\n`);
+    ctx.log("fatal", `Invalid ${kind} ID: ${text}`);
     return undefined;
 };
 
@@ -123,17 +135,24 @@ const readLibraryClient = (client: Client): LibraryClient => ({
     contentSource: () => client.contentSource(),
 });
 
-// Open a library for a single point command: the aggressive background precache
+// Open a library for a single point command. The aggressive background
-// (issue #48) is off — a one-shot `collections` or `get` must not start
+// precache (issue #48) is off, because a one-shot `collections` or `get` must
-// downloading the whole account — and the refresh interval is long so no second
+// not start downloading the whole account. No refresh starts as the library
-// refresh fires mid-command.
+// opens or on a timer. Each command asks for the refresh it needs.
+// `collections` and `files` with a collection ID need none (issues #210 and
+// #217). Nor do `get` and `get-thumb` on an empty cache (issue #222). Nor does
+// `helper fix-missing-thumbnails --file` on an empty cache (issue #223). While
+// a refresh runs, each collection whose files it has fetched is logged, so a
+// large account shows progress rather than nothing.
 const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
     Library.open({
         client: readLibraryClient(client),
         cacheDirectory: ctx.cacheDir,
-        refreshIntervalSeconds: 3600,
+        refreshOnOpen: false,
+        refreshIntervalSeconds: Infinity,
         precacheThumbnails: false,
         precacheOriginals: false,
+        onProgress: logRefresh(ctx.log),
     });
 
 export const loginCommand = async (ctx: CliContext): Promise<number> => {
@@ -142,7 +161,7 @@ export const loginCommand = async (ctx: CliContext): Promise<number> => {
     const password =
         process.env.QUAK_PASSWORD ?? (await ctx.promptSecret("Password"));
 
-    ctx.stderr.write("Authenticating...\n");
+    ctx.log("info", "Authenticating...");
     try {
         const client = await ctx.login({
             email,
@@ -153,11 +172,12 @@ export const loginCommand = async (ctx: CliContext): Promise<number> => {
 
         saveSession(ctx.sessionDir, client.toJSON());
         const info = client.whoami();
-        ctx.stderr.write(`Logged in as ${info.email} (user ${info.userID})\n`);
+        ctx.log("info", `Logged in as ${info.email} (user ${info.userID})`);
-        ctx.stderr.write(`Session saved to ${sessionPath(ctx)}\n`);
+        ctx.log("info", `Session saved to ${sessionPath(ctx)}`);
     } catch (err) {
-        ctx.stderr.write(
+        ctx.log(
-            `Login failed: ${err instanceof Error ? err.message : err}\n`,
+            "fatal",
+            `Login failed: ${err instanceof Error ? err.message : err}`,
         );
         return 1;
     }
@@ -179,7 +199,7 @@ export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
 export const logoutCommand = async (ctx: CliContext): Promise<number> => {
     const path = sessionPath(ctx);
     if (!existsSync(path)) {
-        ctx.stderr.write("No session found.\n");
+        ctx.log("info", "No session found.");
         return 0;
     }
     await init();
@@ -197,16 +217,15 @@ export const logoutCommand = async (ctx: CliContext): Promise<number> => {
     }
     unlinkSync(path);
     if (failure === undefined) {
-        ctx.stderr.write("Session ended on the server.\n");
+        ctx.log("info", "Session ended on the server.");
     } else {
-        ctx.stderr.write(
+        ctx.log("fatal", `Could not end the session on the server: ${failure}`);
-            `Could not end the session on the server: ${failure}\n`,
-        );
     }
-    ctx.stderr.write("Session deleted.\n");
+    ctx.log("info", "Session deleted.");
     if (cacheDir !== undefined) {
-        ctx.stderr.write(
+        ctx.log(
-            `Cache directory ${cacheDir} still holds decrypted data; delete it to remove that data.\n`,
+            "warning",
+            `Cache directory ${cacheDir} still holds decrypted data; delete it to remove that data.`,
         );
     }
     return failure === undefined ? 0 : 1;
@@ -221,9 +240,10 @@ export const collectionsCommand = async (
     if (!client) return 3;
     const lib = await openReadLibrary(ctx, client);
     try {
-        // Force a server round-trip and list in enumeration order (issue #36
+        // Ask the server for the collection list alone and list it in its
-        // amendment, issue #52): the pre-library CLI printed current state in
+        // order (issue #36 amendment, issues #52 and #210): the pre-library
-        // this order, not the albums projection's newest-first order.
+        // CLI printed current state in this order, not the albums
+        // projection's newest-first order.
         const collections = await freshCollections(lib);
 
         if (opts.json) {
@@ -254,26 +274,32 @@ export const collectionsCommand = async (
     }
 };
 
+// Lists the files of the collection `collectionIDStr` names, or without it every
+// file in the account, each once (issue #211).
 export const filesCommand = async (
     ctx: CliContext,
-    opts: { collection: string; json?: boolean },
+    collectionIDStr: string | undefined,
+    opts: { json?: boolean },
 ): Promise<number> => {
     await init();
     const client = requireSession(ctx);
     if (!client) return 3;
-    const collectionID = parseID(ctx, "collection", opts.collection);
+    const collectionID =
-    if (collectionID === undefined) return 1;
+        collectionIDStr === undefined
+            ? undefined
+            : parseID(ctx, "collection", collectionIDStr);
+    if (collectionIDStr !== undefined && collectionID === undefined) return 1;
 
     const lib = await openReadLibrary(ctx, client);
     try {
-        // Force a server round-trip and list in enumeration order (issue #36
+        // Ask the server for current state (issue #36 amendment, issue #52).
-        // amendment, issue #52). Each file prints from its own decrypted
+        // Each file prints from its own decrypted metadata (raw title,
-        // metadata (raw title, microsecond creationTime) via cli-output, and in
+        // microsecond creationTime) via cli-output. The files print in the
-        // the pre-library CLI's enumeration order, not the projection's
+        // order `freshFiles` returns them, not the projection's newest-first
-        // newest-first order.
+        // order.
         const files = await freshFiles(lib, collectionID);
         if (!files) {
-            ctx.stderr.write(`Collection ${collectionID} not found\n`);
+            ctx.log("fatal", `Collection ${collectionID} not found`);
             return 1;
         }
 
@@ -307,40 +333,42 @@ export const getCommand = async (
     try {
         // Force a server round-trip so the file resolves against current state
         // (issue #36 amendment, issue #52).
-        const resolved = await freshFile(lib, fileID);
+        const file = await freshFile(lib, fileID);
-        if (!resolved) {
+        if (!file) {
-            ctx.stderr.write(`File ${fileID} not found\n`);
+            ctx.log("fatal", `File ${fileID} not found`);
             return 1;
         }
-        const { photo, file } = resolved;
 
-        const result = await photo.original();
+        const result = await lib.originalOf(file);
         // Default name is the file's own title, as the pre-library CLI used
         // (not the editedName-preferring projection title) (issue #52).
-        const outPath = opts.out ?? originalName(file);
         if (result.videoPath === undefined) {
+            const outPath = opts.out ?? originalName(file);
             copyFileSync(result.path, outPath);
-            ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
+            ctx.log("info", `${result.bytes} bytes -> ${outPath}`);
             return 0;
         }
         // A live photo is written as its image and its video, each named after
         // the title with its own extension, as Ente's clients name them. With
         // --out, the image goes there and the video beside it.
-        const imageOut =
+        const imageExt = extname(result.path);
-            opts.out ?? withExtension(outPath, extname(result.path));
+        const videoExt = extname(result.videoPath);
-        const videoOut = withExtension(outPath, extname(result.videoPath));
+        const imageOut = opts.out ?? originalName(file, imageExt);
+        const videoOut =
+            opts.out === undefined
+                ? originalName(file, videoExt)
+                : withExtension(opts.out, videoExt);
         if (imageOut.toLowerCase() === videoOut.toLowerCase()) {
-            ctx.stderr.write(
+            ctx.log(
-                `File ${fileID} is a live photo, and its video would also be written to ${imageOut}\n`,
+                "fatal",
+                `File ${fileID} is a live photo, and its video would also be written to ${imageOut}`,
             );
             return 1;
         }
         copyFileSync(result.path, imageOut);
         copyFileSync(result.videoPath, videoOut);
-        ctx.stderr.write(
+        ctx.log("info", `${result.bytes} bytes -> ${imageOut}`);
-            `${result.bytes} bytes -> ${imageOut}\n` +
+        ctx.log("info", `${statSync(videoOut).size} bytes -> ${videoOut}`);
-                `${statSync(videoOut).size} bytes -> ${videoOut}\n`,
-        );
         return 0;
     } finally {
         await lib.close();
@@ -362,19 +390,18 @@ export const getThumbCommand = async (
     try {
         // Force a server round-trip so the file resolves against current state
         // (issue #36 amendment, issue #52).
-        const resolved = await freshFile(lib, fileID);
+        const file = await freshFile(lib, fileID);
-        if (!resolved) {
+        if (!file) {
-            ctx.stderr.write(`File ${fileID} not found\n`);
+            ctx.log("fatal", `File ${fileID} not found`);
             return 1;
         }
-        const { photo, file } = resolved;
 
-        const result = await photo.thumbnail();
+        const result = await lib.thumbnailOf(file);
         // Default name is thumb_<file's own title>, as the pre-library CLI
         // used (not the projection title) (issue #52).
         const outPath = opts.out ?? thumbnailName(file);
         copyFileSync(result.path, outPath);
-        ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
+        ctx.log("info", `${result.bytes} bytes -> ${outPath}`);
         return 0;
     } finally {
         await lib.close();
@@ -394,11 +421,16 @@ export const backupMetadataCommand = async (
         // Refresh first so the dump holds current account state, not what the
         // cache last held; a failed refresh throws.
         await lib.fresh();
-        const { failedMLBatches } = await runMetadataBackup(lib, client, dir, {
+        const result = await runMetadataBackup(lib, client, dir, {
             exif: opts.exif || opts.all,
-            onProgress: (msg) => ctx.stderr.write(msg + "\n"),
+            onProgress: (msg, level) => ctx.log(level ?? "info", msg),
         });
-        return failedMLBatches > 0 ? 1 : 0;
+        if (result.filesWithMLDataError === 0) return 0;
+        ctx.log(
+            "fatal",
+            `quak: the ML data of ${result.filesWithMLDataError} file(s) could not be read; see mlDataError in their JSON in ${dir}`,
+        );
+        return 1;
     } finally {
         await lib.close();
     }
@@ -413,7 +445,7 @@ export const backupCommand = async (
     const client = requireSession(ctx);
     if (!client) return 3;
 
-    ctx.stderr.write("Starting backup...\n");
+    ctx.log("info", "Starting backup...");
     // The lock is taken before the library opens and starts its refresh, so a
     // run refused while another backup of `dir` runs sends no request.
     let lock: BackupLock;
@@ -421,7 +453,7 @@ export const backupCommand = async (
         lock = await lockBackupDirectory(dir);
     } catch (err) {
         if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;
-        ctx.stderr.write(`quak: ${(err as Error).message}\n`);
+        ctx.log("fatal", `quak: ${(err as Error).message}`);
         return 2;
     }
     try {
@@ -436,43 +468,48 @@ export const backupCommand = async (
             refreshIntervalSeconds: Infinity,
             precacheThumbnails: false,
             precacheOriginals: false,
+            onProgress: logRefresh(ctx.log),
         });
         try {
             const result = await lib.backup({
                 downloadDirectory: dir,
                 lock,
                 verify: opts.verify,
-                onProgress: (msg) => {
+                onProgress: (msg, level) => ctx.log(level ?? "info", msg),
-                    if (!opts.json) ctx.stderr.write(msg + "\n");
-                },
             });
 
             if (opts.json) {
                 ctx.stdout.write(JSON.stringify(result, null, 2) + "\n");
             } else {
-                ctx.stderr.write("\n--- Backup complete ---\n");
+                ctx.log("info", "\n--- Backup complete ---");
-                ctx.stderr.write(`  Total files: ${result.totalFiles}\n`);
+                ctx.log("info", `  Total files: ${result.totalFiles}`);
-                ctx.stderr.write(`  Downloaded:  ${result.downloaded}\n`);
+                ctx.log("info", `  Downloaded:  ${result.downloaded}`);
-                ctx.stderr.write(`  Skipped:     ${result.skipped}\n`);
+                ctx.log("info", `  Skipped:     ${result.skipped}`);
                 if (opts.verify) {
-                    ctx.stderr.write(`  Verified:    ${result.verified}\n`);
+                    ctx.log("info", `  Verified:    ${result.verified}`);
-                    ctx.stderr.write(`  Mismatched:  ${result.mismatched}\n`);
+                    ctx.log("info", `  Mismatched:  ${result.mismatched}`);
-                    ctx.stderr.write(`  Unchecked:   ${result.unchecked}\n`);
+                    ctx.log("info", `  Unchecked:   ${result.unchecked}`);
                 }
-                ctx.stderr.write(`  Failed:      ${result.failed}\n`);
+                ctx.log("info", `  Failed:      ${result.failed}`);
                 if (result.errors.length > 0) {
-                    ctx.stderr.write("\nFailed files:\n");
+                    ctx.log("error", "\nFailed files:");
                     for (const e of result.errors) {
-                        ctx.stderr.write(
+                        ctx.log(
-                            `  [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
+                            "error",
+                            `  [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}`,
                         );
                     }
                 }
             }
 
-            return result.failed > 0 || result.undecryptableAlbums.length > 0
+            const albums =
-                ? 1
+                result.undecryptableAlbums.length + result.failedAlbums.length;
-                : 0;
+            if (result.failed === 0 && albums === 0) return 0;
+            ctx.log(
+                "fatal",
+                `quak: ${result.failed} file(s) and ${albums} album(s) failed; see ${join(dir, "failures.json")} and the FAILED lines`,
+            );
+            return 1;
         } finally {
             await lib.close();
         }
@@ -493,18 +530,19 @@ export const listMissingThumbnailsCommand = async (
         // Refresh first so files added since the cache was written are
         // checked; a failed refresh throws.
         await lib.fresh();
-        const missing = await listMissingThumbnails(lib, client, (msg) => {
+        const missing = await listMissingThumbnails(lib, client, (msg, level) =>
-            if (!opts.json) ctx.stderr.write(msg + "\n");
+            ctx.log(level ?? "info", msg),
-        });
+        );
 
         if (opts.json) {
             ctx.stdout.write(JSON.stringify(missing, null, 2) + "\n");
         } else {
             if (missing.length === 0) {
-                ctx.stderr.write("No missing thumbnails found.\n");
+                ctx.log("info", "No missing thumbnails found.");
             } else {
-                ctx.stderr.write(
+                ctx.log(
-                    `\n${missing.length} file(s) with missing thumbnails:\n`,
+                    "info",
+                    `\n${missing.length} file(s) with missing thumbnails:`,
                 );
                 for (const m of missing) {
                     ctx.stdout.write(
@@ -535,31 +573,37 @@ export const fixMissingThumbnailsCommand = async (
     }
     const lib = await openReadLibrary(ctx, client);
     try {
-        // Refresh first so files added since the cache was written are found;
+        let found: Map<number, { file: EnteFile; collection: Collection }>;
-        // a failed refresh throws.
+        if (fileIDs.length > 0) {
+            found = await freshFilesByID(lib, fileIDs);
+        } else {
+            // Refresh first so files added since the cache was written are
+            // checked; a failed refresh throws.
             await lib.fresh();
-        if (fileIDs.length === 0) {
+            ctx.log("info", "Scanning for missing thumbnails...");
-            ctx.stderr.write("Scanning for missing thumbnails...\n");
+            const missing = await listMissingThumbnails(
-            const missing = await listMissingThumbnails(lib, client, (msg) => {
+                lib,
-                if (!opts.json) ctx.stderr.write(msg + "\n");
+                client,
-            });
+                (msg, level) => ctx.log(level ?? "info", msg),
+            );
             fileIDs = missing.map((m) => m.fileID);
             if (fileIDs.length === 0) {
-                ctx.stderr.write("No missing thumbnails found.\n");
+                ctx.log("info", "No missing thumbnails found.");
                 return 0;
             }
-            ctx.stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
+            ctx.log("info", `Found ${fileIDs.length} file(s) to fix.`);
+            found = cachedFilesByID(lib, fileIDs);
         }
 
         const results = await fixMissingThumbnails(
             lib,
             client,
             fileIDs,
-            (msg) => {
+            found,
-                if (!opts.json) ctx.stderr.write(msg + "\n");
+            (msg, level) => ctx.log(level ?? "info", msg),
-            },
         );
 
+        const failed = results.filter((r) => r.status === "failed").length;
         if (opts.json) {
             ctx.stdout.write(JSON.stringify(results, null, 2) + "\n");
         } else {
@@ -567,30 +611,30 @@ export const fixMissingThumbnailsCommand = async (
             const skipped = results.filter(
                 (r) => r.status === "skipped",
             ).length;
-            const failed = results.filter((r) => r.status === "failed").length;
+            ctx.log("info", `\n--- Done ---`);
-            ctx.stderr.write(`\n--- Done ---\n`);
+            ctx.log("info", `  Fixed:   ${fixed}`);
-            ctx.stderr.write(`  Fixed:   ${fixed}\n`);
+            ctx.log("info", `  Skipped: ${skipped}`);
-            ctx.stderr.write(`  Skipped: ${skipped}\n`);
+            ctx.log("info", `  Failed:  ${failed}`);
-            ctx.stderr.write(`  Failed:  ${failed}\n`);
             if (skipped > 0) {
-                ctx.stderr.write("\nSkipped:\n");
+                ctx.log("info", "\nSkipped:");
                 for (const r of results.filter((r) => r.status === "skipped")) {
-                    ctx.stderr.write(
+                    ctx.log("info", `  ${r.fileID}\t${r.title}\t${r.reason}`);
-                        `  ${r.fileID}\t${r.title}\t${r.reason}\n`,
-                    );
                 }
             }
             if (failed > 0) {
-                ctx.stderr.write("\nFailed files:\n");
+                ctx.log("error", "\nFailed files:");
                 for (const r of results.filter((r) => r.status === "failed")) {
-                    ctx.stderr.write(
+                    ctx.log("error", `  ${r.fileID}\t${r.title}\t${r.reason}`);
-                        `  ${r.fileID}\t${r.title}\t${r.reason}\n`,
-                    );
                 }
             }
         }
 
-        return results.some((r) => r.status === "failed") ? 1 : 0;
+        if (failed === 0) return 0;
+        ctx.log(
+            "fatal",
+            `quak: ${failed} file(s) failed; see the "Failed files:" list, or the JSON with --json`,
+        );
+        return 1;
     } finally {
         await lib.close();
     }

src/cli-log.ts

@@ -0,0 +1,102 @@
+// The CLI's log level, and the one function the commands write stderr lines
+// through.
+//
+// Every line quak logs goes to stderr, at one of six levels, from the
+// quietest: fatal, error, warning, info, debug, trace. A run prints the lines
+// at its own level and at every quieter one. Its level is info unless `-v` or
+// `-q` asks for another (see "What quak prints" in the README). A command that
+// runs and exits non-zero logs a fatal line saying why, so even `-q`, which
+// prints only fatal lines, never leaves such an exit unexplained. An error
+// line reports something that failed, such as a file or an album, and a
+// warning something that did not fail but needs the user's attention. stdout
+// carries a command's output and the prompts of `login`, and is not a log.
+// Two things on stderr are not log lines: the `error:` message for a
+// command line that quak cannot parse, and the usage printed when no command
+// is given.
+//
+// The library itself writes no lines: its layers report through callbacks, and
+// the CLI logs what they report. `logRequests` gives the API client options
+// that log each request and each retry, and `logRefresh` the library's
+// `onProgress` that logs what a refresh and an ML data fetch report.
+
+import type { ApiClientOptions } from "./api/client.js";
+import { describeError } from "./errors.js";
+import type { RefreshEvent } from "./library/index.js";
+import type { RetryOptions } from "./retry.js";
+
+const LEVELS = ["fatal", "error", "warning", "info", "debug", "trace"] as const;
+
+export type LogLevel = (typeof LEVELS)[number];
+
+// Writes `line` on stderr when the run's level prints lines at `level`.
+export type Log = (level: LogLevel, line: string) => void;
+
+// The level asked for by `-v` given `verbose` times, or by `-q`: info without
+// either, debug for `-v`, trace for `-vv` and more, fatal for `-q`.
+// `bin/quak.ts` refuses `-q` and `-v` together.
+export const logLevel = (verbose: number, quiet: boolean): LogLevel => {
+    if (quiet) return "fatal";
+    if (verbose === 0) return "info";
+    return verbose === 1 ? "debug" : "trace";
+};
+
+// The log that writes each line at `level` or quieter to `stderr`, ending it
+// with a newline.
+export const stderrLog =
+    (stderr: { write(text: string): unknown }, level: LogLevel): Log =>
+    (lineLevel, line) => {
+        if (LEVELS.indexOf(lineLevel) <= LEVELS.indexOf(level)) {
+            stderr.write(line + "\n");
+        }
+    };
+
+// API client options that log each request at trace, with the status of its
+// response or the error it failed with, and each retry at debug, with why and
+// how long it waits. `retry` is the retry policy, the default when not given.
+export const logRequests = (
+    log: Log,
+    retry?: RetryOptions,
+): ApiClientOptions => ({
+    retry: {
+        ...retry,
+        onRetry: (err, attempt, delayMs) =>
+            log(
+                "debug",
+                `Attempt ${attempt} failed, retrying in ${Math.round(delayMs)} ms: ${describeError(err)}`,
+            ),
+    },
+    onRequest: ({ method, url, status, error }) =>
+        log(
+            "trace",
+            `${method} ${url}: ${error === undefined ? `HTTP ${status}` : describeError(error)}`,
+        ),
+});
+
+// How a log line names each operation the library reports the start and end
+// of.
+const OPERATION_NAMES: Record<RefreshEvent["operation"], string> = {
+    refresh: "Refresh",
+    fetchMLData: "ML data fetch",
+    precacheThumbnails: "Thumbnail precache",
+    precacheOriginals: "Original precache",
+};
+
+// A library `onProgress` that logs, at info, each collection whose files a
+// refresh has fetched, and at debug the start and end of each refresh and ML
+// data fetch, with the error of one that failed.
+export const logRefresh =
+    (log: Log) =>
+    (event: RefreshEvent): void => {
+        if (event.status === "progress") {
+            log(
+                "info",
+                `Fetched the files of ${event.fetched} of ${event.total} collections`,
+            );
+            return;
+        }
+        const error = event.error === undefined ? "" : `: ${event.error}`;
+        log(
+            "debug",
+            `${OPERATION_NAMES[event.operation]} ${event.status}${error}`,
+        );
+    };

src/cli-output.ts

@@ -9,7 +9,7 @@
 // `metadata.title`, and issue #52 requires that output stay byte-identical, so
 // the commands shape their output from the raw `EnteFile` through here.
 
-import { sanitizeFileName } from "./filename.js";
+import { defaultFileName } from "./filename.js";
 import type { EnteFile, FileType, Microseconds } from "./model/types.js";
 
 // One row of `quak files --json`.
@@ -33,11 +33,13 @@ export const fileListRow = (file: EnteFile): FileListRow => ({
 export const fileListLine = (file: EnteFile): string =>
     `${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
 
-// Default output path for `quak get` when `--out` is not given. The title comes
+// Default output path for `quak get` when `--out` is not given, made from the
-// from the server, so it is sanitized; `--out` is the user's and is used as is.
+// title (see `defaultFileName`), with `ext` in place of its extension for a
-export const originalName = (file: EnteFile): string =>
+// live photo's image or video; `--out` is the user's and is used as is.
-    sanitizeFileName(file.metadata.title, `file-${file.id}`);
+export const originalName = (file: EnteFile, ext?: string): string =>
+    defaultFileName(file.metadata.title, file.id, { ext });
 
-// Default output path for `quak get-thumb` when `--out` is not given.
+// Default output path for `quak get-thumb` when `--out` is not given: `thumb_`
+// and the title.
 export const thumbnailName = (file: EnteFile): string =>
-    `thumb_${originalName(file)}`;
+    defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });

src/cli-read.ts

@@ -1,62 +1,150 @@
 // How the CLI's read commands obtain current data.
 //
-// `collections`, `files --collection`, `get`, and `get-thumb` must answer for
+// `collections`, `files`, `get`, and `get-thumb` must answer for
 // the account's state at the moment the command runs, not for whatever the
-// local cache last happened to hold (owner amendment, issue #36). Each helper
+// local cache last happened to hold (owner amendment, issue #36). `get`,
-// therefore forces a server round-trip through `Library.fresh()` and only then
+// `get-thumb` and `files` without a collection ID therefore force a server
-// reads — so a collection, file, or metadata change made elsewhere is visible.
+// round-trip through `Library.fresh()` and only then read, so a change made
+// elsewhere is visible. `collections` asks the library for the collection list
+// alone through `Library.freshCollections()` (issue #210): a refresh would also
+// fetch the files of every collection, which on a large account takes long and
+// which `collections` does not print. `files` with a collection ID reads that
+// list too, then that one collection's files through `Library.freshFiles()`
+// (issue #217). On an empty cache, `get` and `get-thumb` read that list, then
+// one collection's files at a time, until they find the file (issue #222).
+// `helper fix-missing-thumbnails --file` does the same until it finds every
+// file it names (issue #223).
 //
-// `collections` and `files` also list in the library's own enumeration order —
+// `collections` and `files` with a collection ID list in the server's order.
-// `listCollections()`/`listFiles()`, the order the pre-library CLI printed —
+// `files` without one lists in the library's own enumeration order
-// rather than the `albums`/`photos` projection's newest-first order, which
+// (`listFiles()`). Neither command uses the `albums`/`photos` projection's
-// re-sorts the rows. The field values still come from each record's raw
+// newest-first order, which re-sorts the rows. The field values still come
-// metadata via `cli-output.ts`.
+// from each record's raw metadata via `cli-output.ts`. `files` without a
+// collection ID lists every file in the account once (issue #211): the library
+// keeps a record of a file for each collection that holds it, and only the
+// first collection's record is listed.
 
 import type { Collection, EnteFile } from "./model/types.js";
-import type { Photo, PhotosAPI } from "./library/index.js";
 
 // The slice of `Library` these helpers read. `Library` satisfies it
-// structurally; a test can drive them with a stand-in that records the
+// structurally; a test can drive them with a stand-in that records each fresh
-// `fresh()` call and serves records in a known enumeration order.
+// read and serves records in a known enumeration order.
 export interface FreshReadLibrary {
     fresh(): Promise<unknown>;
+    freshCollections(): Promise<Collection[]>;
+    freshFiles(collection: Collection): Promise<EnteFile[]>;
     listCollections(): Collection[];
-    getCollection(id: number): Collection | undefined;
     listFiles(collectionID: number): EnteFile[];
     getFileByID(fileID: number): EnteFile | undefined;
-    photos: Pick<PhotosAPI, "byID">;
 }
 
-// Every live collection, current as of a forced refresh, in enumeration order.
+// Every live collection, current as of one collections request, in the
-export const freshCollections = async (
+// server's order.
+export const freshCollections = (
     lib: FreshReadLibrary,
-): Promise<Collection[]> => {
+): Promise<Collection[]> => lib.freshCollections();
-    await lib.fresh();
-    return lib.listCollections();
-};
 
-// The files of one collection, current as of a forced refresh, in enumeration
+// The files of one collection, as the server holds them now, in the server's
 // order. `undefined` (not an empty list) when the collection does not exist, so
-// the caller can tell "no such collection" from "an empty collection".
+// the caller can tell "no such collection" from "an empty collection". Without
+// a collection ID, every file in the account as of a forced refresh, each once,
+// as held by the first collection that holds it.
 export const freshFiles = async (
     lib: FreshReadLibrary,
-    collectionID: number,
+    collectionID?: number,
 ): Promise<EnteFile[] | undefined> => {
+    if (collectionID !== undefined) {
+        const collections = await lib.freshCollections();
+        const collection = collections.find((c) => c.id === collectionID);
+        return collection ? lib.freshFiles(collection) : undefined;
+    }
     await lib.fresh();
-    if (!lib.getCollection(collectionID)) return undefined;
+    const listed = new Set<number>();
-    return lib.listFiles(collectionID);
+    const files: EnteFile[] = [];
+    for (const collection of lib.listCollections()) {
+        for (const file of lib.listFiles(collection.id)) {
+            if (listed.has(file.id)) continue;
+            listed.add(file.id);
+            files.push(file);
+        }
+    }
+    return files;
 };
 
-// One file, current as of a forced refresh, resolved to both its content
+// One file as the server holds it now, or `undefined` when no collection holds
-// handle (`Photo`, for fetching bytes) and its raw record (`EnteFile`, for the
+// it. When the cache holds the account's collections, it refreshes and looks
-// default output name and field values). `undefined` when the file is unknown.
+// the file up in the cache. On an empty cache, it reads the collection list,
+// then one collection's files at a time, and stops at the first collection
+// that holds the file. That search stores nothing.
 export const freshFile = async (
     lib: FreshReadLibrary,
     fileID: number,
-): Promise<{ photo: Photo; file: EnteFile } | undefined> => {
+): Promise<EnteFile | undefined> => {
+    // Only a refresh stores collections, so a cache holding any holds them all.
+    if (lib.listCollections().length > 0) {
         await lib.fresh();
-    const photo = lib.photos.byID({ fileID });
+        return lib.getFileByID(fileID);
-    const file = lib.getFileByID(fileID);
+    }
-    if (!photo || !file) return undefined;
+    for (const collection of await lib.freshCollections()) {
-    return { photo, file };
+        const files = await lib.freshFiles(collection);
+        const file = files.find((f) => f.id === fileID);
+        if (file) return file;
+    }
+    return undefined;
+};
+
+// The most recently updated collection first, and the higher ID first on a
+// tie. `Library.albums.list()` lists albums in this order.
+const byNewestCollection = (a: Collection, b: Collection): number =>
+    b.updationTime - a.updationTime || b.id - a.id;
+
+// Each file named by `fileIDs` that the cache holds, mapped to its record and
+// the collection it was found in. Collections are searched in
+// `byNewestCollection` order, and a file is found in the first that holds it.
+// So `helper fix-missing-thumbnails` names a file after the same album as
+// `helper list-missing-thumbnails`. It reads the cache alone, so the caller
+// refreshes first.
+export const cachedFilesByID = (
+    lib: FreshReadLibrary,
+    fileIDs: number[],
+): Map<number, { file: EnteFile; collection: Collection }> => {
+    const wanted = new Set(fileIDs);
+    const found = new Map<number, { file: EnteFile; collection: Collection }>();
+    const collections = [...lib.listCollections()].sort(byNewestCollection);
+    for (const collection of collections) {
+        for (const file of lib.listFiles(collection.id)) {
+            if (wanted.has(file.id) && !found.has(file.id)) {
+                found.set(file.id, { file, collection });
+            }
+        }
+    }
+    return found;
+};
+
+// Each file named by `fileIDs` as the server holds it now, mapped to its record
+// and collection as `cachedFilesByID` maps it. When the cache holds the
+// account's collections, it refreshes and looks the files up in the cache. On
+// an empty cache, it reads the collection list, then one collection's files at
+// a time, and stops once it has found every file. That search stores nothing.
+export const freshFilesByID = async (
+    lib: FreshReadLibrary,
+    fileIDs: number[],
+): Promise<Map<number, { file: EnteFile; collection: Collection }>> => {
+    // Only a refresh stores collections, so a cache holding any holds them all.
+    if (lib.listCollections().length > 0) {
+        await lib.fresh();
+        return cachedFilesByID(lib, fileIDs);
+    }
+    const wanted = new Set(fileIDs);
+    const found = new Map<number, { file: EnteFile; collection: Collection }>();
+    const collections = await lib.freshCollections();
+    for (const collection of [...collections].sort(byNewestCollection)) {
+        if (found.size === wanted.size) break;
+        for (const file of await lib.freshFiles(collection)) {
+            if (wanted.has(file.id) && !found.has(file.id)) {
+                found.set(file.id, { file, collection });
+            }
+        }
+    }
+    return found;
 };

src/cli-run.ts

@@ -2,18 +2,23 @@
 
 import type { Writable } from "node:stream";
 import { ApiError } from "./api/client.js";
+import type { Log } from "./cli-log.js";
+import { describeError } from "./errors.js";
 
 // Run a command and exit with its code once stdout/stderr have drained.
 // Exiting before the drain can truncate piped output, and the library can keep
 // the event loop alive after a command returns, so a plain return could hang.
-// An error the command throws is printed as one `quak: MESSAGE` line, without
+// An error the command throws is logged as one fatal `quak: MESSAGE` line,
-// the stack trace, and exits 1. A 401 from the server means it no longer
+// without the stack trace, and exits 1. MESSAGE is the error as
-// accepts the saved session: that prints one line saying to log in again and
+// `describeError` gives it, so it carries the reason Node's `fetch` puts on
-// exits 3, as a missing or corrupt session file does.
+// `cause`. A 401 from the server means it no longer accepts the saved session:
+// that logs one fatal line saying to log in again and exits 3, as a missing or
+// corrupt session file does.
 export const run = async (
     command: Promise<number>,
     stdout: Writable,
     stderr: Writable,
+    log: Log,
     exit: (code: number) => void,
 ): Promise<void> => {
     let code: number;
@@ -21,14 +26,13 @@ export const run = async (
         code = await command;
     } catch (err) {
         if (err instanceof ApiError && err.status === 401) {
-            stderr.write(
+            log(
-                `quak: the saved session is no longer valid; run "quak login"\n`,
+                "fatal",
+                `quak: the saved session is no longer valid; run "quak login"`,
             );
             code = 3;
         } else {
-            stderr.write(
+            log("fatal", `quak: ${describeError(err)}`);
-                `quak: ${err instanceof Error ? err.message : String(err)}\n`,
-            );
             code = 1;
         }
     }

src/client.ts

@@ -373,13 +373,17 @@ export class Client {
     }
 
     // Fetch machine-learning data (face detections + CLIP embeddings) for up
-    // to a batch of files, each decrypted with its own key. One request; the
+    // to a batch of files, each decrypted with its own key, and for each file
-    // library batches at `MLDATA_BATCH_SIZE` and schedules each batch through
+    // whose ML data did not decrypt, gunzip or parse, why not. One request;
-    // its metadata request pool.
+    // the library batches at `MLDATA_BATCH_SIZE` and schedules each batch
+    // through its metadata request pool.
     async fetchMLData(args: {
         fileIDs: number[];
         fileKeys: Map<number, Uint8Array>;
-    }): Promise<Map<number, MLData>> {
+    }): Promise<{
+        payloads: Map<number, MLData>;
+        errors: Map<number, unknown>;
+    }> {
         this.assertLoggedIn();
         return fetchMLDataBatch(this.api, args.fileIDs, args.fileKeys);
     }

src/clip-keywords.ts

@@ -0,0 +1,57 @@
+// Keywords for a file from its CLIP embedding.
+//
+// Ente stores no keywords. A file's ML data holds a CLIP embedding, the 512
+// numbers Ente's image model gave the photo, and Ente's photos app sorts files
+// into the categories of its Discover section with it: a file falls in a
+// category when the cosine similarity of its CLIP embedding and the text
+// embedding of the category's query is at least the category's `minScore`,
+// unless the file is a video and the category leaves out videos. A file's
+// keywords are the titles of the categories it falls in. The categories and
+// their text embeddings are in `clip-keyword-vectors.ts`.
+
+import { DISCOVER_CATEGORIES } from "./clip-keyword-vectors.js";
+import { clipEmbedding } from "./library/mldata.js";
+import type { MLData } from "./mldata-fetch.js";
+
+// How many numbers a CLIP embedding from Ente's image model has.
+const EMBEDDING_LENGTH = 512;
+
+const norm = (vector: number[]): number => Math.hypot(...vector);
+
+const cosineSimilarity = (a: number[], b: number[]): number => {
+    let dot = 0;
+    a.forEach((value, i) => {
+        dot += value * (b[i] ?? 0);
+    });
+    return dot / (norm(a) * norm(b));
+};
+
+// The titles of the Discover categories a file falls in, highest score first,
+// or undefined when its ML data has no CLIP embedding of 512 finite numbers
+// that are not all zero. Tests pass their own `categories`.
+export const clipKeywords = (
+    mlData: MLData,
+    isVideo: boolean,
+    categories = DISCOVER_CATEGORIES,
+): string[] | undefined => {
+    const embedding = clipEmbedding(mlData);
+    if (
+        !embedding ||
+        embedding.length !== EMBEDDING_LENGTH ||
+        norm(embedding) === 0
+    ) {
+        return undefined;
+    }
+
+    const matches: { title: string; score: number }[] = [];
+    for (const category of categories) {
+        if (isVideo && !category.showVideo) continue;
+        const score = cosineSimilarity(embedding, category.textEmbedding);
+        if (score >= category.minScore) {
+            matches.push({ title: category.title, score });
+        }
+    }
+    return matches
+        .sort((a, b) => b.score - a.score)
+        .map((match) => match.title);
+};

src/download/index.ts

@@ -16,7 +16,7 @@ import {
     streamTagFinal,
 } from "../crypto/index.js";
 import { TruncatedStreamError } from "../errors.js";
-import { safeExtension, sanitizeFileName, withExtension } from "../filename.js";
+import { defaultFileName, safeExtension, withExtension } from "../filename.js";
 import { withRetry } from "../retry.js";
 import type { ApiClient } from "../api/client.js";
 import type { EnteFile } from "../model/types.js";
@@ -135,15 +135,17 @@ const streamDecrypt = async (
                     // cannot tell a partial chunk from a corrupt one, so this
                     // is reported as the truncation it almost always is, with
                     // the authentication failure kept as the error's cause.
-                    // Only the pull is guarded: a sink failure on a chunk
+                    // The message leaves that failure to the cause, which
-                    // that did authenticate is a disk error, not a
+                    // `describeError` appends to what `quak backup` prints and
+                    // records. Only the pull is guarded: a sink failure on a
+                    // chunk that did authenticate is a disk error, not a
                     // truncation.
                     let pulled;
                     try {
                         pulled = pullStreamChunk(state, buffer);
                     } catch (err) {
                         throw new TruncatedStreamError(
-                            `download: stream truncated: response body ended with ${buffer.length} trailing bytes that did not authenticate as a final chunk (transfer stopped mid-chunk, or the data is corrupt)`,
+                            `download: stream truncated: response body ended with ${buffer.length} trailing bytes (transfer stopped mid-chunk, or the data is corrupt)`,
                             { cause: err },
                         );
                     }
@@ -197,11 +199,13 @@ const isRunning = (pid: number): boolean => {
 };
 
 // Delete the temp files a killed process left in `dir`: the writer's
-// `.quak-<pid>-<random>.tmp` and the backup copy's
+// `.quak-<pid>-<random>.tmp`, the backup copy's
-// `.quak-backup-<name>-<pid>-<random>.tmp`. Only files whose process is no
+// `.quak-backup-<name>-<pid>-<random>.tmp`, and the image and video of a live
-// longer running are removed, so another process writing into the same
+// photo downloaded to replace a stored one, `.quak-<pid>-<random>.<ext>` (see
-// directory keeps its own. A reused process ID can only keep a leftover a while
+// `replaceOriginal`). Only files whose process is no longer running are
-// longer, never remove a live one.
+// removed, so another process writing into the same directory keeps its own. A
+// reused process ID can only keep a leftover a while longer, never remove a
+// live one.
 export const removeLeftoverTempFiles = (dir: string): void => {
     let names: string[];
     try {
@@ -210,7 +214,9 @@ export const removeLeftoverTempFiles = (dir: string): void => {
         return;
     }
     for (const name of names) {
-        const match = /^\.quak-(?:.*-)?(\d+)-[0-9a-z]*\.tmp$/.exec(name);
+        const match = /^\.quak-(?:.*-)?(\d+)-[0-9a-z]*\.[0-9A-Za-z]+$/.exec(
+            name,
+        );
         if (match && !isRunning(Number(match[1]))) {
             rmSync(join(dir, name), { force: true });
         }
@@ -359,10 +365,14 @@ const openPart = async (
 // name starts with `image` as the image and the one whose name starts with
 // `video` as the video, and refuses a ZIP holding a second of either. It is
 // written unpacked: each part is named `destination` with the extension
-// replaced by its own entry's, and the two must differ ignoring case. When the
+// replaced by its own entry's, and the two must differ ignoring case. When
-// file records a hash, `<imageHash>:<videoHash>` must match it, each over that
+// `defaultName` is set, `destination` is the name `defaultFileName` made from
-// part's own bytes. Only then are the image, then the video, renamed into
+// the title rather than a path the caller gave, and each part is named as
-// place; on any failure neither is stored.
+// `defaultFileName` names it with its own extension, so that the name is cut
+// short to fit with that extension. When the file records a hash,
+// `<imageHash>:<videoHash>` must match it, each over that part's own bytes.
+// Only then are the image, then the video, renamed into place; on any failure
+// neither is stored.
 //
 // The ZIP is chosen by its uploader and may expand enormously, so each part is
 // written as it decompresses and never held, and the ZIP is refused once the
@@ -382,6 +392,7 @@ const decryptLivePhoto = async (
     key: Uint8Array,
     onProgress: ProgressCallback | undefined,
     file: EnteFile,
+    defaultName: boolean,
 ): Promise<DownloadResult> => {
     const sliceSize = 4096;
     const dir = dirname(destination);
@@ -480,8 +491,12 @@ const decryptLivePhoto = async (
                 `live photo's image and video have the same extension, ${video.ext}`,
             );
         }
-        const path = withExtension(destination, image.ext);
+        const partPath = (ext: string): string =>
-        const videoPath = withExtension(destination, video.ext);
+            defaultName
+                ? defaultFileName(file.metadata.title, file.id, { ext })
+                : withExtension(destination, ext);
+        const path = partPath(image.ext);
+        const videoPath = partPath(video.ext);
         for (const part of parts) {
             await part.handle.sync();
             await part.handle.close();
@@ -525,6 +540,9 @@ const decryptLivePhoto = async (
 // and then removes its own temp file, and only the attempt that reaches
 // TAG_FINAL renames one into place, so a download that needed three tries still
 // performs exactly one rename over the destination.
+//
+// `defaultName` says `destination` is the name `defaultFileName` made, not a
+// path the caller gave (see `decryptLivePhoto`).
 const fetchAndDecrypt = async (
     api: ApiClient,
     openStream: () => Promise<ReadableStream<Uint8Array>>,
@@ -533,6 +551,7 @@ const fetchAndDecrypt = async (
     destination: string,
     onProgress?: ProgressCallback,
     original?: EnteFile,
+    defaultName = false,
 ): Promise<DownloadResult> =>
     withRetry(async () => {
         const stream = await openStream();
@@ -545,6 +564,7 @@ const fetchAndDecrypt = async (
                     key,
                     onProgress,
                     original,
+                    defaultName,
                 );
             }
             const bytesWritten = await decryptToTemp(
@@ -576,10 +596,11 @@ export const downloadFile = async (
     outPath?: string,
     onProgress?: ProgressCallback,
 ): Promise<DownloadResult> => {
-    // `outPath` is the caller's and is used as is; the title is the server's
+    // `outPath` is the caller's and is used as is. Without it, the file is
-    // and is sanitized so it can only name a file in the current directory.
+    // written in the current directory under the name `defaultFileName` makes
+    // from the title, which is the server's.
     const resolvedPath =
-        outPath ?? sanitizeFileName(file.metadata.title, `file-${file.id}`);
+        outPath ?? defaultFileName(file.metadata.title, file.id);
     const header = fromBase64(file.file.decryptionHeader);
     return fetchAndDecrypt(
         api,
@@ -589,6 +610,7 @@ export const downloadFile = async (
         resolvedPath,
         onProgress,
         file,
+        outPath === undefined,
     );
 };
 
@@ -598,9 +620,10 @@ export const downloadThumbnail = async (
     outPath?: string,
     onProgress?: ProgressCallback,
 ): Promise<DownloadResult> => {
+    // As in `downloadFile`, with `thumb_` before the title.
     const resolvedPath =
         outPath ??
-        `thumb_${sanitizeFileName(file.metadata.title, `file-${file.id}`)}`;
+        defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
     const header = fromBase64(file.thumbnail.decryptionHeader);
     return fetchAndDecrypt(
         api,

src/errors.ts

@@ -1,10 +1,13 @@
-// Error types that more than one layer of quak needs to recognise.
+// Error types that more than one layer of quak needs to recognise, and how an
+// error is described in what quak prints and in `failures.json`.
 //
-// They live here rather than beside the code that throws them so that the
+// The types live here rather than beside the code that throws them so that the
 // retry classifier can identify them without importing the HTTP client or the
 // download layer — both of which import the classifier. `ApiError` is
 // re-exported from `src/api/client.ts`, which is where callers have always
-// imported it from and where it still belongs conceptually.
+// imported it from and where it still belongs conceptually. The walk along an
+// error's `cause` chain is here too, because the retry classifier reads the
+// chain for errnos and `describeError` for messages.
 
 export class ApiError extends Error {
     readonly status: number;
@@ -43,3 +46,68 @@ export class TruncatedStreamError extends Error {
         this.name = "TruncatedStreamError";
     }
 }
+
+// `cause` is an arbitrary user-settable property and nothing prevents it from
+// forming a cycle, so the walk is bounded. Hanging the process would be worse
+// than misclassifying an error or describing only part of it.
+const MAX_CAUSE_DEPTH = 8;
+
+// The error and each error on its `cause` chain, top first, at most
+// `MAX_CAUSE_DEPTH` of them. `complete` is false when the walk stopped at the
+// depth limit with more of the chain still below it.
+export const causeChain = (
+    err: unknown,
+): { chain: object[]; complete: boolean } => {
+    const chain: object[] = [];
+    let current: unknown = err;
+    for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
+        if (current === null || typeof current !== "object") {
+            return { chain, complete: true };
+        }
+        chain.push(current);
+        const { cause } = current as { cause?: unknown };
+        if (cause === current) return { chain, complete: true };
+        current = cause;
+    }
+    return { chain, complete: current === null || typeof current !== "object" };
+};
+
+// An error as quak prints it and records it in `failures.json`: its message,
+// then the message of each error on its `cause` chain, joined with ": ". Node's
+// `fetch` rejects with just "fetch failed" and puts the reason on `cause`, so
+// this gives "fetch failed: connect ECONNREFUSED 1.2.3.4:443". When the host
+// has several addresses and every one fails, the cause is an `AggregateError`
+// with an empty message and one error per address in `errors`, so an
+// `AggregateError` is given as the messages of its errors, joined with ", ".
+// A cause that is not an object, such as a thrown string, is given as
+// `String(cause)`. Each message is trimmed of the white space around it, as
+// OpenSSL ends its own with a newline. A message the text already holds is
+// left out.
+export const describeError = (err: unknown): string => {
+    if (!(err instanceof Error)) return String(err);
+    const { chain } = causeChain(err);
+    const messages = chain
+        .slice(1)
+        .map((cause) =>
+            cause instanceof AggregateError
+                ? cause.errors
+                      .map((e: unknown) =>
+                          (e instanceof Error ? e.message : String(e)).trim(),
+                      )
+                      .join(", ")
+                : (cause as { message?: unknown }).message,
+        );
+    // `causeChain` stops at a cause that is not an object, so that one is read
+    // from the last error on the chain.
+    const last = chain[chain.length - 1] as { cause?: unknown };
+    if (last.cause !== undefined && typeof last.cause !== "object") {
+        messages.push(String(last.cause));
+    }
+    let text = err.message.trim();
+    for (const message of messages) {
+        if (typeof message !== "string") continue;
+        const trimmed = message.trim();
+        if (!text.includes(trimmed)) text += `: ${trimmed}`;
+    }
+    return text;
+};

src/filename.ts

@@ -4,7 +4,10 @@
 // hands us, and quak does not trust the server. Any name taken from them and
 // used on disk goes through here, so it can only ever name one file inside the
 // directory the caller chose: never a path, never `..`, never hidden, never a
-// Windows device name.
+// Windows device name. Such a name can still be too long for a file name, and
+// `shortenToBytes` or `shortenKeepingExtension` cuts one short.
+// `defaultFileName` does both for the name a file is saved under when the
+// caller gives no path.
 //
 // A path the user typed (`--out`, `outPath`) is not passed through here: the
 // caller is trusted, the server is not.
@@ -29,13 +32,80 @@ export const sanitizeFileName = (name: string, fallback: string): string => {
     return RESERVED_DEVICE_NAME.test(cleaned) ? `_${cleaned}` : cleaned;
 };
 
-// The extension of `title` (".jpg"), or ".bin" when it has none or it holds
+// The extension of `title` (".jpg"), or ".bin" when it has none, it holds
-// anything but letters and digits.
+// anything but letters and digits, or it is more than 16 of them long.
 export const safeExtension = (title: string): string => {
     const ext = extname(title);
-    return /^\.[A-Za-z0-9]+$/.test(ext) ? ext : ".bin";
+    return /^\.[A-Za-z0-9]{1,16}$/.test(ext) ? ext : ".bin";
 };
 
 // `name` with its extension, if it has one, replaced by `ext` (".mov").
 export const withExtension = (name: string, ext: string): string =>
     name.slice(0, name.length - extname(name).length) + ext;
+
+// The longest file name, in bytes, that ext4 and most other file systems allow.
+export const MAX_NAME_BYTES = 255;
+
+// `name` cut short to at most `maxBytes` bytes of UTF-8, the encoding a file
+// name is written to disk in, between two characters, never inside one. A name
+// that fits comes back unchanged.
+export const shortenToBytes = (name: string, maxBytes: number): string => {
+    let bytes = 0;
+    let end = 0;
+    for (const character of name) {
+        bytes += Buffer.byteLength(character);
+        if (bytes > maxBytes) break;
+        end += character.length;
+    }
+    return name.slice(0, end);
+};
+
+// `name` cut short to fit in `maxBytes` bytes with `insert` put right before
+// its extension (`IMG_0001 (12).jpg`), each cut made as `shortenToBytes`
+// makes it. The part before the extension is cut first, so the extension is
+// kept whenever it fits, but that part always keeps its first character:
+// when the extension leaves no room for `insert` and that character, the
+// extension is cut short too, and the part before it keeps the bytes that cut
+// leaves over. So a name that does not start with a dot never comes back
+// starting with one, which would hide it. `insert` is never cut. With no
+// `insert`, a name that fits comes back unchanged.
+export const shortenKeepingExtension = (
+    name: string,
+    maxBytes: number,
+    insert = "",
+): string => {
+    const fullExt = extname(name);
+    const stem = name.slice(0, name.length - fullExt.length);
+    const first = Array.from(stem)[0] ?? "";
+    const ext = shortenToBytes(
+        fullExt,
+        maxBytes - Buffer.byteLength(insert + first),
+    );
+    const room = maxBytes - Buffer.byteLength(insert + ext);
+    return shortenToBytes(stem, room) + insert + ext;
+};
+
+// The name a file is saved under in the working directory when the caller
+// gives no path (`quak get` without `--out`, `downloadFile` without
+// `outPath`): `prefix` (`thumb_` for a thumbnail), then the file's title with
+// unsafe characters replaced, or `file-<id>` when the title is empty, cut
+// short so that the whole fits in a file name, keeping the title's extension
+// whenever it fits. `ext` (".mov"), when given, takes the place of the
+// title's extension before the name is cut short; a live photo's image and
+// video are each named with their own. The cut can leave a device name
+// (`CONSOLE.<long extension>` becomes `CON.<long extension>`); that name gets
+// a leading `_`, as `sanitizeFileName` gives one, and is cut a byte shorter to
+// leave room for it.
+export const defaultFileName = (
+    title: string,
+    fileID: number,
+    opts: { prefix?: string; ext?: string } = {},
+): string => {
+    const { prefix = "", ext } = opts;
+    const safe = sanitizeFileName(title, `file-${fileID}`);
+    const name = ext === undefined ? safe : withExtension(safe, ext);
+    const room = MAX_NAME_BYTES - Buffer.byteLength(prefix);
+    const cut = prefix + shortenKeepingExtension(name, room);
+    if (!RESERVED_DEVICE_NAME.test(cut)) return cut;
+    return `_${prefix}${shortenKeepingExtension(name, room - 1)}`;
+};

src/library/content.ts

@@ -13,6 +13,8 @@
 // originals, the thumbnail pool for thumbnails. The pool limits concurrency,
 // orders on-demand work ahead of background, and dedups by key so a fileID
 // requested twice while the first is still in flight downloads once.
+// `fetchOriginal` passes no key, so it is not shared with a fetch of the same
+// file already in flight.
 //
 // Integrity. The reused streaming decrypt is the enforced guarantee: every
 // chunk is authenticated and the writer renames the file into place only once
@@ -27,9 +29,10 @@
 // It also names each original's save path under the download directory
 // (`savePath`), where `Photo.download()` and `lib.backup()` put it. A copy in
 // the cache does not count as saved there, but is copied there rather than
-// fetched again.
+// fetched again. An original stored there whose bytes do not match its content
+// hash is replaced only by a download that does (`replaceOriginal`).
 
-import { existsSync, readFileSync, statSync } from "node:fs";
+import { createReadStream, existsSync, readFileSync, statSync } from "node:fs";
 import {
     chmod,
     copyFile,
@@ -44,6 +47,12 @@ import {
 import { basename, dirname, extname, join } from "node:path";
 
 import type { ApiClient } from "../api/client.js";
+import {
+    chunkHashFinal,
+    chunkHashInit,
+    chunkHashUpdate,
+    init,
+} from "../crypto/index.js";
 import {
     downloadFile,
     downloadThumbnail,
@@ -349,6 +358,26 @@ export const storedAtSavePath = (
     return storedOriginal(dirname(path), basename(path, extname(path)), file);
 };
 
+// The content hash of the original stored at `stored`, computed as the download
+// check computes it: over each file's bytes, read in chunks, and for a live
+// photo `<imageHash>:<videoHash>`.
+export const storedHash = async (stored: {
+    path: string;
+    videoPath?: string;
+}): Promise<string> => {
+    await init();
+    const hashFile = async (path: string): Promise<string> => {
+        const state = chunkHashInit();
+        for await (const chunk of createReadStream(path)) {
+            chunkHashUpdate(state, chunk as Buffer);
+        }
+        return chunkHashFinal(state);
+    };
+    const hash = await hashFile(stored.path);
+    if (stored.videoPath === undefined) return hash;
+    return `${hash}:${await hashFile(stored.videoPath)}`;
+};
+
 // Copy bytes into `dest` via a temp file in the same directory plus rename, so
 // `dest` appears only once it is whole ("present means complete"). As in the
 // download writer, the temp file is fsynced before the rename and the directory
@@ -405,6 +434,62 @@ export const placeOriginal = async (
     return { path, videoPath };
 };
 
+// Replace the original of `file` stored at its save path under `root` with a
+// new copy, keeping the stored one until the copy is complete and its hash,
+// computed as `storedHash` computes it, is the content hash the file's metadata
+// records. `get` is given a temporary file beside the save path and writes the
+// copy there, a live photo as its image and its video beside that file, each
+// with its own extension. Only a copy that matches is renamed over the stored
+// one, and a live photo's image and video are both hashed before either is
+// renamed. The image is renamed first: should the run stop, or the video's
+// rename fail, before the video is renamed, the new image is left beside the
+// old video, and a later backup with `verify` replaces the video if the video
+// was the part that did not match. When `get` fails or the copy does not match,
+// the temporary files are removed and the stored original stays as it was.
+export const replaceOriginal = async (
+    root: string,
+    file: EnteFile,
+    get: (dest: string) => Promise<{ path: string; videoPath?: string }>,
+): Promise<{ path: string; videoPath?: string }> => {
+    const dest = savePath(root, file);
+    const dir = dirname(dest);
+    // Named as the download writer names its temporary files; a live photo's
+    // image and video get this name with their own extensions. The next
+    // backup removes any that a killed run leaves (`removeLeftoverTempFiles`).
+    const tmp = join(
+        dir,
+        `.quak-${process.pid}-${Math.random().toString(36).slice(2)}.tmp`,
+    );
+    let got: { path: string; videoPath?: string } | undefined;
+    try {
+        got = await get(tmp);
+        if ((await storedHash(got)) !== file.metadata.hash) {
+            throw new Error(
+                "the downloaded original does not match its content hash either",
+            );
+        }
+        if (got.videoPath === undefined) {
+            await rename(got.path, dest);
+            await fsyncPath(dir);
+            return { path: dest };
+        }
+        const path = withExtension(dest, extname(got.path));
+        const videoPath = withExtension(dest, extname(got.videoPath));
+        await rename(got.path, path);
+        await rename(got.videoPath, videoPath);
+        const name = basename(dest, extname(dest));
+        await writeLivePhotoJSON(dir, name, { path, videoPath });
+        return { path, videoPath };
+    } finally {
+        // Whatever was not renamed into place. A failure to remove it must
+        // not replace the error that explains what went wrong.
+        for (const path of [tmp, got?.path, got?.videoPath]) {
+            if (path === undefined) continue;
+            await rm(path, { force: true }).catch(() => undefined);
+        }
+    }
+};
+
 export class ContentCache implements PhotoContent, ThumbnailsAPI {
     private readonly pools: RequestPools;
     private readonly source: ContentSource;
@@ -496,14 +581,34 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         fileID: number,
         opts?: ContentOptions,
     ): Promise<ContentResult> {
-        return this.get(fileID, "original", "on-demand", opts?.onProgress);
+        const file = this.fileOrThrow(fileID);
+        return this.get(file, "original", "on-demand", opts?.onProgress);
     }
 
     async thumbnail(
         fileID: number,
         opts?: ContentOptions,
     ): Promise<ContentResult> {
-        return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
+        const file = this.fileOrThrow(fileID);
+        return this.get(file, "thumbnail", "on-demand", opts?.onProgress);
+    }
+
+    // The original of `file`, fetched and cached as `original` does. `file` is
+    // used as given, so `getFile` need not know it.
+    async originalOf(
+        file: EnteFile,
+        opts?: ContentOptions,
+    ): Promise<ContentResult> {
+        return this.get(file, "original", "on-demand", opts?.onProgress);
+    }
+
+    // The thumbnail of `file`, fetched and cached as `thumbnail` does. `file`
+    // is used as given, so `getFile` need not know it.
+    async thumbnailOf(
+        file: EnteFile,
+        opts?: ContentOptions,
+    ): Promise<ContentResult> {
+        return this.get(file, "thumbnail", "on-demand", opts?.onProgress);
     }
 
     // Put the original at the save path of `file` under the download directory
@@ -530,7 +635,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         destination: string,
     ): Promise<ContentResult> {
         const result = await this.acquire(
-            fileID,
+            this.fileOrThrow(fileID),
             "original",
             "on-demand",
             undefined,
@@ -543,6 +648,34 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         };
     }
 
+    // Fetch an original through the content pool to `destination`, a live
+    // photo as its image and its video beside it, and return where it landed.
+    // Unlike `backupOriginal`, it fetches even when the cache or the save path
+    // already holds the original, and it records nothing: what it writes is
+    // the caller's. It is not shared with a fetch of the same file already
+    // under way, which writes somewhere else.
+    async fetchOriginal(
+        fileID: number,
+        destination: string,
+    ): Promise<{ path: string; videoPath?: string }> {
+        const file = this.fileOrThrow(fileID);
+        return this.pools.content.run(
+            async () => {
+                const fetched = await this.fetchInto(
+                    file,
+                    destination,
+                    "original",
+                    undefined,
+                );
+                for (const path of [fetched.path, fetched.videoPath]) {
+                    if (path !== undefined) await chmod(path, FILE_MODE);
+                }
+                return fetched;
+            },
+            { priority: "on-demand" },
+        );
+    }
+
     async ensure(args: EnsureOptions): Promise<EnsureResult[]> {
         return this.ensureThumbnails(args);
     }
@@ -607,7 +740,8 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         onProgress: ((event: EnsureEvent) => void) | undefined,
     ): Promise<EnsureResult> {
         try {
-            const result = await this.acquire(fileID, kind, priority, signal);
+            const file = this.fileOrThrow(fileID);
+            const result = await this.acquire(file, kind, priority, signal);
             const status = result.cached ? "skipped" : "done";
             onProgress?.({ fileID, status, path: result.path });
             return { fileID, path: result.path };
@@ -622,8 +756,16 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         }
     }
 
+    // The record `getFile` gives for `fileID`. It throws for a file `getFile`
+    // does not know.
+    private fileOrThrow(fileID: number): EnteFile {
+        const file = this.getFile(fileID);
+        if (!file) throw new Error(`content cache: unknown file ${fileID}`);
+        return file;
+    }
+
     private async get(
-        fileID: number,
+        file: EnteFile,
         kind: Kind,
         priority: Priority,
         onProgress: ((event: ContentEvent) => void) | undefined,
@@ -631,7 +773,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         const onByte: ProgressCallback | undefined = onProgress
             ? (bytesDone) => onProgress({ status: "downloading", bytesDone })
             : undefined;
-        const result = await this.acquire(fileID, kind, priority, undefined, {
+        const result = await this.acquire(file, kind, priority, undefined, {
             onByte,
         });
         onProgress?.(
@@ -653,7 +795,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
     // photo's original is present only with its video, and is returned with
     // it.
     private async acquire(
-        fileID: number,
+        file: EnteFile,
         kind: Kind,
         priority: Priority,
         signal: AbortSignal | undefined,
@@ -664,8 +806,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
         videoPath?: string;
         cached: boolean;
     }> {
-        const file = this.getFile(fileID);
+        const fileID = file.id;
-        if (!file) throw new Error(`content cache: unknown file ${fileID}`);
         const isLivePhoto =
             kind === "original" && file.metadata.fileType === "livePhoto";
 

src/library/index.ts

@@ -18,6 +18,17 @@
 // the caller (the default reads stay silent and serve the last good copy). The
 // default methods and the background loop are unchanged.
 //
+// `freshCollections()` is the fresh read of the collection list alone (issue
+// #210): one collections request, no collection's files, and nothing stored,
+// for a caller such as `quak collections` that needs no more. `freshFiles()`
+// is the fresh read of one collection's files, and stores nothing either
+// (issue #217). A caller that asks for each refresh it needs can also open the
+// library with `refreshOnOpen: false`, so that `open()` starts none.
+//
+// A refresh reports through `onProgress` each collection whose files it has
+// fetched, so a large account, whose first refresh fetches the files of every
+// collection, shows progress.
+//
 // A refresh stages all of its network work first and only mutates the store
 // once every fetch has succeeded. A refresh that fails partway therefore never
 // becomes visible to reads: the last good snapshot stays in place, and the
@@ -55,6 +66,8 @@ import {
     ContentCache,
     savePath,
     storedAtSavePath,
+    type ContentOptions,
+    type ContentResult,
     type ContentSource,
     type ThumbnailsAPI,
     type EnsureOptions,
@@ -89,6 +102,7 @@ export {
 } from "./content.js";
 export { type MLDataAPI, type SimilarResult } from "./mlsearch.js";
 import type { CollectionsPage, FilesPage } from "../client.js";
+import { describeError } from "../errors.js";
 import { MLDATA_BATCH_SIZE, type MLData } from "../mldata-fetch.js";
 import type { Collection, EnteFile } from "../model/types.js";
 import { runBackup, type BackupOptions, type BackupResult } from "../backup.js";
@@ -138,12 +152,16 @@ export interface LibraryClient {
         sinceTime: number;
     }): Promise<FilesPage>;
     // Fetch ML data (face detections + CLIP embeddings) for up to a batch of
-    // files. Optional: a client without it simply disables ML fetching, leaving
+    // files, and for each file whose ML data did not decrypt, gunzip or parse,
-    // the metadata refresh untouched.
+    // why not. Optional: a client without it simply disables ML fetching,
+    // leaving the metadata refresh untouched.
     fetchMLData?(args: {
         fileIDs: number[];
         fileKeys: Map<number, Uint8Array>;
-    }): Promise<Map<number, MLData>>;
+    }): Promise<{
+        payloads: Map<number, MLData>;
+        errors: Map<number, unknown>;
+    }>;
     // The byte source for the on-disk content cache. Optional so a mock client
     // that only serves metadata still satisfies the interface; when absent (and
     // no explicit `contentSource` is passed to `open`) the content cache is
@@ -154,7 +172,11 @@ export interface LibraryClient {
 // A progress event for one unit of background work. A metadata "refresh" or an
 // ML "fetchMLData" pass each fire "started" before their network work and then
 // exactly one of "done" or "failed"; "failed" carries the error message and
-// an ML "done" reports how many payloads it stored. The precache fills
+// an ML "done" reports how many payloads it stored and, in `mlDataErrors`,
+// each file whose ML data did not decrypt, gunzip or parse, with the reason.
+// In between, a refresh fires "progress" after each collection whose files it
+// fetched: `fetched` is how many collections it has fetched the files of so
+// far, `total` how many it will. The precache fills
 // ("precacheThumbnails"/"precacheOriginals", #48) fire "started"/"done" around
 // each sweep that has work, "done" reporting the count newly cached.
 export interface RefreshEvent {
@@ -163,9 +185,11 @@ export interface RefreshEvent {
         | "fetchMLData"
         | "precacheThumbnails"
         | "precacheOriginals";
-    status: "started" | "done" | "failed";
+    status: "started" | "progress" | "done" | "failed";
     error?: string;
     fetched?: number;
+    total?: number;
+    mlDataErrors?: { fileID: number; error: string }[];
 }
 
 export type RefreshProgressCallback = (event: RefreshEvent) => void;
@@ -184,6 +208,10 @@ export interface LibraryOptions {
     // `Infinity` starts none: the library then refreshes only as it opens and
     // when `fresh()` or `backup()` asks.
     refreshIntervalSeconds?: number;
+    // Whether `open()` starts a refresh. Defaults to true. With false, `open()`
+    // returns at once onto what the cache holds, empty or not, and a refresh
+    // starts only when `fresh()` or `backup()` asks or the timer fires.
+    refreshOnOpen?: boolean;
     onProgress?: RefreshProgressCallback;
     // The bounded request pools (issue #45), shared by the ML-data fetch (the
     // metadata pool) and the content cache. Defaults to a fresh set at the
@@ -225,6 +253,11 @@ export interface LibraryStatus {
     lastMLFetchAt?: number;
     // The most recent ML fetch pass's error, set only while it failed.
     lastMLError?: string;
+    // Each file whose ML data the last ML fetch pass that succeeded got but
+    // could not decrypt, gunzip or parse, with the reason. A pass still
+    // running leaves it as it is. A pass that fails leaves it as it was. Such
+    // ML data is not cached, so the next pass asks for it again.
+    mlDataErrors: { fileID: number; error: string }[];
     // ML payloads stored on disk and CLIP embeddings in the index; undefined
     // when ML fetching is disabled.
     mlStored?: number;
@@ -292,6 +325,12 @@ export class Library {
     private lastError?: string;
     private lastMLFetchAt?: number;
     private lastMLError?: string;
+    // For each file whose ML data the last ML fetch pass that succeeded got
+    // but could not decrypt, gunzip or parse, why not, by fileID. A pass
+    // gathers its own and puts them here when it ends without failing, so
+    // `status()` and `backup()` never see the list of a pass that is still
+    // running or that failed. Kept in RAM only, never in the cache.
+    private mlDataErrors = new Map<number, unknown>();
     // The plain-record projection as of the last refresh, and the GUI change
     // subscribers. A refresh that alters the projection notifies each with the
     // delta; `lastRecords` is kept current every refresh so a subscriber that
@@ -363,7 +402,8 @@ export class Library {
     // then opens empty with the failure recorded in `status()`. With an
     // existing cache the first refresh runs in the background and `open()`
     // returns as soon as the cached data is ready — an unreachable server does
-    // not block opening.
+    // not block opening. With `refreshOnOpen: false` there is no first
+    // refresh: `open()` returns onto the cache as it is.
     static async open(opts: LibraryOptions): Promise<Library> {
         const { userID } = opts.client.whoami();
         const cacheDirectory =
@@ -466,7 +506,10 @@ export class Library {
         // refresh.
         precache?.start();
 
-        if (store.loadedFromDisk) {
+        if (opts.refreshOnOpen === false) {
+            // The caller asks for each refresh it needs.
+            lib.scheduleNext();
+        } else if (store.loadedFromDisk) {
             // An existing copy already answers reads; refresh in the background
             // and start the interval once that first cycle settles.
             void lib.runRefresh().then(() => lib.scheduleNext());
@@ -542,6 +585,7 @@ export class Library {
             lastError: this.lastError,
             lastMLFetchAt: this.lastMLFetchAt,
             lastMLError: this.lastMLError,
+            mlDataErrors: this.describeMLDataErrors(),
             mlStored: ml?.stored,
             mlIndexed: ml?.indexed,
             originalsUsedBytes: originals?.usedBytes,
@@ -569,6 +613,69 @@ export class Library {
         };
     }
 
+    // The fresh read of the collection list alone (issue #210): every live
+    // collection that decrypts, in the server's order, from one collections
+    // request. Unlike a refresh, it fetches no collection's files, which on a
+    // large account take long. Nothing is stored: a refresh fetches a
+    // collection's files only when the collection is new or has changed since
+    // the stored copy, so a collection stored here without its files would
+    // never have them fetched.
+    async freshCollections(): Promise<Collection[]> {
+        if (this.closed) throw new Error("the library is closed");
+        const { collections } = await this.client.collectionsSince({
+            sinceTime: 0,
+        });
+        return collections;
+    }
+
+    // The fresh read of one collection's files (issue #217): every live file
+    // in it that decrypts, in the server's order. `collection` is a record
+    // `freshCollections()` returned, whose key decrypts the files. It fetches
+    // no other collection's files. Like `freshCollections()`, it stores
+    // nothing. It returns each file once, as the latest copy the server sent,
+    // where the first copy was.
+    async freshFiles(collection: Collection): Promise<EnteFile[]> {
+        if (this.closed) throw new Error("the library is closed");
+        const { files } = await this.client.filesSince({
+            collectionID: collection.id,
+            collectionKey: collection.key,
+            sinceTime: 0,
+        });
+        const byID = new Map<number, EnteFile>();
+        for (const file of files) byID.set(file.id, file);
+        return [...byID.values()];
+    }
+
+    // The original of `file`, fetched and cached as `Photo.original()` does.
+    // `file` is used as given, so the store need not hold it. On an empty
+    // cache, `quak get` passes a file that `freshFiles()` returned.
+    async originalOf(
+        file: EnteFile,
+        opts?: ContentOptions,
+    ): Promise<ContentResult> {
+        if (!this.cache) {
+            throw new Error(
+                "originalOf requires a library opened with a content cache",
+            );
+        }
+        return this.cache.originalOf(file, opts);
+    }
+
+    // The thumbnail of `file`, fetched and cached as `Photo.thumbnail()` does.
+    // `file` is used as given, so the store need not hold it. On an empty
+    // cache, `quak get-thumb` passes a file that `freshFiles()` returned.
+    async thumbnailOf(
+        file: EnteFile,
+        opts?: ContentOptions,
+    ): Promise<ContentResult> {
+        if (!this.cache) {
+            throw new Error(
+                "thumbnailOf requires a library opened with a content cache",
+            );
+        }
+        return this.cache.thumbnailOf(file, opts);
+    }
+
     // Back up every in-scope file to `opts.downloadDirectory`, or else the
     // library's, each original at its save path, with a durable failure
     // ledger (issue #51). Takes the lock in that directory first, unless the
@@ -606,10 +713,13 @@ export class Library {
                 undecryptable: () => this.store.listUndecryptable(),
                 original: (fileID, destination) =>
                     cache!.backupOriginal(fileID, destination),
+                fetchOriginal: (fileID, destination) =>
+                    cache!.fetchOriginal(fileID, destination),
                 thumbnail: (fileID) => cache!.thumbnail(fileID),
                 contentConcurrency: this.pools.content.concurrency,
                 fetchMLData: () => this.fetchMLDataNow(),
                 mlData: (fileID) => this.mldata.forFile({ fileID }),
+                mlDataError: (fileID) => this.mlDataErrors.get(fileID),
             },
             { ...opts, downloadDirectory },
         );
@@ -719,18 +829,28 @@ export class Library {
         // Stage per-collection file diffs. A collection's files are
         // re-enumerated only when its updationTime has advanced past the cached
         // copy; an unchanged album's file list cannot have changed. New
-        // collections enumerate from the beginning of time.
+        // collections enumerate from the beginning of time. Each collection
+        // whose files are fetched is reported, with how many there are to
+        // fetch.
+        const changedCollections = page.collections.filter((collection) => {
+            const known = this.store.getCollection(collection.id);
+            return !known || collection.updationTime > known.updationTime;
+        });
         const filePages: { collectionID: number; page: FilesPage }[] = [];
-        for (const collection of page.collections) {
+        for (const collection of changedCollections) {
             const known = this.store.getCollection(collection.id);
-            if (known && collection.updationTime <= known.updationTime)
-                continue;
             const filePage = await this.client.filesSince({
                 collectionID: collection.id,
                 collectionKey: collection.key,
                 sinceTime: known ? known.updationTime : 0,
             });
             filePages.push({ collectionID: collection.id, page: filePage });
+            this.emit({
+                operation: "refresh",
+                status: "progress",
+                fetched: filePages.length,
+                total: changedCollections.length,
+            });
         }
 
         // Network work done; commit to the store and persist only if something
@@ -836,8 +956,11 @@ export class Library {
 
     // One ML fetch pass: fetch, decrypt and store the ML data for every file
     // the store knows about that is not cached (or whose `updationTime` has
-    // advanced), through the metadata pool, and update the CLIP index. A
+    // advanced), through the metadata pool, and update the CLIP index. For a
-    // failure is reported through `status()` and `onProgress`, then thrown.
+    // file whose ML data does not decrypt, gunzip or parse, it gathers the
+    // reason instead, and puts the reasons it gathered in `mlDataErrors` when
+    // it ends without failing. A failure leaves `mlDataErrors` as it was and
+    // is reported through `status()` and `onProgress`, then thrown.
     private async runMLFetch(): Promise<void> {
         const mldata = this.mlStore;
         // Bind so the call keeps the client as its receiver when invoked
@@ -847,7 +970,14 @@ export class Library {
 
         const files = this.uniqueFiles();
         const needed = mldata.neededFor(files);
-        if (needed.length === 0) return;
+        // This pass's reasons, put in `this.mlDataErrors` when it ends without
+        // failing. A file whose ML data did not decode in the pass before is
+        // still among `needed`, so this pass gives its reason anew.
+        const mlDataErrors = new Map<number, unknown>();
+        if (needed.length === 0) {
+            this.mlDataErrors = mlDataErrors;
+            return;
+        }
 
         this.emit({ operation: "fetchMLData", status: "started" });
         try {
@@ -862,20 +992,25 @@ export class Library {
             for (let i = 0; i < needed.length; i += MLDATA_BATCH_SIZE) {
                 if (this.closed) break;
                 const batch = needed.slice(i, i + MLDATA_BATCH_SIZE);
-                const payloads = await this.pools.metadata.run(
+                const { payloads, errors } = await this.pools.metadata.run(
                     () => fetchMLData({ fileIDs: batch, fileKeys }),
                     { priority: "background" },
                 );
+                for (const [id, error] of errors) {
+                    mlDataErrors.set(id, error);
+                }
                 stored += (await mldata.storeFetched(payloads, updation))
                     .stored;
             }
 
+            this.mlDataErrors = mlDataErrors;
             this.lastMLFetchAt = Date.now();
             this.lastMLError = undefined;
             this.emit({
                 operation: "fetchMLData",
                 status: "done",
                 fetched: stored,
+                mlDataErrors: this.describeMLDataErrors(),
             });
         } catch (err) {
             const error = err instanceof Error ? err.message : String(err);
@@ -885,6 +1020,14 @@ export class Library {
         }
     }
 
+    // `mlDataErrors` as `status()` and the ML "done" event give it.
+    private describeMLDataErrors(): { fileID: number; error: string }[] {
+        return [...this.mlDataErrors].map(([fileID, err]) => ({
+            fileID,
+            error: describeError(err),
+        }));
+    }
+
     // The distinct files the store holds, one entry per fileID (a file in
     // several collections shares its ML data), each carrying the key and the
     // newest `updationTime` seen across its memberships.

src/library/mldata.ts

@@ -61,7 +61,7 @@ export interface MLIndex {
 // Pull the CLIP embedding out of a payload, or undefined when it is absent or
 // misshapen. Kept strict so a bad payload is skipped rather than corrupting the
 // packed index.
-const clipEmbedding = (payload: MLData): number[] | undefined => {
+export const clipEmbedding = (payload: MLData): number[] | undefined => {
     const clip = payload.clip;
     if (typeof clip !== "object" || clip === null) return undefined;
     const embedding = (clip as { embedding?: unknown }).embedding;

src/metadata-backup.ts

@@ -1,10 +1,16 @@
 import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
 import { join } from "node:path";
 import * as jpeg from "jpeg-js";
+import { clipKeywords } from "./clip-keywords.js";
 import type { Client } from "./client.js";
+import { describeError } from "./errors.js";
 import { readExifTags } from "./exif.js";
 import type { Library, Photo } from "./library/index.js";
-import { sanitizeFileName } from "./filename.js";
+import {
+    MAX_NAME_BYTES,
+    sanitizeFileName,
+    shortenToBytes,
+} from "./filename.js";
 import {
     fetchMLDataBatch,
     MLDATA_BATCH_SIZE,
@@ -12,7 +18,10 @@ import {
 } from "./mldata-fetch.js";
 import type { EnteFile } from "./model/types.js";
 
-export type ProgressCallback = (message: string) => void;
+// Reports one line of the dump's progress. `level` is "error" when the line
+// reports ML data that could not be read, and is left out for every other
+// line.
+export type ProgressCallback = (message: string, level?: "error") => void;
 
 export interface MetadataBackupOptions {
     exif?: boolean;
@@ -89,16 +98,18 @@ const extractExif = async (
 
 // Dump every decrypted metadata layer the account holds into a directory tree
 // of plain JSON: account, per-collection, and per-file records including the
-// private and public magic metadata and (by default) the ML data. Collections
+// private and public magic metadata and the ML data, with the keywords of its
+// CLIP embedding in place of the embedding (see clip-keywords.ts). Collections
 // and files are enumerated from the library's cache, which the caller refreshes
-// first. Returns how many ML data requests failed; their files are still
+// first. Returns how many files have no ML data because their ML data request
+// failed or their ML data did not decrypt, gunzip or parse; they are still
 // written, with `mlDataError` in place of `mlData`.
 export const runMetadataBackup = async (
     lib: Library,
     client: Client,
     outDir: string,
     opts?: MetadataBackupOptions,
-): Promise<{ failedMLBatches: number }> => {
+): Promise<{ filesWithMLDataError: number }> => {
     const log = opts?.onProgress ?? (() => {});
     const wantExif = opts?.exif ?? false;
 
@@ -125,7 +136,15 @@ export const runMetadataBackup = async (
         const col = lib.getCollection(album.collectionID);
         if (!col) continue;
 
-        const dirName = `${col.id}-${sanitizeFileName(col.name, "unnamed")}`;
+        // The album's ID keeps the folders apart, so the name after it can be
+        // cut short to fit in a file name.
+        const prefix = `${col.id}-`;
+        const dirName =
+            prefix +
+            shortenToBytes(
+                sanitizeFileName(col.name, "unnamed"),
+                MAX_NAME_BYTES - prefix.length,
+            );
         const colDir = join(outDir, "collections", dirName);
         mkdirSync(colDir, { recursive: true });
 
@@ -164,26 +183,34 @@ export const runMetadataBackup = async (
     }
 
     // One failed request (retries exhausted) must not end the dump: its files
-    // get the reason in `mlDataError` and the other batches go on.
+    // get the reason in `mlDataError` and the other batches go on. So does a
+    // file whose ML data does not decrypt, gunzip or parse.
     log("Fetching ML data (face detections, CLIP embeddings)...");
     const mlDataMap = new Map<number, MLData>();
     const mlDataErrors = new Map<number, string>();
-    let failedMLBatches = 0;
     const fileIDs = [...fileKeys.keys()];
     for (let i = 0; i < fileIDs.length; i += MLDATA_BATCH_SIZE) {
         const batch = fileIDs.slice(i, i + MLDATA_BATCH_SIZE);
         try {
-            const result = await fetchMLDataBatch(
+            const { payloads, errors } = await fetchMLDataBatch(
                 client.getApiClient(),
                 batch,
                 fileKeys,
             );
-            for (const [id, payload] of result) mlDataMap.set(id, payload);
+            for (const [id, payload] of payloads) mlDataMap.set(id, payload);
+            for (const [id, err] of errors) {
+                const reason = describeError(err);
+                log(
+                    `ML data of file ${id} could not be read: ${reason}`,
+                    "error",
+                );
+                mlDataErrors.set(id, reason);
+            }
         } catch (err) {
             const reason = err instanceof Error ? err.message : String(err);
-            failedMLBatches++;
             log(
                 `ML data request for ${batch.length} file(s) failed: ${reason}`,
+                "error",
             );
             for (const id of batch) mlDataErrors.set(id, reason);
         }
@@ -205,8 +232,16 @@ export const runMetadataBackup = async (
         if (file.pubMagicMetadata)
             fileMeta.pubMagicMetadata = file.pubMagicMetadata;
 
+        // The ML data without the CLIP embedding, and the keywords that
+        // embedding gives.
         const ml = mlDataMap.get(file.id);
-        if (ml) fileMeta.mlData = ml;
+        if (ml) {
+            const { clip: _clip, ...mlDataWithoutClip } = ml;
+            fileMeta.mlData = mlDataWithoutClip;
+            const isVideo = file.metadata.fileType === "video";
+            const keywords = clipKeywords(ml, isVideo);
+            if (keywords) fileMeta.keywords = keywords;
+        }
         const mlError = mlDataErrors.get(file.id);
         if (mlError) fileMeta.mlDataError = mlError;
 
@@ -229,5 +264,5 @@ export const runMetadataBackup = async (
     }
 
     log("Metadata backup complete.");
-    return { failedMLBatches };
+    return { filesWithMLDataError: mlDataErrors.size };
 };

src/mldata-fetch.ts

@@ -28,43 +28,52 @@ interface RawRemoteFileData {
     updatedAt?: number;
 }
 
-// Decrypt one entry with its file key and gunzip the JSON payload. Returns
+// Decrypt one entry with its file key and gunzip the JSON payload. Throws why
-// undefined when the key is unknown or the entry does not decrypt/parse, so one
+// when the entry does not decrypt, gunzip or parse, or parses to something
-// corrupt file never fails a whole batch.
+// other than a JSON object, such as `null`, a number or an array.
-const decodeEntry = (
+const decodeEntry = (entry: RawRemoteFileData, key: Uint8Array): MLData => {
-    entry: RawRemoteFileData,
-    key: Uint8Array | undefined,
-): MLData | undefined => {
-    if (!key) return undefined;
-    try {
     const decrypted = decryptBlob(
         fromBase64(entry.encryptedData),
         fromBase64(entry.decryptionHeader),
         key,
     );
     const json = gunzipSync(Buffer.from(decrypted)).toString("utf-8");
-        return JSON.parse(json) as MLData;
+    const payload: unknown = JSON.parse(json);
-    } catch {
+    if (
-        return undefined;
+        typeof payload !== "object" ||
+        payload === null ||
+        Array.isArray(payload)
+    ) {
+        throw new Error("not a JSON object");
     }
+    return payload as MLData;
 };
 
 // Fetch ML data for up to `MLDATA_BATCH_SIZE` ids in a single request. This is
 // the unit the request pools schedule; callers with more ids split them into
-// batches and submit each batch to the pool.
+// batches and submit each batch to the pool. Returns the payload of each file
+// whose entry decodes and, in `errors`, why not for each file whose entry does
+// not, so one corrupt file never fails a whole batch. An entry for a file
+// whose key is not known, which quak did not ask for, is skipped.
 export const fetchMLDataBatch = async (
     api: ApiClient,
     fileIDs: number[],
     fileKeys: Map<number, Uint8Array>,
-): Promise<Map<number, MLData>> => {
+): Promise<{ payloads: Map<number, MLData>; errors: Map<number, unknown> }> => {
     const { data } = await api.postJSON<{ data: RawRemoteFileData[] }>(
         "/files/data/fetch",
         { type: "mldata", fileIDs },
     );
-    const result = new Map<number, MLData>();
+    const payloads = new Map<number, MLData>();
+    const errors = new Map<number, unknown>();
     for (const entry of data ?? []) {
-        const payload = decodeEntry(entry, fileKeys.get(entry.fileID));
+        const key = fileKeys.get(entry.fileID);
-        if (payload) result.set(entry.fileID, payload);
+        if (!key) continue;
+        try {
+            payloads.set(entry.fileID, decodeEntry(entry, key));
+        } catch (err) {
+            errors.set(entry.fileID, err);
         }
-    return result;
+    }
+    return { payloads, errors };
 };

src/retry.ts

@@ -10,7 +10,7 @@
 // declining to retry a transient one costs a single file that the next run
 // picks up anyway. When the evidence is ambiguous, fail fast.
 
-import { ApiError, TruncatedStreamError } from "./errors.js";
+import { ApiError, causeChain, TruncatedStreamError } from "./errors.js";
 
 export interface RetryOptions {
     // Total calls, not retries: `attempts: 1` disables retrying.
@@ -24,6 +24,9 @@ export interface RetryOptions {
     sleep?: (ms: number) => Promise<void>;
     // Injected so the jitter is reproducible under test.
     random?: () => number;
+    // Called before each wait with the error that failed attempt number
+    // `attempt` and how long the wait is, for a caller that logs retries.
+    onRetry?: (err: unknown, attempt: number, delayMs: number) => void;
 }
 
 export type ResolvedRetryOptions = Required<RetryOptions>;
@@ -35,6 +38,7 @@ export const DEFAULT_RETRY_OPTIONS: ResolvedRetryOptions = {
     sleep: (ms: number): Promise<void> =>
         new Promise((resolve) => setTimeout(resolve, ms)),
     random: Math.random,
+    onRetry: () => {},
 };
 
 // For a run nobody is watching, such as `quak backup` from cron. A request that
@@ -55,9 +59,18 @@ export const resolveRetryOptions = (
     maxDelayMs: opts?.maxDelayMs ?? DEFAULT_RETRY_OPTIONS.maxDelayMs,
     sleep: opts?.sleep ?? DEFAULT_RETRY_OPTIONS.sleep,
     random: opts?.random ?? DEFAULT_RETRY_OPTIONS.random,
+    onRetry: opts?.onRetry ?? DEFAULT_RETRY_OPTIONS.onRetry,
 });
 
 // Transport failures: the request did not complete, for reasons below HTTP.
+//
+// The errnos are Node's. The binary `make install` builds runs under bun,
+// whose `fetch` rejects with a plain `Error` that has no `cause` and a `code`
+// of bun's own. Bun gives `ConnectionRefused` for a refused connection and for
+// a TLS handshake that failed, and `ConnectionRefused` or `FailedToOpenSocket`
+// for a name that does not resolve. For a connection reset or closed before
+// the response ended, it gives Node's `ECONNRESET`. These were found by
+// running that binary, built with bun 1.3.13, against listeners on 127.0.0.1.
 const TRANSPORT_CODES = new Set([
     "ECONNRESET",
     "ECONNABORTED",
@@ -70,6 +83,9 @@ const TRANSPORT_CODES = new Set([
     "ENETUNREACH",
     "ENETRESET",
     "ENETDOWN",
+    // Bun's.
+    "ConnectionRefused",
+    "FailedToOpenSocket",
 ]);
 
 // The subset of the above that can only be reported before a TCP connection
@@ -87,13 +103,14 @@ const TRANSPORT_CODES = new Set([
 // ambiguity this set exists to exclude. They stay in `TRANSPORT_CODES`, so
 // they remain retryable for idempotent calls; only replay eligibility is
 // narrowed. See `isSafeToReplay`.
+//
+// Bun's `ConnectionRefused` and `FailedToOpenSocket` are absent too, so a
+// `POST` or `PUT` is not replayed on them. When a connection bun kept open from
+// an earlier request closes before any answer, bun sends the request again on
+// a new connection, and if that one fails it gives one of these codes although
+// the server may already have the request.
 const CONNECT_CODES = new Set(["ENOTFOUND", "EAI_AGAIN", "ECONNREFUSED"]);
 
-// `cause` is an arbitrary user-settable property and nothing prevents it from
-// forming a cycle, so the walk is bounded. Hanging the process would be a
-// worse outcome than any misclassification.
-const MAX_CAUSE_DEPTH = 8;
-
 // Collect every `code` in an error's cause chain. undici does not put the
 // errno on the error it throws — it hangs the underlying socket error off
 // `cause`, sometimes more than one level down — so a classifier that only read
@@ -101,20 +118,21 @@ const MAX_CAUSE_DEPTH = 8;
 // connection permanent. `complete` is false when the walk stopped at the
 // depth limit with more of the chain still below it.
 const causeCodes = (err: unknown): { codes: string[]; complete: boolean } => {
+    const { chain, complete } = causeChain(err);
     const codes: string[] = [];
-    let current: unknown = err;
+    for (const link of chain) {
-    for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
+        const { code } = link as { code?: unknown };
-        if (current === null || typeof current !== "object") {
-            return { codes, complete: true };
-        }
-        const { code, cause } = current as { code?: unknown; cause?: unknown };
         if (typeof code === "string") codes.push(code);
-        if (cause === current) return { codes, complete: true };
-        current = cause;
     }
-    return { codes, complete: current === null || typeof current !== "object" };
+    return { codes, complete };
 };
 
+// Does the error's cause chain hold the code of a transport failure, one of
+// `TRANSPORT_CODES`? `isRetryable` retries such an error, and `classify` in
+// `src/backup.ts` calls it transient.
+export const hasTransportCode = (err: unknown): boolean =>
+    causeCodes(err).codes.some((code) => TRANSPORT_CODES.has(code));
+
 const isAbort = (err: unknown): boolean => {
     if (err === null || typeof err !== "object") return false;
     const { name } = err as { name?: unknown };
@@ -156,9 +174,10 @@ export const isRetryable = (err: unknown): boolean => {
     // deliberately literal. Demanding a recognised `cause` instead would
     // classify real network failures as permanent and fail backups that should
     // have succeeded; the cost of the imprecision is bounded by the attempt
-    // count.
+    // count. Bun's fetch rejects with a plain `Error` instead, which is
+    // recognised by its code (see `TRANSPORT_CODES`).
     if (err instanceof TypeError) return true;
-    return causeCodes(err).codes.some((code) => TRANSPORT_CODES.has(code));
+    return hasTransportCode(err);
 };
 
 // Could the first attempt already have taken effect on the server?
@@ -174,11 +193,13 @@ export const isRetryable = (err: unknown): boolean => {
 //
 // Everything else is ambiguous. A 5xx proves the server did process the
 // request. A reset or a broken pipe can arrive after it was fully sent and
-// acted on. A routing errno can be delivered on an established socket. A
+// acted on. A routing errno can be delivered on an established socket. Bun's
-// deadline says nothing at all about the server's state. So every errno in the
+// `ConnectionRefused` and `FailedToOpenSocket` can come from a new connection
-// cause chain must be a connect errno: one other errno anywhere in the chain
+// bun opened to resend a request that already went out on a connection it kept
-// is doubt, and doubt is not replayed. A chain longer than the walk is doubt
+// open. A deadline says nothing at all about the server's state. So every code in the
-// too: the links below the limit were never read.
+// cause chain must be one of `CONNECT_CODES`: one other code anywhere in the
+// chain is doubt, and doubt is not replayed. A chain longer than the walk is
+// doubt too: the links below the limit were never read.
 export const isSafeToReplay = (err: unknown): boolean => {
     const { codes, complete } = causeCodes(err);
     return (
@@ -218,7 +239,9 @@ export const withRetry = async <T>(
                 // and its status intact.
                 throw err;
             }
-            await policy.sleep(backoffMs(attempt, policy));
+            const delayMs = backoffMs(attempt, policy);
+            policy.onRetry(err, attempt, delayMs);
+            await policy.sleep(delayMs);
         }
     }
 };

src/thumbnails.ts

@@ -5,7 +5,7 @@ import type { Client } from "./client.js";
 import type { Library } from "./library/index.js";
 import { ApiError } from "./api/client.js";
 import { encryptBlob, toBase64 } from "./crypto/index.js";
-import type { EnteFile } from "./model/types.js";
+import type { Collection, EnteFile } from "./model/types.js";
 
 // The server refuses a thumbnail larger than the one it already records for the
 // file (`thumbnail.size`, the encrypted size), so these encodings are tried
@@ -49,7 +49,10 @@ export interface ThumbnailFixResult {
     reason?: string;
 }
 
-export type ProgressCallback = (message: string) => void;
+// Reports one line of progress. `level` is "error" when the line reports a
+// file that could not be checked or fixed, and is left out for every other
+// line.
+export type ProgressCallback = (message: string, level?: "error") => void;
 
 // Enumerate every file the library knows about, newest album first, each file
 // once, and report those whose server-side thumbnail is missing. "Missing" is
@@ -111,6 +114,7 @@ export const listMissingThumbnails = async (
                 } else {
                     log(
                         `[${album.name}] Could not check ${photo.title}: ${err instanceof Error ? err.message : String(err)} (not reported as missing)`,
+                        "error",
                     );
                 }
             }
@@ -222,16 +226,18 @@ const encryptThumbnailWithin = (
     return undefined;
 };
 
-// Regenerate and upload a thumbnail for each requested file. Originals are read
+// Regenerate and upload a thumbnail for each requested file. `found` holds the
-// through the library's content cache (`photo.original()`); the generated
+// record of each requested file that a collection holds, and the collection it
-// thumbnail is JPEG-encoded, encrypted under the file's own key, and registered
+// was found in. Each original is read through the library's content cache, by
-// with the server — the encrypt-and-upload path is unchanged. Each file is
+// `lib.originalOf(file)`. The new thumbnail is a JPEG, encrypted under the
+// file's own key. It is uploaded, then registered with the server. Each file is
 // resolved to one outcome (fixed / skipped / failed) and a failure on one file
 // never stops the others.
 export const fixMissingThumbnails = async (
     lib: Library,
     client: Client,
     fileIDs: number[],
+    found: Map<number, { file: EnteFile; collection: Collection }>,
     onProgress?: ProgressCallback,
 ): Promise<ThumbnailFixResult[]> => {
     const log = onProgress ?? (() => {});
@@ -239,31 +245,8 @@ export const fixMissingThumbnails = async (
     const api = client.getApiClient();
     const { userID } = client.whoami();
 
-    // Resolve each requested fileID to its file record and owning album by
-    // enumerating the library, each file taken from the first album that holds
-    // it. The raw `EnteFile` carries the per-file key the thumbnail is
-    // encrypted under, which the projected records deliberately do not.
-    const wanted = new Set(fileIDs);
-    const fileMap = new Map<
-        number,
-        { file: EnteFile; collectionName: string }
-    >();
-    for (const album of lib.albums.list()) {
-        for (const photo of album.photos.list()) {
-            if (!wanted.has(photo.fileID) || fileMap.has(photo.fileID))
-                continue;
-            const file = lib.getFile(album.collectionID, photo.fileID);
-            if (file) {
-                fileMap.set(photo.fileID, {
-                    file,
-                    collectionName: album.name,
-                });
-            }
-        }
-    }
-
     for (const fileID of fileIDs) {
-        const entry = fileMap.get(fileID);
+        const entry = found.get(fileID);
         if (!entry) {
             results.push({
                 fileID,
@@ -275,7 +258,8 @@ export const fixMissingThumbnails = async (
             continue;
         }
 
-        const { file, collectionName } = entry;
+        const { file } = entry;
+        const collectionName = entry.collection.name;
         const title = file.metadata.title;
 
         const skipReason = reasonToSkip(file, userID);
@@ -293,15 +277,10 @@ export const fixMissingThumbnails = async (
         const maxSize = file.thumbnail.size!;
 
         try {
-            const photo = lib.photos.byID({ fileID });
-            if (!photo) {
-                throw new Error("file not present in the library cache");
-            }
-
             log(
                 `[${collectionName}] Downloading ${title} for thumbnail generation...`,
             );
-            const { path } = await photo.original();
+            const { path } = await lib.originalOf(file);
             const fileBytes = new Uint8Array(readFileSync(path));
 
             if (!isJpeg(fileBytes)) {
@@ -346,7 +325,7 @@ export const fixMissingThumbnails = async (
                 ciphertext.length,
                 md5,
             );
-            await api.putFile(url, ciphertext);
+            await api.putFile(url, ciphertext, md5);
             await api.updateThumbnail(file.id, objectKey, toBase64(header));
 
             log(`[${collectionName}] Thumbnail uploaded for ${title}`);
@@ -359,6 +338,7 @@ export const fixMissingThumbnails = async (
         } catch (err) {
             log(
                 `[${collectionName}] FAILED ${title}: ${err instanceof Error ? err.message : err}`,
+                "error",
             );
             results.push({
                 fileID,

test/api/client.test.ts

@@ -780,7 +780,7 @@ describe("ApiClient timeouts", () => {
 
         await client.getJSON("/a");
         await client.postJSON("/b", {});
-        await client.putFile("https://s3.example/x", new Uint8Array([1]));
+        await client.putFile("https://s3.example/x", new Uint8Array([1]), "m");
         await client.putJSON("/c", {});
 
         for (const call of calls) {
@@ -825,7 +825,8 @@ describe("ApiClient timeouts", () => {
             [
                 "putFile",
                 () => new Response(null, { status: 200 }),
-                (c) => c.putFile("https://s3.example/x", new Uint8Array([1])),
+                (c) =>
+                    c.putFile("https://s3.example/x", new Uint8Array([1]), "m"),
             ],
             [
                 "getFileStream",
@@ -1049,7 +1050,7 @@ describe("ApiClient error typing", () => {
         const client = new ApiClient({ fetch, retry: noWait });
 
         const err: unknown = await client
-            .putFile("https://s3.example/obj", new Uint8Array([1, 2]))
+            .putFile("https://s3.example/obj", new Uint8Array([1, 2]), "m")
             .catch((e: unknown) => e);
 
         expect(err).toBeInstanceOf(ApiError);
@@ -1067,7 +1068,11 @@ describe("ApiClient error typing", () => {
         );
         const client = new ApiClient({ fetch, retry: noWait });
 
-        await client.putFile("https://s3.example/obj", new Uint8Array([1, 2]));
+        await client.putFile(
+            "https://s3.example/obj",
+            new Uint8Array([1, 2]),
+            "m",
+        );
         expect(calls).toHaveLength(2);
     });
 
@@ -1222,3 +1227,66 @@ describe("ApiClient non-idempotent requests", () => {
         }
     });
 });
+
+// `onRequest` hears of each request the client sends, as its response arrives
+// or as it fails. The CLI logs these with `-vv`. A retried request is reported
+// once for each attempt.
+describe("ApiClient onRequest", () => {
+    it("reports each request with its status, or with the error it failed with", async () => {
+        const refused = new TypeError("fetch failed");
+        const { fetch } = scriptedFetch(
+            refused,
+            textResponse("unavailable", 503),
+            jsonResponse({ collections: [] }),
+            jsonResponse({}),
+            streamResponse(new Uint8Array([1, 2, 3])),
+        );
+        const requests: unknown[] = [];
+        const client = new ApiClient({
+            fetch,
+            retry: noWait,
+            onRequest: (request) => requests.push(request),
+        });
+
+        await client.getJSON("/collections/v2", { sinceTime: 0 });
+        await client.postJSON("/users/logout", {});
+        await readAll(await client.getFileStream(7));
+
+        const collections = "https://api.ente.io/collections/v2?sinceTime=0";
+        expect(requests).toEqual([
+            { method: "GET", url: collections, error: refused },
+            { method: "GET", url: collections, status: 503 },
+            { method: "GET", url: collections, status: 200 },
+            {
+                method: "POST",
+                url: "https://api.ente.io/users/logout",
+                status: 200,
+            },
+            {
+                method: "GET",
+                url: "https://files.ente.io/?fileID=7",
+                status: 200,
+            },
+        ]);
+    });
+
+    it("reports a presigned upload without its query, which holds the upload's signature", async () => {
+        const { fetch } = scriptedFetch(new Response(null, { status: 200 }));
+        const requests: unknown[] = [];
+        const client = new ApiClient({
+            fetch,
+            retry: noWait,
+            onRequest: (request) => requests.push(request),
+        });
+
+        await client.putFile(
+            "https://s3.example/obj?X-Amz-Signature=abc",
+            new Uint8Array([1, 2]),
+            "m",
+        );
+
+        expect(requests).toEqual([
+            { method: "PUT", url: "https://s3.example/obj", status: 200 },
+        ]);
+    });
+});

test/api/upload.test.ts

@@ -80,13 +80,20 @@ describe("ApiClient.putFile", () => {
         const client = new ApiClient({ fetch, authToken: "secret-tok" });
         const data = new Uint8Array([1, 2, 3, 4, 5]);
 
-        await client.putFile("https://s3.example.com/presigned?sig=abc", data);
+        await client.putFile(
+            "https://s3.example.com/presigned?sig=abc",
+            data,
+            "md5base64==",
+        );
 
         expect(calls[0]!.url).toBe("https://s3.example.com/presigned?sig=abc");
         expect(calls[0]!.init?.method).toBe("PUT");
         const headers = new Headers(calls[0]!.init?.headers as HeadersInit);
         expect(headers.get("Content-Type")).toBe("application/octet-stream");
         expect(headers.get("Content-Length")).toBe("5");
+        // The server signs the MD5 into the URL; storage refuses the PUT
+        // without it.
+        expect(headers.get("Content-MD5")).toBe("md5base64==");
         // Must NOT leak auth headers to S3
         expect(headers.has("X-Auth-Token")).toBe(false);
         expect(headers.has("X-Client-Package")).toBe(false);
@@ -99,7 +106,11 @@ describe("ApiClient.putFile", () => {
         const client = new ApiClient({ fetch });
 
         await expect(
-            client.putFile("https://s3.example.com/bad", new Uint8Array(10)),
+            client.putFile(
+                "https://s3.example.com/bad",
+                new Uint8Array(10),
+                "md5base64==",
+            ),
         ).rejects.toThrow(/403/);
     });
 });

test/cli/backup-end-to-end.test.ts

@@ -9,7 +9,7 @@
  * data this file encrypts with libsodium, as Ente's clients encrypt it. So
  * quak's own code fetches each original, decrypts it, checks it against the
  * hash its uploader recorded and unpacks a live photo, and quak's own retries
- * repeat a failed request. Two tests run `lib.backup()` instead, as a program
+ * repeat a failed request. Three tests run `lib.backup()` instead, as a program
  * using quak as a library does.
  *
  * The account, user 42, has three albums:
@@ -54,6 +54,7 @@ import {
     saveSession,
     type CliContext,
 } from "../../src/cli-commands.js";
+import { stderrLog } from "../../src/cli-log.js";
 import { run } from "../../src/cli-run.js";
 import { loadSession } from "../../src/cli-session.js";
 import { Client } from "../../src/client.js";
@@ -85,10 +86,17 @@ const DAY = "2024/2024-03/2024-03-01";
 const TOAST = new TextEncoder().encode("a few seconds of the wedding toast");
 
 // Ente's ML data for beach.jpg: a face it found, and the photo's CLIP
-// embedding.
+// embedding, here that of a photo of a sunset (see clip-keywords.test.ts).
 const ML_DATA = {
     face: { faces: [{ faceID: "100_1", score: 0.98 }] },
-    clip: { embedding: [0.5, 0.6, 0.7] },
+    clip: {
+        embedding: JSON.parse(
+            readFileSync(
+                new URL("../clip-keywords-sunset.json", import.meta.url),
+                "utf-8",
+            ),
+        ).embedding,
+    },
 };
 
 // Each file's ID, its owner, its original as uploaded, and the metadata its
@@ -274,15 +282,17 @@ afterAll(() => {
 
 // Run `quak backup <dir>` as `bin/quak.ts` runs it, sending its requests
 // through `fetch`, and return its exit code and everything it printed on
-// stderr. Its requests are retried as those of `quak backup` are, except that
+// stderr and on stdout. Its requests are retried as those of `quak backup`
-// a retry does not wait. Its cache is beside `dir`, so each run of one
+// are, except that a retry does not wait. A download is given up after
-// directory finds the cache the run before it left, as cron's runs find the
+// `downloadTimeoutMs` with no bytes received, 60 seconds unless given. Its
-// user's.
+// cache is beside `dir`, so each run of one directory finds the cache the run
+// before it left, as cron's runs find the user's.
 const backup = async (
     dir: string,
-    opts: { verify?: boolean } = {},
+    opts: { verify?: boolean; json?: boolean } = {},
     fetch = server,
-): Promise<{ code: number; stderr: string }> => {
+    downloadTimeoutMs?: number,
+): Promise<{ code: number; stderr: string; stdout: string }> => {
     let printed = "";
     const stderr = new Writable({
         write(chunk: Buffer, _encoding, done) {
@@ -290,8 +300,10 @@ const backup = async (
             done();
         },
     });
+    let output = "";
     const stdout = new Writable({
-        write(_chunk, _encoding, done) {
+        write(chunk: Buffer, _encoding, done) {
+            output += chunk.toString();
             done();
         },
     });
@@ -299,24 +311,26 @@ const backup = async (
     const prompted = async (): Promise<never> => {
         throw new Error("quak backup prompted");
     };
+    const log = stderrLog(stderr, "info");
     const ctx: CliContext = {
         stdout,
-        stderr,
+        log,
         sessionDir,
         cacheDir: `${dir}-cache`,
         loadSession: (path) =>
             loadSession(path, {
                 fetch,
                 retry: { ...UNATTENDED_RETRY_OPTIONS, sleep: async () => {} },
+                downloadTimeoutMs,
             }),
         login: prompted,
         prompt: prompted,
         promptSecret: prompted,
     };
     const code = await new Promise<number>((resolve) => {
-        void run(backupCommand(ctx, dir, opts), stdout, stderr, resolve);
+        void run(backupCommand(ctx, dir, opts), stdout, stderr, log, resolve);
     });
-    return { code, stderr: printed };
+    return { code, stderr: printed, stdout: output };
 };
 
 // The ID of the file whose original a request downloads, or undefined for any
@@ -329,6 +343,41 @@ const downloadOf = (input: RequestInfo | URL): number | undefined => {
         : undefined;
 };
 
+// The error Node gives for a connection to `address` that was refused.
+const refusedAt = (address: string): Error =>
+    Object.assign(new Error(`connect ECONNREFUSED ${address}`), {
+        code: "ECONNREFUSED",
+    });
+
+// What Node's `fetch` rejects with when the server refuses the connection, as
+// for every failure below HTTP: a TypeError that says only "fetch failed",
+// with the reason, and its errno, as its cause. When the server has several
+// addresses and each one refuses, the cause is an AggregateError with an empty
+// message, one error per address and the errno of the first. `reason` is the
+// error as quak prints it and records it in `failures.json`.
+const REFUSED = [
+    {
+        addresses: "its one address",
+        error: (): TypeError =>
+            new TypeError("fetch failed", { cause: refusedAt("1.2.3.4:443") }),
+        reason: "fetch failed: connect ECONNREFUSED 1.2.3.4:443",
+    },
+    {
+        addresses: "each of its two addresses",
+        error: (): TypeError =>
+            new TypeError("fetch failed", {
+                cause: Object.assign(
+                    new AggregateError([
+                        refusedAt("2001:db8::1:443"),
+                        refusedAt("1.2.3.4:443"),
+                    ]),
+                    { code: "ECONNREFUSED" },
+                ),
+            }),
+        reason: "fetch failed: connect ECONNREFUSED 2001:db8::1:443, connect ECONNREFUSED 1.2.3.4:443",
+    },
+];
+
 // The fake server, pushing onto `downloads` the ID of each file whose original
 // is downloaded.
 const recordingDownloads =
@@ -369,7 +418,7 @@ describe("the first backup of the account", () => {
         expect(downloads.sort()).toEqual([100, 200, 300]);
     });
 
-    // The tree "Backup layout" in the README describes. Each original is at
+    // The tree under "The backup directory" in the README. Each original is at
     // its save path, `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, with
     // the extension of its title; the live photo is its image and its video,
     // each with the extension it had inside the ZIP, and a `.livephoto.json`
@@ -444,7 +493,10 @@ describe("the first backup of the account", () => {
             },
             updationTime: TAKEN,
         });
-        expect(beach.mlData).toEqual(ML_DATA);
+        // The JSON holds the ML data without its CLIP embedding, and the
+        // keywords the embedding gives in its place.
+        expect(beach.mlData).toEqual({ face: ML_DATA.face });
+        expect(beach.keywords).toEqual(["Sunset"]);
         expect(beach.imageMetadata.exif.Make.description).toBe("Canon");
         expect(beach.imageMetadata.exif.Model.description).toBe("EOS R5");
 
@@ -455,6 +507,7 @@ describe("the first backup of the account", () => {
             metadata: { title: "toast.mp4", fileType: "video" },
         });
         expect(toast).not.toHaveProperty("mlData");
+        expect(toast).not.toHaveProperty("keywords");
         expect(toast).not.toHaveProperty("imageMetadata");
         expect(toast).not.toHaveProperty("imageMetadataError");
 
@@ -592,6 +645,40 @@ describe("quak backup", () => {
         ).toEqual([]);
     });
 
+    // The user has put a file in collections/ with the name of the Trips
+    // folder, so quak cannot make that folder. It says so on one line, writes
+    // the two albums after it, and exits 1, and --json lists the album's ID
+    // in `failedAlbums`. No file failed, so there is no failures.json.
+    it("exits 1 when an album's folder cannot be made, writes the other albums, and lists it in --json", async () => {
+        const dir = mkdtempSync(join(root, "backup-"));
+        mkdirSync(join(dir, "collections"));
+        writeFileSync(join(dir, "collections", "Trips"), "mine");
+
+        const first = await backup(dir);
+        const { code, stdout } = await backup(dir, { json: true });
+
+        expect(first.code).toBe(1);
+        expect(first.stderr).toMatch(/^FAILED album Trips: EEXIST: /m);
+        expect(code).toBe(1);
+        expect(JSON.parse(stdout)).toMatchObject({
+            failed: 0,
+            undecryptableAlbums: [],
+            failedAlbums: [10],
+        });
+        const collections = join(dir, "collections");
+        expect(readJSON(join(collections, "Family.json")).id).toBe(20);
+        expect(readdirSync(join(collections, "Family")).sort()).toEqual([
+            "IMG_0002.heic",
+            "IMG_0002.mov",
+            "beach.jpg",
+        ]);
+        expect(readJSON(join(collections, "Friend's Wedding.json")).id).toBe(
+            30,
+        );
+        expect(existsSync(join(collections, "Trips.json"))).toBe(false);
+        expect(existsSync(join(dir, "failures.json"))).toBe(false);
+    });
+
     // The server no longer accepts the saved session's token, as after
     // `quak logout` elsewhere, and answers every request with 401. The run
     // stops at the refresh it starts with, says to log in again, and exits 3.
@@ -619,6 +706,317 @@ describe("quak backup", () => {
         expect(readdirSync(dir)).toEqual([]);
     });
 
+    // The server refuses every connection, at its one address or at each of
+    // several. quak tries each request again, here without waiting, and then
+    // the run stops at the refresh it starts with. The line it prints gives
+    // the reason Node's `fetch` put on the error's cause, so a cron log tells
+    // a refused connection from a name that did not resolve, and the exit
+    // code is 1.
+    it.each(REFUSED)(
+        "prints why the refresh could not reach the server at $addresses, and exits 1",
+        async ({ error, reason }) => {
+            const dir = mkdtempSync(join(root, "backup-"));
+            const fetch: typeof globalThis.fetch = async () => {
+                throw error();
+            };
+
+            const { code, stderr } = await backup(dir, {}, fetch);
+
+            expect(code).toBe(1);
+            expect(stderr).toBe(
+                "Starting backup...\n" +
+                    "Refreshing library...\n" +
+                    `quak: ${reason}\n`,
+            );
+        },
+    );
+
+    // The files CDN refuses every connection for toast.mp4, at its one
+    // address or at each of several. Once its retries are spent, the file
+    // goes into `failures.json` with the reason, as a transient failure,
+    // since a later run may well reach the CDN, and the lines the backup
+    // prints for it give the same reason. The rest of the account is backed
+    // up, and the exit code is 1.
+    it.each(REFUSED)(
+        "records why a download could not reach the server at $addresses in failures.json, as transient, and exits 1",
+        async ({ error, reason }) => {
+            let attempts = 0;
+            const fetch: typeof globalThis.fetch = async (input, init) => {
+                if (downloadOf(input) === 200) {
+                    attempts++;
+                    throw error();
+                }
+                return server(input, init);
+            };
+            const dir = mkdtempSync(join(root, "backup-"));
+
+            const { code, stderr } = await backup(dir, {}, fetch);
+
+            expect(code).toBe(1);
+            expect(attempts).toBe(UNATTENDED_RETRY_OPTIONS.attempts);
+            const failures = readJSON(join(dir, "failures.json"));
+            expect(Object.keys(failures.files)).toEqual(["200"]);
+            expect(failures.files["200"]).toMatchObject({
+                fileID: 200,
+                title: "toast.mp4",
+                classification: "transient",
+                attempts: 1,
+                error: reason,
+            });
+            expect(stderr).toContain(`FAILED original toast.mp4: ${reason}\n`);
+            expect(stderr).toContain(
+                `[Friend's Wedding] toast.mp4 (id 200): ${reason}\n`,
+            );
+            expect(readFileSync(join(dir, DAY, "2024-03-01.100.jpg"))).toEqual(
+                Buffer.from(JPEG_WITH_EXIF),
+            );
+            expect(existsSync(join(dir, DAY, "2024-03-01.200.mp4"))).toBe(
+                false,
+            );
+        },
+    );
+
+    // The download of toast.mp4 fails as bun's `fetch` reports a failure, in
+    // the binary `make install` builds: a plain `Error` with bun's own code
+    // and message, and no cause. These are what bun gave for a refused
+    // connection, a name that did not resolve and a reset connection. The
+    // file is tried as often as under Node, and goes into `failures.json`
+    // with that message, as a transient failure.
+    it.each([
+        [
+            "ConnectionRefused",
+            "Unable to connect. Is the computer able to access the url?",
+        ],
+        ["FailedToOpenSocket", "Was there a typo in the url or port?"],
+        [
+            "ECONNRESET",
+            "The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()",
+        ],
+    ])(
+        "records a download that failed with bun's %s in failures.json, as transient",
+        async (bunCode, message) => {
+            let attempts = 0;
+            const fetch: typeof globalThis.fetch = async (input, init) => {
+                if (downloadOf(input) === 200) {
+                    attempts++;
+                    throw Object.assign(new Error(message), {
+                        code: bunCode,
+                        errno: 0,
+                    });
+                }
+                return server(input, init);
+            };
+            const dir = mkdtempSync(join(root, "backup-"));
+
+            const { code } = await backup(dir, {}, fetch);
+
+            expect(code).toBe(1);
+            expect(attempts).toBe(UNATTENDED_RETRY_OPTIONS.attempts);
+            const failures = readJSON(join(dir, "failures.json"));
+            expect(failures.files["200"]).toMatchObject({
+                classification: "transient",
+                error: message,
+            });
+        },
+    );
+
+    // The download of toast.mp4 stalls: the server sends the headers and then
+    // never a byte of the body. Each attempt is given up after 50 ms, quak's
+    // own deadline for this run, with a `TimeoutError` whose message says
+    // only that the download stalled. The file goes into `failures.json` as a
+    // transient failure, as a request that ran out of time is retried.
+    it("records a stalled download in failures.json, as transient", async () => {
+        let attempts = 0;
+        const fetch: typeof globalThis.fetch = async (input, init) => {
+            if (downloadOf(input) === 200) {
+                attempts++;
+                return new Response(
+                    new ReadableStream<Uint8Array>({
+                        pull: () => new Promise<void>(() => {}),
+                    }),
+                );
+            }
+            return server(input, init);
+        };
+        const dir = mkdtempSync(join(root, "backup-"));
+
+        const { code } = await backup(dir, {}, fetch, 50);
+
+        expect(code).toBe(1);
+        expect(attempts).toBe(UNATTENDED_RETRY_OPTIONS.attempts);
+        const failures = readJSON(join(dir, "failures.json"));
+        expect(failures.files["200"]).toMatchObject({
+            classification: "transient",
+            error: "download stalled: no bytes received for 50 ms",
+        });
+    });
+
+    // The name of Ente's API server does not resolve when quak fetches the ML
+    // data, after every original has been backed up. No file has ML data
+    // cached yet, so each goes into `failures.json` with the reason, as a
+    // transient failure, as a download that could not reach the server does,
+    // and the exit code is 1.
+    it("records why the ML data fetch could not reach the server in failures.json, as transient, and exits 1", async () => {
+        const fetch: typeof globalThis.fetch = async (input, init) => {
+            if (new URL(String(input)).pathname === "/files/data/fetch") {
+                throw new TypeError("fetch failed", {
+                    cause: Object.assign(
+                        new Error("getaddrinfo ENOTFOUND api.ente.io"),
+                        { code: "ENOTFOUND" },
+                    ),
+                });
+            }
+            return server(input, init);
+        };
+        const dir = mkdtempSync(join(root, "backup-"));
+
+        const { code, stderr } = await backup(dir, {}, fetch);
+
+        expect(code).toBe(1);
+        const error =
+            "ML data: fetch failed: getaddrinfo ENOTFOUND api.ente.io";
+        expect(stderr).toContain(`FAILED ${error}\n`);
+        const failures = readJSON(join(dir, "failures.json"));
+        expect(
+            Object.values(failures.files).map((f) => [
+                f.fileID,
+                f.classification,
+                f.error,
+            ]),
+        ).toEqual([
+            [100, "transient", error],
+            [200, "transient", error],
+            [300, "transient", error],
+        ]);
+    });
+
+    // The server sends beach.jpg's ML data encrypted under a key that is not
+    // the file's, so quak cannot decrypt it. The file's JSON has the reason
+    // in place of the ML data, and the file goes into `failures.json` with
+    // it, so the exit code is 1. The other two files have no ML data, which
+    // is no failure.
+    it("records a file whose ML data does not decrypt in its JSON and in failures.json, and exits 1", async () => {
+        const otherKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+        const mlData = encryptStreamBlob(
+            gzipSync(JSON.stringify(ML_DATA)),
+            otherKey,
+        );
+        const fetch: typeof globalThis.fetch = async (input, init) => {
+            if (new URL(String(input)).pathname !== "/files/data/fetch") {
+                return server(input, init);
+            }
+            const data = [
+                {
+                    fileID: BEACH.id,
+                    encryptedData: toBase64(mlData.ciphertext),
+                    decryptionHeader: toBase64(mlData.header),
+                },
+            ];
+            return new Response(JSON.stringify({ data }), {
+                headers: { "content-type": "application/json" },
+            });
+        };
+        const dir = mkdtempSync(join(root, "backup-"));
+
+        const { code, stderr } = await backup(dir, {}, fetch);
+
+        expect(code).toBe(1);
+        const reason = "secretstream chunk authentication failed";
+        const beachJSON = readJSON(join(dir, DAY, "2024-03-01.100.json"));
+        expect(beachJSON).not.toHaveProperty("mlData");
+        expect(beachJSON.mlDataError).toBe(reason);
+        const failures = readJSON(join(dir, "failures.json"));
+        expect(Object.keys(failures.files)).toEqual(["100"]);
+        expect(failures.files["100"]).toMatchObject({
+            fileID: 100,
+            title: "beach.jpg",
+            error: `ML data: ${reason}`,
+        });
+        expect(stderr).toContain(`beach.jpg (id 100): ML data: ${reason}\n`);
+        for (const fileID of [200, 300]) {
+            const json = readJSON(join(dir, DAY, `2024-03-01.${fileID}.json`));
+            expect(json).not.toHaveProperty("mlData");
+            expect(json).not.toHaveProperty("mlDataError");
+        }
+    });
+
+    // A program using quak as a library backs the account up, and another ML
+    // data fetch, as one the library's refresh starts, fails while the backup
+    // writes the JSON files. The backup's own fetch got beach.jpg's ML data
+    // under a key that is not the file's; the fetch that fails leaves that
+    // reason in place, so beach.jpg's JSON and `failures.json` still give it.
+    it("keeps the reason in beach.jpg's JSON and in failures.json when an ML data fetch fails while lib.backup() writes the JSON files", async () => {
+        const otherKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+        const mlData = encryptStreamBlob(
+            gzipSync(JSON.stringify(ML_DATA)),
+            otherKey,
+        );
+        let mlDataFetchFails = false;
+        const fetch: typeof globalThis.fetch = async (input, init) => {
+            if (new URL(String(input)).pathname !== "/files/data/fetch") {
+                return server(input, init);
+            }
+            if (mlDataFetchFails) throw new Error("the connection was reset");
+            const data = [
+                {
+                    fileID: BEACH.id,
+                    encryptedData: toBase64(mlData.ciphertext),
+                    decryptionHeader: toBase64(mlData.header),
+                },
+            ];
+            return new Response(JSON.stringify({ data }), {
+                headers: { "content-type": "application/json" },
+            });
+        };
+        const dir = mkdtempSync(join(root, "backup-"));
+        const lib = await Library.open({
+            client: loadSession(join(sessionDir, "session.json"), {
+                fetch,
+                retry: { ...UNATTENDED_RETRY_OPTIONS, sleep: async () => {} },
+            })!,
+            cacheDirectory: `${dir}-cache`,
+            downloadDirectory: dir,
+            refreshIntervalSeconds: Infinity,
+            precacheThumbnails: false,
+            precacheOriginals: false,
+        });
+        try {
+            // The backup reads a file's cached ML data just before it writes
+            // that file's JSON. The first time, a refresh starts an ML data
+            // fetch whose request fails, and the backup goes on once that
+            // fetch has failed.
+            const forFile = lib.mldata.forFile.bind(lib.mldata);
+            vi.spyOn(lib.mldata, "forFile").mockImplementationOnce(
+                async (args) => {
+                    mlDataFetchFails = true;
+                    await lib.fresh();
+                    await vi.waitFor(
+                        () => expect(lib.status().lastMLError).toBeDefined(),
+                        { timeout: 2000, interval: 5 },
+                    );
+                    return forFile(args);
+                },
+            );
+
+            const result = await lib.backup();
+
+            expect(result.failed).toBe(1);
+            const reason = "secretstream chunk authentication failed";
+            const beachJSON = readJSON(join(dir, DAY, "2024-03-01.100.json"));
+            expect(beachJSON).not.toHaveProperty("mlData");
+            expect(beachJSON.mlDataError).toBe(reason);
+            const failures = readJSON(join(dir, "failures.json"));
+            expect(Object.keys(failures.files)).toEqual(["100"]);
+            expect(failures.files["100"]).toMatchObject({
+                fileID: 100,
+                title: "beach.jpg",
+                error: `ML data: ${reason}`,
+            });
+        } finally {
+            await lib.close();
+        }
+    });
+
     // Something changed beach.jpg on disk after it was backed up. A run with
     // --verify hashes every stored original, finds that one no longer matches
     // the hash Ente recorded, says so, and downloads it again; it put back the
@@ -719,6 +1117,9 @@ describe("a backup whose lock is lost", () => {
         expect(code).toBe(1);
         expect(stderr).toBe(
             "Starting backup...\n" +
+                "Fetched the files of 1 of 3 collections\n" +
+                "Fetched the files of 2 of 3 collections\n" +
+                "Fetched the files of 3 of 3 collections\n" +
                 "Refreshing library...\n" +
                 "Fetching original beach.jpg (100)...\n" +
                 "Fetching original IMG_0002.HEIC (300)...\n" +

test/cli/backup-undecryptable.test.ts

@@ -42,6 +42,7 @@ import {
     saveSession,
     type CliContext,
 } from "../../src/cli-commands.js";
+import { stderrLog } from "../../src/cli-log.js";
 import { loadSession } from "../../src/cli-session.js";
 import { Client, type ClientSnapshot } from "../../src/client.js";
 import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
@@ -304,7 +305,7 @@ const context = (ente: FakeEnte): CliContext => {
     saveSession(join(root, "session"), ente.snapshot());
     return {
         stdout,
-        stderr,
+        log: stderrLog(stderr, "info"),
         sessionDir: join(root, "session"),
         cacheDir: join(root, "cache"),
         loadSession: (path) =>
@@ -370,6 +371,24 @@ describe("quak backup with records it cannot decrypt", () => {
         expect(stderr.text).toContain("  Failed:      2\n");
     });
 
+    // The album is an error line. The last line is fatal and counts the files
+    // and the album, so `-q`, which prints fatal lines only, still says why
+    // the backup exits 1.
+    it("logs the album at error, and ends with one fatal line counting the files and the album", async () => {
+        const ctx = {
+            ...context(brokenAccount()),
+            log: stderrLog(stderr, "error"),
+        };
+        expect(await backupCommand(ctx, backupDir(), {})).toBe(1);
+
+        expect(stderr.text).toMatch(
+            /^FAILED album 2: could not be decrypted: /,
+        );
+        expect(stderr.text.split("\n").at(-2)).toBe(
+            `quak: 2 file(s) and 1 album(s) failed; see ${join(backupDir(), "failures.json")} and the FAILED lines`,
+        );
+    });
+
     it("names each album it cannot decrypt in the --json result", async () => {
         expect(await backup(brokenAccount(), { json: true })).toBe(1);
 
@@ -548,7 +567,7 @@ describe("the read commands with records quak cannot decrypt", () => {
         expect(await collectionsCommand(context(ente), {})).toBe(0);
         expect(stdout.text).toBe("1\talbum\tTrip\n");
 
-        expect(await filesCommand(context(ente), { collection: "1" })).toBe(0);
+        expect(await filesCommand(context(ente), "1", {})).toBe(0);
         expect(stdout.text).toBe("10\timage\tphoto-10.jpg\n");
     });
 });

test/cli/backup.test.ts

@@ -52,6 +52,7 @@ import { tmpdir } from "node:os";
 import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
 
 import { runBackup, type BackupLibrary } from "../../src/backup.js";
+import { DISCOVER_CATEGORIES } from "../../src/clip-keyword-vectors.js";
 import { Library } from "../../src/library/index.js";
 import type { ContentSource } from "../../src/library/content.js";
 import type { CollectionsPage, FilesPage } from "../../src/client.js";
@@ -191,9 +192,12 @@ class HostileClient extends MockClient {
 }
 
 // A content source that writes byte buffers of the expected length and can be
-// told to fail one fileID's original, to exercise per-file resilience.
+// told to fail one fileID's original, as a server error or as a 404, or to
+// write other bytes for it, to exercise per-file resilience.
 interface StubSource extends ContentSource {
     failID?: number;
+    notFoundID?: number;
+    otherBytesID?: number;
     failThumbID?: number;
     originalCalls: number;
 }
@@ -204,8 +208,10 @@ const stubSource = (): StubSource => {
         original: async ({ file: f, destination }) => {
             s.originalCalls++;
             if (s.failID === f.id) throw new Error("HTTP 500 from server");
+            if (s.notFoundID === f.id) throw new Error("HTTP 404 from server");
             const size = SIZE_BY_ID[f.id] ?? 10;
-            writeFileSync(destination, Buffer.alloc(size));
+            const fill = s.otherBytesID === f.id ? 2 : 0;
+            writeFileSync(destination, Buffer.alloc(size, fill));
             return { bytesWritten: size };
         },
         thumbnail: async ({ file: f, destination }) => {
@@ -696,17 +702,26 @@ describe("lib.backup", () => {
         // A child that has already exited: its process ID is not running.
         const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
         const leftover = `.quak-backup-2026-03-01.100.jpg-${exitedPID}-abc123.tmp`;
+        // A live photo's image and video downloaded to replace a stored one,
+        // before they were renamed into place.
+        const replacement = [
+            `.quak-${exitedPID}-abc789.heic`,
+            `.quak-${exitedPID}-abc789.mov`,
+        ];
         // This test's own process stands in for a backup running at the same
         // time.
         const inProgress = `.quak-backup-2026-03-01.101.jpg-${process.pid}-def456.tmp`;
-        writeFileSync(join(day, leftover), "partial");
+        for (const name of [leftover, ...replacement, inProgress]) {
-        writeFileSync(join(day, inProgress), "partial");
+            writeFileSync(join(day, name), "partial");
+        }
         const lib = await openLibrary(stubSource());
 
         await lib.backup({ downloadDirectory: outDir });
 
         const names = readdirSync(day);
         expect(names).not.toContain(leftover);
+        expect(names).not.toContain(replacement[0]);
+        expect(names).not.toContain(replacement[1]);
         expect(names).toContain(inProgress);
         lib.close();
     });
@@ -932,19 +947,30 @@ describe("the backup lock", () => {
     });
 });
 
-// Also serves ML data: file 100 has `ML_PAYLOAD`, the other two have none.
+// Also serves ML data: file 100 has `ML_PAYLOAD`, whose CLIP embedding is that
-// While `mlError` is set, every ML data request fails with it.
+// of a photo of a sunset (see clip-keywords.test.ts), and the other two have
-const ML_PAYLOAD = { face: { faces: [] }, clip: { embedding: [0.5, 0.25] } };
+// none. While `mlError` is set, every ML data request fails with it.
+const SUNSET_EMBEDDING: number[] = JSON.parse(
+    readFileSync(
+        new URL("../clip-keywords-sunset.json", import.meta.url),
+        "utf-8",
+    ),
+).embedding;
+const ML_PAYLOAD = {
+    face: { faces: [] },
+    clip: { embedding: SUNSET_EMBEDDING },
+};
 
 class MLClient extends MockClient {
     mlError?: string;
-    async fetchMLData(args: {
+    async fetchMLData(args: { fileIDs: number[] }): Promise<{
-        fileIDs: number[];
+        payloads: Map<number, MLData>;
-    }): Promise<Map<number, MLData>> {
+        errors: Map<number, unknown>;
+    }> {
         if (this.mlError) throw new Error(this.mlError);
-        const result = new Map<number, MLData>();
+        const payloads = new Map<number, MLData>();
-        if (args.fileIDs.includes(100)) result.set(100, ML_PAYLOAD);
+        if (args.fileIDs.includes(100)) payloads.set(100, ML_PAYLOAD);
-        return result;
+        return { payloads, errors: new Map() };
     }
 }
 
@@ -953,16 +979,99 @@ const fileJSON = (outDir: string, fileID: number): Record<string, unknown> =>
     JSON.parse(readFileSync(saved(outDir, `${fileID}.json`), "utf-8"));
 
 describe("ML data in each file's JSON", () => {
-    it("writes a file's ML data, and no ML field for a file that has none", async () => {
+    // The JSON holds the ML data without its CLIP embedding, and in its place
+    // the keywords the embedding gives.
+    it("writes a file's ML data and keywords, and no ML field for a file that has none", async () => {
         const lib = await openLibrary(stubSource(), new MLClient());
         const outDir = join(root, "backup");
 
         const result = await lib.backup({ downloadDirectory: outDir });
 
         expect(result.failed).toBe(0);
-        expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
+        expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } });
+        expect(fileJSON(outDir, 100).keywords).toEqual(["Sunset"]);
         expect(fileJSON(outDir, 101)).not.toHaveProperty("mlData");
         expect(fileJSON(outDir, 101)).not.toHaveProperty("mlDataError");
+        expect(fileJSON(outDir, 101)).not.toHaveProperty("keywords");
+        await lib.close();
+    });
+
+    // A file's ML data can hold faces and no CLIP embedding. Its JSON then has
+    // the ML data and no keywords.
+    it("writes no keywords for ML data without a CLIP embedding", async () => {
+        const faces = { face: { faces: [{ faceID: "101_1", score: 0.9 }] } };
+
+        // Gives file 101 ML data holding only faces.
+        class FacesClient extends MockClient {
+            async fetchMLData(args: { fileIDs: number[] }): Promise<{
+                payloads: Map<number, MLData>;
+                errors: Map<number, unknown>;
+            }> {
+                const payloads = new Map<number, MLData>();
+                if (args.fileIDs.includes(101)) payloads.set(101, faces);
+                return { payloads, errors: new Map() };
+            }
+        }
+        const lib = await openLibrary(stubSource(), new FacesClient());
+        const outDir = join(root, "backup");
+
+        const result = await lib.backup({ downloadDirectory: outDir });
+
+        expect(result.failed).toBe(0);
+        expect(fileJSON(outDir, 101).mlData).toEqual(faces);
+        expect(fileJSON(outDir, 101)).not.toHaveProperty("keywords");
+        await lib.close();
+    });
+
+    // Ente's photos app keeps videos out of some Discover categories, Identity
+    // among them. Here a photo and a video share one CLIP embedding, Identity's
+    // own text embedding, so the photo gets the keyword Identity and the video
+    // does not.
+    it("keeps a video out of a category that leaves out videos", async () => {
+        const identity = DISCOVER_CATEGORIES.find(
+            (c) => c.title === "Identity",
+        )!;
+        expect(identity.showVideo).toBe(false);
+        const toast = file(102, 1, "toast.mp4");
+        const video = {
+            ...toast,
+            metadata: { ...toast.metadata, fileType: "video" as const },
+        };
+
+        // Serves beach.jpg and the video in the Vacation album, and gives
+        // each ML data holding only that embedding.
+        class VideoClient extends MockClient {
+            override async filesSince(args: {
+                collectionID: number;
+            }): Promise<FilesPage> {
+                const files =
+                    args.collectionID === 1
+                        ? [file(100, 1, "beach.jpg"), video]
+                        : [];
+                return { files, deleted: [], cursor: 1 };
+            }
+            async fetchMLData(args: { fileIDs: number[] }): Promise<{
+                payloads: Map<number, MLData>;
+                errors: Map<number, unknown>;
+            }> {
+                const payloads = new Map<number, MLData>();
+                for (const id of args.fileIDs) {
+                    payloads.set(id, {
+                        clip: { embedding: identity.textEmbedding },
+                    });
+                }
+                return { payloads, errors: new Map() };
+            }
+        }
+        const lib = await openLibrary(stubSource(), new VideoClient());
+        const outDir = join(root, "backup");
+
+        const result = await lib.backup({ downloadDirectory: outDir });
+
+        expect(result.failed).toBe(0);
+        expect(fileJSON(outDir, 100).keywords).toContain("Identity");
+        expect(fileJSON(outDir, 102)).toHaveProperty("keywords");
+        expect(fileJSON(outDir, 102).keywords).not.toContain("Identity");
         await lib.close();
     });
 
@@ -978,7 +1087,7 @@ describe("ML data in each file's JSON", () => {
         const result = await lib.backup({ downloadDirectory: outDir });
 
         // File 100's ML data was cached by the first run and is kept.
-        expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
+        expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } });
         expect(fileJSON(outDir, 100)).not.toHaveProperty("mlDataError");
         for (const fileID of [101, 200]) {
             expect(fileJSON(outDir, fileID).mlDataError).toBe(
@@ -1011,7 +1120,7 @@ describe("ML data in each file's JSON", () => {
 
         expect(result.failed).toBe(0);
         expect(result.skipped).toBe(3);
-        expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
+        expect(fileJSON(outDir, 100).mlData).toEqual({ face: { faces: [] } });
         for (const fileID of [100, 101, 200]) {
             expect(fileJSON(outDir, fileID)).not.toHaveProperty("mlDataError");
         }
@@ -1294,7 +1403,8 @@ describe("image metadata in each file's JSON", () => {
 });
 
 // With `verify`, each original already stored is hashed, and one that does not
-// match the content hash its metadata records is downloaded again.
+// match the content hash its metadata records is downloaded again, and replaced
+// only by a download that matches.
 describe("backup with verify", () => {
     // MockClient's files, each recording the hash of what `stubSource` writes
     // for it, except diagram.png (200), which records none.
@@ -1337,6 +1447,12 @@ describe("backup with verify", () => {
     const good = Buffer.alloc(SIZE_BY_ID[100]!);
     const corrupt = Buffer.alloc(SIZE_BY_ID[100]!, 1);
 
+    // The temporary files left in the date folder of the backup in `outDir`.
+    const tempFiles = (outDir: string): string[] =>
+        readdirSync(join(outDir, DAY)).filter((name) =>
+            name.startsWith(".quak-"),
+        );
+
     it("leaves an original that matches its hash, and one with no hash, as they are", async () => {
         const { lib, source, outDir } = await backedUp();
         const calls = source.originalCalls;
@@ -1381,13 +1497,16 @@ describe("backup with verify", () => {
         expect(log.filter((msg) => msg.startsWith("MISMATCH"))).toEqual([
             "MISMATCH original beach.jpg (100): its bytes do not match its content hash",
         ]);
+        expect(tempFiles(outDir)).toEqual([]);
         await lib.close();
     });
 
-    it("records a failed download in failures.json, with the original removed", async () => {
+    // A copy whose hash does not match is still mostly the photo, so the
+    // backup keeps it until it holds a better one.
+    it("keeps an original that does not match its hash when its download answers 404", async () => {
         const { lib, source, outDir } = await backedUp();
         writeFileSync(saved(outDir, "100.jpg"), corrupt);
-        source.failID = 100;
+        source.notFoundID = 100;
 
         const result = await lib.backup({
             downloadDirectory: outDir,
@@ -1400,18 +1519,18 @@ describe("backup with verify", () => {
             failed: 1,
         });
         expect(Object.keys(readLedger(outDir).files)).toEqual(["100"]);
-        expect(existsSync(saved(outDir, "100.jpg"))).toBe(false);
+        expect(readFileSync(saved(outDir, "100.jpg"))).toEqual(corrupt);
+        expect(
+            readlinkSync(join(outDir, "collections", "Vacation", "beach.jpg")),
+        ).toBe(linkTo("100.jpg"));
+        expect(tempFiles(outDir)).toEqual([]);
         await lib.close();
     });
 
-    it("records in failures.json an original the cache puts back with the same bad bytes", async () => {
+    it("keeps an original that does not match its hash when its download does not match either", async () => {
-        const lib = await openLibrary(stubSource(), new HashedClient());
+        const { lib, source, outDir } = await backedUp();
-        const outDir = join(root, "backup");
+        writeFileSync(saved(outDir, "100.jpg"), corrupt);
-        // The cache holds a bad copy, and a backup copies an original the
+        source.otherBytesID = 100;
-        // cache holds to its save path.
-        const cached = await lib.photos.byID({ fileID: 100 })!.original();
-        writeFileSync(cached.path, corrupt);
-        await lib.backup({ downloadDirectory: outDir });
 
         const result = await lib.backup({
             downloadDirectory: outDir,
@@ -1424,10 +1543,39 @@ describe("backup with verify", () => {
             failed: 1,
         });
         expect(result.errors.map((e) => e.error)).toEqual([
-            "the original put back does not match its content hash either",
+            "the downloaded original does not match its content hash either",
         ]);
         expect(Object.keys(readLedger(outDir).files)).toEqual(["100"]);
         expect(readFileSync(saved(outDir, "100.jpg"))).toEqual(corrupt);
+        expect(tempFiles(outDir)).toEqual([]);
+        await lib.close();
+    });
+
+    // The cache can hold the same bad bytes, as a copy from it is not checked
+    // as a download is, so the replacement is always downloaded.
+    it("downloads the replacement even when the cache holds the original", async () => {
+        const source = stubSource();
+        const lib = await openLibrary(source, new HashedClient());
+        const outDir = join(root, "backup");
+        // The cache holds a bad copy, and a backup copies an original the
+        // cache holds to its save path.
+        const cached = await lib.photos.byID({ fileID: 100 })!.original();
+        writeFileSync(cached.path, corrupt);
+        await lib.backup({ downloadDirectory: outDir });
+        const calls = source.originalCalls;
+
+        const result = await lib.backup({
+            downloadDirectory: outDir,
+            verify: true,
+        });
+
+        expect(result).toMatchObject({
+            downloaded: 1,
+            mismatched: 1,
+            failed: 0,
+        });
+        expect(source.originalCalls).toBe(calls + 1);
+        expect(readFileSync(saved(outDir, "100.jpg"))).toEqual(good);
         await lib.close();
     });
 
@@ -1498,23 +1646,29 @@ describe("backup album folders", () => {
         files: EnteFile[];
     }
 
-    const libraryOf = (albums: Album[]): BackupLibrary => ({
+    // The original of the file `notFound`, if given, always fails to download.
+    const libraryOf = (albums: Album[], notFound?: number): BackupLibrary => ({
         refresh: async () => {},
         listCollections: () => albums.map((a) => a.collection),
         listFiles: (id) =>
             albums.find((a) => a.collection.id === id)?.files ?? [],
         undecryptable: () => [],
         original: async (fileID) => {
+            if (fileID === notFound) throw new Error("HTTP 404");
             const path = join(root, `source-${fileID}`);
             writeFileSync(path, `original ${fileID}`);
             return { path };
         },
+        fetchOriginal: async () => {
+            throw new Error("no verify in this stand-in");
+        },
         thumbnail: async () => {
             throw new Error("no thumbnails in this stand-in");
         },
         contentConcurrency: 5,
         fetchMLData: async () => {},
         mlData: async () => undefined,
+        mlDataError: () => undefined,
         whoami: () => ({ email: "backup@example.com", userID: USER_ID }),
     });
 
@@ -1600,6 +1754,251 @@ describe("backup album folders", () => {
         expect(albumID(outDir, "Trip (11) (12).json")).toBe(12);
     });
 
+    // The next three tests end with the album "ok" (14), whose one file
+    // always fails to download, so each run has a failure to record in
+    // failures.json after the album folders are written.
+    const ok: Album = {
+        collection: collection(14, "ok"),
+        files: [file(5, 14, "e.jpg")],
+    };
+
+    // The folder of an album named "x.json" would be the JSON file of the
+    // album "x", and once ended every run there. Two runs, as on two nights,
+    // write the same tree, and the failed file's attempt count goes up.
+    it("adds the ID to an album whose folder would have another album's JSON name", async () => {
+        const outDir = join(root, "backup");
+        const lib = libraryOf(
+            [
+                {
+                    collection: collection(10, "x"),
+                    files: [file(1, 10, "a.jpg")],
+                },
+                {
+                    collection: collection(11, "x.json"),
+                    files: [file(2, 11, "b.jpg")],
+                },
+                {
+                    collection: collection(12, "Trip"),
+                    files: [file(3, 12, "c.jpg")],
+                },
+                {
+                    collection: collection(13, "TRIP.json"),
+                    files: [file(4, 13, "d.jpg")],
+                },
+                ok,
+            ],
+            5,
+        );
+
+        await runBackup(lib, { downloadDirectory: outDir });
+        const result = await runBackup(lib, { downloadDirectory: outDir });
+
+        expect(result.failedAlbums).toEqual([]);
+        expect(tree(outDir)).toEqual([
+            "TRIP.json (13)/",
+            `TRIP.json (13)/d.jpg -> ${linkTo("4.jpg")}`,
+            "TRIP.json (13).json",
+            "Trip/",
+            `Trip/c.jpg -> ${linkTo("3.jpg")}`,
+            "Trip.json",
+            "ok/",
+            "ok.json",
+            "x/",
+            `x/a.jpg -> ${linkTo("1.jpg")}`,
+            "x.json",
+            "x.json (11)/",
+            `x.json (11)/b.jpg -> ${linkTo("2.jpg")}`,
+            "x.json (11).json",
+        ]);
+        expect(albumID(outDir, "x.json")).toBe(10);
+        expect(albumID(outDir, "x.json (11).json")).toBe(11);
+        expect(albumID(outDir, "TRIP.json (13).json")).toBe(13);
+        expect(albumID(outDir, "ok.json")).toBe(14);
+        expect(result.failed).toBe(1);
+        expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 2 });
+    });
+
+    // A name over 255 bytes is refused as a file name, and an album folder's
+    // name leaves room for `.json` in its JSON's. "é" takes two bytes, and a
+    // name is cut between two characters.
+    it("cuts an album name and a title too long for a file name short and adds the ID", async () => {
+        const outDir = join(root, "backup");
+        const lib = libraryOf(
+            [
+                {
+                    collection: collection(10, "L".repeat(300)),
+                    files: [file(1, 10, `${"é".repeat(150)}.jpg`)],
+                },
+                ok,
+            ],
+            5,
+        );
+
+        await runBackup(lib, { downloadDirectory: outDir });
+        const result = await runBackup(lib, { downloadDirectory: outDir });
+
+        const folder = `${"L".repeat(245)} (10)`;
+        const link = `${"é".repeat(123)} (1).jpg`;
+        expect(Buffer.byteLength(`${folder}.json`)).toBe(255);
+        expect(Buffer.byteLength(link)).toBe(254);
+        expect(result.failedAlbums).toEqual([]);
+        expect(tree(outDir)).toEqual([
+            `${folder}/`,
+            `${folder}/${link} -> ${linkTo("1.jpg")}`,
+            `${folder}.json`,
+            "ok/",
+            "ok.json",
+        ]);
+        expect(albumID(outDir, `${folder}.json`)).toBe(10);
+        expect(albumID(outDir, "ok.json")).toBe(14);
+        expect(result.failed).toBe(1);
+        expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 2 });
+    });
+
+    // The user has put a file in collections/ with the name of the album
+    // Trip's folder, so the folder cannot be made.
+    it("goes on to the next album when an album's folder cannot be made, and lists it as failed", async () => {
+        const outDir = join(root, "backup");
+        const lib = libraryOf(
+            [
+                {
+                    collection: collection(10, "Trip"),
+                    files: [file(1, 10, "a.jpg")],
+                },
+                ok,
+            ],
+            5,
+        );
+        mkdirSync(join(outDir, "collections"), { recursive: true });
+        writeFileSync(join(outDir, "collections", "Trip"), "mine");
+        const logged: string[] = [];
+
+        const result = await runBackup(lib, {
+            downloadDirectory: outDir,
+            onProgress: (message) => logged.push(message),
+        });
+
+        expect(result.failedAlbums).toEqual([10]);
+        expect(logged).toContainEqual(
+            expect.stringMatching(/^FAILED album Trip: EEXIST: /),
+        );
+        expect(tree(outDir)).toEqual(["Trip", "ok/", "ok.json"]);
+        expect(albumID(outDir, "ok.json")).toBe(14);
+        expect(existsSync(saved(outDir, "1.jpg"))).toBe(true);
+        expect(result.failed).toBe(1);
+        expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 1 });
+    });
+
+    // The extensions of these titles leave no room for the ID and the "a"
+    // before them, so each is cut short to 250 bytes, which leaves one, for
+    // the "a". File 1's extension is 303 bytes and file 2's 271. Neither
+    // extension is only letters and digits, so both originals are saved with
+    // `.bin`, a name that fits.
+    it("cuts the extension of a title short when it leaves no room for the ID", async () => {
+        const outDir = join(root, "backup");
+        const lib = libraryOf([
+            {
+                collection: collection(10, "Trip"),
+                files: [
+                    file(1, 10, `a.${"L".repeat(300)}-x`),
+                    file(2, 10, `a.${"€".repeat(90)}`),
+                ],
+            },
+        ]);
+
+        const result = await runBackup(lib, { downloadDirectory: outDir });
+
+        const link = `a (1).${"L".repeat(249)}`;
+        const link2 = `a (2).${"€".repeat(83)}`;
+        expect(Buffer.byteLength(link)).toBe(255);
+        expect(Buffer.byteLength(link2)).toBe(255);
+        expect(result.failed).toBe(0);
+        expect(tree(outDir)).toEqual([
+            "Trip/",
+            `Trip/${link} -> ${linkTo("1.bin")}`,
+            `Trip/${link2} -> ${linkTo("2.bin")}`,
+            "Trip.json",
+        ]);
+    });
+
+    // Files 1 and 3 share a title, so each gets its ID before the
+    // extension. File 2's title is too long, and its extension leaves no
+    // room for the ID and the "x" before it. Were the ID put at the end of
+    // file 2's title cut short, its link would have file 1's name, and the
+    // link made second would replace the first.
+    it("keeps names unique when a title's extension leaves no room for the ID", async () => {
+        const outDir = join(root, "backup");
+        const shared = `x.${"Q".repeat(245)} (2)`;
+        const lib = libraryOf([
+            {
+                collection: collection(10, "Trip"),
+                files: [
+                    file(1, 10, shared),
+                    file(2, 10, `x (1).${"Q".repeat(250)}-`),
+                    file(3, 10, shared),
+                ],
+            },
+        ]);
+
+        const result = await runBackup(lib, { downloadDirectory: outDir });
+
+        expect(result.failed).toBe(0);
+        expect(tree(outDir)).toEqual([
+            "Trip/",
+            `Trip/x (1).${"Q".repeat(245)} (2) -> ${linkTo("1.bin")}`,
+            `Trip/x (2).${"Q".repeat(249)} -> ${linkTo("2.bin")}`,
+            `Trip/x (3).${"Q".repeat(245)} (2) -> ${linkTo("3.bin")}`,
+            "Trip.json",
+        ]);
+    });
+
+    // A title of 255 bytes is the longest file name allowed, and an album
+    // name of 250 bytes the longest whose JSON name fits; both are kept as
+    // they are. A name one byte longer gets the ID and is cut short. "é"
+    // takes two bytes, and in both names the room left ends inside one, so
+    // the cut falls before it, one byte short of the room.
+    it("keeps a name at the byte limit and adds the ID to one a byte over it", async () => {
+        const outDir = join(root, "backup");
+        const album = "A".repeat(250);
+        const title = `${"t".repeat(251)}.jpg`;
+        const longAlbum = `${"é".repeat(125)}B`;
+        const longTitle = `${"é".repeat(126)}.jpg`;
+        const lib = libraryOf([
+            {
+                collection: collection(10, album),
+                files: [file(1, 10, title), file(2, 10, longTitle)],
+            },
+            {
+                collection: collection(11, longAlbum),
+                files: [file(3, 11, "c.jpg")],
+            },
+        ]);
+
+        const result = await runBackup(lib, { downloadDirectory: outDir });
+
+        const folder = `${"é".repeat(122)} (11)`;
+        const link = `${"é".repeat(123)} (2).jpg`;
+        expect(Buffer.byteLength(`${album}.json`)).toBe(255);
+        expect(Buffer.byteLength(title)).toBe(255);
+        expect(Buffer.byteLength(longAlbum)).toBe(251);
+        expect(Buffer.byteLength(longTitle)).toBe(256);
+        expect(Buffer.byteLength(`${folder}.json`)).toBe(254);
+        expect(Buffer.byteLength(link)).toBe(254);
+        expect(result.failed).toBe(0);
+        expect(result.failedAlbums).toEqual([]);
+        expect(tree(outDir)).toEqual([
+            `${album}/`,
+            `${album}/${title} -> ${linkTo("1.jpg")}`,
+            `${album}/${link} -> ${linkTo("2.jpg")}`,
+            `${album}.json`,
+            `${folder}/`,
+            `${folder}/c.jpg -> ${linkTo("3.jpg")}`,
+            `${folder}.json`,
+        ]);
+        expect(albumID(outDir, `${album}.json`)).toBe(10);
+        expect(albumID(outDir, `${folder}.json`)).toBe(11);
+    });
+
     it("changes nothing on a second run over an unchanged account", async () => {
         const outDir = join(root, "backup");
         const lib = libraryOf([
@@ -1996,11 +2395,12 @@ describe("backup of live photos", () => {
             Buffer.from(IMAGE),
         );
         expect(readFileSync(video)).toEqual(Buffer.from(VIDEO));
+        expect(readdirSync(join(outDir, DAY)).sort()).toEqual(stored);
         expect(tree(outDir)).toEqual(linked);
         await lib.close();
     });
 
-    it("removes both files of a live photo that does not match its hash when downloading it again fails", async () => {
+    it("keeps both files of a live photo that does not match its hash when downloading it again fails", async () => {
         const { file: live, body } = await asLivePhoto(
             file(500, 10, "IMG_0500.HEIC"),
         );
@@ -2018,8 +2418,14 @@ describe("backup of live photos", () => {
 
         expect(result).toMatchObject({ mismatched: 1, failed: 1 });
         expect(Object.keys(readLedger(outDir).files)).toEqual(["500"]);
-        expect(existsSync(saved(outDir, "500.heic"))).toBe(false);
+        expect(readFileSync(saved(outDir, "500.heic"))).toEqual(
-        expect(existsSync(saved(outDir, "500.mov"))).toBe(false);
+            Buffer.from(IMAGE),
+        );
+        expect(readFileSync(saved(outDir, "500.mov"), "utf-8")).toBe(
+            "another few seconds of video",
+        );
+        expect(readdirSync(join(outDir, DAY)).sort()).toEqual(stored);
+        expect(tree(outDir)).toEqual(linked);
         await lib.close();
     });
 

test/cli/bin.test.ts

@@ -1,10 +1,18 @@
 /**
- * Tests for the retry options `bin/quak.ts` loads the saved session with:
+ * Tests for the retry options `bin/quak.ts` loads the saved session with
- * `quak backup` gets the unattended ones, every other command the default.
+ * (`quak backup` gets the unattended ones, every other command the default),
+ * each with the callbacks that log its requests and retries, for `quak files`
+ * taking its collection ID as an optional argument, and for `quak get` and
+ * `quak get-thumb` refusing `--collection`.
  *
  * Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
  * replaced by one that records the options it is given and reports no session,
  * so the command stops before it makes any request.
+ *
+ * The first load of the sources `bin/quak.ts` imports compiles them, and the
+ * first `init()` starts libsodium. On a loaded host that takes longer than a
+ * test's timeout, so both happen here, before the tests run. Each test's
+ * `vi.resetModules()` then only runs the compiled sources again.
  */
 
 import { mkdtempSync, rmSync } from "node:fs";
@@ -12,8 +20,13 @@ import { tmpdir } from "node:os";
 import { join } from "node:path";
 import { afterAll, afterEach, describe, expect, it, vi } from "vitest";
 import type { ApiClientOptions } from "../../src/api/client.js";
+import "../../src/cli-commands.js";
+import { init } from "../../src/crypto/index.js";
+import "../../src/index.js";
 import { UNATTENDED_RETRY_OPTIONS } from "../../src/retry.js";
 
+await init();
+
 const loaded = vi.hoisted(() => [] as (ApiClientOptions | undefined)[]);
 
 vi.mock("../../src/cli-session.js", () => ({
@@ -26,14 +39,19 @@ vi.mock("../../src/cli-session.js", () => ({
 const argv = process.argv;
 const dir = mkdtempSync(join(tmpdir(), "quak-bin-test-"));
 
-// Run `quak <args>` to completion. The "Not logged in" message and the exit
+// Run `quak <args>` to completion and return what it wrote on stderr, which is
-// are swallowed.
+// kept off the terminal. The exit is swallowed.
-const quak = async (...args: string[]): Promise<void> => {
+const quak = async (...args: string[]): Promise<string> => {
-    vi.spyOn(process.stderr, "write").mockImplementation(() => true);
+    let written = "";
+    vi.spyOn(process.stderr, "write").mockImplementation((text) => {
+        written += String(text);
+        return true;
+    });
     vi.spyOn(process, "exit").mockImplementation(() => undefined as never);
     process.argv = ["node", "quak", ...args];
     vi.resetModules();
     await import("../../bin/quak.js");
+    return written;
 };
 
 afterEach(() => {
@@ -59,13 +77,43 @@ describe("bin/quak.ts session loading", () => {
                     attempts,
                     baseDelayMs,
                     maxDelayMs,
+                    onRetry: expect.any(Function),
                 }),
+                onRequest: expect.any(Function),
             },
         ]);
     });
 
-    it("loads the session for another command with the default options", async () => {
+    it("loads the session for another command with the default retry options", async () => {
         await quak("collections");
-        expect(loaded).toEqual([undefined]);
+        expect(loaded).toEqual([
+            {
+                retry: { onRetry: expect.any(Function) },
+                onRequest: expect.any(Function),
+            },
+        ]);
     });
 });
+
+// The command line takes `files` with or without a collection ID, and the
+// command runs as far as its session check either way (issue #211).
+describe("bin/quak.ts files", () => {
+    it("runs without a collection ID", async () => {
+        expect(await quak("files")).toMatch(/^Not logged in/);
+    });
+
+    it("runs with a collection ID", async () => {
+        expect(await quak("files", "7")).toMatch(/^Not logged in/);
+    });
+});
+
+describe("bin/quak.ts get and get-thumb", () => {
+    it.each(["get", "get-thumb"])(
+        "%s refuses --collection as an unknown option",
+        async (command) => {
+            expect(await quak(command, "1", "--collection", "2")).toMatch(
+                /^error: unknown option '--collection'/,
+            );
+        },
+    );
+});

test/cli/commands.test.ts

@@ -21,7 +21,7 @@ import {
     utimesSync,
     writeFileSync,
 } from "node:fs";
-import { join } from "node:path";
+import { basename, join } from "node:path";
 import { tmpdir } from "node:os";
 import { PassThrough } from "node:stream";
 import * as jpegJs from "jpeg-js";
@@ -51,13 +51,24 @@ import {
     listMissingThumbnailsCommand,
     fixMissingThumbnailsCommand,
 } from "../../src/cli-commands.js";
+import { stderrLog } from "../../src/cli-log.js";
 import { run } from "../../src/cli-run.js";
 import { loadSession } from "../../src/cli-session.js";
-import type { Client, ClientSnapshot, LoginOptions } from "../../src/client.js";
+import {
+    Client,
+    type ClientSnapshot,
+    type LoginOptions,
+} from "../../src/client.js";
 import { savePath, type ContentSource } from "../../src/library/content.js";
 import type { Collection, EnteFile } from "../../src/model/types.js";
 import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
 import { defaultCacheDirectory } from "../../src/library/index.js";
+import {
+    buildAccount,
+    buildMetaFetch,
+    encryptSecretbox,
+    encryptStreamBlob,
+} from "../ente-server.js";
 import { HEIC_WITH_EXIF } from "../exif-heic.js";
 import {
     asLivePhoto,
@@ -111,25 +122,31 @@ const FILES: Record<number, EnteFile[]> = {
 // An original is 7 bytes and a thumbnail 3. `failID` makes that file's
 // original fail; `emptyThumbID` makes the server report that file's
 // thumbnail as empty. `withNewFile` adds new.jpg (102) to Vacation, advancing
-// the collection's updationTime as the server does, and `refreshError` makes
+// the collection's updationTime as the server does. `beachInWork` adds
-// listing collections fail with that message.
+// beach.jpg (100) to Work as well, so that one file is in both albums.
+// `refreshError` makes listing collections fail with that message.
 const fakeClient = (
     opts: {
         failID?: number;
         emptyThumbID?: number;
         withNewFile?: boolean;
+        beachInWork?: boolean;
         refreshError?: string;
     } = {},
 ) => {
     const collections = opts.withNewFile
         ? [{ ...COLLECTIONS[0], updationTime: 2 }, COLLECTIONS[1]]
         : COLLECTIONS;
-    const files = opts.withNewFile
+    let files = FILES;
-        ? {
+    if (opts.withNewFile) {
-              ...FILES,
+        files = {
-              1: [...FILES[1], { ...file(102, 1, "new.jpg"), updationTime: 2 }],
+            ...files,
+            1: [...files[1], { ...file(102, 1, "new.jpg"), updationTime: 2 }],
+        };
+    }
+    if (opts.beachInWork) {
+        files = { ...files, 2: [...files[2], file(100, 2, "beach.jpg")] };
     }
-        : FILES;
     const source: ContentSource = {
         original: async ({ file: f, destination }) => {
             if (f.id === opts.failID) throw new Error("HTTP 500 from server");
@@ -171,6 +188,12 @@ const fakeClient = (
     return fake as unknown as Client;
 };
 
+// What a read command prints on stderr while the refresh it waits for fetches
+// the files of each of the fake's two albums, on an empty cache.
+const FETCHED_BOTH_ALBUMS =
+    "Fetched the files of 1 of 2 collections\n" +
+    "Fetched the files of 2 of 2 collections\n";
+
 // Collects everything written to it.
 class Output {
     text = "";
@@ -183,9 +206,11 @@ let root: string;
 let stdout: Output;
 let stderr: Output;
 
+// The commands log at the default level, info, as `quak` without `-v` or `-q`
+// does; the levels have their own tests, in `log.test.ts`.
 const context = (client: Client | null = fakeClient()): CliContext => ({
     stdout,
-    stderr,
+    log: stderrLog(stderr, "info"),
     sessionDir: join(root, "session"),
     cacheDir: join(root, "cache"),
     loadSession: () => client,
@@ -238,7 +263,7 @@ describe("session file", () => {
         const dir = join(root, "backup");
         expect(await whoamiCommand(ctx)).toBe(3);
         expect(await collectionsCommand(ctx, {})).toBe(3);
-        expect(await filesCommand(ctx, { collection: "1" })).toBe(3);
+        expect(await filesCommand(ctx, "1", {})).toBe(3);
         expect(await getCommand(ctx, "100", {})).toBe(3);
         expect(await getThumbCommand(ctx, "100", {})).toBe(3);
         expect(await backupMetadataCommand(ctx, dir, {})).toBe(3);
@@ -485,17 +510,14 @@ describe("collections", () => {
 
 describe("files", () => {
     it("prints one tab-separated line per file", async () => {
-        expect(await filesCommand(context(), { collection: "1" })).toBe(0);
+        expect(await filesCommand(context(), "1", {})).toBe(0);
         expect(stdout.text).toBe(
             "100\timage\tbeach.jpg\n" + "101\timage\tsunset.jpg\n",
         );
     });
 
     it("prints a JSON array with --json", async () => {
-        const code = await filesCommand(context(), {
+        const code = await filesCommand(context(), "2", { json: true });
-            collection: "2",
-            json: true,
-        });
         expect(code).toBe(0);
         expect(JSON.parse(stdout.text)).toEqual([
             {
@@ -509,20 +531,52 @@ describe("files", () => {
     });
 
     it("exits 1 for an unknown collection", async () => {
-        expect(await filesCommand(context(), { collection: "9" })).toBe(1);
+        expect(await filesCommand(context(), "9", {})).toBe(1);
         expect(stderr.text).toBe("Collection 9 not found\n");
     });
+
+    // Without a collection ID it lists every file in the account once (issue
+    // #211). beach.jpg is in Vacation and Work, and Vacation comes first, so it
+    // is listed once, as Vacation's.
+    it("without a collection ID prints every file once", async () => {
+        const ctx = context(fakeClient({ beachInWork: true }));
+        expect(await filesCommand(ctx, undefined, {})).toBe(0);
+        expect(stdout.text).toBe(
+            "100\timage\tbeach.jpg\n" +
+                "101\timage\tsunset.jpg\n" +
+                "200\timage\tdiagram.png\n",
+        );
+        expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS);
+    });
+
+    it("without a collection ID gives each file the first album holding it with --json", async () => {
+        const ctx = context(fakeClient({ beachInWork: true }));
+        expect(await filesCommand(ctx, undefined, { json: true })).toBe(0);
+        expect(
+            JSON.parse(stdout.text).map(
+                (row: { id: number; collectionID: number }) => [
+                    row.id,
+                    row.collectionID,
+                ],
+            ),
+        ).toEqual([
+            [100, 1],
+            [101, 1],
+            [200, 2],
+        ]);
+        expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS);
+    });
 });
 
 describe("get and get-thumb", () => {
-    it("get finds a file in any album without --collection", async () => {
+    it("get finds a file in any album", async () => {
         const out = join(root, "diagram.png");
         expect(await getCommand(context(), "200", { out })).toBe(0);
         expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
         expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
     });
 
-    it("get-thumb finds a file in any album without --collection", async () => {
+    it("get-thumb finds a file in any album", async () => {
         const out = join(root, "thumb.jpg");
         expect(await getThumbCommand(context(), "200", { out })).toBe(0);
         expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
@@ -544,14 +598,109 @@ describe("get and get-thumb", () => {
     });
 });
 
+// A file system refuses a file name over 255 bytes. The account here is one
+// album, whose name is 300 bytes, holding one file, 700, with the title given.
+describe("names from an album name or a title over 255 bytes", () => {
+    const longNames = (title: string): Client =>
+        ({
+            ...fakeClient(),
+            collectionsSince: async () => ({
+                collections: [collection(7, "A".repeat(300))],
+                deleted: [],
+                cursor: 1,
+            }),
+            filesSince: async () => ({
+                files: [file(700, 7, title)],
+                deleted: [],
+                cursor: 1,
+            }),
+        }) as unknown as Client;
+    const longTitle = `${"t".repeat(296)}.jpg`;
+
+    // Run `command` in a new directory, where `get` and `get-thumb` without
+    // `--out` write, and return its exit code and the directory.
+    const inNewDirectory = async (
+        command: () => Promise<number>,
+    ): Promise<{ code: number; dir: string }> => {
+        const dir = mkdtempSync(join(root, "cwd-"));
+        const previous = process.cwd();
+        process.chdir(dir);
+        try {
+            return { code: await command(), dir };
+        } finally {
+            process.chdir(previous);
+        }
+    };
+
+    it("get cuts the title short to 255 bytes, keeping its extension, and exits 0", async () => {
+        const { code, dir } = await inNewDirectory(() =>
+            getCommand(context(longNames(longTitle)), "700", {}),
+        );
+
+        const name = `${"t".repeat(251)}.jpg`;
+        expect(Buffer.byteLength(name)).toBe(255);
+        expect(code).toBe(0);
+        expect(readdirSync(dir)).toEqual([name]);
+        expect(stderr.text).toBe(`7 bytes -> ${name}\n`);
+    });
+
+    it("get-thumb cuts the title short so that thumb_ and it fit in 255 bytes, and exits 0", async () => {
+        const { code, dir } = await inNewDirectory(() =>
+            getThumbCommand(context(longNames(longTitle)), "700", {}),
+        );
+
+        const name = `thumb_${"t".repeat(245)}.jpg`;
+        expect(Buffer.byteLength(name)).toBe(255);
+        expect(code).toBe(0);
+        expect(readdirSync(dir)).toEqual([name]);
+        expect(stderr.text).toBe(`3 bytes -> ${name}\n`);
+    });
+
+    it("backup-metadata cuts the album name short so that its folder's name fits in 255 bytes, and exits 0", async () => {
+        const dir = join(root, "dump");
+
+        const code = await backupMetadataCommand(
+            context(longNames("beach.jpg")),
+            dir,
+            {},
+        );
+
+        const folder = `7-${"A".repeat(253)}`;
+        expect(Buffer.byteLength(folder)).toBe(255);
+        expect(code).toBe(0);
+        expect(readdirSync(join(dir, "collections"))).toEqual([folder]);
+        expect(readdirSync(join(dir, "collections", folder)).sort()).toEqual([
+            "700.json",
+            "_collection.json",
+        ]);
+    });
+
+    // The title's extension is 300 letters, too long for the save path's
+    // name, so the original is saved with `.bin`.
+    it("backup saves a title with a 300-letter extension as .bin, and exits 0", async () => {
+        const dir = join(root, "backup");
+        const title = `b.${"e".repeat(300)}`;
+
+        const code = await backupCommand(context(longNames(title)), dir, {});
+
+        const path = savePath(dir, file(700, 7, title));
+        expect(code).toBe(0);
+        expect(basename(path)).toMatch(/^\d{4}-\d{2}-\d{2}\.700\.bin$/);
+        expect(readFileSync(path)).toEqual(Buffer.alloc(7, 700 & 0xff));
+    });
+});
+
 // A live photo, which Ente stores as one ZIP, is written as its image and its
 // video, which a photo viewer can open. The account here is Vacation holding
 // one live photo, 300, downloaded through the real download layer
 // (test/live-photo.ts).
 describe("a live photo", () => {
-    const livePhotoClient = async (image = IMAGE): Promise<Client> => {
+    const livePhotoClient = async (
+        image = IMAGE,
+        title = "IMG_0300.HEIC",
+    ): Promise<Client> => {
         const { file: live, body } = await asLivePhoto(
-            file(300, 1, "IMG_0300.HEIC"),
+            file(300, 1, title),
             livePhotoZip({ "image.heic": image, "video.mov": VIDEO }),
             livePhotoHash(image, VIDEO),
         );
@@ -598,6 +747,44 @@ describe("a live photo", () => {
         );
     });
 
+    // Run `get` without `--out` in a new directory for a live photo titled
+    // `title`, and return its exit code and the names it wrote.
+    const getInNewDirectory = async (
+        title: string,
+    ): Promise<{ code: number; names: string[] }> => {
+        const ctx = context(await livePhotoClient(IMAGE, title));
+        const dir = mkdtempSync(join(root, "cwd-"));
+        const previous = process.cwd();
+        process.chdir(dir);
+        try {
+            const code = await getCommand(ctx, "300", {});
+            return { code, names: readdirSync(dir).sort() };
+        } finally {
+            process.chdir(previous);
+        }
+    };
+    const imageName = `${"t".repeat(250)}.heic`;
+    const videoName = `${"t".repeat(251)}.mov`;
+
+    // The title fits, but ".heic" is one byte longer than ".jpg", so the
+    // image's name is cut short once its extension is in it.
+    it("get cuts its image's name short for a 255-byte title ending in .jpg, and exits 0", async () => {
+        const title = `${"t".repeat(251)}.jpg`;
+
+        const result = await getInNewDirectory(title);
+
+        expect(Buffer.byteLength(title)).toBe(255);
+        expect(Buffer.byteLength(imageName)).toBe(255);
+        expect(Buffer.byteLength(videoName)).toBe(255);
+        expect(result).toEqual({ code: 0, names: [imageName, videoName] });
+    });
+
+    it("get cuts the names of its image and video short for a 300-byte title with no extension, and exits 0", async () => {
+        const result = await getInNewDirectory("t".repeat(300));
+
+        expect(result).toEqual({ code: 0, names: [imageName, videoName] });
+    });
+
     it("get --out writes the image there and the video beside it", async () => {
         const out = join(root, "photo.jpg");
         const video = join(root, "photo.mov");
@@ -718,6 +905,40 @@ describe("backup", () => {
         );
     });
 
+    // A failed file is an error line both where the backup fails it and in
+    // the list at the end. The last line is fatal, so `-q`, which prints
+    // fatal lines only, still says why the backup exits 1.
+    it("logs a failed file at error, and ends with one fatal line counting the failures", async () => {
+        const dir = join(root, "backup");
+        const last = `quak: 1 file(s) and 0 album(s) failed; see ${join(dir, "failures.json")} and the FAILED lines\n`;
+        const ctx = context(fakeClient({ failID: 101 }));
+        expect(
+            await backupCommand(
+                { ...ctx, log: stderrLog(stderr, "error") },
+                dir,
+                {},
+            ),
+        ).toBe(1);
+        expect(stderr.text).toBe(
+            "FAILED original sunset.jpg: HTTP 500 from server\n" +
+                "\nFailed files:\n" +
+                "  [Vacation] sunset.jpg (id 101): HTTP 500 from server\n" +
+                last,
+        );
+
+        stderr.text = "";
+        expect(
+            await backupCommand(
+                { ...ctx, log: stderrLog(stderr, "fatal") },
+                dir,
+                { json: true },
+            ),
+        ).toBe(1);
+        expect(stderr.text).toBe(last);
+    });
+
+    // `--json` shapes stdout only: the log lines stay on stderr, and the JSON
+    // takes the place of the summary.
     it("prints the result as JSON with --json, still exiting 1 on a failure", async () => {
         const ctx = context(fakeClient({ failID: 101 }));
         const code = await backupCommand(ctx, join(root, "backup"), {
@@ -732,7 +953,31 @@ describe("backup", () => {
             failed: 1,
         });
         expect(result.errors[0].fileID).toBe(101);
-        expect(stderr.text).toBe("Starting backup...\n");
+        expect(stderr.text).toContain(
+            "Fetching original sunset.jpg (101)...\n",
+        );
+        expect(stderr.text).not.toContain("Backup complete");
+    });
+
+    // On an empty cache the library's first refresh fetches the files of
+    // every album before the backup can start, which on a large account takes
+    // a while; each album it has fetched the files of is counted on stderr,
+    // with `--json` too.
+    it("counts on stderr each album whose files its first refresh fetched, with --json too", async () => {
+        const start =
+            "Starting backup...\n" +
+            FETCHED_BOTH_ALBUMS +
+            "Refreshing library...\n";
+        for (const json of [false, true]) {
+            stderr.text = "";
+            const dir = mkdtempSync(join(root, "backup-"));
+            const ctx = {
+                ...context(),
+                cacheDir: mkdtempSync(join(root, "cache-")),
+            };
+            expect(await backupCommand(ctx, dir, { json })).toBe(0);
+            expect(stderr.text.slice(0, start.length)).toBe(start);
+        }
     });
 
     it("--verify downloads again an original that does not match its hash, prints the counts, and exits 0", async () => {
@@ -822,6 +1067,7 @@ describe("backup", () => {
                 backupCommand(ctx, dir, {}),
                 new PassThrough(),
                 runStderr,
+                stderrLog(runStderr, "info"),
                 resolve,
             );
         });
@@ -1062,7 +1308,9 @@ describe("helper list-missing-thumbnails", () => {
         expect(stderr.text).toContain("No missing thumbnails found.\n");
     });
 
-    it("prints a JSON array with --json and no progress", async () => {
+    // `--json` shapes stdout only: the progress stays on stderr, and the JSON
+    // takes the place of the count and the list.
+    it("prints a JSON array with --json, and its progress on stderr", async () => {
         const ctx = context(fakeClient({ emptyThumbID: 200 }));
         expect(await listMissingThumbnailsCommand(ctx, { json: true })).toBe(0);
         expect(JSON.parse(stdout.text)).toEqual([
@@ -1073,7 +1321,54 @@ describe("helper list-missing-thumbnails", () => {
                 reason: "empty thumbnail (0 bytes)",
             },
         ]);
-        expect(stderr.text).toBe("");
+        expect(stderr.text).toContain(FETCHED_BOTH_ALBUMS);
+        expect(stderr.text).toContain("[Work] Checking thumbnails...\n");
+        expect(stderr.text).not.toContain("with missing thumbnails");
+    });
+});
+
+describe("helper fix-missing-thumbnails", () => {
+    // File 999 is in no album, so fixing it fails. The list of failed files
+    // is at error, and the last line is fatal, so `-q`, which prints fatal
+    // lines only, still says why the command exits 1.
+    it("lists a failed file at error, and ends with one fatal line counting the failures", async () => {
+        const last = `quak: 1 file(s) failed; see the "Failed files:" list, or the JSON with --json\n`;
+        const ctx = { ...context(), log: stderrLog(stderr, "error") };
+        expect(await fixMissingThumbnailsCommand(ctx, { file: ["999"] })).toBe(
+            1,
+        );
+        expect(stderr.text).toBe(
+            "\nFailed files:\n" +
+                "  999\tunknown\tfile not found in any collection\n" +
+                last,
+        );
+
+        stderr.text = "";
+        const quiet = { ...context(), log: stderrLog(stderr, "fatal") };
+        expect(
+            await fixMissingThumbnailsCommand(quiet, {
+                file: ["999"],
+                json: true,
+            }),
+        ).toBe(1);
+        expect(stderr.text).toBe(last);
+    });
+
+    // beach.jpg (100) is in Vacation (1) and Work (2), both updated at the same
+    // time. The fixer names a file after the first album holding it in the
+    // order of `Library.albums.list()`. On a tie, the album with the higher ID
+    // comes first, so beach.jpg is named after Work.
+    it("names a photo in two albums after the same album on an empty cache as on a cache that holds the account", async () => {
+        const ctx = context(fakeClient({ beachInWork: true }));
+        const opts = { file: ["100"], json: true };
+        const inWork = [{ fileID: 100, collection: "Work", status: "skipped" }];
+        expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
+        expect(JSON.parse(stdout.text)).toMatchObject(inWork);
+
+        expect(await filesCommand(ctx, undefined, {})).toBe(0);
+        stdout.text = "";
+        expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
+        expect(JSON.parse(stdout.text)).toMatchObject(inWork);
     });
 });
 
@@ -1103,11 +1398,12 @@ describe("backup-metadata --exif", () => {
     });
 });
 
-// Each test first runs `collections` so the cache holds the account as it was,
+// Each test first runs `files` so the cache holds the account as it was, then
-// then changes the server under it.
+// changes the server under it. (`collections` and `files` with a collection ID
+// would not do: they store nothing.)
 describe("backup-metadata and the thumbnail helpers refresh first", () => {
     beforeEach(async () => {
-        expect(await collectionsCommand(context(), {})).toBe(0);
+        expect(await filesCommand(context(), undefined, {})).toBe(0);
         stdout.text = "";
         stderr.text = "";
     });
@@ -1166,10 +1462,10 @@ describe("backup-metadata and the thumbnail helpers refresh first", () => {
     });
 });
 
-// `files --collection`, `get`, `get-thumb` and `helper fix-missing-thumbnails
+// `files`, `get`, `get-thumb` and `helper fix-missing-thumbnails --file` take
-// --file` take an ID only when it is written in decimal digits and is a safe
+// an ID only when it is written in decimal digits and is a safe integer.
-// integer. `Number()` alone reads `0x10` as 16, `1e3` as 1000 and an empty
+// `Number()` alone reads `0x10` as 16, `1e3` as 1000 and an empty string as 0,
-// string as 0, so the command would act on a file the user never named.
+// so the command would act on a file the user never named.
 // Anything else is refused with one line naming it, and exit 1, before any
 // request.
 //
@@ -1223,21 +1519,18 @@ describe("an ID given on the command line", () => {
 
     it("files looks up a decimal collection ID", async () => {
         const ctx = emptyAccount();
-        expect(await filesCommand(ctx, { collection: "1" })).toBe(1);
+        expect(await filesCommand(ctx, "1", {})).toBe(1);
         expect(stderr.text).toBe("Collection 1 not found\n");
         expect(requests).not.toHaveLength(0);
     });
 
-    it.each(NOT_IDS)(
+    it.each(NOT_IDS)("files refuses %j before any request", async (arg) => {
-        "files refuses --collection %j before any request",
-        async (arg) => {
         const ctx = emptyAccount();
-            expect(await filesCommand(ctx, { collection: arg })).toBe(1);
+        expect(await filesCommand(ctx, arg, {})).toBe(1);
         expect(stderr.text).toBe(`Invalid collection ID: ${arg}\n`);
         expect(stdout.text).toBe("");
         expect(requests).toHaveLength(0);
-        },
+    });
-    );
 
     it("get looks up a decimal file ID", async () => {
         expect(await getCommand(emptyAccount(), "100", {})).toBe(1);
@@ -1292,3 +1585,385 @@ describe("an ID given on the command line", () => {
         },
     );
 });
+
+// An account with many albums (issues #210, #217 and #222). On an empty cache
+// `collections`, `files` with a collection ID and `get` used to wait for a
+// refresh that fetched and decrypted the files of every album, one album after
+// another. So did `helper fix-missing-thumbnails --file` (issue #223). On such
+// an account they seemed to hang. The account here has 40 albums of one photo
+// each, encrypted as Ente's clients encrypt them, on the fake Ente server of
+// `test/ente-server.ts`. The tests record the path and query of each request
+// quak sends. The session is the one logging in to that server saves.
+describe("an account with many albums", () => {
+    const ALBUMS = 40;
+    const PASSWORD = "many-albums";
+    const requests: string[] = [];
+    let server: typeof globalThis.fetch;
+    let snapshot: ClientSnapshot;
+
+    // What `files` without a collection ID prints on an empty cache: every
+    // album's photo on stdout, and on stderr each album whose files its
+    // refresh fetched.
+    let everyPhoto = "";
+    let progress = "";
+    for (let n = 1; n <= ALBUMS; n++) {
+        everyPhoto += `${n * 100}\timage\tphoto-${n}.jpg\n`;
+        progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`;
+    }
+
+    // The key of each album, by album ID.
+    const albumKeys = new Map<number, Uint8Array>();
+
+    // A photo in album `collectionID` as the server sends it, and its original
+    // encrypted as the server stores it. The original's bytes are the title.
+    // These tests fetch no thumbnail. The server records a thumbnail size, so
+    // `helper fix-missing-thumbnails` downloads the original before it skips
+    // the photo as not a JPEG.
+    const photoRow = (photo: {
+        id: number;
+        collectionID: number;
+        title: string;
+        updationTime: number;
+    }): { row: Record<string, unknown>; original: Uint8Array } => {
+        const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+        const photoKeyBox = encryptSecretbox(
+            photoKey,
+            albumKeys.get(photo.collectionID)!,
+        );
+        const metadata = encryptStreamBlob(
+            new TextEncoder().encode(
+                JSON.stringify({
+                    title: photo.title,
+                    fileType: 0,
+                    creationTime: 1000,
+                    modificationTime: 1000,
+                }),
+            ),
+            photoKey,
+        );
+        const original = encryptStreamBlob(
+            new TextEncoder().encode(photo.title),
+            photoKey,
+        );
+        const row = {
+            id: photo.id,
+            collectionID: photo.collectionID,
+            ownerID: USER_ID,
+            encryptedKey: toBase64(photoKeyBox.ciphertext),
+            keyDecryptionNonce: toBase64(photoKeyBox.nonce),
+            metadata: {
+                encryptedData: toBase64(metadata.ciphertext),
+                decryptionHeader: toBase64(metadata.header),
+            },
+            file: { decryptionHeader: toBase64(original.header) },
+            thumbnail: { decryptionHeader: "" },
+            info: { thumbSize: 1000 },
+            updationTime: photo.updationTime,
+        };
+        return { row, original: original.ciphertext };
+    };
+
+    // Album n is named `Album n` and holds one photo, n * 100, titled
+    // `photo-n.jpg`.
+    beforeAll(async () => {
+        await sodium.ready;
+        const account = await buildAccount(PASSWORD);
+        const collections: Record<string, unknown>[] = [];
+        const filesByCollection: Record<number, Record<string, unknown>[]> = {};
+        const fileCiphertexts: Record<number, Uint8Array> = {};
+        for (let id = 1; id <= ALBUMS; id++) {
+            const albumKey = sodium.crypto_secretbox_keygen();
+            albumKeys.set(id, albumKey);
+            const albumKeyBox = encryptSecretbox(albumKey, account.masterKey);
+            const name = encryptSecretbox(
+                new TextEncoder().encode(`Album ${id}`),
+                albumKey,
+            );
+            collections.push({
+                id,
+                owner: { id: USER_ID },
+                encryptedKey: toBase64(albumKeyBox.ciphertext),
+                keyDecryptionNonce: toBase64(albumKeyBox.nonce),
+                encryptedName: toBase64(name.ciphertext),
+                nameDecryptionNonce: toBase64(name.nonce),
+                type: "album",
+                updationTime: 1,
+            });
+            const { row, original } = photoRow({
+                id: id * 100,
+                collectionID: id,
+                title: `photo-${id}.jpg`,
+                updationTime: 1,
+            });
+            filesByCollection[id] = [row];
+            fileCiphertexts[id * 100] = original;
+        }
+        const ente = buildMetaFetch({
+            verifier: account.verifier,
+            srpAttributes: account.srpAttributes,
+            keyAttributes: account.keyAttributes,
+            encryptedToken: account.encryptedToken,
+            collections,
+            filesByCollection,
+            encryptedMLData: {},
+            fileCiphertexts,
+        });
+        server = (async (input: RequestInfo | URL, init?: RequestInit) => {
+            const url =
+                typeof input === "string"
+                    ? input
+                    : input instanceof URL
+                      ? input.href
+                      : input.url;
+            const { pathname, search } = new URL(url);
+            requests.push(pathname + search);
+            return ente(input, init);
+        }) as typeof globalThis.fetch;
+        const client = await Client.login({
+            email: "many@example.com",
+            password: PASSWORD,
+            apiOptions: { fetch: server },
+        });
+        snapshot = client.toJSON();
+    });
+
+    beforeEach(() => {
+        requests.length = 0;
+    });
+
+    // A context whose saved session sends its requests to `fetch`, by default
+    // the fake server.
+    const accountContext = (fetch = server): CliContext => {
+        const ctx = context();
+        saveSession(ctx.sessionDir, snapshot);
+        return {
+            ...ctx,
+            loadSession: (path) => loadSession(path, { fetch }),
+        };
+    };
+
+    it("collections lists every album from one request, fetching no album's files", async () => {
+        const ctx = accountContext();
+        expect(await collectionsCommand(ctx, {})).toBe(0);
+
+        let albums = "";
+        for (let id = 1; id <= ALBUMS; id++)
+            albums += `${id}\talbum\tAlbum ${id}\n`;
+        expect(stdout.text).toBe(albums);
+        expect(stderr.text).toBe("");
+        expect(requests).toEqual(["/collections/v2?sinceTime=0"]);
+
+        // It stored nothing that would make a refresh skip an album's files:
+        // `files` after it still lists every album's photo.
+        stdout.text = "";
+        expect(await filesCommand(ctx, undefined, {})).toBe(0);
+        expect(stdout.text).toBe(everyPhoto);
+    });
+
+    it("files with a collection ID asks for the album list and that album's files alone", async () => {
+        expect(await filesCommand(accountContext(), "7", {})).toBe(0);
+
+        expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
+        expect(stderr.text).toBe("");
+        expect(requests).toEqual([
+            "/collections/v2?sinceTime=0",
+            "/collections/v2/diff?collectionID=7&sinceTime=0",
+        ]);
+    });
+
+    // Here the server sends album 7's files in two pages. Photo 700 changed
+    // between them, so both pages hold a copy of it.
+    it("files with a collection ID lists only the later copy of a photo sent in two pages", async () => {
+        const firstPage = [
+            photoRow({
+                id: 700,
+                collectionID: 7,
+                title: "photo-7.jpg",
+                updationTime: 1,
+            }).row,
+            photoRow({
+                id: 701,
+                collectionID: 7,
+                title: "photo-701.jpg",
+                updationTime: 1,
+            }).row,
+        ];
+        const secondPage = [
+            photoRow({
+                id: 700,
+                collectionID: 7,
+                title: "photo-7-edited.jpg",
+                updationTime: 2,
+            }).row,
+        ];
+        const paged = (async (input: RequestInfo | URL, init?: RequestInit) => {
+            const url = new URL(input instanceof Request ? input.url : input);
+            if (
+                url.pathname !== "/collections/v2/diff" ||
+                url.searchParams.get("collectionID") !== "7"
+            ) {
+                return server(input, init);
+            }
+            const first = url.searchParams.get("sinceTime") === "0";
+            return new Response(
+                JSON.stringify({
+                    diff: first ? firstPage : secondPage,
+                    hasMore: first,
+                }),
+            );
+        }) as typeof globalThis.fetch;
+
+        expect(await filesCommand(accountContext(paged), "7", {})).toBe(0);
+
+        // Photo 700 is listed once, with the later title, before photo 701.
+        expect(stdout.text).toBe(
+            "700\timage\tphoto-7-edited.jpg\n701\timage\tphoto-701.jpg\n",
+        );
+    });
+
+    // The cache stays empty, so a later `files` without a collection ID still
+    // fetches the files of all 40 albums.
+    it("files with a collection ID stores nothing in the cache", async () => {
+        const ctx = accountContext();
+        expect(await filesCommand(ctx, "7", {})).toBe(0);
+        expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
+
+        stdout.text = "";
+        expect(await filesCommand(ctx, undefined, {})).toBe(0);
+        expect(stdout.text).toBe(everyPhoto);
+        expect(stderr.text).toBe(progress);
+    });
+
+    // Photo 200 is in album 2, so `get` asks for no file of album 3 or later.
+    // The last request downloads the photo's original.
+    it("get on an empty cache asks for the albums' files only until one holds the photo", async () => {
+        const out = join(root, "photo.jpg");
+        expect(await getCommand(accountContext(), "200", { out })).toBe(0);
+
+        expect(readFileSync(out, "utf-8")).toBe("photo-2.jpg");
+        expect(stderr.text).toBe(`11 bytes -> ${out}\n`);
+        expect(requests).toEqual([
+            "/collections/v2?sinceTime=0",
+            "/collections/v2/diff?collectionID=1&sinceTime=0",
+            "/collections/v2/diff?collectionID=2&sinceTime=0",
+            "/?fileID=200",
+        ]);
+        expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
+    });
+
+    // `files` first fills the cache, so `get` waits for a refresh. No album has
+    // changed since, so the refresh asks for no album's files.
+    it("get on a cache that holds the account asks for no unchanged album's files, also for a photo in no album", async () => {
+        const ctx = accountContext();
+        expect(await filesCommand(ctx, undefined, {})).toBe(0);
+        requests.length = 0;
+        stderr.text = "";
+
+        const out = join(root, "photo.jpg");
+        expect(await getCommand(ctx, "4000", { out })).toBe(0);
+        expect(readFileSync(out, "utf-8")).toBe("photo-40.jpg");
+        expect(stderr.text).toBe(`12 bytes -> ${out}\n`);
+        expect(requests).toEqual([
+            "/collections/v2?sinceTime=1",
+            "/?fileID=4000",
+        ]);
+
+        requests.length = 0;
+        stderr.text = "";
+        expect(await getCommand(ctx, "999", { out })).toBe(1);
+        expect(stderr.text).toBe("File 999 not found\n");
+        expect(requests).toEqual(["/collections/v2?sinceTime=1"]);
+    });
+
+    // The helper searches the most recently updated album first. All 40 were
+    // updated at the same time, and on a tie the higher ID comes first. Photo
+    // 3900 is in album 39, so the helper asks for no file of album 38 or
+    // earlier. The last request downloads the photo's original.
+    it("fix-missing-thumbnails --file on an empty cache asks for the albums' files only until one holds the photo", async () => {
+        const opts = { file: ["3900"], json: true };
+        expect(await fixMissingThumbnailsCommand(accountContext(), opts)).toBe(
+            0,
+        );
+
+        const reason =
+            "unsupported image format (only baseline JPEG can be regenerated)";
+        expect(JSON.parse(stdout.text)).toEqual([
+            {
+                fileID: 3900,
+                title: "photo-39.jpg",
+                collection: "Album 39",
+                status: "skipped",
+                reason,
+            },
+        ]);
+        expect(stderr.text).toBe(
+            "[Album 39] Downloading photo-39.jpg for thumbnail generation...\n" +
+                `[Album 39] Skipping photo-39.jpg: ${reason}\n`,
+        );
+        expect(requests).toEqual([
+            "/collections/v2?sinceTime=0",
+            "/collections/v2/diff?collectionID=40&sinceTime=0",
+            "/collections/v2/diff?collectionID=39&sinceTime=0",
+            "/?fileID=3900",
+        ]);
+        expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
+    });
+
+    // `files` first fills the cache, so the helper waits for a refresh. No
+    // album has changed since, so the refresh asks for no album's files.
+    it("fix-missing-thumbnails --file on a cache that holds the account asks for no unchanged album's files, also for a photo in no album", async () => {
+        const ctx = accountContext();
+        expect(await filesCommand(ctx, undefined, {})).toBe(0);
+        requests.length = 0;
+        stdout.text = "";
+        stderr.text = "";
+
+        const opts = { file: ["3900"], json: true };
+        expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
+        const reason =
+            "unsupported image format (only baseline JPEG can be regenerated)";
+        expect(JSON.parse(stdout.text)).toEqual([
+            {
+                fileID: 3900,
+                title: "photo-39.jpg",
+                collection: "Album 39",
+                status: "skipped",
+                reason,
+            },
+        ]);
+        expect(stderr.text).toBe(
+            "[Album 39] Downloading photo-39.jpg for thumbnail generation...\n" +
+                `[Album 39] Skipping photo-39.jpg: ${reason}\n`,
+        );
+        expect(requests).toEqual([
+            "/collections/v2?sinceTime=1",
+            "/?fileID=3900",
+        ]);
+
+        requests.length = 0;
+        stdout.text = "";
+        const notFound = { file: ["999"], json: true };
+        expect(await fixMissingThumbnailsCommand(ctx, notFound)).toBe(1);
+        expect(JSON.parse(stdout.text)).toMatchObject([
+            { fileID: 999, reason: "file not found in any collection" },
+        ]);
+        expect(requests).toEqual(["/collections/v2?sinceTime=1"]);
+    });
+
+    it("files counts on stderr each album whose files its refresh fetched", async () => {
+        expect(await filesCommand(accountContext(), undefined, {})).toBe(0);
+
+        expect(stdout.text).toBe(everyPhoto);
+        expect(stderr.text).toBe(progress);
+    });
+
+    // `--json` shapes stdout only, so the refresh's progress is still counted
+    // on stderr.
+    it("files --json counts on stderr each album whose files its refresh fetched", async () => {
+        const opts = { json: true };
+        expect(await filesCommand(accountContext(), undefined, opts)).toBe(0);
+
+        expect(JSON.parse(stdout.text)).toHaveLength(ALBUMS);
+        expect(stderr.text).toBe(progress);
+    });
+});

test/cli/log.test.ts

@@ -0,0 +1,359 @@
+/**
+ * Tests for the CLI's log levels, set with `-v`/`--verbose` and `-q`/`--quiet`
+ * (`src/cli-log.ts`, issue #212).
+ *
+ * Every line quak prints on stderr is a log line at one of six levels, from
+ * the quietest: fatal, error, warning, info, debug, trace. A run prints the
+ * lines at its own level and at every quieter one. The level is info unless
+ * asked otherwise: `-v` raises it to debug, `-vv` to trace, and more repeats
+ * stay at trace. `-q` lowers it to fatal, so a run prints only the line a
+ * command prints as it fails with a non-zero exit code. stdout holds the
+ * command's output at every level.
+ *
+ * Most tests run `bin/quak.ts`, as the smoke test does, against an account of
+ * one album holding one photo: the saved session is in a temporary directory,
+ * and `fetch` is a fake of the Ente server that serves the album and the photo
+ * encrypted as Ente's clients encrypt them. Each run has a cache directory of
+ * its own, so its refresh fetches the album's files, as on a first run.
+ */
+
+import { mkdtempSync, rmSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import sodium from "libsodium-wrappers-sumo";
+import {
+    afterAll,
+    afterEach,
+    beforeAll,
+    describe,
+    expect,
+    it,
+    vi,
+} from "vitest";
+
+import { ApiClient } from "../../src/api/client.js";
+import { saveSession } from "../../src/cli-commands.js";
+import { logRefresh, logRequests, type Log } from "../../src/cli-log.js";
+import { init, toBase64 } from "../../src/crypto/index.js";
+import { encryptSecretbox, encryptStreamBlob } from "../ente-server.js";
+
+// `bin/quak.ts` reads the saved session from the data directory env-paths
+// gives it, which here is a temporary directory made before the first run.
+const paths = vi.hoisted(() => ({ data: "" }));
+vi.mock("env-paths", () => ({ default: () => paths }));
+
+const USER_ID = 42;
+const root = mkdtempSync(join(tmpdir(), "quak-log-test-"));
+const argv = process.argv;
+
+class ExitCalled extends Error {}
+
+// Run `quak <args>` to its exit, with a cache of its own, and return its exit
+// code and what it wrote on stdout and on stderr, which are kept off the
+// terminal. `process.exit` is stubbed to throw, so nothing runs after it, as
+// nothing does after a real exit.
+const quak = async (
+    ...args: string[]
+): Promise<{ code: number; stdout: string; stderr: string }> => {
+    let stdout = "";
+    let stderr = "";
+    let code = NaN;
+    vi.spyOn(process.stdout, "write").mockImplementation((text) => {
+        stdout += String(text);
+        return true;
+    });
+    vi.spyOn(process.stderr, "write").mockImplementation((text) => {
+        stderr += String(text);
+        return true;
+    });
+    vi.spyOn(process, "exit").mockImplementation((exitCode) => {
+        code = Number(exitCode ?? 0);
+        throw new ExitCalled();
+    });
+    const cache = mkdtempSync(join(root, "cache-"));
+    process.argv = ["node", "quak", "--cache-dir", cache, ...args];
+    vi.resetModules();
+    await expect(import("../../bin/quak.js")).rejects.toBeInstanceOf(
+        ExitCalled,
+    );
+    return { code, stdout, stderr };
+};
+
+// Album 1, "Trips", of user 42, holding photo 100, "beach.jpg". The album's
+// key is encrypted with the account's master key, the photo's key with the
+// album's, and the photo's metadata with its own key. Only the session and
+// these two records are needed: the tests never log in.
+beforeAll(async () => {
+    await init();
+    await sodium.ready;
+    const masterKey = sodium.crypto_secretbox_keygen();
+    const keyPair = sodium.crypto_box_keypair();
+    paths.data = join(root, "session");
+    saveSession(paths.data, {
+        email: "log@example.com",
+        userID: USER_ID,
+        token: "token",
+        masterKey: toBase64(masterKey),
+        secretKey: toBase64(keyPair.privateKey),
+        publicKey: toBase64(keyPair.publicKey),
+    });
+
+    const albumKey = sodium.crypto_secretbox_keygen();
+    const albumKeyBox = encryptSecretbox(albumKey, masterKey);
+    const name = encryptSecretbox(new TextEncoder().encode("Trips"), albumKey);
+    const album = {
+        id: 1,
+        owner: { id: USER_ID },
+        encryptedKey: toBase64(albumKeyBox.ciphertext),
+        keyDecryptionNonce: toBase64(albumKeyBox.nonce),
+        encryptedName: toBase64(name.ciphertext),
+        nameDecryptionNonce: toBase64(name.nonce),
+        type: "album",
+        updationTime: 1,
+    };
+    const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+    const photoKeyBox = encryptSecretbox(photoKey, albumKey);
+    const metadata = encryptStreamBlob(
+        new TextEncoder().encode(
+            JSON.stringify({
+                title: "beach.jpg",
+                fileType: 0,
+                creationTime: 1000,
+                modificationTime: 1000,
+            }),
+        ),
+        photoKey,
+    );
+    const photo = {
+        id: 100,
+        collectionID: 1,
+        ownerID: USER_ID,
+        encryptedKey: toBase64(photoKeyBox.ciphertext),
+        keyDecryptionNonce: toBase64(photoKeyBox.nonce),
+        metadata: {
+            encryptedData: toBase64(metadata.ciphertext),
+            decryptionHeader: toBase64(metadata.header),
+        },
+        // These tests fetch no original and no thumbnail.
+        file: { decryptionHeader: "" },
+        thumbnail: { decryptionHeader: "" },
+        updationTime: 1,
+    };
+
+    const json = (body: unknown): Response =>
+        new Response(JSON.stringify(body), {
+            headers: { "content-type": "application/json" },
+        });
+    vi.stubGlobal("fetch", async (input: string) => {
+        const { pathname } = new URL(input);
+        if (pathname === "/collections/v2") {
+            return json({ collections: [album] });
+        }
+        if (pathname === "/collections/v2/diff") {
+            return json({ diff: [photo], hasMore: false });
+        }
+        return new Response("not found", { status: 404 });
+    });
+});
+
+afterEach(() => {
+    process.argv = argv;
+    vi.restoreAllMocks();
+});
+
+afterAll(() => {
+    vi.unstubAllGlobals();
+    rmSync(root, { recursive: true, force: true });
+});
+
+// What `quak files` writes on stdout at every level.
+const FILES = "100\timage\tbeach.jpg\n";
+
+// What it writes on stderr at each level. Info is the refresh's progress, one
+// line for the one album whose files it fetched. Debug adds the start and end
+// of the refresh, and trace each request with the status of its answer.
+const INFO = "Fetched the files of 1 of 1 collections\n";
+const DEBUG = "Refresh started\n" + INFO + "Refresh done\n";
+const TRACE =
+    "Refresh started\n" +
+    "GET https://api.ente.io/collections/v2?sinceTime=0: HTTP 200\n" +
+    "GET https://api.ente.io/collections/v2/diff?collectionID=1&sinceTime=0: HTTP 200\n" +
+    INFO +
+    "Refresh done\n";
+
+describe("the log level", () => {
+    it("is info by default", async () => {
+        expect(await quak("files")).toEqual({
+            code: 0,
+            stdout: FILES,
+            stderr: INFO,
+        });
+    });
+
+    it("is debug with -v", async () => {
+        expect(await quak("-v", "files")).toEqual({
+            code: 0,
+            stdout: FILES,
+            stderr: DEBUG,
+        });
+    });
+
+    it("is trace with -vv", async () => {
+        expect(await quak("-vv", "files")).toEqual({
+            code: 0,
+            stdout: FILES,
+            stderr: TRACE,
+        });
+    });
+
+    it("stays at trace with -vvv", async () => {
+        expect((await quak("-vvv", "files")).stderr).toBe(TRACE);
+    });
+
+    it("counts --verbose --verbose as -vv", async () => {
+        expect((await quak("--verbose", "--verbose", "files")).stderr).toBe(
+            TRACE,
+        );
+    });
+
+    // Both are options of the program, taken before or after the command's
+    // name, as `--cache-dir` is.
+    it("is set by an option after the command's name too", async () => {
+        expect((await quak("files", "-v")).stderr).toBe(DEBUG);
+        expect((await quak("files", "--json", "-vv")).stderr).toBe(TRACE);
+        expect((await quak("files", "-q")).stderr).toBe("");
+    });
+});
+
+describe("-q", () => {
+    it("prints nothing on stderr when the command works", async () => {
+        expect(await quak("-q", "files")).toEqual({
+            code: 0,
+            stdout: FILES,
+            stderr: "",
+        });
+    });
+
+    // No album holds file 999, so fixing it fails. The summary from
+    // `--- Done ---` to the end of the `Failed files:` list is left out. The
+    // line saying why the command failed is the fatal one, and it is printed.
+    it("prints only the line a failing command exits with", async () => {
+        const fix = ["helper", "fix-missing-thumbnails", "--file", "999"];
+        expect(await quak("-q", ...fix)).toEqual({
+            code: 1,
+            stdout: "",
+            stderr: `quak: 1 file(s) failed; see the "Failed files:" list, or the JSON with --json\n`,
+        });
+    });
+
+    it("is refused with -v, in one line, before the command runs", async () => {
+        for (const args of [
+            ["-q", "-v", "files"],
+            ["files", "--verbose", "--quiet"],
+        ]) {
+            const { code, stdout, stderr } = await quak(...args);
+            expect(code).toBe(1);
+            expect(stdout).toBe("");
+            expect(stderr).toBe(
+                "error: option '-q, --quiet' cannot be used with option '-v, --verbose'\n",
+            );
+        }
+    });
+});
+
+// `--help` lists both options, and each command's help lists them too, since
+// each command takes them.
+describe("--help", () => {
+    it("lists -v and -q for the program and for each command", async () => {
+        for (const args of [["--help"], ["files", "--help"]]) {
+            const { stdout } = await quak(...args);
+            expect(stdout).toContain("-v, --verbose");
+            expect(stdout).toContain("-q, --quiet");
+        }
+    });
+});
+
+// The library writes no lines itself. Its layers report through callbacks,
+// and the CLI logs what they report.
+describe("what the library reports", () => {
+    const collect = (): { log: Log; lines: string[] } => {
+        const lines: string[] = [];
+        return { log: (level, line) => lines.push(`${level}: ${line}`), lines };
+    };
+
+    // A request answered 503 and retried: each attempt is a request, logged
+    // at trace with its status, and the retry is logged at debug with how
+    // long it waits and why.
+    it("logs each request at trace and each retry at debug", async () => {
+        const { log, lines } = collect();
+        const answers = [
+            new Response("unavailable", { status: 503 }),
+            new Response("{}", {
+                headers: { "content-type": "application/json" },
+            }),
+        ];
+        const client = new ApiClient({
+            ...logRequests(log, {
+                sleep: async () => {},
+                random: () => 0.5,
+            }),
+            fetch: async () => answers.shift()!,
+        });
+
+        await client.getJSON("/collections/v2", { sinceTime: 0 });
+
+        const url = "https://api.ente.io/collections/v2?sinceTime=0";
+        expect(lines).toEqual([
+            `trace: GET ${url}: HTTP 503`,
+            "debug: Attempt 1 failed, retrying in 250 ms: HTTP 503",
+            `trace: GET ${url}: HTTP 200`,
+        ]);
+    });
+
+    it("logs a request that failed with its error", async () => {
+        const { log, lines } = collect();
+        const client = new ApiClient({
+            ...logRequests(log, { attempts: 1 }),
+            fetch: async () => {
+                throw new TypeError("fetch failed", {
+                    cause: new Error("connect ECONNREFUSED 1.2.3.4:443"),
+                });
+            },
+        });
+
+        await expect(client.getJSON("/collections/v2")).rejects.toThrow();
+
+        expect(lines).toEqual([
+            "trace: GET https://api.ente.io/collections/v2: fetch failed: connect ECONNREFUSED 1.2.3.4:443",
+        ]);
+    });
+
+    // Each operation is named in plain words.
+    it("logs a refresh's progress at info, and its start and end at debug", () => {
+        const { log, lines } = collect();
+        const onProgress = logRefresh(log);
+
+        onProgress({ operation: "refresh", status: "started" });
+        onProgress({
+            operation: "refresh",
+            status: "progress",
+            fetched: 1,
+            total: 2,
+        });
+        onProgress({
+            operation: "refresh",
+            status: "failed",
+            error: "HTTP 503",
+        });
+        onProgress({ operation: "fetchMLData", status: "started" });
+        onProgress({ operation: "fetchMLData", status: "done", fetched: 3 });
+
+        expect(lines).toEqual([
+            "debug: Refresh started",
+            "info: Fetched the files of 1 of 2 collections",
+            "debug: Refresh failed: HTTP 503",
+            "debug: ML data fetch started",
+            "debug: ML data fetch done",
+        ]);
+    });
+});

test/cli/metadata-backup.test.ts

@@ -41,12 +41,14 @@ import { beforeAll, afterAll, describe, expect, it, vi } from "vitest";
 import { init, toBase64, encryptBlob } from "../../src/crypto/index.js";
 import * as jpegJs from "jpeg-js";
 import { Client } from "../../src/client.js";
+import { DISCOVER_CATEGORIES } from "../../src/clip-keyword-vectors.js";
 import { Library } from "../../src/library/index.js";
 import {
     runMetadataBackup,
     type MetadataBackupOptions,
 } from "../../src/metadata-backup.js";
 import { backupMetadataCommand } from "../../src/cli-commands.js";
+import { stderrLog, type LogLevel } from "../../src/cli-log.js";
 import {
     buildAccount,
     buildMetaFetch,
@@ -55,8 +57,8 @@ import {
     type MetaMockState,
 } from "../ente-server.js";
 
-// One file per ML data request, so the two files of the mock account are
+// One file per ML data request, so each file of the mock account is fetched in
-// fetched in two requests and one of them can fail on its own.
+// a request of its own and one of them can fail on its own.
 vi.mock("../../src/mldata-fetch.js", async (importOriginal) => ({
     ...(await importOriginal<typeof import("../../src/mldata-fetch.js")>()),
     MLDATA_BATCH_SIZE: 1,
@@ -65,6 +67,21 @@ vi.mock("../../src/mldata-fetch.js", async (importOriginal) => ({
 const TEST_EMAIL = "metabackup@example.com";
 const TEST_PASSWORD = "metapass";
 
+// The CLIP embedding of a photo of a sunset (see clip-keywords.test.ts), which
+// the ML data of beach.jpg holds.
+const SUNSET_EMBEDDING: number[] = JSON.parse(
+    readFileSync(
+        new URL("../clip-keywords-sunset.json", import.meta.url),
+        "utf-8",
+    ),
+).embedding;
+
+// Identity, a Discover category of Ente's photos app that leaves out videos.
+const IDENTITY = DISCOVER_CATEGORIES.find((c) => c.title === "Identity")!;
+
+// ML data holding faces and no CLIP embedding, which file 302 has.
+const FACES_ONLY = { face: { faces: [{ faceID: "face-def", score: 0.9 }] } };
+
 let mock: MetaMockState;
 let testDir: string;
 
@@ -203,6 +220,65 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
         updationTime: 1710000000000000,
     };
 
+    // A file in coll 10 with the ML data given. `fileType` is Ente's number: 0
+    // an image, 1 a video.
+    const vacationFile = (
+        id: number,
+        title: string,
+        fileType: number,
+        mlData: object,
+    ) => {
+        const fileKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+        const { ciphertext: encKey, nonce } = encryptSecretbox(fileKey, ck1);
+        const meta = encryptStreamBlob(
+            new TextEncoder().encode(
+                JSON.stringify({
+                    title,
+                    fileType,
+                    creationTime: 1700000000000000,
+                    modificationTime: 1700000000000000,
+                }),
+            ),
+            fileKey,
+        );
+        const ml = encryptBlob(
+            new Uint8Array(gzipSync(Buffer.from(JSON.stringify(mlData)))),
+            fileKey,
+        );
+        return {
+            raw: {
+                id,
+                collectionID: 10,
+                ownerID: 42,
+                encryptedKey: toBase64(encKey),
+                keyDecryptionNonce: toBase64(nonce),
+                metadata: {
+                    encryptedData: toBase64(meta.ciphertext),
+                    decryptionHeader: toBase64(meta.header),
+                },
+                file: {
+                    decryptionHeader: toBase64(sodium.randombytes_buf(24)),
+                },
+                thumbnail: {
+                    decryptionHeader: toBase64(sodium.randombytes_buf(24)),
+                },
+                updationTime: 1700000000000000,
+            },
+            mlData: {
+                encryptedData: toBase64(ml.ciphertext),
+                decryptionHeader: toBase64(ml.header),
+            },
+        };
+    };
+    // Files 300 and 301 in coll 10: a photo and a video whose ML data holds
+    // only a CLIP embedding, Identity's own text embedding.
+    const identityML = { clip: { embedding: IDENTITY.textEmbedding } };
+    const identityPhoto = vacationFile(300, "passport.jpg", 0, identityML);
+    const identityVideo = vacationFile(301, "passport.mp4", 1, identityML);
+    // File 302 in coll 10: a photo whose ML data holds faces and no CLIP
+    // embedding.
+    const facesPhoto = vacationFile(302, "party.jpg", 0, FACES_ONLY);
+
     // Encrypt ML data for file 100 (gzipped JSON, encrypted with file key)
     const mlPayload = JSON.stringify({
         face: {
@@ -229,7 +305,7 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
         clip: {
             version: 1,
             client: "test",
-            embedding: [0.5, 0.6, 0.7],
+            embedding: SUNSET_EMBEDDING,
         },
     });
     const gzipped = gzipSync(Buffer.from(mlPayload));
@@ -269,12 +345,23 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
         keyAttributes: account.keyAttributes,
         encryptedToken: account.encryptedToken,
         collections: [rawColl1, rawColl2],
-        filesByCollection: { 10: [rawFile1], 20: [rawFile2] },
+        filesByCollection: {
+            10: [
+                rawFile1,
+                identityPhoto.raw,
+                identityVideo.raw,
+                facesPhoto.raw,
+            ],
+            20: [rawFile2],
+        },
         encryptedMLData: {
             100: {
                 encryptedData: toBase64(mlCiphertext),
                 decryptionHeader: toBase64(mlHeader),
             },
+            300: identityPhoto.mlData,
+            301: identityVideo.mlData,
+            302: facesPhoto.mlData,
         },
         fileCiphertexts: { 100: encFileBody1 },
     };
@@ -431,7 +518,9 @@ describe("quak backup-metadata", () => {
         expect(account.email).toBe(TEST_EMAIL);
     });
 
-    it("fetches and decrypts ML data by default", async () => {
+    // The CLIP embedding is left out of the ML data, and the keywords it gives
+    // are written in its place.
+    it("fetches and decrypts ML data by default, with keywords for the CLIP embedding", async () => {
         const outDir = join(testDir, "ml-data");
         await runBackup(outDir);
 
@@ -451,8 +540,41 @@ describe("quak backup-metadata", () => {
         expect(fileMeta.mlData.face.faces[0].faceID).toBe("face-abc");
         expect(fileMeta.mlData.face.faces[0].score).toBeCloseTo(0.98);
         expect(fileMeta.mlData.face.faces[0].detection.box.x).toBeCloseTo(0.1);
-        expect(fileMeta.mlData.clip).toBeDefined();
+        expect(fileMeta.mlData).not.toHaveProperty("clip");
-        expect(fileMeta.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]);
+        expect(fileMeta.keywords).toEqual(["Sunset"]);
+    });
+
+    // Ente's photos app keeps videos out of some Discover categories, Identity
+    // among them. Files 300 and 301, a photo and a video, share one CLIP
+    // embedding, Identity's own text embedding, so the photo gets the keyword
+    // Identity and the video does not.
+    it("keeps a video out of a category that leaves out videos", async () => {
+        expect(IDENTITY.showVideo).toBe(false);
+        const outDir = join(testDir, "video-keywords");
+        await runBackup(outDir);
+
+        const vacation = join(outDir, "collections", "10-Vacation");
+        const fileJSON = (fileID: number) =>
+            JSON.parse(readFileSync(join(vacation, `${fileID}.json`), "utf-8"));
+        expect(fileJSON(300).keywords).toContain("Identity");
+        expect(fileJSON(301)).toHaveProperty("keywords");
+        expect(fileJSON(301).keywords).not.toContain("Identity");
+    });
+
+    // A file's ML data can hold faces and no CLIP embedding. File 302's JSON
+    // then has the ML data and no keywords.
+    it("writes no keywords for ML data without a CLIP embedding", async () => {
+        const outDir = join(testDir, "no-clip");
+        await runBackup(outDir);
+
+        const fileMeta = JSON.parse(
+            readFileSync(
+                join(outDir, "collections", "10-Vacation", "302.json"),
+                "utf-8",
+            ),
+        );
+        expect(fileMeta.mlData).toEqual(FACES_ONLY);
+        expect(fileMeta).not.toHaveProperty("keywords");
     });
 
     it("extracts EXIF from downloaded files when --exif is set", async () => {
@@ -489,15 +611,18 @@ describe("quak backup-metadata", () => {
     });
 });
 
-describe("quak backup-metadata when an ML data request fails", () => {
+describe("quak backup-metadata when a file's ML data cannot be read", () => {
-    // Run the CLI command against the mock and return its exit code, stderr
+    // Run the CLI command against `fetch`, the mock unless given, logging at
-    // and output directory.
+    // `level`, and return its exit code, stderr and output directory.
-    const runCommand = async (failMLDataFor?: number) => {
+    const runCommand = async (
+        fetch = buildMetaFetch(mock),
+        level: LogLevel = "info",
+    ) => {
         const client = await Client.login({
             email: TEST_EMAIL,
             password: TEST_PASSWORD,
             apiOptions: {
-                fetch: buildMetaFetch(mock, failMLDataFor),
+                fetch,
                 retry: { sleep: async () => {} },
             },
         });
@@ -506,7 +631,10 @@ describe("quak backup-metadata when an ML data request fails", () => {
         const code = await backupMetadataCommand(
             {
                 stdout: { write: () => true },
-                stderr: { write: (text: string) => (stderr += text) },
+                log: stderrLog(
+                    { write: (text: string) => (stderr += text) },
+                    level,
+                ),
                 sessionDir: testDir,
                 cacheDir: mkdtempSync(join(testDir, "cache-")),
                 loadSession: () => client,
@@ -517,11 +645,16 @@ describe("quak backup-metadata when an ML data request fails", () => {
         return { code, stderr, outDir };
     };
 
+    // The failed request is logged at error, so it is the first line printed
+    // at that level.
     it("writes every file, marks the failed batch's files, and exits 1", async () => {
-        const { code, stderr, outDir } = await runCommand(200);
+        const { code, stderr, outDir } = await runCommand(
+            buildMetaFetch(mock, 200),
+            "error",
+        );
 
         expect(code).toBe(1);
-        expect(stderr).toContain("ML data request for 1 file(s) failed");
+        expect(stderr).toMatch(/^ML data request for 1 file\(s\) failed: /);
 
         const ok = JSON.parse(
             readFileSync(
@@ -529,7 +662,7 @@ describe("quak backup-metadata when an ML data request fails", () => {
                 "utf-8",
             ),
         );
-        expect(ok.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]);
+        expect(ok.keywords).toEqual(["Sunset"]);
         expect(ok.mlDataError).toBeUndefined();
 
         const failed = JSON.parse(
@@ -543,6 +676,53 @@ describe("quak backup-metadata when an ML data request fails", () => {
         expect(failed.mlDataError).toContain("500");
     });
 
+    // The server sends ML data for diagram.png encrypted under a key that is
+    // not the file's, so quak cannot decrypt it. That is logged at error, and
+    // the last line is fatal, so `-q`, which prints fatal lines only, still
+    // says why the command exits 1.
+    it("writes the reason for ML data that does not decrypt, and exits 1", async () => {
+        const otherKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+        const { header, ciphertext } = encryptBlob(
+            new Uint8Array(gzipSync(Buffer.from("{}"))),
+            otherKey,
+        );
+        const { code, stderr, outDir } = await runCommand(
+            buildMetaFetch({
+                ...mock,
+                encryptedMLData: {
+                    ...mock.encryptedMLData,
+                    200: {
+                        encryptedData: toBase64(ciphertext),
+                        decryptionHeader: toBase64(header),
+                    },
+                },
+            }),
+            "error",
+        );
+
+        expect(code).toBe(1);
+        const reason = "secretstream chunk authentication failed";
+        expect(stderr).toBe(
+            `ML data of file 200 could not be read: ${reason}\n` +
+                `quak: the ML data of 1 file(s) could not be read; see mlDataError in their JSON in ${outDir}\n`,
+        );
+        const failed = JSON.parse(
+            readFileSync(
+                join(outDir, "collections", "20-__Work", "200.json"),
+                "utf-8",
+            ),
+        );
+        expect(failed.mlData).toBeUndefined();
+        expect(failed.mlDataError).toBe(reason);
+        const ok = JSON.parse(
+            readFileSync(
+                join(outDir, "collections", "10-Vacation", "100.json"),
+                "utf-8",
+            ),
+        );
+        expect(ok.keywords).toEqual(["Sunset"]);
+    });
+
     it("exits 0 when every ML data request succeeds", async () => {
         const { code } = await runCommand();
         expect(code).toBe(0);

test/cli/read.test.ts

@@ -3,17 +3,25 @@
  * issue #36, issue #52).
  *
  * The `collections`, `files`, `get`, and `get-thumb` commands must answer for
- * current server state, not the local cache, so each helper forces a
+ * current server state, not the local cache. `get`, `get-thumb` and `files`
- * `Library.fresh()` round-trip before it reads. The stand-in library below
+ * without a collection ID force a `Library.fresh()` round-trip before they
- * serves nothing until `fresh()` has been awaited, so a helper that read
+ * read. The stand-in library below serves no cached record until `fresh()` has
- * without refreshing would come back empty and fail here.
+ * been awaited, so a helper that read without refreshing would come back empty
+ * and fail here. `collections` asks the library for the collection list alone
+ * (`freshCollections()`, issue #210). `files` with a collection ID asks for
+ * that list, then for that one collection's files (`freshFiles()`, issue
+ * #217). Neither may wait for a refresh, which would fetch the files of every
+ * collection. On an empty cache, `get` and `get-thumb` ask for that list, then
+ * for one collection's files at a time, until they find the file (issue #222).
+ * `helper fix-missing-thumbnails --file` does the same until it finds every
+ * file it names, reading the most recently updated collection first (issue
+ * #223).
  *
- * `collections` and `files` also list in the library's enumeration order
+ * `collections` and `files` also list in the order the library gives them, not
- * (`listCollections`/`listFiles`) — the order the pre-library CLI printed — not
  * the albums/photos projection's newest-first order. The fixtures are seeded in
- * an enumeration order that a newest-first sort would rearrange, so a
+ * an order that a newest-first sort would rearrange, so a regression to the
- * regression to the projection order would fail here too. Field values still
+ * projection order would fail here too. Field values still come from the raw
- * come from the raw metadata via `cli-output.ts`.
+ * metadata via `cli-output.ts`.
  */
 
 import { describe, it, expect } from "vitest";
@@ -22,10 +30,10 @@ import {
     freshCollections,
     freshFiles,
     freshFile,
+    freshFilesByID,
     type FreshReadLibrary,
 } from "../../src/cli-read.js";
 import { fileListRow } from "../../src/cli-output.js";
-import type { Photo } from "../../src/library/index.js";
 import type { Collection, EnteFile } from "../../src/model/types.js";
 
 const collection = (id: number, updationTime: number): Collection => ({
@@ -59,11 +67,15 @@ const file = (
     updationTime: creationTime,
 });
 
-// A library that reveals its records only after `fresh()` has been awaited, and
+// A library that reveals its cached records only after `fresh()` has been
-// serves them in the enumeration order it was given. `photos.byID` returns a
+// awaited, and serves them in the enumeration order it was given.
-// stand-in `Photo` carrying just the fileID the helper passes through.
+// `freshCollections()` and `freshFiles()` answer in that order without a
+// refresh.
 class FakeLibrary implements FreshReadLibrary {
     freshCalls = 0;
+    freshCollectionsCalls = 0;
+    // The ID of each collection `freshFiles()` was asked for.
+    freshFilesCalls: number[] = [];
     private refreshed = false;
 
     constructor(
@@ -77,12 +89,18 @@ class FakeLibrary implements FreshReadLibrary {
         return {};
     }
 
-    listCollections(): Collection[] {
+    async freshCollections(): Promise<Collection[]> {
-        return this.refreshed ? this.collections : [];
+        this.freshCollectionsCalls++;
+        return this.collections;
     }
 
-    getCollection(id: number): Collection | undefined {
+    async freshFiles(collection: Collection): Promise<EnteFile[]> {
-        return this.listCollections().find((c) => c.id === id);
+        this.freshFilesCalls.push(collection.id);
+        return this.files.filter((f) => f.collectionID === collection.id);
+    }
+
+    listCollections(): Collection[] {
+        return this.refreshed ? this.collections : [];
     }
 
     listFiles(collectionID: number): EnteFile[] {
@@ -95,18 +113,10 @@ class FakeLibrary implements FreshReadLibrary {
         if (!this.refreshed) return undefined;
         return this.files.find((f) => f.id === fileID);
     }
-
-    photos = {
-        byID: ({ fileID }: { fileID: number }): Photo | undefined => {
-            if (!this.refreshed) return undefined;
-            if (!this.files.some((f) => f.id === fileID)) return undefined;
-            return { fileID } as unknown as Photo;
-        },
-    };
 }
 
 describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
-    it("freshCollections refreshes first, then lists in enumeration order", async () => {
+    it("freshCollections reads the collection list alone, in enumeration order", async () => {
         // Enumeration order 2, 1, 3; a newest-first sort would be 3, 2, 1.
         const lib = new FakeLibrary(
             [collection(2, 200), collection(1, 300), collection(3, 100)],
@@ -114,7 +124,8 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
         );
         const rows = await freshCollections(lib);
 
-        expect(lib.freshCalls).toBe(1);
+        expect(lib.freshCollectionsCalls).toBe(1);
+        expect(lib.freshCalls).toBe(0);
         expect(rows.map((c) => c.id)).toEqual([2, 1, 3]);
         // The projection's newest-first order is a different sequence, so this
         // is not accidentally that order.
@@ -125,18 +136,24 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
         expect(rows.map((c) => c.id)).not.toEqual(newestFirst);
     });
 
-    it("freshFiles refreshes first, lists in enumeration order, keeps raw fields", async () => {
+    it("freshFiles with a collection ID reads that collection's files alone and keeps their order and raw fields", async () => {
-        // Enumeration order by id 10, 11, 12; creationTimes ascending, so a
+        // Given in the order 10, 11, 12; creationTimes ascending, so a
         // newest-first sort would reverse them.
         const files = [
             file(10, 1, 1_700_000_000_000_000),
             file(11, 1, 1_700_000_000_000_001),
             file(12, 1, 1_700_000_000_000_002),
+            file(20, 2, 1_700_000_000_000_003),
         ];
-        const lib = new FakeLibrary([collection(1, 100)], files);
+        const lib = new FakeLibrary(
+            [collection(1, 100), collection(2, 200)],
+            files,
+        );
 
         const rows = await freshFiles(lib, 1);
-        expect(lib.freshCalls).toBe(1);
+        expect(lib.freshCalls).toBe(0);
+        expect(lib.freshCollectionsCalls).toBe(1);
+        expect(lib.freshFilesCalls).toEqual([1]);
         expect(rows?.map((f) => f.id)).toEqual([10, 11, 12]);
 
         // Field values come from raw metadata: microsecond creationTime and the
@@ -166,30 +183,144 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
         ]);
     });
 
+    // Without a collection ID, `files` lists every file in the account once
+    // (issue #211). File 11 is in collections 1 and 2; collection 2 is
+    // enumerated first, so 11 is listed once, with collection 2's record.
+    it("freshFiles without a collection ID lists every file once, from the first collection holding it", async () => {
+        const files = [
+            file(10, 1, 1_700_000_000_000_000),
+            file(11, 1, 1_700_000_000_000_001),
+            file(20, 2, 1_700_000_000_000_002),
+            file(11, 2, 1_700_000_000_000_001),
+        ];
+        const lib = new FakeLibrary(
+            [collection(2, 100), collection(1, 200)],
+            files,
+        );
+
+        const rows = await freshFiles(lib);
+        expect(lib.freshCalls).toBe(1);
+        expect(rows?.map((f) => [f.id, f.collectionID])).toEqual([
+            [20, 2],
+            [11, 2],
+            [10, 1],
+        ]);
+    });
+
     it("freshFiles returns undefined for an unknown collection", async () => {
         const lib = new FakeLibrary([collection(1, 100)], []);
         const rows = await freshFiles(lib, 999);
-        expect(lib.freshCalls).toBe(1);
+        expect(lib.freshCalls).toBe(0);
+        expect(lib.freshFilesCalls).toEqual([]);
         expect(rows).toBeUndefined();
     });
 
-    it("freshFile refreshes first, then resolves the photo and its raw record", async () => {
+    // Three collections, each holding one file.
-        const f = file(10, 1, 1_700_000_000_000_000);
+    const threeCollections = (): FakeLibrary =>
-        const lib = new FakeLibrary([collection(1, 100)], [f]);
+        new FakeLibrary(
-
+            [collection(1, 100), collection(2, 200), collection(3, 300)],
-        const resolved = await freshFile(lib, 10);
+            [
-        expect(lib.freshCalls).toBe(1);
+                file(10, 1, 1_700_000_000_000_000),
-        expect(resolved?.photo.fileID).toBe(10);
+                file(20, 2, 1_700_000_000_000_001),
-        expect(resolved?.file.metadata.title).toBe("file-10.jpg");
+                file(30, 3, 1_700_000_000_000_002),
-        expect(resolved?.file.metadata.creationTime).toBe(
+            ],
-            1_700_000_000_000_000,
         );
+
+    it("freshFile on an empty cache reads each collection's files in the server's order until one holds the file", async () => {
+        const lib = threeCollections();
+
+        const found = await freshFile(lib, 20);
+        expect(lib.freshCalls).toBe(0);
+        expect(lib.freshCollectionsCalls).toBe(1);
+        expect(lib.freshFilesCalls).toEqual([1, 2]);
+        expect(found?.metadata.title).toBe("file-20.jpg");
+        expect(found?.metadata.creationTime).toBe(1_700_000_000_000_001);
     });
 
-    it("freshFile returns undefined for an unknown file", async () => {
+    // The stand-in's cache holds what the server holds once `fresh()` has run.
-        const lib = new FakeLibrary([collection(1, 100)], []);
+    it("freshFile on a cache that holds the collections refreshes, then looks the file up in the cache", async () => {
-        const resolved = await freshFile(lib, 404);
+        const lib = threeCollections();
-        expect(lib.freshCalls).toBe(1);
+        await lib.fresh();
-        expect(resolved).toBeUndefined();
+
+        const found = await freshFile(lib, 30);
+        expect(lib.freshCalls).toBe(2);
+        expect(lib.freshCollectionsCalls).toBe(0);
+        expect(lib.freshFilesCalls).toEqual([]);
+        expect(found?.collectionID).toBe(3);
+    });
+
+    it("freshFile on an empty cache returns undefined for an unknown file, after reading every collection's files", async () => {
+        const lib = threeCollections();
+
+        const found = await freshFile(lib, 404);
+        expect(lib.freshCalls).toBe(0);
+        expect(lib.freshFilesCalls).toEqual([1, 2, 3]);
+        expect(found).toBeUndefined();
+    });
+
+    // The server lists collections 1, 2 and 3, and collection 3 was updated
+    // most recently. File 11 is in collections 1 and 2.
+    const fileInTwoCollections = (): FakeLibrary =>
+        new FakeLibrary(
+            [collection(1, 100), collection(2, 200), collection(3, 300)],
+            [
+                file(10, 1, 1_700_000_000_000_000),
+                file(11, 1, 1_700_000_000_000_001),
+                file(11, 2, 1_700_000_000_000_001),
+                file(20, 2, 1_700_000_000_000_002),
+                file(30, 3, 1_700_000_000_000_003),
+            ],
+        );
+
+    it("freshFilesByID on an empty cache reads the most recently updated collection's files first, until it has found every file", async () => {
+        const lib = fileInTwoCollections();
+
+        const found = await freshFilesByID(lib, [11, 20]);
+        expect(lib.freshCalls).toBe(0);
+        expect(lib.freshCollectionsCalls).toBe(1);
+        expect(lib.freshFilesCalls).toEqual([3, 2]);
+        expect(found.get(11)?.collection.id).toBe(2);
+        expect(found.get(11)?.file.collectionID).toBe(2);
+        expect(found.get(20)?.collection.id).toBe(2);
+    });
+
+    // Collection 1 was updated last, though collections 2 and 3 have higher
+    // IDs. File 11 is in collections 1 and 2.
+    it("freshFilesByID on an empty cache reads the most recently updated collection first, though its ID is the lowest", async () => {
+        const lib = new FakeLibrary(
+            [collection(1, 400), collection(2, 200), collection(3, 300)],
+            [
+                file(11, 1, 1_700_000_000_000_001),
+                file(11, 2, 1_700_000_000_000_001),
+            ],
+        );
+
+        const found = await freshFilesByID(lib, [11]);
+        expect(lib.freshFilesCalls).toEqual([1]);
+        expect(found.get(11)?.collection.id).toBe(1);
+        expect(found.get(11)?.file.collectionID).toBe(1);
+    });
+
+    // The stand-in's cache holds what the server holds once `fresh()` has run.
+    it("freshFilesByID on a cache that holds the collections refreshes, then finds each file in the most recently updated collection that holds it", async () => {
+        const lib = fileInTwoCollections();
+        await lib.fresh();
+
+        const found = await freshFilesByID(lib, [11, 404]);
+        expect(lib.freshCalls).toBe(2);
+        expect(lib.freshCollectionsCalls).toBe(0);
+        expect(lib.freshFilesCalls).toEqual([]);
+        expect(found.get(11)?.collection.id).toBe(2);
+        expect(found.get(11)?.file.collectionID).toBe(2);
+        expect(found.has(404)).toBe(false);
+    });
+
+    it("freshFilesByID on an empty cache reads every collection's files when a file is in none of them", async () => {
+        const lib = fileInTwoCollections();
+
+        const found = await freshFilesByID(lib, [10, 404]);
+        expect(lib.freshFilesCalls).toEqual([3, 2, 1]);
+        expect(found.get(10)?.collection.id).toBe(1);
+        expect(found.has(404)).toBe(false);
     });
 });

test/cli/run.test.ts

@@ -6,6 +6,7 @@ import { PassThrough } from "node:stream";
 import { describe, it, expect } from "vitest";
 
 import { ApiError } from "../../src/api/client.js";
+import { stderrLog } from "../../src/cli-log.js";
 import { run } from "../../src/cli-run.js";
 
 // A stream whose written text is kept in `text`; writes finish at once, so
@@ -23,7 +24,13 @@ const runToExit = async (
     const stdout = collector();
     const stderr = collector();
     const code = await new Promise<number>((resolve) => {
-        void run(command, stdout.stream, stderr.stream, resolve);
+        void run(
+            command,
+            stdout.stream,
+            stderr.stream,
+            stderrLog(stderr.stream, "info"),
+            resolve,
+        );
     });
     return { code, stdout: stdout.text(), stderr: stderr.text() };
 };
@@ -48,6 +55,78 @@ describe("run", () => {
         );
     });
 
+    // Node's `fetch` rejects every failure below HTTP with a TypeError that
+    // says only "fetch failed", and hangs the reason, with its errno, off
+    // `cause`. The printed line carries the reason, so a cron log tells a
+    // refused connection from a name that did not resolve.
+    it("prints the reason a fetch failed, from the error's cause", async () => {
+        const refused = new TypeError("fetch failed", {
+            cause: Object.assign(
+                new Error("connect ECONNREFUSED 1.2.3.4:443"),
+                { code: "ECONNREFUSED" },
+            ),
+        });
+        const result = await runToExit(Promise.reject(refused));
+        expect(result).toEqual({
+            code: 1,
+            stdout: "",
+            stderr: "quak: fetch failed: connect ECONNREFUSED 1.2.3.4:443\n",
+        });
+    });
+
+    // When the host has several addresses and every one refuses, Node's
+    // `fetch` gives as the cause an AggregateError with an empty message and
+    // one error per address. The printed line gives each address's reason.
+    it("prints the reason for each address a fetch could not reach", async () => {
+        const refusedAt = (address: string): Error =>
+            Object.assign(new Error(`connect ECONNREFUSED ${address}`), {
+                code: "ECONNREFUSED",
+            });
+        const refused = new TypeError("fetch failed", {
+            cause: Object.assign(
+                new AggregateError([
+                    refusedAt("2001:db8::1:443"),
+                    refusedAt("1.2.3.4:443"),
+                ]),
+                { code: "ECONNREFUSED" },
+            ),
+        });
+        const result = await runToExit(Promise.reject(refused));
+        expect(result).toEqual({
+            code: 1,
+            stdout: "",
+            stderr: "quak: fetch failed: connect ECONNREFUSED 2001:db8::1:443, connect ECONNREFUSED 1.2.3.4:443\n",
+        });
+    });
+
+    // An error that already quotes its cause's message is printed as it is.
+    it("does not print a cause's message twice", async () => {
+        const reset = new Error("read ECONNRESET");
+        const wrapped = new Error(`download: file 7: ${reset.message}`, {
+            cause: reset,
+        });
+        const result = await runToExit(Promise.reject(wrapped));
+        expect(result.stderr).toBe("quak: download: file 7: read ECONNRESET\n");
+    });
+
+    // OpenSSL ends its message for a failed TLS connection, such as one to a
+    // server that answers in plain HTTP, with a newline. The printed line is
+    // still one line.
+    it("prints a cause's message without the white space around it", async () => {
+        const wrongVersion = new TypeError("fetch failed", {
+            cause: Object.assign(
+                new Error(
+                    "C0B73A55A47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:\n",
+                ),
+                { code: "ERR_SSL_WRONG_VERSION_NUMBER" },
+            ),
+        });
+        const result = await runToExit(Promise.reject(wrongVersion));
+        expect(result.stderr).toBe(
+            "quak: fetch failed: C0B73A55A47F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:354:\n",
+        );
+    });
+
     it("prints a thrown value that is not an Error", async () => {
         const result = await runToExit(Promise.reject("offline"));
         expect(result).toEqual({
@@ -57,6 +136,14 @@ describe("run", () => {
         });
     });
 
+    // A cause that is not an Error, such as a thrown string, is printed as a
+    // thrown value that is not an Error is.
+    it("prints a cause that is not an Error", async () => {
+        const wrapped = new Error("ML data", { cause: "offline" });
+        const result = await runToExit(Promise.reject(wrapped));
+        expect(result.stderr).toBe("quak: ML data: offline\n");
+    });
+
     it("on a 401 from the server says to run quak login, on one line, and exits 3", async () => {
         const result = await runToExit(
             Promise.reject(new ApiError("unauthorized", 401)),

test/clip-keywords-sunset.json

@@ -0,0 +1,109 @@
+{
+    "photo": "https://upload.wikimedia.org/wikipedia/commons/d/db/Sunset_off_the_Massachusetts_coast_-_NOAA.jpg",
+    "sha256": "be62f3289c9bf7b75af68cd6978021d54e8a531d5d4cc493f6e9fbf5e1ab74d3",
+    "embedding": [
+        -0.0021123015, -0.04437615, -0.062354308, 0.006249992, 0.022141185,
+        0.011159019, 0.01227392, -0.00030532264, 0.03186437, 0.09196922,
+        0.039111234, -0.037084445, -0.032962628, -0.019820206, -0.013141825,
+        0.0047183544, 0.017137334, 0.046581306, 0.0047622235, 0.04982521,
+        0.023128154, 0.032080866, 0.037709303, 0.019241406, 0.0032100384,
+        -0.014275843, 0.049337693, 0.025953151, 0.014263995, -0.0062328726,
+        -0.040504098, -0.015083303, -0.008123875, -0.006233639, -0.05352683,
+        -0.0048083956, 0.0763486, -0.013155067, 0.025114346, 0.022016674,
+        0.0001588023, -0.010341572, -0.0037518279, -0.019181008, 0.04549337,
+        -0.0010725886, 0.004636866, 0.017177915, -0.0117317885, 0.0026097358,
+        0.030738998, 0.013158572, -0.08856403, -0.0121391015, 0.011477229,
+        -0.0033104012, 0.035524756, -0.018607879, -0.0071739843, -0.031906743,
+        0.0056713386, -0.0044195866, 0.00292471, 0.00061686034, -0.043075223,
+        0.09040427, 0.001522537, -0.011445652, 0.009609886, -0.0053201183,
+        -0.0057452377, 0.00972853, -0.0024397348, -0.021255083, 0.015893811,
+        -0.006020548, -0.09428023, 0.014277583, -0.017975299, -0.011878315,
+        0.011825521, 0.00014203975, -0.0020437292, -0.05305343, -0.024840152,
+        -0.011336872, -0.002796541, -0.00036171873, 0.004530499, 0.044486605,
+        0.021602718, -0.01292857, 0.024530916, 0.019559948, -0.039928272,
+        -0.0681174, 0.049101, 0.02461532, 0.0365531, -0.04070358, -0.011011939,
+        -0.0027188526, -0.020370876, 0.012904856, 0.043157287, -0.051658224,
+        -0.3726012, 0.019549074, -0.0499252, -0.04800559, -0.011493859,
+        0.02893256, 0.060341954, 0.0013929688, -0.051283866, -0.024286803,
+        0.044124287, -0.005995806, -0.062817656, 0.007369614, -0.021470634,
+        -0.028147545, 0.00411659, 0.04234773, 0.03787647, 0.008579682,
+        -0.038568925, -0.027498025, -0.028416378, 0.0022012168, 0.010906191,
+        -0.013745787, 0.018267838, 0.013531097, -0.061559845, 0.004682517,
+        -0.005993303, 0.0028113858, -0.0028286583, 0.0026029332, -0.06693403,
+        -0.0072381413, -0.02419584, 0.0037009937, -0.0011296495, -0.030310456,
+        0.0005999609, 0.009945354, 0.024915686, 0.038672045, 0.023009432,
+        -0.05093051, -0.09101409, -0.06686167, -0.06211481, -0.022600474,
+        0.0024902367, -0.0067171026, -0.016769778, -0.0032803628, 0.008623083,
+        -0.0016110836, 0.014925073, 0.033811048, 0.012829449, -0.052575592,
+        0.03728281, -0.0051201633, 0.021238064, -0.031086605, -0.0043402985,
+        0.25054494, -0.028396713, -0.010725523, -0.023445267, 0.021751683,
+        0.003130971, 0.012270548, -0.014836475, 0.014466885, 0.0021478906,
+        -0.088153616, 0.015356315, 0.039639276, -0.040283255, 0.014923924,
+        -0.029303031, -0.004475201, 0.047299225, -0.06726505, -0.012491676,
+        -0.002080582, 0.03506341, 0.04146718, 0.0074114986, -0.009582633,
+        0.006623749, -0.05454898, -0.0034387247, -0.014780821, 0.03698328,
+        -0.034441486, 0.22032379, -0.021792699, 0.008001907, 0.0341019,
+        0.0025784625, -0.0071689896, -0.014151225, -0.050547987, -0.030372301,
+        -0.03288388, -0.042987447, -0.0033656165, 0.021502906, -0.0030374376,
+        -0.024413, -0.03506603, 0.030891743, 0.0069994032, 0.004887366,
+        0.030707076, 0.0065348223, 0.0008240555, 0.016809713, -0.024151647,
+        -0.0040458064, 0.044567548, 0.03908212, -0.061782435, 0.0026002831,
+        0.02003242, -0.0115451, -0.045135964, -0.0663542, 0.005240975,
+        0.0010777109, -0.009276002, -0.011919081, 0.027654119, -0.013634551,
+        0.0129154315, -0.013195512, 0.040571272, -0.099290505, -0.04656693,
+        -0.20979854, 0.04181004, -0.06769166, -0.04121946, 0.018260444,
+        0.005617506, -0.15694222, 0.032472793, 0.014189598, 0.008345103,
+        -0.041118417, 0.0074527003, -0.011494104, 0.004906474, -0.0027100458,
+        0.0009994594, -0.041061398, -0.045274835, 0.015187793, 0.034275036,
+        -0.02813289, 0.02098019, -0.015043198, 0.03716667, 0.013542642,
+        -0.008104835, 0.014965486, -0.027978558, 0.021163663, -0.006942737,
+        -0.034969814, -0.038961593, 0.016802894, -0.012318175, -0.047821593,
+        -0.01933216, 0.0064432374, 0.0322358, -0.020437784, 0.02659447,
+        -0.03022022, -0.019443465, -0.094178624, -0.017009107, 0.03860671,
+        -0.00023695534, -0.033089854, -0.014677462, 0.012356399, 0.018040499,
+        -0.19360113, 0.011710817, -0.019871691, 0.020124774, 0.024948105,
+        -0.024573864, -0.028349072, -0.041731454, 0.032646254, 0.015098165,
+        0.0018447012, -0.01129152, -0.02481687, 0.036048, 0.20797811,
+        0.04317757, -0.017239755, 0.033431605, -0.011384422, 0.0099174455,
+        -0.19739223, -0.008901062, -0.056426458, 0.014591864, 0.0044474266,
+        0.008209326, -0.012241819, 0.042553645, -0.021552453, 0.008611734,
+        0.048944127, -0.013872049, 0.019179989, 0.008183264, -0.053223096,
+        -0.027327595, -0.0033542586, -0.012353858, -0.010941209, 0.010212546,
+        0.029714134, -0.011385593, -0.0013529479, 0.014505385, 0.01148663,
+        0.012613636, -0.0074430504, -0.014008101, -0.028146038, -0.042778917,
+        0.02143165, 0.016812252, -0.06286215, 0.02085532, -0.035128277,
+        0.0241282, 0.01397908, -0.034992814, 0.013153766, 0.0068236147,
+        0.013594184, -0.06859393, 0.051849406, 0.029675119, -0.0037699796,
+        -0.00013216575, 0.02916647, 0.031859048, -0.015006051, 0.0001439966,
+        0.0026584554, -0.0046948595, 0.039754897, 0.034873154, 0.0012870439,
+        -0.011516082, -0.074089095, 0.019241294, 0.00076195505, 0.0027334464,
+        -0.036206458, -0.0011109262, -0.048838835, -0.00204685, 0.01179991,
+        0.0068136845, 0.009852495, 0.017308988, 0.03159545, 0.058803353,
+        0.038116634, 0.008836143, -0.010460637, -0.010994099, -0.0012926942,
+        -0.048370842, 0.037696566, -0.08093239, -0.0041242186, -0.005866373,
+        0.02048607, -0.12549639, 0.011705739, -0.0021425558, 0.018078592,
+        -0.038228087, 0.004245971, -0.11560507, 0.01381203, -0.02734893,
+        0.029595908, 0.101082794, -0.042842254, -0.08397301, -0.1546237,
+        -0.037624564, 0.0069606556, -0.008020479, 0.0043268185, 0.016504597,
+        0.044554908, -0.05240209, 0.020519335, -0.050994243, -0.019861093,
+        0.022066997, -0.0188911, 0.012919696, -0.022872692, -0.012961374,
+        -0.020071112, 0.0074553066, 0.06402014, 0.04065211, 0.014139608,
+        0.06221546, -0.02574805, 0.010114545, 0.020619567, 0.032264773,
+        -0.009148828, 0.022849955, -0.016593235, -0.032456633, -0.006737303,
+        -0.0037493357, -0.065912865, -0.0024181998, -0.0037048983, -0.019304488,
+        0.009085037, 0.057529982, -0.0378338, 0.01792042, 0.01885012,
+        -0.0043600523, 0.017382368, -0.027792772, 0.006749118, -0.0053297766,
+        -0.011628587, 0.015736576, 0.024801968, 0.011543913, -0.01159873,
+        -0.000759976, 0.042183932, 0.05764386, -0.06422416, 0.010577374,
+        -0.02261249, 0.02282793, -0.05141075, -0.11287078, -0.020670144,
+        -0.020953743, 0.030885734, -0.012597732, -0.121472664, 0.028447151,
+        -0.0415664, -0.010061708, 0.06441338, 0.0271207, -0.0016693748,
+        -0.0045694243, 0.020342957, 0.030528478, 0.007129517, -0.0029600926,
+        -0.007061397, 0.06036486, -0.004736018, -0.0090786135, -0.07601116,
+        -0.039556272, -0.051269546, -0.019473685, 0.029858151, 0.028464556,
+        0.012424476, 0.013742254, 0.008742927, 0.006065751, -0.009892663,
+        0.07699999, -0.003443197, 0.0027863425, 0.029197995, 0.0211013,
+        0.01098379, 0.016684106, -0.028917195, 0.008122037, 0.0265241,
+        -0.04461223
+    ]
+}

test/clip-keywords.test.ts

@@ -0,0 +1,129 @@
+// `clipKeywords`: the keywords quak writes for a file in place of its CLIP
+// embedding.
+//
+// Ente's ML data for a file holds a CLIP embedding, 512 numbers its image
+// model gave the photo. Ente's photos app puts a file in a category of its
+// Discover section when the cosine similarity of that embedding and the text
+// embedding of the category's query is at least the category's minimum score,
+// and keeps videos out of some categories. `clipKeywords` applies that rule
+// and returns the titles of the categories a file falls in, highest score
+// first; `quak backup` and `quak backup-metadata` write them as `keywords`.
+
+import { readFileSync } from "node:fs";
+import { describe, expect, it } from "vitest";
+
+import { DISCOVER_CATEGORIES } from "../src/clip-keyword-vectors.js";
+import { clipKeywords } from "../src/clip-keywords.js";
+
+// 512 numbers: the ones given, then zeros.
+const vector = (...start: number[]): number[] => [
+    ...start,
+    ...new Array<number>(512 - start.length).fill(0),
+];
+
+const withEmbedding = (embedding: unknown) => ({
+    face: { faces: [] },
+    clip: { embedding },
+});
+
+// Two made-up categories, each with a text embedding along one axis. The
+// embedding [3, 4] has a cosine similarity of 3/5 with "Across" and 4/5 with
+// "Up". Videos are kept out of "Across".
+const ACROSS = {
+    title: "Across",
+    query: "across",
+    minScore: 0.6,
+    showVideo: false,
+    textEmbedding: vector(1),
+};
+const UP = {
+    title: "Up",
+    query: "up",
+    minScore: 0.5,
+    showVideo: true,
+    textEmbedding: vector(0, 1),
+};
+
+describe("clipKeywords", () => {
+    it("gives each category whose minimum score the similarity reaches, highest score first", () => {
+        expect(
+            clipKeywords(withEmbedding(vector(3, 4)), false, [ACROSS, UP]),
+        ).toEqual(["Up", "Across"]);
+    });
+
+    it("leaves out a category when the similarity is just below its minimum score", () => {
+        const stricter = { ...ACROSS, minScore: 0.6 + 1e-9 };
+
+        expect(
+            clipKeywords(withEmbedding(vector(3, 4)), false, [stricter, UP]),
+        ).toEqual(["Up"]);
+    });
+
+    it("keeps a video out of a category that leaves out videos", () => {
+        expect(
+            clipKeywords(withEmbedding(vector(3, 4)), true, [ACROSS, UP]),
+        ).toEqual(["Up"]);
+    });
+
+    it("gives an empty list for an embedding that falls in no category", () => {
+        expect(
+            clipKeywords(withEmbedding(vector(0, 0, 1)), false, [ACROSS, UP]),
+        ).toEqual([]);
+    });
+
+    it.each([
+        ["no CLIP embedding", { face: { faces: [] } }],
+        ["a CLIP part that is not an object", { clip: null }],
+        ["a CLIP part without an embedding", { clip: {} }],
+        ["an embedding of 511 numbers", withEmbedding(vector(3, 4).slice(1))],
+        ["an embedding with NaN in it", withEmbedding(vector(3, NaN))],
+        ["an embedding with Infinity in it", withEmbedding(vector(Infinity))],
+        ["an embedding of strings", withEmbedding(vector(3, 4).map(String))],
+        ["an embedding of zeros", withEmbedding(vector())],
+    ])("gives no keywords for ML data with %s", (_, mlData) => {
+        expect(clipKeywords(mlData, false, [ACROSS, UP])).toBeUndefined();
+    });
+});
+
+// The CLIP embedding Ente's image model gives a public-domain NOAA photo of a
+// sunset, made by bin/clip-keyword-vectors from the photo at `photo`.
+const SUNSET: { photo: string; sha256: string; embedding: number[] } =
+    JSON.parse(
+        readFileSync(
+            new URL("clip-keywords-sunset.json", import.meta.url),
+            "utf-8",
+        ),
+    );
+
+describe("Ente's Discover categories", () => {
+    it("are the 16 of Ente's photos app, each with a text embedding of norm 1", () => {
+        expect(DISCOVER_CATEGORIES.map((c) => c.title)).toEqual([
+            "Identity",
+            "Screenshots",
+            "QR Codes",
+            "Receipts",
+            "Notes",
+            "Memes",
+            "Visiting Cards",
+            "Babies",
+            "Pets",
+            "Selfies",
+            "Wallpapers",
+            "Food",
+            "Celebrations",
+            "Sunset",
+            "Hills",
+            "Greenery",
+        ]);
+        for (const category of DISCOVER_CATEGORIES) {
+            expect(category.textEmbedding).toHaveLength(512);
+            expect(Math.hypot(...category.textEmbedding)).toBeCloseTo(1, 5);
+        }
+    });
+
+    it("give a photo of a sunset the keyword Sunset and no other", () => {
+        expect(
+            clipKeywords({ clip: { embedding: SUNSET.embedding } }, false),
+        ).toEqual(["Sunset"]);
+    });
+});

test/download/download.test.ts

@@ -74,7 +74,11 @@ import {
 } from "vitest";
 import { init, toBase64, STREAM_CHUNK_SIZE } from "../../src/crypto/index.js";
 import { ApiClient } from "../../src/api/client.js";
-import { ApiError, TruncatedStreamError } from "../../src/errors.js";
+import {
+    ApiError,
+    describeError,
+    TruncatedStreamError,
+} from "../../src/errors.js";
 import type { RetryOptions } from "../../src/retry.js";
 import {
     downloadFile,
@@ -654,6 +658,26 @@ describe("downloadFile", () => {
         expect(readdirSync(parent)).toEqual(["cwd"]);
     });
 
+    // A file system refuses a file name over 255 bytes.
+    it("cuts a title too long for a file name short, keeping its extension", async () => {
+        const plaintext = new Uint8Array([1, 2, 3]);
+        const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+        const { header, ciphertext } = encryptFileBody(plaintext, key);
+        const thumbPush =
+            sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
+        const file = buildMockEnteFile(key, header, thumbPush.header);
+        file.metadata.title = `${"t".repeat(296)}.jpg`;
+        const dir = mkdtempSync(join(testDir, "long-title-"));
+
+        const api = new ApiClient({ fetch: mockFetchForBody(ciphertext) });
+        const result = await inDirectory(dir, () => downloadFile(api, file));
+
+        const name = `${"t".repeat(251)}.jpg`;
+        expect(Buffer.byteLength(name)).toBe(255);
+        expect(result.path).toBe(name);
+        expect(readdirSync(dir)).toEqual([name]);
+    });
+
     it("uses an explicit outPath verbatim, even one with ..", async () => {
         // The caller is trusted: its path is not sanitized.
         const { api, file } = fixtureFor(
@@ -747,6 +771,28 @@ describe("downloadThumbnail", () => {
         expect(result.path).toBe("thumb__etc_passwd");
         expect(readdirSync(dir)).toEqual(["thumb__etc_passwd"]);
     });
+
+    it("leaves room for thumb_ when it cuts a title too long for a file name short", async () => {
+        const plaintext = new Uint8Array([0xff, 0xd8, 0xff, 0xe0]);
+        const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+        const filePush =
+            sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
+        const { header: thumbHeader, ciphertext: thumbCipher } =
+            encryptFileBody(plaintext, key);
+        const file = buildMockEnteFile(key, filePush.header, thumbHeader);
+        file.metadata.title = `${"t".repeat(296)}.jpg`;
+        const dir = mkdtempSync(join(testDir, "long-thumb-title-"));
+
+        const api = new ApiClient({ fetch: mockFetchForBody(thumbCipher) });
+        const result = await inDirectory(dir, () =>
+            downloadThumbnail(api, file),
+        );
+
+        const name = `thumb_${"t".repeat(245)}.jpg`;
+        expect(Buffer.byteLength(name)).toBe(255);
+        expect(result.path).toBe(name);
+        expect(readdirSync(dir)).toEqual([name]);
+    });
 });
 
 // ---------------------------------------------------------------------------
@@ -834,6 +880,11 @@ describe.each(entryPoints)(
             expect(((err as Error).cause as Error).message).toMatch(
                 /authentication failed/i,
             );
+            // As quak prints it and records it in `failures.json`, the error
+            // states the authentication failure once, from its cause.
+            expect(describeError(err)).toMatch(
+                /^download: stream truncated: response body ended with \d+ trailing bytes \(transfer stopped mid-chunk, or the data is corrupt\): secretstream chunk authentication failed$/,
+            );
 
             // And, as with every other failure, the destination is untouched
             // and no staged temp file survives.
@@ -1872,6 +1923,53 @@ describe("downloadFile live photos", () => {
         ]);
     });
 
+    // A file system refuses a file name over 255 bytes. Each name is cut
+    // short once its own extension is in it.
+    const imageName = `${"t".repeat(250)}.heic`;
+    const videoName = `${"t".repeat(251)}.mov`;
+
+    // The title fits, but ".heic" is one byte longer than ".jpg".
+    it("cuts the image's name short for a 255-byte title ending in .jpg when no outPath is given", async () => {
+        const title = `${"t".repeat(251)}.jpg`;
+        const t = setup(livePhotoZip(), { ...livePhoto, title });
+
+        const result = await inDirectory(t.dir, () =>
+            downloadFile(t.api, t.file),
+        );
+
+        expect(Buffer.byteLength(title)).toBe(255);
+        expect(Buffer.byteLength(imageName)).toBe(255);
+        expect(Buffer.byteLength(videoName)).toBe(255);
+        expect(result.path).toBe(imageName);
+        expect(result.videoPath).toBe(videoName);
+        expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]);
+    });
+
+    it("cuts the names of the image and the video short for a 300-byte title with no extension when no outPath is given", async () => {
+        const t = setup(livePhotoZip(), {
+            ...livePhoto,
+            title: "t".repeat(300),
+        });
+
+        const result = await inDirectory(t.dir, () =>
+            downloadFile(t.api, t.file),
+        );
+
+        expect(result.path).toBe(imageName);
+        expect(result.videoPath).toBe(videoName);
+        expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]);
+    });
+
+    // The caller's path is used as is; only the extension is each part's own.
+    it("keeps an outPath too long for a file name as given", async () => {
+        const t = setup(livePhotoZip(), livePhoto);
+        const outPath = join(t.dir, `${"t".repeat(251)}.jpg`);
+
+        await expect(downloadFile(t.api, t.file, outPath)).rejects.toThrow(
+            /ENAMETOOLONG/,
+        );
+    });
+
     it("gives each part its own extension from the ZIP, letters and digits only", async () => {
         const zip = livePhotoZip({ "image.JPG": IMAGE, "video.m-4v": VIDEO });
         const t = setup(zip, livePhoto);

test/ente-server.ts

@@ -19,8 +19,11 @@ import { SRP, SrpServer } from "fast-srp-hap";
 import { toBase64, deriveKEK, deriveLoginSubkey } from "../src/crypto/index.js";
 import type { KeyAttributes } from "../src/auth/types.js";
 
-const TEST_OPS = 2;
+// The account's password is stretched at libsodium's lowest Argon2id cost.
-const TEST_MEM = 64 * 1024 * 1024;
+// Every login stretches it again, and at a realistic cost that takes seconds on
+// a loaded host.
+const TEST_OPS = 1;
+const TEST_MEM = 8192;
 
 export interface MetaMockState {
     verifier: Buffer;

test/filename/filename.test.ts

@@ -14,7 +14,13 @@
 
 import { describe, expect, it } from "vitest";
 
-import { safeExtension, sanitizeFileName } from "../../src/filename.js";
+import {
+    defaultFileName,
+    safeExtension,
+    sanitizeFileName,
+    shortenKeepingExtension,
+    shortenToBytes,
+} from "../../src/filename.js";
 
 const FALLBACK = "file-42";
 
@@ -90,4 +96,133 @@ describe("safeExtension", () => {
         expect(safeExtension("x.jp g")).toBe(".bin");
         expect(safeExtension("x.jpg\0")).toBe(".bin");
     });
+
+    it("keeps an extension of 16 letters and digits, and uses .bin for a longer one", () => {
+        expect(safeExtension(`x.${"e".repeat(16)}`)).toBe(`.${"e".repeat(16)}`);
+        expect(safeExtension(`x.${"e".repeat(17)}`)).toBe(".bin");
+        expect(safeExtension(`b.${"e".repeat(300)}`)).toBe(".bin");
+    });
+});
+
+describe("shortenToBytes", () => {
+    it("returns a name that fits unchanged", () => {
+        expect(shortenToBytes("beach", 5)).toBe("beach");
+        expect(shortenToBytes("café", 5)).toBe("café");
+        expect(shortenToBytes("", 0)).toBe("");
+    });
+
+    it("cuts a name to the number of bytes given", () => {
+        expect(shortenToBytes("L".repeat(300), 255)).toBe("L".repeat(255));
+    });
+
+    // In UTF-8 "é" takes two bytes and an emoji four, which a JavaScript
+    // string holds as two halves.
+    it("cuts between two characters, never inside one", () => {
+        expect(shortenToBytes("ééé", 5)).toBe("éé");
+        expect(shortenToBytes("a\u{1F600}b", 4)).toBe("a");
+        expect(shortenToBytes("a\u{1F600}b", 5)).toBe("a\u{1F600}");
+        expect(Buffer.byteLength(shortenToBytes("é".repeat(200), 255))).toBe(
+            254,
+        );
+    });
+});
+
+describe("shortenKeepingExtension", () => {
+    it("returns a name that fits unchanged", () => {
+        expect(shortenKeepingExtension("beach.jpg", 9)).toBe("beach.jpg");
+        expect(shortenKeepingExtension("beach", 5)).toBe("beach");
+    });
+
+    it("cuts the part before the extension short and keeps the extension", () => {
+        expect(shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255)).toBe(
+            `${"t".repeat(251)}.jpg`,
+        );
+        expect(shortenKeepingExtension("é".repeat(150), 255)).toBe(
+            "é".repeat(127),
+        );
+    });
+
+    // A name cut down to its extension would start with a dot, and so be
+    // hidden.
+    it("keeps the first character before the extension, cutting the extension short to leave room for it", () => {
+        expect(shortenKeepingExtension(`b.${"e".repeat(300)}`, 255)).toBe(
+            `b.${"e".repeat(253)}`,
+        );
+        expect(shortenKeepingExtension(`ab.${"e".repeat(300)}`, 255)).toBe(
+            `a.${"e".repeat(253)}`,
+        );
+        expect(shortenKeepingExtension(`é.${"e".repeat(300)}`, 255)).toBe(
+            `é.${"e".repeat(252)}`,
+        );
+        // "€" takes three bytes, so the extension is cut to 253 and the
+        // part before it keeps the two bytes left over, for the "a".
+        expect(shortenKeepingExtension(`a.${"€".repeat(90)}`, 255)).toBe(
+            `a.${"€".repeat(84)}`,
+        );
+    });
+
+    it("puts `insert` right before the extension and leaves room for it", () => {
+        expect(shortenKeepingExtension("beach.jpg", 255, " (7)")).toBe(
+            "beach (7).jpg",
+        );
+        expect(
+            shortenKeepingExtension(`${"t".repeat(296)}.jpg`, 255, " (7)"),
+        ).toBe(`${"t".repeat(247)} (7).jpg`);
+        expect(
+            shortenKeepingExtension(`b.${"e".repeat(300)}`, 255, " (7)"),
+        ).toBe(`b (7).${"e".repeat(249)}`);
+    });
+});
+
+describe("defaultFileName", () => {
+    it("is the title when it is safe and fits", () => {
+        expect(defaultFileName("IMG_0001.HEIC", 42)).toBe("IMG_0001.HEIC");
+    });
+
+    it("replaces unsafe characters, and is file-<id> for an empty title", () => {
+        expect(defaultFileName("../x.jpg", 42)).toBe("__x.jpg");
+        expect(defaultFileName("", 42)).toBe("file-42");
+    });
+
+    it("cuts a title over 255 bytes short, keeping its extension", () => {
+        expect(defaultFileName(`${"t".repeat(296)}.jpg`, 42)).toBe(
+            `${"t".repeat(251)}.jpg`,
+        );
+        expect(defaultFileName(`b.${"e".repeat(300)}`, 42)).toBe(
+            `b.${"e".repeat(253)}`,
+        );
+    });
+
+    it("puts `prefix` before the title and leaves room for it", () => {
+        expect(defaultFileName("beach.jpg", 42, { prefix: "thumb_" })).toBe(
+            "thumb_beach.jpg",
+        );
+        expect(
+            defaultFileName(`${"t".repeat(296)}.jpg`, 42, { prefix: "thumb_" }),
+        ).toBe(`thumb_${"t".repeat(245)}.jpg`);
+    });
+
+    // A live photo's image and video are each named with their own
+    // extension, and the name is cut short once that extension is in it.
+    it("puts `ext` in place of the title's extension before cutting the name short", () => {
+        expect(defaultFileName("IMG_0001.HEIC", 42, { ext: ".mov" })).toBe(
+            "IMG_0001.mov",
+        );
+        expect(defaultFileName("", 42, { ext: ".mov" })).toBe("file-42.mov");
+        expect(
+            defaultFileName(`${"t".repeat(251)}.jpg`, 42, { ext: ".heic" }),
+        ).toBe(`${"t".repeat(250)}.heic`);
+        expect(defaultFileName("t".repeat(300), 42, { ext: ".heic" })).toBe(
+            `${"t".repeat(250)}.heic`,
+        );
+    });
+
+    // Cutting `CONSOLE` or `NULL` short before a long extension leaves `CON`
+    // or `NUL`, so the name gets the `_` `sanitizeFileName` gives a device
+    // name, and the part before the extension gives up a byte for it.
+    it("never returns a Windows device name, even once cut short", () => {
+        const ext = `.${"e".repeat(251)}`;
+        expect(defaultFileName(`CONSOLE${ext}`, 42)).toBe(`_CO${ext}`);
+        expect(defaultFileName(`NULL${ext}`, 42)).toBe(`_NU${ext}`);
+    });
 });

test/library/content-library.test.ts

@@ -253,6 +253,12 @@ describe("Library content wiring", () => {
         await expect(
             lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
         ).rejects.toThrow(/content cache/i);
+        await expect(lib.originalOf(file(1, 1))).rejects.toThrow(
+            /content cache/i,
+        );
+        await expect(lib.thumbnailOf(file(1, 1))).rejects.toThrow(
+            /content cache/i,
+        );
         await lib.close();
     });
 });

test/library/content.test.ts

@@ -375,6 +375,25 @@ describe("ContentCache.original / thumbnail", () => {
         await cache.open();
         await expect(cache.original(999)).rejects.toThrow(/unknown file/i);
     });
+
+    // `getFile` knows no file here, so the cache can fetch only the file it is
+    // given.
+    it("originalOf and thumbnailOf fetch and cache the file they are given", async () => {
+        const { cache, source } = buildCache({ files: [] });
+        await cache.open();
+
+        const original = await cache.originalOf(file(5));
+        const thumbnail = await cache.thumbnailOf(file(5));
+
+        expect(source.originalCalls).toEqual([5]);
+        expect(source.thumbnailCalls).toEqual([5]);
+        expect(original.path).toBe(join(cacheDir, "originals", "5.jpg"));
+        expect(thumbnail.path).toBe(join(cacheDir, "thumbnails", "5.jpg"));
+        expect(cache.pathsFor(5)).toEqual({
+            originalPath: original.path,
+            thumbnailPath: thumbnail.path,
+        });
+    });
 });
 
 describe("ContentCache.ensureThumbnails", () => {

test/library/fresh.test.ts

@@ -14,6 +14,9 @@
  *     three.
  *  3. A refresh that fails rejects the fresh read (currency was unavailable),
  *     while the default reads stay silent and keep serving the last good copy.
+ *  4. `freshCollections()` reads the collection list alone, with one request
+ *     and no collection's files, and stores nothing, so the next refresh still
+ *     fetches the files of every collection (issue #210).
  *
  * The client is the same metadata-only mock the background-refresh tests use:
  * no crypto, no network, scripted pages, and a record of each call. A long
@@ -270,3 +273,70 @@ describe("Library.fresh", () => {
         }
     });
 });
+
+describe("Library.freshCollections", () => {
+    let dir: string;
+    let cacheDirectory: string;
+
+    beforeEach(() => {
+        dir = mkdtempSync(join(tmpdir(), "quak-fresh-collections-"));
+        cacheDirectory = join(dir, "cache");
+    });
+
+    afterEach(() => {
+        rmSync(dir, { recursive: true, force: true });
+    });
+
+    // A refresh fetches a collection's files only when the collection is new
+    // or its updationTime has advanced past the stored copy's. Had the
+    // collection list been stored, the next refresh would take every
+    // collection for one it already holds the files of, and fetch none.
+    it("asks for the collection list alone and stores nothing", async () => {
+        const client = new MockClient();
+        const page = {
+            collections: [collection(2, 100), collection(1, 110)],
+            deleted: [],
+            cursor: 110,
+        };
+        client.collectionsQueue.push(page, page);
+        client.filesFor(1, {
+            files: [file(1001, 1, 90)],
+            deleted: [],
+            cursor: 90,
+        });
+        client.filesFor(2, {
+            files: [file(2001, 2, 95)],
+            deleted: [],
+            cursor: 95,
+        });
+
+        const lib = await Library.open({
+            client,
+            cacheDirectory,
+            refreshOnOpen: false,
+            refreshIntervalSeconds: SLOW_INTERVAL,
+        });
+        try {
+            // One request, the whole list in the server's order, and no
+            // collection's files.
+            const collections = await lib.freshCollections();
+            expect(collections.map((c) => c.id)).toEqual([2, 1]);
+            expect(collections.map((c) => c.name)).toEqual([
+                "album-2",
+                "album-1",
+            ]);
+            expect(client.collectionsSinceTimes).toEqual([0]);
+            expect(client.filesCalls).toEqual([]);
+            expect(lib.listCollections()).toEqual([]);
+
+            // The next refresh still asks from the beginning and fetches the
+            // files of both collections.
+            const reads = await lib.fresh();
+            expect(client.collectionsSinceTimes).toEqual([0, 0]);
+            expect(reads.photos.byID({ fileID: 1001 })?.fileID).toBe(1001);
+            expect(reads.photos.byID({ fileID: 2001 })?.fileID).toBe(2001);
+        } finally {
+            await lib.close();
+        }
+    });
+});

test/library/library.test.ts

@@ -26,6 +26,9 @@
  *    opening.
  * 8. A save failure that leaves RAM ahead of disk keeps `status().lastError`
  *    set and keeps retrying the write; a later empty refresh does not clear it.
+ * 9. With `refreshOnOpen: false`, `open()` starts no refresh at all, and a
+ *    refresh reports through `onProgress` each collection whose files it
+ *    fetched.
  *
  * The client is a mock: no crypto, no network. It serves scripted pages and
  * records the `sinceTime` each call carried so cursor threading is provable.
@@ -581,6 +584,107 @@ describe("Library.open and background refresh", () => {
         }
     });
 
+    // A caller that asks for each refresh it needs, as the CLI's read commands
+    // do, opens without one: on an empty cache the refresh `open()` would
+    // await fetches the files of every collection, which `quak collections`
+    // never needs (issue #210).
+    it("starts no refresh as it opens when refreshOnOpen is false", async () => {
+        const client = new MockClient();
+        client.collectionsQueue.push({
+            collections: [collection(1, 100)],
+            deleted: [],
+            cursor: 100,
+        });
+
+        const lib = await Library.open({
+            client,
+            cacheDirectory,
+            refreshOnOpen: false,
+            refreshIntervalSeconds: Infinity,
+        });
+        try {
+            expect(client.collectionsSinceTimes).toEqual([]);
+            expect(lib.listCollections()).toEqual([]);
+            expect(lib.status().lastRefreshAt).toBeUndefined();
+
+            // The refresh comes when the caller asks for it.
+            await lib.fresh();
+            expect(client.collectionsSinceTimes).toEqual([0]);
+            expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
+        } finally {
+            await lib.close();
+        }
+    });
+
+    // A refresh fetches the files of each collection that changed, one
+    // collection after another, and stores nothing until it has them all. On
+    // a large account that takes a while, so it reports each collection whose
+    // files it has fetched, and how many it will fetch in all (issue #210).
+    it("reports each collection whose files a refresh fetched", async () => {
+        const events: RefreshEvent[] = [];
+        const client = new MockClient();
+        client.collectionsQueue.push({
+            collections: [
+                collection(1, 100),
+                collection(2, 100),
+                collection(3, 100),
+            ],
+            deleted: [],
+            cursor: 100,
+        });
+
+        const lib = await Library.open({
+            client,
+            cacheDirectory,
+            refreshIntervalSeconds: Infinity,
+            onProgress: (e) => events.push(e),
+        });
+        try {
+            expect(events).toEqual([
+                { operation: "refresh", status: "started" },
+                {
+                    operation: "refresh",
+                    status: "progress",
+                    fetched: 1,
+                    total: 3,
+                },
+                {
+                    operation: "refresh",
+                    status: "progress",
+                    fetched: 2,
+                    total: 3,
+                },
+                {
+                    operation: "refresh",
+                    status: "progress",
+                    fetched: 3,
+                    total: 3,
+                },
+                { operation: "refresh", status: "done" },
+            ]);
+
+            // Only collection 2 changed since, so the next refresh fetches
+            // its files alone and counts one.
+            events.length = 0;
+            client.collectionsQueue.push({
+                collections: [collection(2, 200)],
+                deleted: [],
+                cursor: 200,
+            });
+            await lib.fresh();
+            expect(events.filter((e) => e.status === "progress")).toEqual([
+                {
+                    operation: "refresh",
+                    status: "progress",
+                    fetched: 1,
+                    total: 1,
+                },
+            ]);
+        } finally {
+            await lib.close();
+        }
+    });
+
     it("keeps a save failure visible until a save actually succeeds", async () => {
         const saveSpy = vi
             .spyOn(MetadataStore.prototype, "save")

test/library/mldata.test.ts

@@ -12,9 +12,11 @@
  *
  *   2. `Library` wiring: after each refresh the library fetches ML data through
  *      the metadata pool for every known file not yet cached, is incremental on
- *      later refreshes, and refetches a file whose `updationTime` advanced.
+ *      later refreshes, and refetches a file whose `updationTime` advanced. A
- *      Opening a cache directory written by another account starts empty,
+ *      file whose ML data did not decode is reported with the reason, which
- *      its ML data included (issue #104).
+ *      stays listed until the next pass ends, and asked for again. Opening a
+ *      cache directory written by another account starts empty, its ML data
+ *      included (issue #104).
  *
  * Embedding values are chosen to be exactly representable as float32 so the
  * round-trip through `clip.f32` compares equal.
@@ -26,7 +28,7 @@ import { tmpdir } from "node:os";
 import { join } from "node:path";
 
 import { MLDataStore } from "../../src/library/mldata.js";
-import { Library } from "../../src/library/index.js";
+import { Library, type RefreshEvent } from "../../src/library/index.js";
 import type { CollectionsPage, FilesPage } from "../../src/client.js";
 import type { MLData } from "../../src/mldata-fetch.js";
 import type { Collection, EnteFile } from "../../src/model/types.js";
@@ -256,12 +258,14 @@ const file = (
 
 // A mock client that serves scripted collection/file pages and per-file ML
 // payloads, recording every ML fetch request so incremental behaviour is
-// provable.
+// provable. A file in `mlErrorByFile` has ML data that did not decode, for
+// the reason given there.
 class MLMockClient {
     userID = USER_ID;
     collectionsQueue: CollectionsPage[] = [];
     filesByCollection = new Map<number, FilesPage[]>();
     mlByFile = new Map<number, MLData>();
+    mlErrorByFile = new Map<number, Error>();
     mlFetchCalls: number[][] = [];
 
     whoami(): { email: string; userID: number } {
@@ -298,14 +302,20 @@ class MLMockClient {
     async fetchMLData(args: {
         fileIDs: number[];
         fileKeys: Map<number, Uint8Array>;
-    }): Promise<Map<number, MLData>> {
+    }): Promise<{
+        payloads: Map<number, MLData>;
+        errors: Map<number, unknown>;
+    }> {
         this.mlFetchCalls.push([...args.fileIDs]);
-        const result = new Map<number, MLData>();
+        const payloads = new Map<number, MLData>();
+        const errors = new Map<number, unknown>();
         for (const id of args.fileIDs) {
             const p = this.mlByFile.get(id);
-            if (p) result.set(id, p);
+            if (p) payloads.set(id, p);
+            const error = this.mlErrorByFile.get(id);
+            if (error) errors.set(id, error);
         }
-        return result;
+        return { payloads, errors };
     }
 
     filesFor(collectionID: number, ...pages: FilesPage[]): void {
@@ -449,6 +459,137 @@ describe("Library ML-data fetch on refresh", () => {
         }
     });
 
+    it("reports a file whose ML data did not decode through status() and onProgress, and asks for it again", async () => {
+        const client = new MLMockClient();
+        client.collectionsQueue.push({
+            collections: [collection(1, 100)],
+            deleted: [],
+            cursor: 100,
+        });
+        client.filesFor(1, {
+            files: [file(1001, 1, 90), file(1002, 1, 95)],
+            deleted: [],
+            cursor: 95,
+        });
+        client.mlByFile.set(1001, payload([0.5, 0.25, 0.75]));
+        client.mlErrorByFile.set(
+            1002,
+            new Error("secretstream chunk authentication failed"),
+        );
+        const events: RefreshEvent[] = [];
+
+        const lib = await Library.open({
+            client,
+            cacheDirectory,
+            refreshIntervalSeconds: Infinity,
+            onProgress: (event) => events.push(event),
+        });
+        try {
+            await vi.waitFor(
+                () => expect(lib.status().lastMLFetchAt).toBeGreaterThan(0),
+                { timeout: 2000, interval: 5 },
+            );
+            const reasons = [
+                {
+                    fileID: 1002,
+                    error: "secretstream chunk authentication failed",
+                },
+            ];
+            expect(lib.status().mlDataErrors).toEqual(reasons);
+            expect(lib.status().mlStored).toBe(1);
+            expect(events.filter((e) => e.operation === "fetchMLData")).toEqual(
+                [
+                    { operation: "fetchMLData", status: "started" },
+                    {
+                        operation: "fetchMLData",
+                        status: "done",
+                        fetched: 1,
+                        mlDataErrors: reasons,
+                    },
+                ],
+            );
+
+            // Nothing was cached for 1002, so the pass a refresh starts asks
+            // for it again, and once its ML data decodes the reason is gone.
+            client.mlErrorByFile.clear();
+            client.mlByFile.set(1002, payload([1, -2, 0.5]));
+            await lib.fresh();
+            await vi.waitFor(
+                () => {
+                    expect(lib.status().mlStored).toBe(2);
+                    expect(lib.status().mlDataErrors).toEqual([]);
+                },
+                { timeout: 2000, interval: 5 },
+            );
+            expect(client.mlFetchCalls.at(-1)).toEqual([1002]);
+        } finally {
+            await lib.close();
+        }
+    });
+
+    it("still lists the last pass's reasons while the next pass runs", async () => {
+        const client = new MLMockClient();
+        client.collectionsQueue.push({
+            collections: [collection(1, 100)],
+            deleted: [],
+            cursor: 100,
+        });
+        client.filesFor(1, {
+            files: [file(1002, 1, 95)],
+            deleted: [],
+            cursor: 95,
+        });
+        client.mlErrorByFile.set(
+            1002,
+            new Error("secretstream chunk authentication failed"),
+        );
+        const reasons = [
+            { fileID: 1002, error: "secretstream chunk authentication failed" },
+        ];
+        let release!: () => void;
+        const held = new Promise<void>((r) => (release = r));
+
+        const lib = await Library.open({
+            client,
+            cacheDirectory,
+            refreshIntervalSeconds: Infinity,
+        });
+        try {
+            await vi.waitFor(
+                () => expect(lib.status().lastMLFetchAt).toBeGreaterThan(0),
+                { timeout: 2000, interval: 5 },
+            );
+            expect(lib.status().mlDataErrors).toEqual(reasons);
+
+            // Hold the next ML data fetch open until the test releases it.
+            let fetchStarted!: () => void;
+            const started = new Promise<void>((r) => (fetchStarted = r));
+            const realFetch = client.fetchMLData.bind(client);
+            client.fetchMLData = async (args) => {
+                fetchStarted();
+                await held;
+                return realFetch(args);
+            };
+            client.mlErrorByFile.clear();
+            client.mlByFile.set(1002, payload([1, -2, 0.5]));
+
+            // The pass a refresh starts asks for 1002 again. While it is
+            // held, the reason from the pass before is still listed; once it
+            // ends, its own empty list takes the place of that one.
+            await lib.fresh();
+            await started;
+            expect(lib.status().mlDataErrors).toEqual(reasons);
+            release();
+            await vi.waitFor(
+                () => expect(lib.status().mlDataErrors).toEqual([]),
+                { timeout: 2000, interval: 5 },
+            );
+        } finally {
+            release();
+            await lib.close();
+        }
+    });
+
     it("close() resolves only after a running ML data fetch has stored its payloads", async () => {
         const client = new MLMockClient();
         client.collectionsQueue.push({

test/mldata-fetch/mldata-fetch.test.ts

@@ -0,0 +1,54 @@
+// `fetchMLDataBatch`, against a server that sends one file's ML data.
+//
+// ML data that decrypts and gunzips but is not a JSON object is reported with
+// the reason, as ML data that does not parse is, and is never returned as a
+// payload: the library would fail to store it.
+
+import { gzipSync } from "node:zlib";
+import sodium from "libsodium-wrappers-sumo";
+import { beforeAll, describe, expect, it } from "vitest";
+
+import { ApiClient } from "../../src/api/client.js";
+import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
+import { fetchMLDataBatch } from "../../src/mldata-fetch.js";
+
+beforeAll(async () => {
+    await init();
+    await sodium.ready;
+});
+
+// Fetch the ML data of file 1 from a server that sends `json`, gzipped and
+// encrypted with the file's key.
+const fetchServing = async (json: string) => {
+    const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+    const { header, ciphertext } = encryptBlob(
+        new Uint8Array(gzipSync(json)),
+        key,
+    );
+    const data = [
+        {
+            fileID: 1,
+            encryptedData: toBase64(ciphertext),
+            decryptionHeader: toBase64(header),
+        },
+    ];
+    const api = new ApiClient({
+        fetch: async () =>
+            new Response(JSON.stringify({ data }), {
+                headers: { "content-type": "application/json" },
+            }),
+    });
+    return fetchMLDataBatch(api, [1], new Map([[1, key]]));
+};
+
+describe("fetchMLDataBatch", () => {
+    it.each(["null", "5", "[]"])(
+        "gives a reason for ML data that is %s",
+        async (json) => {
+            const { payloads, errors } = await fetchServing(json);
+
+            expect(payloads.size).toBe(0);
+            expect(errors.get(1)).toEqual(new Error("not a JSON object"));
+        },
+    );
+});

test/packaging/build-bin.test.ts

@@ -0,0 +1,241 @@
+// `script/build-bin` compiles `bin/quak`, the single binary that the README's
+// cron setup runs. bun compiles it from a copy of its own executable, so the
+// binary loads the same libraries as the bun that built it. nix's bun loads
+// libraries from the nix store, which garbage collection can delete: on Linux
+// its C library, on macOS its ICU. The script therefore downloads bun's own
+// release build, which loads only the system's libraries.
+//
+// What the script promises, and why each test matters:
+//
+// - Each system, machine and C library that bun publishes a build for gets its
+//   own archive: Linux on x86-64 or arm64, with glibc or with musl, as on
+//   Alpine, and macOS on Intel or Apple silicon. A bun built for another
+//   system, machine or C library does not start, and `make install` fails.
+// - The archive's SHA-256 must match the one in the script before anything in
+//   the archive runs. Without that check, a tampered download would build the
+//   binary that cron runs.
+// - Any other system or machine stops with an error, and never falls back to
+//   nix's bun. Building with nix's bun would bring back the dependency on the
+//   nix store.
+//
+// Each test copies the script into a fresh directory, which the script then
+// treats as the checkout, and runs it there. Stubs first on PATH stand in for
+// `uname`, `ldd`, `curl`, `unzip`, `nix-shell` and `shasum`, so no test
+// downloads anything or runs a real bun.
+import { afterEach, beforeEach, describe, expect, it } from "vitest";
+import { execFileSync, spawnSync } from "node:child_process";
+import {
+    chmodSync,
+    copyFileSync,
+    existsSync,
+    mkdirSync,
+    mkdtempSync,
+    readFileSync,
+    rmSync,
+    symlinkSync,
+    writeFileSync,
+} from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import { fileURLToPath } from "node:url";
+
+const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
+
+// The stubs, keyed by the command each replaces. `uname` and `ldd` describe the
+// system that runBuildBin passes in the environment. `curl`, `unzip`,
+// `nix-shell` and `shasum` write the arguments they were given, one per line,
+// to a file named after the command in $STUB_LOG_DIR, which also shows that
+// they ran.
+const stubs: Record<string, string> = {
+    uname: `
+        case "$1" in
+            -s) echo "$STUB_UNAME_S" ;;
+            -m) echo "$STUB_UNAME_M" ;;
+        esac`,
+    // The script looks only at the start of the line: "musl libc" or not.
+    ldd: `
+        if [ "$STUB_LIBC" = musl ]; then
+            echo "musl libc (x86_64)"
+        else
+            echo "ldd (GNU libc) 2.36"
+        fi`,
+    // Writes a file that is not the archive to the path after -o.
+    curl: `
+        printf '%s\\n' "$@" > "$STUB_LOG_DIR/curl"
+        while [ $# -gt 1 ]; do
+            if [ "$1" = -o ]; then echo "not the archive" > "$2"; fi
+            shift
+        done`,
+    // Unpacks a stub bun where the real archive puts bun: in a directory named
+    // after the archive, inside the directory after -d. That bun writes
+    // bin/quak, as a real compile would.
+    unzip: `
+        printf '%s\\n' "$@" > "$STUB_LOG_DIR/unzip"
+        for arg in "$@"; do
+            case "$arg" in *.zip) archive="$arg" ;; esac
+            if [ "$previous" = -d ]; then directory="$arg"; fi
+            previous="$arg"
+        done
+        bun="$directory/$(basename "$archive" .zip)/bun"
+        mkdir -p "$(dirname "$bun")"
+        echo '#!/bin/sh' > "$bun"
+        echo 'mkdir -p bin && echo binary > bin/quak' >> "$bun"
+        chmod +x "$bun"`,
+    "nix-shell": `
+        printf '%s\\n' "$@" > "$STUB_LOG_DIR/nix-shell"`,
+    // The script checks the hash with `shasum` where there is no `sha256sum`.
+    // The stub fails, as `shasum` does when the hash does not match.
+    shasum: `
+        printf '%s\\n' "$@" > "$STUB_LOG_DIR/shasum"
+        exit 1`,
+};
+
+let checkout = "";
+
+const writeScript = (path: string, body: string): void => {
+    writeFileSync(path, `#!/bin/sh\n${body}\n`);
+    chmodSync(path, 0o755);
+};
+
+// A checkout holding the script, a stub script/build that does nothing, the
+// stubs, and the directory they log to.
+beforeEach(() => {
+    checkout = mkdtempSync(join(tmpdir(), "quak-build-bin-"));
+    mkdirSync(join(checkout, "script"));
+    copyFileSync(
+        join(repoRoot, "script/build-bin"),
+        join(checkout, "script/build-bin"),
+    );
+    chmodSync(join(checkout, "script/build-bin"), 0o755);
+    writeScript(join(checkout, "script/build"), "exit 0");
+    mkdirSync(join(checkout, "stubs"));
+    for (const [command, body] of Object.entries(stubs)) {
+        writeScript(join(checkout, "stubs", command), body);
+    }
+    mkdirSync(join(checkout, "log"));
+});
+
+afterEach(() => {
+    rmSync(checkout, { recursive: true, force: true });
+});
+
+// Runs the script with nothing in its environment but the stubs' settings and a
+// PATH. Unless the caller passes one, the PATH finds the stubs first and then
+// the host's commands.
+const runBuildBin = (
+    system: string,
+    machine: string,
+    {
+        libc = "glibc",
+        path = `${join(checkout, "stubs")}:${process.env.PATH}`,
+    } = {},
+) =>
+    spawnSync(join(checkout, "script/build-bin"), {
+        cwd: checkout,
+        encoding: "utf-8",
+        env: {
+            PATH: path,
+            STUB_UNAME_S: system,
+            STUB_UNAME_M: machine,
+            STUB_LIBC: libc,
+            STUB_LOG_DIR: join(checkout, "log"),
+        },
+    });
+
+// The real commands that the script and the stubs run.
+const hostCommands = [
+    "basename",
+    "chmod",
+    "dirname",
+    "grep",
+    "mkdir",
+    "mktemp",
+    "rm",
+];
+
+// A Mac has no `ldd`, and not every Mac has `sha256sum`. This makes a PATH that
+// finds neither: the stubs other than `ldd`, then links to hostCommands.
+const makeMacPath = (): string => {
+    rmSync(join(checkout, "stubs", "ldd"));
+    mkdirSync(join(checkout, "host-commands"));
+    for (const command of hostCommands) {
+        const target = execFileSync("sh", ["-c", `command -v ${command}`], {
+            encoding: "utf-8",
+        }).trim();
+        symlinkSync(target, join(checkout, "host-commands", command));
+    }
+    return `${join(checkout, "stubs")}:${join(checkout, "host-commands")}`;
+};
+
+const ran = (command: string): boolean =>
+    existsSync(join(checkout, "log", command));
+
+const argumentsOf = (command: string): string[] =>
+    readFileSync(join(checkout, "log", command), "utf-8")
+        .trimEnd()
+        .split("\n");
+
+describe("script/build-bin", () => {
+    // The stub curl writes a file that is not the archive, so the hash check
+    // must stop the script before unzip runs. A script without the check would
+    // run unzip, and the stub bun it unpacks would write bin/quak. On Linux the
+    // check must use `sha256sum`, since a stock Alpine has no `shasum`.
+    it.each([
+        ["Linux", "x86_64", "glibc", "bun-linux-x64-baseline.zip"],
+        ["Linux", "x86_64", "musl", "bun-linux-x64-musl-baseline.zip"],
+        ["Linux", "aarch64", "glibc", "bun-linux-aarch64.zip"],
+        ["Linux", "aarch64", "musl", "bun-linux-aarch64-musl.zip"],
+    ])(
+        "on %s %s with %s, downloads %s and stops when its hash does not match",
+        (system, machine, libc, archive) => {
+            const result = runBuildBin(system, machine, { libc });
+            expect(argumentsOf("curl")).toContain(
+                "https://github.com/oven-sh/bun/releases/download/bun-v1.3.13/" +
+                    archive,
+            );
+            expect(ran("shasum")).toBe(false);
+            expect(result.status).not.toBe(0);
+            expect(ran("unzip")).toBe(false);
+            expect(ran("nix-shell")).toBe(false);
+            expect(existsSync(join(checkout, "bin/quak"))).toBe(false);
+        },
+    );
+
+    // On the PATH from makeMacPath the script finds no `sha256sum` and checks
+    // the hash with the stub `shasum`. That stub fails, so the script must stop
+    // before unzip runs.
+    // On Apple silicon `uname -m` says arm64, where bun's archive says aarch64.
+    it.each([
+        ["Darwin", "x86_64", "bun-darwin-x64-baseline.zip"],
+        ["Darwin", "arm64", "bun-darwin-aarch64.zip"],
+    ])(
+        "on %s %s, downloads %s and stops when shasum rejects its hash",
+        (system, machine, archive) => {
+            const result = runBuildBin(system, machine, {
+                path: makeMacPath(),
+            });
+            expect(argumentsOf("curl")).toContain(
+                "https://github.com/oven-sh/bun/releases/download/bun-v1.3.13/" +
+                    archive,
+            );
+            expect(argumentsOf("shasum")).toEqual(["-a", "256", "-c", "-"]);
+            expect(result.status).not.toBe(0);
+            expect(ran("unzip")).toBe(false);
+            expect(ran("nix-shell")).toBe(false);
+            expect(existsSync(join(checkout, "bin/quak"))).toBe(false);
+        },
+    );
+
+    it.each([
+        ["Linux", "riscv64"],
+        ["FreeBSD", "amd64"],
+    ])(
+        "stops on %s %s, which bun publishes no build for",
+        (system, machine) => {
+            const result = runBuildBin(system, machine);
+            expect(result.status).not.toBe(0);
+            expect(ran("curl")).toBe(false);
+            expect(ran("nix-shell")).toBe(false);
+        },
+    );
+});

test/retry/retry.test.ts

@@ -77,6 +77,14 @@ const recordingSleep = (): {
 const errnoError = (code: string, message = code): Error =>
     Object.assign(new Error(message), { code });
 
+/**
+ * An error shaped like what bun's `fetch` rejects with for a failure below
+ * HTTP: a plain `Error` with bun's own message, a `code`, an `errno` of 0, and
+ * no `cause`.
+ */
+const bunError = (code: string, message: string): Error =>
+    Object.assign(new Error(message), { code, errno: 0 });
+
 // ---------------------------------------------------------------------------
 // Classification
 // ---------------------------------------------------------------------------
@@ -244,6 +252,82 @@ describe("isRetryable: transport failures", () => {
     });
 });
 
+describe("isRetryable and isSafeToReplay: bun's fetch errors", () => {
+    // The binary `make install` builds runs quak under bun, whose `fetch`
+    // does not reject as Node's does. Each shape here was found by running
+    // that binary, built with bun 1.3.13, against listeners on 127.0.0.1 and
+    // a name under `.invalid`.
+
+    it("retries a refused connection, and does not replay it", () => {
+        // Bun gives the same error when the TLS handshake fails, and when a
+        // new connection it opened to resend a request is refused. That
+        // request may already have reached the server on a connection bun
+        // kept open from an earlier request.
+        const refused = bunError(
+            "ConnectionRefused",
+            "Unable to connect. Is the computer able to access the url?",
+        );
+        expect(isRetryable(refused)).toBe(true);
+        expect(isSafeToReplay(refused)).toBe(false);
+    });
+
+    it("retries a name that does not resolve, with either code bun gives, and does not replay it", () => {
+        // For the same name bun gave `ConnectionRefused` on some requests and
+        // `FailedToOpenSocket` on others. Either can also come from a new
+        // connection bun opened to resend a request, as above.
+        for (const unresolved of [
+            bunError(
+                "ConnectionRefused",
+                "Unable to connect. Is the computer able to access the url?",
+            ),
+            bunError(
+                "FailedToOpenSocket",
+                "Was there a typo in the url or port?",
+            ),
+        ]) {
+            expect(isRetryable(unresolved)).toBe(true);
+            expect(isSafeToReplay(unresolved)).toBe(false);
+        }
+    });
+
+    it("retries a connection closed before the response ended, and does not replay it", () => {
+        // Bun gives Node's errno, `ECONNRESET`, for a reset and for a close,
+        // before the response or part-way through the body. The request may
+        // already have reached the server.
+        const reset = bunError(
+            "ECONNRESET",
+            "The socket connection was closed unexpectedly. For more information, pass `verbose: true` in the second argument to fetch()",
+        );
+        expect(isRetryable(reset)).toBe(true);
+        expect(isSafeToReplay(reset)).toBe(false);
+    });
+
+    it("retries a request that ran out of time, and does not replay it", () => {
+        // What bun's fetch rejects with when `AbortSignal.timeout()`, which
+        // bounds each JSON request, fires.
+        const timedOut = new DOMException(
+            "The operation timed out.",
+            "TimeoutError",
+        );
+        expect(isRetryable(timedOut)).toBe(true);
+        expect(isSafeToReplay(timedOut)).toBe(false);
+    });
+
+    it("does not retry a certificate bun rejects", () => {
+        // A certificate that fails verification fails the same way however
+        // often it is asked. Under Node it is retried only because every
+        // `TypeError` is.
+        expect(
+            isRetryable(
+                bunError(
+                    "DEPTH_ZERO_SELF_SIGNED_CERT",
+                    "self signed certificate",
+                ),
+            ),
+        ).toBe(false);
+    });
+});
+
 describe("isRetryable: stream truncation versus corruption", () => {
     it("retries a truncated stream", () => {
         // Truncation is a transfer that stopped early. The bytes that did
@@ -471,6 +555,36 @@ describe("withRetry", () => {
         expect(delays).toHaveLength(2);
     });
 
+    it("tells `onRetry` of each attempt it retries, before the wait", async () => {
+        // A caller that logs retries, as the CLI does with `-v`, learns why
+        // each attempt failed and how long the next one is put off. The last
+        // attempt is not retried, so it is not reported.
+        const { sleep, delays } = recordingSleep();
+        const retries: [string, number, number][] = [];
+        let calls = 0;
+        const failure = withRetry(
+            () => {
+                calls++;
+                return Promise.reject(new ApiError(`attempt ${calls}`, 503));
+            },
+            {
+                attempts: 3,
+                sleep,
+                random: () => 0.5,
+                onRetry: (err, attempt, delayMs) => {
+                    retries.push([(err as Error).message, attempt, delayMs]);
+                },
+            },
+        );
+
+        await expect(failure).rejects.toThrow("attempt 3");
+        expect(retries).toEqual([
+            ["attempt 1", 1, 250],
+            ["attempt 2", 2, 500],
+        ]);
+        expect(delays).toEqual([250, 500]);
+    });
+
     it("does not retry at all when attempts is 1", async () => {
         const { sleep, delays } = recordingSleep();
         let calls = 0;
@@ -624,6 +738,7 @@ describe("retry defaults", () => {
         expect(resolved.maxDelayMs).toBe(DEFAULT_RETRY_OPTIONS.maxDelayMs);
         expect(typeof resolved.sleep).toBe("function");
         expect(typeof resolved.random).toBe("function");
+        expect(typeof resolved.onRetry).toBe("function");
     });
 
     it("resolves to the defaults when given nothing", () => {

test/thumbnails/thumbnails.test.ts

@@ -6,10 +6,12 @@
  * working thumbnails, others return 404 or empty bodies. The tests
  * verify that the detection and repair logic handles each case correctly.
  *
- * As of issue #52 both helpers take an open `Library` for enumeration and the
+ * `listMissingThumbnails` takes an open `Library` to enumerate the files. Both
- * `Client` for the API operations that stay unchanged (the thumbnail existence
+ * helpers take the `Client` for the API operations: the thumbnail existence
- * check, and the encrypt-and-upload path). `fixMissingThumbnails` reads each
+ * check, and the encrypt-and-upload path. `fixMissingThumbnails` takes the
- * original through the library's content cache (`photo.original()`).
+ * record of each file, which these tests look up with `cachedFilesByID`. It
+ * reads each original through the library's content cache, by
+ * `lib.originalOf(file)`.
  *
  * `fixMissingThumbnails` is the most complex function in quak: it
  * downloads the original file, generates a JPEG thumbnail with jpeg-js,
@@ -41,6 +43,7 @@ import {
 import { SRP, SrpServer } from "fast-srp-hap";
 import { Client } from "../../src/client.js";
 import { Library } from "../../src/library/index.js";
+import { cachedFilesByID } from "../../src/cli-read.js";
 import {
     listMissingThumbnails,
     fixMissingThumbnails,
@@ -54,8 +57,11 @@ import type { RetryOptions } from "../../src/retry.js";
 
 const TEST_EMAIL = "thumb@example.com";
 const TEST_PASSWORD = "thumbpass";
-const TEST_OPS = 2;
+// The password is stretched at libsodium's lowest Argon2id cost. Each test
-const TEST_MEM = 64 * 1024 * 1024;
+// builds an account and logs in, and at a realistic cost that takes seconds on
+// a loaded host.
+const TEST_OPS = 1;
+const TEST_MEM = 8192;
 const TEST_TIME = 1700000000000000;
 
 interface ThumbMockState {
@@ -267,6 +273,7 @@ const buildThumbMock = async (opts?: {
 
 const buildThumbFetch = (m: ThumbMockState) => {
     let srpServer: SrpServer;
+    let signedMD5: string | undefined;
 
     return (async (
         input: RequestInfo | URL,
@@ -359,8 +366,9 @@ const buildThumbFetch = (m: ThumbMockState) => {
             return new Response("not found", { status: 404 });
         }
 
-        // Upload URL minting
+        // Upload URL minting. The real server signs the MD5 into the URL.
         if (path === "/files/upload-url") {
+            signedMD5 = JSON.parse(init?.body as string).contentMD5;
             return json({
                 objectKey: `42/thumb-${Date.now()}`,
                 url: "https://s3.mock.test/presigned-put",
@@ -369,6 +377,11 @@ const buildThumbFetch = (m: ThumbMockState) => {
 
         // Presigned PUT (S3 upload)
         if (url.startsWith("https://s3.mock.test/")) {
+            // Storage refuses a PUT that lacks the signed `Content-MD5`.
+            const headers = new Headers(init?.headers);
+            if (headers.get("Content-MD5") !== signedMD5) {
+                return new Response("AccessDenied", { status: 400 });
+            }
             const body = init?.body;
             // Store the uploaded bytes for later inspection
             if (body instanceof Uint8Array) {
@@ -544,11 +557,22 @@ describe("listMissingThumbnails", () => {
         const client = await login(counted.fetch, { ...noWait });
         const lib = await openLib(client);
 
-        const missing = await listMissingThumbnails(lib, client);
+        const errors: string[] = [];
+        const missing = await listMissingThumbnails(
+            lib,
+            client,
+            (msg, level) => {
+                if (level === "error") errors.push(msg);
+            },
+        );
         lib.close();
 
-        // Only the genuinely empty thumbnail is reported.
+        // Only the genuinely empty thumbnail is reported. The file that
+        // could not be checked is reported as an error.
         expect(missing.map((m) => m.fileID)).toEqual([101]);
+        expect(errors).toEqual([
+            expect.stringMatching(/^\[.*\] Could not check file-102\.jpg: /),
+        ]);
         // And the 500 was retried rather than accepted as an answer: four
         // attempts is the library default.
         expect(counted.matched()).toBe(4);
@@ -645,7 +669,12 @@ describe("fixMissingThumbnails", () => {
         const client = await login(buildThumbFetch(fixMock));
         const lib = await openLib(client);
 
-        const results = await fixMissingThumbnails(lib, client, [101]);
+        const results = await fixMissingThumbnails(
+            lib,
+            client,
+            [101],
+            cachedFilesByID(lib, [101]),
+        );
         lib.close();
 
         expect(results.length).toBe(1);
@@ -682,7 +711,12 @@ describe("fixMissingThumbnails", () => {
         const client = await login(buildThumbFetch(fixMock));
         const lib = await openLib(client);
 
-        const results = await fixMissingThumbnails(lib, client, [999]);
+        const results = await fixMissingThumbnails(
+            lib,
+            client,
+            [999],
+            cachedFilesByID(lib, [999]),
+        );
         lib.close();
 
         expect(results.length).toBe(1);
@@ -699,7 +733,16 @@ describe("fixMissingThumbnails", () => {
         const client = await login(buildThumbFetch(fixMock));
         const lib = await openLib(client);
 
-        const results = await fixMissingThumbnails(lib, client, [101, 102]);
+        const errors: string[] = [];
+        const results = await fixMissingThumbnails(
+            lib,
+            client,
+            [101, 102],
+            cachedFilesByID(lib, [101, 102]),
+            (msg, level) => {
+                if (level === "error") errors.push(msg);
+            },
+        );
         lib.close();
 
         expect(results.length).toBe(2);
@@ -707,6 +750,10 @@ describe("fixMissingThumbnails", () => {
         const failure = results.find((r) => r.fileID === 102)!;
         expect(success.status).toBe("fixed");
         expect(failure.status).toBe("failed");
+        // The failed file, and only it, is reported as an error.
+        expect(errors).toEqual([
+            expect.stringMatching(/^\[.*\] FAILED file-102\.jpg: /),
+        ]);
     });
 
     it("skips a non-JPEG image and a video as unsupported, not failed (issue #17)", async () => {
@@ -722,6 +769,7 @@ describe("fixMissingThumbnails", () => {
             lib,
             client,
             [101, 103, 104],
+            cachedFilesByID(lib, [101, 103, 104]),
         );
         lib.close();
 
@@ -756,7 +804,12 @@ describe("fixMissingThumbnails", () => {
         const client = await login(counted.fetch);
         const lib = await openLib(client);
 
-        const results = await fixMissingThumbnails(lib, client, [101]);
+        const results = await fixMissingThumbnails(
+            lib,
+            client,
+            [101],
+            cachedFilesByID(lib, [101]),
+        );
         lib.close();
 
         expect(results[0]!.status).toBe("skipped");
@@ -777,7 +830,12 @@ describe("fixMissingThumbnails", () => {
         const client = await login(counted.fetch);
         const lib = await openLib(client);
 
-        const results = await fixMissingThumbnails(lib, client, [101]);
+        const results = await fixMissingThumbnails(
+            lib,
+            client,
+            [101],
+            cachedFilesByID(lib, [101]),
+        );
         lib.close();
 
         expect(results[0]!.status).toBe("skipped");
@@ -818,7 +876,12 @@ describe("fixMissingThumbnails", () => {
         const client = await login(buildThumbFetch(fixMock));
         const lib = await openLib(client);
 
-        const results = await fixMissingThumbnails(lib, client, [101]);
+        const results = await fixMissingThumbnails(
+            lib,
+            client,
+            [101],
+            cachedFilesByID(lib, [101]),
+        );
         lib.close();
 
         expect(results[0]!.status).toBe("fixed");

vitest.config.ts

@@ -3,8 +3,15 @@ import { configDefaults, defineConfig } from "vitest/config";
 // vitest does not read .gitignore when looking for tests. A checkout nested
 // under .claude/ has its own test/ tree, and without this exclude the suite
 // runs once per nested checkout and still reports success.
+//
+// vitest starts one worker per CPU core by default, and a shared host has
+// dozens. Four keep the suite well inside the Dockerfile's time limit. vitest
+// 2 rejects a maxWorkers below its default minWorkers, so minWorkers is set
+// too.
 export default defineConfig({
     test: {
         exclude: [...configDefaults.exclude, ".claude/**"],
+        minWorkers: 1,
+        maxWorkers: 4,
     },
 });