runBackup rejects a contentConcurrency that is not a positive integer

A library object whose contentConcurrency was missing, 0 or negative
started no worker, so the backup stored no original, dropped the earlier
entries of failures.json and reported no error. runBackup now throws a
RangeError for any limit that is not a positive integer, as BoundedPool
does, before it takes the lock.

Model: opus-5-5

README.md

@@ -127,19 +127,21 @@ matters.
 
 ### Exit codes
 
-| Code | Meaning                                                                                                                                                     |
-| ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `0`  | The backup is complete: every original is at its save path, and no file failed.                                                                             |
-| `1`  | The run finished with files in `failures.json`, or it stopped on another error, printed as one line `quak: <message>`. The next run tries again.            |
-| `2`  | Another backup of the same directory is running. This run sent no request and changed nothing.                                                              |
-| `3`  | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. |
+| Code | Meaning                                                                                                                                                                           |
+| ---- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `0`  | The backup is complete: every original is at its save path, and no file failed.                                                                                                   |
+| `1`  | The run finished with files in `failures.json` or an album it could not decrypt, or it stopped on another error, printed as one line `quak: <message>`. The next run tries again. |
+| `2`  | Another backup of the same directory is running. This run sent no request and changed nothing.                                                                                    |
+| `3`  | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as.                       |
 
 After a `1`, the next run fetches each missing original again, fetches the ML
-data that is not cached, and rebuilds the album links. An original that a
-`--verify` run could not read, or put back with bytes that still do not match,
-stays at its save path. Only a later `--verify` run checks it again: a run
-without `--verify` leaves it as it is and takes the file out of `failures.json`,
-so that run can exit 0.
+data that is not cached, and rebuilds the album links. An album quak could not
+decrypt, or a file in `failures.json` because quak could not decrypt it, keeps
+the exit code at `1` until the server sends a copy that decrypts, or deletes it
+(see "Backup layout"). An original that a `--verify` run could not read, or put
+back with bytes that still do not match, stays at its save path. Only a later
+`--verify` run checks it again: a run without `--verify` leaves it as it is and
+takes the file out of `failures.json`, so that run can exit 0.
 
 ## Examples
 
@@ -623,8 +625,10 @@ library. The read commands — `collections`, `files`, `get`, `get-thumb`,
 answer, so they report current account state rather than whatever the cache last
 held. If that round-trip fails, the command prints the error on one line and
 exits 1, or 3 when the server no longer accepts the saved session (see "Session
-handling"). `--cache-dir` overrides where the cache lives; without it each
-account gets its own directory under the per-user cache path.
+handling"). An album or file quak cannot decrypt does not fail it; the command
+answers from what did decrypt (see "Default reads vs. fresh reads").
+`--cache-dir` overrides where the cache lives; without it each account gets its
+own directory under the per-user cache path.
 
 An ID given to `files --collection`, `get`, `get-thumb` or
 `helper fix-missing-thumbnails --file` is written in decimal digits, no larger
@@ -741,6 +745,19 @@ succeeds, or once it is no longer in the library or in the backup's scope. The
 library's `lib.backup({ includeThumbnails: true })` also writes
 `thumbnails/<fileID>.jpg` beside `collections/`; `quak backup` does not.
 
+A file in the backup's scope that the library could not decrypt (see "Default
+reads vs. fresh reads") goes into `failures.json` with an error that starts
+`could not be decrypted:` and gives the reason. If a copy of it in another album
+decrypted, it does not: its original is stored from that copy, and the album
+where it does not decrypt neither links nor lists it. An album the library could
+not decrypt has no name quak can read and no files it can list. A run without
+`--json` prints one line for it,
+`FAILED album <id>: could not be decrypted: <reason>`, and `--json` and the
+`BackupResult` list its ID in `undecryptableAlbums`. Such a file in
+`failures.json`, or such an album, keeps the exit code at 1, run after run,
+until the server sends a copy that decrypts or deletes it; the next run then
+drops it.
+
 `backup.lock` keeps two backups of the same directory from running at once, such
 as a cron run that starts while the previous one is still going. A backup
 creates `<dir>` if it is missing and takes the lock before its refresh, and
@@ -777,9 +794,12 @@ both, each named after the title with that file's extension (`IMG_0001.heic` and
 
 Each run removes the symlinks into the date folders that no longer belong in
 their collection's directory, and the directories (and JSON) of collections that
-were deleted or renamed. Nothing else in `collections/` is touched: a file or a
-symlink you put there stays, and a directory that still holds one after its
-symlinks are removed stays too, with its JSON.
+were deleted or renamed. An album quak can no longer decrypt loses its directory
+and JSON the same way, and a file whose copy in an album quak can no longer
+decrypt loses its symlink there and its entry in that album's JSON; the
+originals stay at their save paths. Nothing else in `collections/` is touched: a
+file or a symlink you put there stays, and a directory that still holds one
+after its symlinks are removed stays too, with its JSON.
 
 Each file is downloaded exactly once regardless of how many collections it
 appears in, and written once: straight to its save path, with no copy left in
@@ -915,6 +935,16 @@ rejects the caller (default reads stay silent and keep serving the last good
 copy). The CLI's read commands use fresh reads (issue
 https://git.eeqj.de/sneak/quak/issues/75).
 
+An album or file in the server's answer that quak cannot decrypt, or whose
+metadata is not JSON, does not fail the refresh. The refresh leaves it out, an
+album with all its files, and drops any older copy the cache held, so reads
+answer from what did decrypt. The library keeps a list of what it left out, with
+the reason, in `metadata.json`: the next refresh asks the server only for what
+changed, so it would not see the record again. A record stays on that list until
+a later refresh gets a copy that decrypts or learns that the server deleted it.
+`lib.backup()` reports each such album, and each such file in its scope that has
+no copy that decrypted (see "Backup layout").
+
 ### Read surface
 
 - `lib.albums.list()` → `Album[]`, newest-updated first.
@@ -1053,6 +1083,11 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
 Each pool orders on-demand work ahead of background/precache work and dedups
 in-flight fetches by key, and an idle pool never lends its slots to a busy one.
 
+`lib.backup()`, and so `quak backup`, works on up to the content pool's limit of
+originals at once, 5 unless the library was opened with other `pools`. The limit
+covers everything it does for one original, hashing it for `verify` included, so
+however large the account is, no more originals than that are open at once.
+
 ### On-disk cache layout
 
 Under `cacheDirectory`:

TODO.md

@@ -25,6 +25,27 @@ declares one.
 
 # Completed Steps
 
+- 2026-10-14: `lib.backup()`, and so `quak backup`, works on up to the content
+  pool's limit of originals at once, 5 by default, instead of one at a time
+  (issue 182). That many workers take files from one shared list, each doing all
+  the work for one original, its `--verify` hashing included, so no more
+  originals than that are open at once. `BackupLibrary` says the limit as
+  `contentConcurrency`. A test holds each download at the fake Ente server to
+  count those in flight.
+
+- 2026-10-14: An album or file quak cannot decrypt, or whose metadata is not
+  JSON, no longer fails the refresh, so one bad record no longer stops
+  `quak backup` from storing anything (issue 181). The refresh leaves it out, an
+  album with its files, and the library keeps it with the reason in
+  `metadata.json` until the server sends a copy that decrypts or deletes it.
+  `quak backup` stores every other file, records each such file in
+  `failures.json` with an error starting `could not be decrypted:`, names each
+  such album's ID in a printed line, or with `--json` in `undecryptableAlbums`,
+  and exits 1 while any remain. A file whose copy in another album decrypts is
+  not recorded: its original is stored from that copy, and the album where it
+  does not decrypt neither links nor lists it. The read commands answer from
+  what decrypted.
+
 - 2026-10-14: `quak backup` no longer asks the server for changes every 3
   seconds while it runs (issue 183). It opens its library with
   `refreshIntervalSeconds: Infinity`, which the library now takes as no

src/backup.ts

@@ -5,9 +5,9 @@
 // lock. It waits for a completed refresh of the library (a failed one fails the
 // backup before any file is touched), then, for every file in scope, puts its
 // original at its save path under `downloadDirectory`, as `Photo.download()`
-// does, waits for an ML data fetch, and rebuilds the derived views (per-file
-// sidecars, per-collection symlink trees, per-collection JSON) from the model.
-// The on-disk layout:
+// does, up to the content pool's limit of them at once, waits for an ML data
+// fetch, and rebuilds the derived views (per-file sidecars, per-collection
+// symlink trees, per-collection JSON) from the model. The on-disk layout:
 //
 //     <downloadDirectory>/
 //         YYYY/YYYY-MM/YYYY-MM-DD/
@@ -47,6 +47,9 @@
 // download, a failed symlink, or ML data missing because the ML data fetch
 // failed is caught, recorded in `failures.json` with a classification, a
 // running attempt count, and the last-tried time, and the run continues.
+// A file the library could not decrypt is recorded there too, unless a copy of
+// it in another album decrypted; an album it could not decrypt is logged by ID
+// and listed in `result.undecryptableAlbums`, as its name is not known.
 // `result.failed` — and thus the CLI's exit code — stays non-zero while any
 // failure remains unresolved and clears once every one succeeds. Each run
 // reconciles the ledger against the files it attempted, so an entry for a file
@@ -86,6 +89,7 @@ import {
     storedAtSavePath,
 } from "./library/content.js";
 import { representative } from "./library/records.js";
+import type { Undecryptable } from "./library/store.js";
 import { extractImageMetadata } from "./metadata-backup.js";
 import type { MLData } from "./mldata-fetch.js";
 import type { Collection, EnteFile, FileMetadata } from "./model/types.js";
@@ -141,6 +145,9 @@ export interface BackupResult {
     failed: number;
     // This run's per-file errors, in encounter order.
     errors: BackupError[];
+    // The IDs of the albums the library could not decrypt, whose files it
+    // could not list; the CLI exits non-zero while there are any.
+    undecryptableAlbums: number[];
 }
 
 // The slice of the library that backup drives. `Library` implements it; a test
@@ -151,6 +158,9 @@ export interface BackupLibrary {
     refresh(): Promise<void>;
     listCollections(): Collection[];
     listFiles(collectionID: number): EnteFile[];
+    // The albums, and files in albums, that the library could not decrypt,
+    // which `listCollections` and `listFiles` leave out.
+    undecryptable(): Undecryptable[];
     // Get an original's bytes onto disk through the content cache/pools,
     // returning where they landed: `destination` when they were fetched now,
     // otherwise wherever they already were (the cache, or the library's save
@@ -161,6 +171,9 @@ export interface BackupLibrary {
         destination: string,
     ): Promise<{ path: string; videoPath?: string }>;
     thumbnail(fileID: number): Promise<{ path: string }>;
+    // How many originals the backup works on at once: the limit of the
+    // library's content pool.
+    contentConcurrency: number;
     // Wait for an ML data fetch to complete, joining one already running or
     // starting one. Rejects with the reason when it fails; resolves at once
     // when the library cannot fetch ML data.
@@ -574,114 +587,149 @@ const runLockedBackup = async (
     let unchecked = 0;
 
     const recordFailure = (
-        file: EnteFile,
+        fileID: number,
+        title: string,
         collection: string,
         err: unknown,
     ): void => {
         // Count at most one attempt per file per run: a file whose original
         // and thumbnail both fail this run must not double its attempt count
         // or appear twice in errors.
-        if (failedThisRun.has(file.id)) return;
+        if (failedThisRun.has(fileID)) return;
         const error = errorMessage(err);
-        errors.push({
-            fileID: file.id,
-            title: file.metadata.title,
-            collection,
-            error,
-        });
-        const prior = ledger.get(file.id);
-        ledger.set(file.id, {
-            fileID: file.id,
-            title: file.metadata.title,
+        errors.push({ fileID, title, collection, error });
+        const prior = ledger.get(fileID);
+        ledger.set(fileID, {
+            fileID,
+            title,
             classification: classify(err),
             attempts: (prior?.attempts ?? 0) + 1,
             lastTriedAt: now,
             error,
         });
-        failedThisRun.add(file.id);
+        failedThisRun.add(fileID);
     };
 
+    // What the library could not decrypt. An album is logged by its ID, as
+    // its name is not known, so a run limited to some albums logs it too. A
+    // file in an album in scope is backed up from a copy of it in another
+    // album that decrypted, even an album out of scope. With no such copy it
+    // is recorded as failed, with an empty title as its metadata is not known.
+    const scope = new Set(collections.map((c) => c.id));
+    const undecryptableAlbums: number[] = [];
+    for (const record of lib.undecryptable()) {
+        const error = `could not be decrypted: ${record.error}`;
+        if (record.fileID === undefined) {
+            log(`FAILED album ${record.collectionID}: ${error}`);
+            undecryptableAlbums.push(record.collectionID);
+        } else if (scope.has(record.collectionID)) {
+            const copies = memberships.get(record.fileID);
+            if (copies) {
+                distinct.set(record.fileID, representative(copies));
+            } else {
+                recordFailure(
+                    record.fileID,
+                    "",
+                    collectionName.get(record.collectionID) ?? "",
+                    new Error(error),
+                );
+            }
+        }
+    }
+
     // Phase 1: get the bytes. Put each pending original at its save path
     // through the content cache/pools, as `Photo.download()` does, and fetch
     // the optional thumbnails; a present file is left as is. With `verify`, a
     // present original is hashed first, and one that does not match the hash
     // its metadata records is removed and fetched like a missing one. One
     // that cannot be read is recorded as failed and left where it is.
-    if (includeOriginals) {
-        for (const [fileID, file] of distinct) {
-            let stored = storedAtSavePath(downloadDirectory, file);
-            let mismatch = false;
-            if (stored !== undefined && verify) {
-                try {
-                    if (file.metadata.hash === undefined) {
-                        unchecked++;
-                    } else if (
-                        (await storedHash(stored)) === file.metadata.hash
-                    ) {
-                        verified++;
-                    } else {
-                        log(
-                            `MISMATCH original ${file.metadata.title} (${fileID}): its bytes do not match its content hash`,
-                        );
-                        mismatched++;
-                        mismatch = true;
-                        rmSync(stored.path);
-                        if (stored.videoPath !== undefined) {
-                            rmSync(stored.videoPath);
-                        }
-                        stored = undefined;
-                    }
-                } catch (err) {
-                    log(
-                        `FAILED verifying original ${file.metadata.title}: ${errorMessage(err)}`,
-                    );
-                    recordFailure(
-                        file,
-                        collectionName.get(file.collectionID) ?? "",
-                        err,
-                    );
-                    continue;
-                }
-            }
-            if (stored !== undefined) {
-                skipped++;
-                continue;
-            }
+    // `storeOriginal` does this for one original, and records its failure.
+    const storeOriginal = async (file: EnteFile): Promise<void> => {
+        let stored = storedAtSavePath(downloadDirectory, file);
+        let mismatch = false;
+        if (stored !== undefined && verify) {
             try {
-                log(`Fetching original ${file.metadata.title} (${fileID})...`);
-                // A fetched original is written straight to its save path (a
-                // live photo beside it); only one that was already cached
-                // elsewhere is copied.
-                const placed = await placeOriginal(
-                    downloadDirectory,
-                    file,
-                    (dest) => lib.original(fileID, dest),
-                );
-                storedThisRun.add(fileID);
-                // A copy from the cache is not checked as a download is and can
-                // hold the same bad bytes, so what is put back after a
-                // mismatch is hashed too. A bad copy stays where it is and the
-                // file fails.
-                if (
-                    mismatch &&
-                    (await storedHash(placed)) !== file.metadata.hash
-                ) {
-                    throw new Error(
-                        "the original put back does not match its content hash either",
+                if (file.metadata.hash === undefined) {
+                    unchecked++;
+                } else if ((await storedHash(stored)) === file.metadata.hash) {
+                    verified++;
+                } else {
+                    log(
+                        `MISMATCH original ${file.metadata.title} (${file.id}): its bytes do not match its content hash`,
                     );
+                    mismatched++;
+                    mismatch = true;
+                    rmSync(stored.path);
+                    if (stored.videoPath !== undefined) {
+                        rmSync(stored.videoPath);
+                    }
+                    stored = undefined;
                 }
-                downloaded++;
             } catch (err) {
                 log(
-                    `FAILED original ${file.metadata.title}: ${errorMessage(err)}`,
+                    `FAILED verifying original ${file.metadata.title}: ${errorMessage(err)}`,
                 );
                 recordFailure(
-                    file,
+                    file.id,
+                    file.metadata.title,
                     collectionName.get(file.collectionID) ?? "",
                     err,
                 );
+                return;
             }
         }
+        if (stored !== undefined) {
+            skipped++;
+            return;
+        }
+        try {
+            log(`Fetching original ${file.metadata.title} (${file.id})...`);
+            // A fetched original is written straight to its save path (a
+            // live photo beside it); only one that was already cached
+            // elsewhere is copied.
+            const placed = await placeOriginal(
+                downloadDirectory,
+                file,
+                (dest) => lib.original(file.id, dest),
+            );
+            storedThisRun.add(file.id);
+            // A copy from the cache is not checked as a download is and can
+            // hold the same bad bytes, so what is put back after a mismatch
+            // is hashed too. A bad copy stays where it is and the file fails.
+            if (mismatch && (await storedHash(placed)) !== file.metadata.hash) {
+                throw new Error(
+                    "the original put back does not match its content hash either",
+                );
+            }
+            downloaded++;
+        } catch (err) {
+            log(`FAILED original ${file.metadata.title}: ${errorMessage(err)}`);
+            recordFailure(
+                file.id,
+                file.metadata.title,
+                collectionName.get(file.collectionID) ?? "",
+                err,
+            );
+        }
+    };
+
+    // Up to the content pool's limit of originals are worked on at once,
+    // their `verify` hashing included, so however large the account is, no
+    // more than that many are open. Each worker takes the next file from the
+    // iterator they all share until none is left. A worker that throws, as
+    // one does when `onProgress` throws, fails the backup only once every
+    // worker has stopped, so none writes into the backup after it has ended.
+    if (includeOriginals) {
+        const pending = distinct.values();
+        const worker = async (): Promise<void> => {
+            for (const file of pending) await storeOriginal(file);
+        };
+        const workers = await Promise.allSettled(
+            Array.from({ length: lib.contentConcurrency }, worker),
+        );
+        for (const w of workers) {
+            if (w.status === "rejected") throw w.reason;
+        }
     }
 
     if (includeThumbnails) {
@@ -693,7 +741,8 @@ const runLockedBackup = async (
                 await copyAtomic(path, dest);
             } catch (err) {
                 recordFailure(
-                    file,
+                    file.id,
+                    file.metadata.title,
                     collectionName.get(file.collectionID) ?? "",
                     err,
                 );
@@ -733,7 +782,8 @@ const runLockedBackup = async (
             if (mlData === undefined && mlDataError !== undefined) {
                 writeSidecar(path, file, { mlDataError }, image);
                 recordFailure(
-                    file,
+                    file.id,
+                    file.metadata.title,
                     collectionName.get(file.collectionID) ?? "",
                     new Error(`ML data: ${mlDataError}`),
                 );
@@ -806,7 +856,12 @@ const runLockedBackup = async (
                 log(
                     `FAILED symlink ${c.name}/${linkName}: ${errorMessage(err)}`,
                 );
-                recordFailure(link.file, c.name, err);
+                recordFailure(
+                    link.file.id,
+                    link.file.metadata.title,
+                    c.name,
+                    err,
+                );
             }
         }
 
@@ -837,6 +892,7 @@ const runLockedBackup = async (
         unchecked,
         failed: ledger.size,
         errors,
+        undecryptableAlbums,
     };
 };
 
@@ -877,6 +933,14 @@ export const runBackup = async (
                 "open the library with one)",
         );
     }
+    // As `BoundedPool` does. A limit below 1, or one that is not a number,
+    // would back up no original and end with no error.
+    const { contentConcurrency } = lib;
+    if (!Number.isInteger(contentConcurrency) || contentConcurrency < 1) {
+        throw new RangeError(
+            `contentConcurrency must be a positive integer, got ${contentConcurrency}`,
+        );
+    }
     if (opts.lockHeld) {
         return runLockedBackup(lib, opts, downloadDirectory);
     }

src/cli-commands.ts

@@ -470,7 +470,9 @@ export const backupCommand = async (
                 }
             }
 
-            return result.failed > 0 ? 1 : 0;
+            return result.failed > 0 || result.undecryptableAlbums.length > 0
+                ? 1
+                : 0;
         } finally {
             await lib.close();
         }

src/client.ts

@@ -44,20 +44,28 @@ export interface ClientSnapshot {
 
 // The result of a resumable enumeration. Live decrypted records and deleted
 // ids are kept apart on purpose: a tombstone carries no key or metadata to
-// decrypt, so it is a bare id rather than a hollowed-out record. `cursor` is
-// the max `updationTime` seen, to pass back into the next call.
+// decrypt, so it is a bare id rather than a hollowed-out record. A live row
+// that cannot be decrypted, or whose metadata cannot be parsed, is in
+// `undecryptable` with the reason instead, so one bad row does not fail the
+// call; absent means there was none. `cursor` is the max `updationTime` seen,
+// those rows included, to pass back into the next call.
 export interface CollectionsPage {
     collections: Collection[];
     deleted: number[];
+    undecryptable?: { id: number; error: string }[];
     cursor: number;
 }
 
 export interface FilesPage {
     files: EnteFile[];
     deleted: number[];
+    undecryptable?: { id: number; error: string }[];
     cursor: number;
 }
 
+const errorMessage = (err: unknown): string =>
+    err instanceof Error ? err.message : String(err);
+
 export class Client {
     private readonly api: ApiClient;
     private readonly email: string;
@@ -237,11 +245,12 @@ export class Client {
     }
 
     // Enumerate collections changed since `sinceTime`. Live collections are
-    // decrypted; tombstoned ones (isDeleted) are surfaced as bare ids. The
-    // returned cursor is the max `updationTime` seen — including tombstones, so
-    // the next sync resumes past them — falling back to `sinceTime` when the
-    // response is empty. `/collections/v2` returns the whole changed set in one
-    // response, so there is no pagination here.
+    // decrypted; tombstoned ones (isDeleted) are surfaced as bare ids, and
+    // ones that do not decrypt as ids with the reason. The returned cursor is
+    // the max `updationTime` seen — including tombstones, so the next sync
+    // resumes past them — falling back to `sinceTime` when the response is
+    // empty. `/collections/v2` returns the whole changed set in one response,
+    // so there is no pagination here.
     async collectionsSince(args: {
         sinceTime: number;
     }): Promise<CollectionsPage> {
@@ -254,31 +263,40 @@ export class Client {
 
         const collections: Collection[] = [];
         const deleted: number[] = [];
+        const undecryptable: { id: number; error: string }[] = [];
         let cursor = args.sinceTime;
         for (const raw of raws) {
             if (raw.isDeleted) {
                 deleted.push(raw.id);
             } else {
-                collections.push(
-                    decryptCollection(
-                        raw,
-                        {
-                            masterKey: this.masterKey,
-                            publicKey: this.publicKey,
-                            secretKey: this.secretKey,
-                        },
-                        this.userID,
-                    ),
-                );
+                try {
+                    collections.push(
+                        decryptCollection(
+                            raw,
+                            {
+                                masterKey: this.masterKey,
+                                publicKey: this.publicKey,
+                                secretKey: this.secretKey,
+                            },
+                            this.userID,
+                        ),
+                    );
+                } catch (err) {
+                    undecryptable.push({
+                        id: raw.id,
+                        error: errorMessage(err),
+                    });
+                }
             }
             if (raw.updationTime > cursor) cursor = raw.updationTime;
         }
-        return { collections, deleted, cursor };
+        return { collections, deleted, undecryptable, cursor };
     }
 
     // Enumerate a collection's files changed since `sinceTime`, paginating the
     // diff from that cursor. Live rows are decrypted; tombstoned ones are
-    // surfaced as bare ids. Returns the final cursor to resume from.
+    // surfaced as bare ids, and ones that do not decrypt or parse as ids with
+    // the reason. Returns the final cursor to resume from.
     async filesSince(args: {
         collectionID: number;
         collectionKey: Uint8Array;
@@ -288,6 +306,7 @@ export class Client {
         const { collectionID, collectionKey } = args;
         const files: EnteFile[] = [];
         const deleted: number[] = [];
+        const undecryptable: { id: number; error: string }[] = [];
         let cursor = args.sinceTime;
         for (;;) {
             const { diff, hasMore } = await this.api.getJSON<{
@@ -300,7 +319,14 @@ export class Client {
                 if (raw.isDeleted) {
                     deleted.push(raw.id);
                 } else {
-                    files.push(decryptFile(raw, collectionKey));
+                    try {
+                        files.push(decryptFile(raw, collectionKey));
+                    } catch (err) {
+                        undecryptable.push({
+                            id: raw.id,
+                            error: errorMessage(err),
+                        });
+                    }
                 }
                 if (raw.updationTime > pageMax) pageMax = raw.updationTime;
             }
@@ -321,18 +347,19 @@ export class Client {
             }
             cursor = pageMax;
         }
-        return { files, deleted, cursor };
+        return { files, deleted, undecryptable, cursor };
     }
 
-    // Whole-account listing: every live collection, deletions hidden. A thin
-    // wrapper over `collectionsSince` from the beginning of time.
+    // Whole-account listing: every live collection that decrypts, deletions
+    // hidden. A thin wrapper over `collectionsSince` from the beginning of
+    // time.
     async listCollections(): Promise<Collection[]> {
         const { collections } = await this.collectionsSince({ sinceTime: 0 });
         return collections;
     }
 
-    // Every live file in a collection, deletions hidden. A thin wrapper over
-    // `filesSince` from the beginning of time.
+    // Every live file in a collection that decrypts, deletions hidden. A thin
+    // wrapper over `filesSince` from the beginning of time.
     async listFiles(
         collectionID: number,
         collectionKey: Uint8Array,

src/library/index.ts

@@ -601,9 +601,11 @@ export class Library {
                 refresh: () => this.refreshNow(),
                 listCollections: () => this.store.listCollections(),
                 listFiles: (id) => this.store.listFiles(id),
+                undecryptable: () => this.store.listUndecryptable(),
                 original: (fileID, destination) =>
                     cache!.backupOriginal(fileID, destination),
                 thumbnail: (fileID) => cache!.thumbnail(fileID),
+                contentConcurrency: this.pools.content.concurrency,
                 fetchMLData: () => this.fetchMLDataNow(),
                 mlData: (fileID) => this.mldata.forFile({ fileID }),
             },
@@ -703,7 +705,10 @@ export class Library {
 
     // Fetch every change since the stored cursor, then commit. All network
     // reads happen before any store mutation, so a fetch that throws leaves the
-    // store untouched and the previous snapshot intact.
+    // store untouched and the previous snapshot intact. A collection or file
+    // that does not decrypt is left out instead, as if deleted, with the reason
+    // kept in the store: the cursor moves past it, so no later refresh sees it
+    // again until the server's copy changes.
     private async refreshOnce(): Promise<void> {
         const page = await this.client.collectionsSince({
             sinceTime: this.store.collectionsSinceTime,
@@ -740,9 +745,17 @@ export class Library {
                 this.store.deleteCollection(id);
                 changed = true;
             }
+            if (this.store.deleteUndecryptable(id)) changed = true;
+        }
+        // A collection left out takes its files with it.
+        for (const { id, error } of page.undecryptable ?? []) {
+            this.store.deleteCollection(id);
+            this.store.putUndecryptable({ collectionID: id, error });
+            changed = true;
         }
         for (const collection of page.collections) {
             this.store.putCollection(collection);
+            this.store.deleteUndecryptable(collection.id);
             changed = true;
         }
 
@@ -752,9 +765,22 @@ export class Library {
                     this.store.deleteFile(collectionID, id);
                     changed = true;
                 }
+                if (this.store.deleteUndecryptable(collectionID, id)) {
+                    changed = true;
+                }
+            }
+            for (const { id, error } of filePage.undecryptable ?? []) {
+                this.store.deleteFile(collectionID, id);
+                this.store.putUndecryptable({
+                    collectionID,
+                    fileID: id,
+                    error,
+                });
+                changed = true;
             }
             for (const f of filePage.files) {
                 this.store.putFile(f);
+                this.store.deleteUndecryptable(collectionID, f.id);
                 changed = true;
             }
         }

src/library/store.ts

@@ -2,10 +2,11 @@
 //
 // The store keeps one `metadata.json` file holding the account's server
 // state: the user id, a schema version, the cursor for the incremental
-// collections listing, and the decrypted collection and file records. The
-// whole file is read into RAM on load and rewritten as a whole on save; there
-// is no partial update and no lock file. A separate refresh unit populates the
-// store from the server — this module only stores what it is given.
+// collections listing, the decrypted collection and file records, and the
+// collections and files the refresh could not decrypt. The whole file is read
+// into RAM on load and rewritten as a whole on save; there is no partial
+// update and no lock file. A separate refresh unit populates the store from
+// the server — this module only stores what it is given.
 //
 // The file is a cache, so it is never trusted to exist or to be intact: a
 // missing or unreadable file loads as an empty store rather than an error, and
@@ -33,12 +34,22 @@ const FILE_MODE = 0o600;
 type StoredCollection = Omit<Collection, "key"> & { key: string };
 type StoredFile = Omit<EnteFile, "key"> & { key: string };
 
+// A collection, or with `fileID` a file in it, that the last refresh to fetch
+// it could not decrypt or parse, and why. It is kept until a later refresh
+// fetches it again, because a refresh fetches only what changed.
+export interface Undecryptable {
+    collectionID: number;
+    fileID?: number;
+    error: string;
+}
+
 interface StoredMetadata {
     schemaVersion: number;
     userID: number;
     collectionsSinceTime: Microseconds;
     collections: StoredCollection[];
     files: StoredFile[];
+    undecryptable: Undecryptable[];
 }
 
 const encodeKey = (key: Uint8Array): string =>
@@ -67,6 +78,7 @@ export class MetadataStore {
 
     private readonly collections = new Map<number, Collection>();
     private readonly files = new Map<string, EnteFile>();
+    private undecryptable: Undecryptable[] = [];
 
     private constructor(path: string) {
         this.path = path;
@@ -107,6 +119,12 @@ export class MetadataStore {
                 };
                 store.files.set(fileKey(file.collectionID, file.id), file);
             }
+            for (const record of parsed.undecryptable) {
+                if (typeof record.collectionID !== "number") {
+                    throw new Error("damaged record of what did not decrypt");
+                }
+                store.undecryptable.push(record);
+            }
         } catch {
             // Any corruption discards the partial result: a half-read cache is
             // worse than an empty one, since the refresh unit will rebuild it.
@@ -134,6 +152,7 @@ export class MetadataStore {
                 ...f,
                 key: encodeKey(f.key),
             })),
+            undecryptable: this.undecryptable,
         };
 
         const dir = dirname(this.path);
@@ -165,7 +184,8 @@ export class MetadataStore {
         this.collections.set(collection.id, collection);
     }
 
-    // Removing a collection also drops its file memberships: a file record is
+    // Removing a collection also drops its file memberships, and any record
+    // that it or one of its files could not be decrypted: a file record is
     // only meaningful as part of a collection the cache still knows about.
     deleteCollection(id: number): void {
         this.collections.delete(id);
@@ -174,6 +194,9 @@ export class MetadataStore {
                 this.files.delete(key);
             }
         }
+        this.undecryptable = this.undecryptable.filter(
+            (record) => record.collectionID !== id,
+        );
     }
 
     getFile(collectionID: number, fileID: number): EnteFile | undefined {
@@ -205,4 +228,26 @@ export class MetadataStore {
     deleteFile(collectionID: number, fileID: number): void {
         this.files.delete(fileKey(collectionID, fileID));
     }
+
+    listUndecryptable(): Undecryptable[] {
+        return [...this.undecryptable];
+    }
+
+    // Replaces any earlier record of the same collection or file.
+    putUndecryptable(record: Undecryptable): void {
+        this.deleteUndecryptable(record.collectionID, record.fileID);
+        this.undecryptable.push(record);
+    }
+
+    // Forgets the record of the collection, or with `fileID` of that file in
+    // it. Returns whether there was one.
+    deleteUndecryptable(collectionID: number, fileID?: number): boolean {
+        const before = this.undecryptable.length;
+        this.undecryptable = this.undecryptable.filter(
+            (record) =>
+                record.collectionID !== collectionID ||
+                record.fileID !== fileID,
+        );
+        return this.undecryptable.length < before;
+    }
 }

test/cli/backup-concurrency.test.ts

@@ -0,0 +1,530 @@
+/**
+ * How many originals `lib.backup()` works on at once.
+ *
+ * The backup puts up to the content pool's limit of originals at their save
+ * paths at the same time: 5 for a library opened with the default
+ * `RequestPools`, as `quak backup` opens it, or the `contentConcurrency` of the
+ * pools a library was opened with. Everything the backup does for one original
+ * counts against that limit, hashing it for `verify` included, so however large
+ * the account is, the backup has no more than that many originals open at once.
+ *
+ * As in `backup-end-to-end.test.ts`, the real `Client` and `Library` run
+ * against `buildMetaFetch` from `test/ente-server.ts`, a fake of the Ente HTTP
+ * API serving data this file encrypts with libsodium. Here a test can also hold
+ * each original's download until it lets it go, to see how many the backup has
+ * in flight at once.
+ *
+ * The account, user 42, has one album of its own, Holiday, of eight photos,
+ * all taken at noon on 1 March 2024, each with an original of its own.
+ */
+
+import {
+    existsSync,
+    lstatSync,
+    mkdtempSync,
+    readdirSync,
+    readFileSync,
+    readlinkSync,
+    rmSync,
+    writeFileSync,
+} from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+import sodium from "libsodium-wrappers-sumo";
+import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
+
+import type { BackupOptions } from "../../src/backup.js";
+import { Client, type ClientSnapshot } from "../../src/client.js";
+import { init, toBase64 } from "../../src/crypto/index.js";
+import { Library } from "../../src/library/index.js";
+import { RequestPools } from "../../src/library/pools.js";
+import {
+    buildAccount,
+    buildMetaFetch,
+    encryptSecretbox,
+    encryptStreamBlob,
+} from "../ente-server.js";
+import { blake2b } from "../live-photo.js";
+
+const EMAIL = "backup@example.com";
+const PASSWORD = "backup-password";
+// The fake server logs every account in as user 42.
+const USER_ID = 42;
+
+// Noon on 1 March 2024 in the time zone of the machine running the test, in
+// microseconds, as Ente records times.
+const TAKEN = new Date(2024, 2, 1, 12).getTime() * 1000;
+
+// The eight photos' IDs. Photo `id` is titled `photo-<id>.jpg`, and its
+// original is a few bytes naming it.
+const PHOTO_IDS = [101, 102, 103, 104, 105, 106, 107, 108];
+const originalOf = (id: number): Uint8Array =>
+    new TextEncoder().encode(`the photo ${id}`);
+
+// Where a backup to `dir` saves photo `id`'s original.
+const savedAt = (dir: string, id: number): string =>
+    join(dir, "2024/2024-03/2024-03-01", `2024-03-01.${id}.jpg`);
+
+// The fake server, the session it logs the account in with, and the
+// directory every backup below is made in.
+let server: typeof globalThis.fetch;
+let session: ClientSnapshot;
+let root: string;
+
+// The fake server, holding the account encrypted as Ente's clients upload it.
+const buildServer = async () => {
+    const account = await buildAccount(PASSWORD);
+
+    // The album as `/collections/v2` lists it: its key encrypted with the
+    // account's master key, its name with its own key.
+    const albumKey = sodium.crypto_secretbox_keygen();
+    const encryptedAlbumKey = encryptSecretbox(albumKey, account.masterKey);
+    const encryptedName = encryptSecretbox(
+        new TextEncoder().encode("Holiday"),
+        albumKey,
+    );
+    const album = {
+        id: 10,
+        owner: { id: USER_ID },
+        encryptedKey: toBase64(encryptedAlbumKey.ciphertext),
+        keyDecryptionNonce: toBase64(encryptedAlbumKey.nonce),
+        encryptedName: toBase64(encryptedName.ciphertext),
+        nameDecryptionNonce: toBase64(encryptedName.nonce),
+        type: "album",
+        updationTime: TAKEN,
+    };
+
+    // Each photo as the album's `/collections/v2/diff` lists it: its key
+    // encrypted with the album's key, and its metadata, holding the hash of
+    // its original, encrypted with its own key. Its original, encrypted with
+    // its own key too, is what the files CDN serves for it.
+    const ciphertexts: Record<number, Uint8Array> = {};
+    const photos = PHOTO_IDS.map((id) => {
+        const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
+        const original = encryptStreamBlob(originalOf(id), key);
+        ciphertexts[id] = original.ciphertext;
+        const encryptedKey = encryptSecretbox(key, albumKey);
+        const metadata = encryptStreamBlob(
+            new TextEncoder().encode(
+                JSON.stringify({
+                    title: `photo-${id}.jpg`,
+                    fileType: 0,
+                    hash: blake2b(originalOf(id)),
+                    creationTime: TAKEN,
+                    modificationTime: TAKEN,
+                }),
+            ),
+            key,
+        );
+        return {
+            id,
+            collectionID: 10,
+            ownerID: USER_ID,
+            encryptedKey: toBase64(encryptedKey.ciphertext),
+            keyDecryptionNonce: toBase64(encryptedKey.nonce),
+            metadata: {
+                encryptedData: toBase64(metadata.ciphertext),
+                decryptionHeader: toBase64(metadata.header),
+            },
+            file: { decryptionHeader: toBase64(original.header) },
+            thumbnail: {
+                decryptionHeader: toBase64(sodium.randombytes_buf(24)),
+            },
+            updationTime: TAKEN,
+        };
+    });
+
+    return buildMetaFetch({
+        verifier: account.verifier,
+        srpAttributes: account.srpAttributes,
+        keyAttributes: account.keyAttributes,
+        encryptedToken: account.encryptedToken,
+        collections: [album],
+        filesByCollection: { 10: photos },
+        encryptedMLData: {},
+        fileCiphertexts: ciphertexts,
+    });
+};
+
+beforeAll(async () => {
+    await init();
+    await sodium.ready;
+    root = mkdtempSync(join(tmpdir(), "quak-backup-concurrency-"));
+    server = await buildServer();
+    const client = await Client.login({
+        email: EMAIL,
+        password: PASSWORD,
+        apiOptions: { fetch: server },
+    });
+    session = client.toJSON();
+});
+
+afterAll(() => {
+    rmSync(root, { recursive: true, force: true });
+});
+
+// The ID of the photo whose original a request downloads, or undefined for any
+// other request. quak downloads an original from
+// `https://files.ente.io/?fileID=<id>`.
+const downloadOf = (input: RequestInfo | URL): number | undefined => {
+    const url = new URL(String(input));
+    return url.hostname === "files.ente.io"
+        ? Number(url.searchParams.get("fileID"))
+        : undefined;
+};
+
+// The fake server, except that it answers 404 to every download of photo
+// `failing`'s original.
+const failingFor =
+    (failing: number): typeof globalThis.fetch =>
+    async (input, init) =>
+        downloadOf(input) === failing
+            ? new Response("not found", { status: 404 })
+            : server(input, init);
+
+// The fake server, holding each download of an original until the test lets
+// it go. `held` lists the downloads waiting, the first to arrive first;
+// `most()` is the largest number that ever waited at once. A held download of
+// photo `failing`'s original is answered 404 once it is let go.
+const holdingServer = (failing?: number) => {
+    const held: { id: number; release: () => void }[] = [];
+    let most = 0;
+    const fetch: typeof globalThis.fetch = async (input, init) => {
+        const id = downloadOf(input);
+        if (id !== undefined) {
+            await new Promise<void>((release) => {
+                held.push({ id, release });
+                most = Math.max(most, held.length);
+            });
+            if (id === failing) {
+                return new Response("not found", { status: 404 });
+            }
+        }
+        return server(input, init);
+    };
+    return { fetch, held, most: () => most };
+};
+
+// Let the held downloads go one at a time, the first to arrive first, each
+// once `limit` of them wait, or all the downloads still to come when there
+// are fewer. Each of the eight photos is downloaded once. With a backup that
+// downloads fewer than `limit` at once, this fails waiting for them. Each wait
+// allows five seconds, for a loaded machine's disk to sync the files written.
+const releaseOneAtATime = async (
+    held: { release: () => void }[],
+    limit: number,
+): Promise<void> => {
+    for (let left = PHOTO_IDS.length; left > 0; left--) {
+        await vi.waitFor(
+            () => expect(held).toHaveLength(Math.min(limit, left)),
+            { interval: 5, timeout: 5000 },
+        );
+        held.shift()!.release();
+    }
+};
+
+// A library of the account opened as `quak backup` opens one, with no
+// precache and no background refresh, sending its requests through `fetch`.
+// It backs up to `dir` and keeps its cache beside it. Its request pools are
+// `pools`, or by default the library's own.
+const openLibrary = (
+    dir: string,
+    fetch: typeof globalThis.fetch,
+    pools?: RequestPools,
+): Promise<Library> =>
+    Library.open({
+        client: Client.fromJSON(session, { fetch }),
+        downloadDirectory: dir,
+        cacheDirectory: `${dir}-cache`,
+        refreshIntervalSeconds: Infinity,
+        precacheThumbnails: false,
+        precacheOriginals: false,
+        pools,
+    });
+
+// Every entry under `dir`, relative to it, sorted, a symlink as
+// `name -> target`.
+const listTree = (dir: string): string[] =>
+    readdirSync(dir, { recursive: true })
+        .sort()
+        .map((name) => {
+            const path = join(dir, name);
+            return lstatSync(path).isSymbolicLink()
+                ? `${name} -> ${readlinkSync(path)}`
+                : name;
+        });
+
+const expectStored = (dir: string, ids: number[]): void => {
+    for (const id of ids) {
+        expect(readFileSync(savedAt(dir, id))).toEqual(
+            Buffer.from(originalOf(id)),
+        );
+    }
+};
+
+describe("lib.backup()", () => {
+    // A library opened with the default pools, as `quak backup` opens one,
+    // downloads up to five originals at once. As soon as one finishes, the
+    // backup starts on the next photo, so five stay in flight until fewer
+    // than five are left.
+    it("downloads five originals at once, and never more, with the default pools", async () => {
+        const dir = mkdtempSync(join(root, "backup-"));
+        const holding = holdingServer();
+        const lib = await openLibrary(dir, holding.fetch);
+        try {
+            const backup = lib.backup();
+            await releaseOneAtATime(holding.held, 5);
+            const result = await backup;
+
+            expect(holding.most()).toBe(5);
+            expect(result).toMatchObject({
+                totalFiles: 8,
+                downloaded: 8,
+                failed: 0,
+            });
+            expectStored(dir, PHOTO_IDS);
+        } finally {
+            await lib.close();
+        }
+    });
+
+    // The limit is the content pool's, so a library opened with pools of
+    // another `contentConcurrency` backs up that many at once.
+    it("downloads as many originals at once as the content pool of the pools it was opened with allows", async () => {
+        const dir = mkdtempSync(join(root, "backup-"));
+        const holding = holdingServer();
+        const lib = await openLibrary(
+            dir,
+            holding.fetch,
+            new RequestPools({ contentConcurrency: 2 }),
+        );
+        try {
+            const backup = lib.backup();
+            await releaseOneAtATime(holding.held, 2);
+            const result = await backup;
+
+            expect(holding.most()).toBe(2);
+            expect(result).toMatchObject({ downloaded: 8, failed: 0 });
+            expectStored(dir, PHOTO_IDS);
+        } finally {
+            await lib.close();
+        }
+    });
+
+    // With `verify`, hashing an original already stored is part of the work
+    // on it, and counts against the limit too: a large backup never has more
+    // than five originals open for hashing. Here every stored original has
+    // changed on disk, so each is hashed, found not to match, and downloaded
+    // again. While five downloads wait, the backup has hashed only those five
+    // photos' originals; it opens the next one once a download finishes.
+    it("with verify, hashes no more originals ahead of their downloads than the limit", async () => {
+        const dir = mkdtempSync(join(root, "backup-"));
+        const first = await openLibrary(dir, server);
+        try {
+            await first.backup();
+        } finally {
+            await first.close();
+        }
+        for (const id of PHOTO_IDS) {
+            writeFileSync(savedAt(dir, id), "not the photo any more");
+        }
+
+        const holding = holdingServer();
+        const lib = await openLibrary(dir, holding.fetch);
+        const lines: string[] = [];
+        const mismatches = () =>
+            lines.filter((line) => line.startsWith("MISMATCH"));
+        try {
+            const backup = lib.backup({
+                verify: true,
+                onProgress: (line) => lines.push(line),
+            });
+            await vi.waitFor(() => expect(holding.held).toHaveLength(5), {
+                interval: 5,
+                timeout: 5000,
+            });
+            expect(mismatches()).toHaveLength(5);
+
+            await releaseOneAtATime(holding.held, 5);
+            const result = await backup;
+
+            expect(holding.most()).toBe(5);
+            expect(mismatches()).toHaveLength(8);
+            expect(result).toMatchObject({
+                downloaded: 8,
+                verified: 0,
+                mismatched: 8,
+                failed: 0,
+            });
+            expectStored(dir, PHOTO_IDS);
+        } finally {
+            await lib.close();
+        }
+    });
+
+    // The server answers 404 for photo 103's original. A 404 is not retried,
+    // so that download fails at once, while other downloads are still in
+    // flight. The backup records the failure and goes on: the downloads
+    // beside it finish, the photos after it are downloaded, and only photo
+    // 103 is missing from the backup and in failures.json.
+    it("goes on with the other originals when one fails", async () => {
+        const dir = mkdtempSync(join(root, "backup-"));
+        const holding = holdingServer(103);
+        const lib = await openLibrary(dir, holding.fetch);
+        try {
+            const backup = lib.backup();
+            await releaseOneAtATime(holding.held, 5);
+            const result = await backup;
+
+            expect(result).toMatchObject({
+                totalFiles: 8,
+                downloaded: 7,
+                failed: 1,
+            });
+            expect(result.errors.map((e) => e.fileID)).toEqual([103]);
+            expectStored(
+                dir,
+                PHOTO_IDS.filter((id) => id !== 103),
+            );
+            expect(existsSync(savedAt(dir, 103))).toBe(false);
+            const failures = JSON.parse(
+                readFileSync(join(dir, "failures.json"), "utf-8"),
+            );
+            expect(Object.keys(failures.files)).toEqual(["103"]);
+        } finally {
+            await lib.close();
+        }
+    });
+
+    // A throw from `onProgress` fails the backup: `lib.backup()` rejects with
+    // it. It rejects only once it has finished with every original it was
+    // working on, so nothing is written into the backup after it has ended
+    // and removed its lock. Here, with a content pool of eight, all eight
+    // downloads wait at once. Photo 103's is let go first and answered 404,
+    // and the line reporting that failure throws while the other seven wait.
+    it("when onProgress throws, rejects only once the originals it was working on are in place", async () => {
+        const dir = mkdtempSync(join(root, "backup-"));
+        const holding = holdingServer(103);
+        const lib = await openLibrary(
+            dir,
+            holding.fetch,
+            new RequestPools({ contentConcurrency: 8 }),
+        );
+        const lines: string[] = [];
+        try {
+            const backup = lib.backup({
+                onProgress: (line) => {
+                    lines.push(line);
+                    if (line.startsWith("FAILED")) {
+                        throw new Error("the progress output is closed");
+                    }
+                },
+            });
+            // The photos whose originals are in place, and whether the lock
+            // is there, at the moment `lib.backup()` rejects.
+            let storedWhenRejected: number[] = [];
+            let lockedWhenRejected = true;
+            const rejection = backup.catch((err: unknown) => {
+                storedWhenRejected = PHOTO_IDS.filter((id) =>
+                    existsSync(savedAt(dir, id)),
+                );
+                lockedWhenRejected = existsSync(join(dir, "backup.lock"));
+                return err;
+            });
+
+            await vi.waitFor(() => expect(holding.held).toHaveLength(8), {
+                interval: 5,
+                timeout: 5000,
+            });
+            const at103 = holding.held.findIndex((h) => h.id === 103);
+            holding.held.splice(at103, 1)[0]!.release();
+            await vi.waitFor(
+                () =>
+                    expect(
+                        lines.some((line) => line.startsWith("FAILED")),
+                    ).toBe(true),
+                { interval: 5, timeout: 5000 },
+            );
+            for (const download of holding.held) download.release();
+
+            expect(await rejection).toEqual(
+                new Error("the progress output is closed"),
+            );
+            expect(storedWhenRejected).toEqual(
+                PHOTO_IDS.filter((id) => id !== 103),
+            );
+            expect(lockedWhenRejected).toBe(false);
+        } finally {
+            await lib.close();
+        }
+    });
+
+    // Working on several originals at once changes only how long a backup
+    // takes. The same two runs, with a content pool of one, which backs up
+    // one original at a time, and with the default pools, end the same way:
+    // the same result, the same progress lines (in another order), the same
+    // failures.json, and the same tree. The first run stores every photo but
+    // 103, whose download is answered 404. Then two stored originals change
+    // on disk, and a run with `verify` puts them back and tries 103 again.
+    it("ends as a backup of one original at a time does", async () => {
+        const twoRuns = async (pools: RequestPools) => {
+            const dir = mkdtempSync(join(root, "backup-"));
+            const lines: string[] = [];
+            const backup = async (opts: BackupOptions) => {
+                const lib = await openLibrary(dir, failingFor(103), pools);
+                try {
+                    return await lib.backup({
+                        ...opts,
+                        onProgress: (line) => lines.push(line),
+                    });
+                } finally {
+                    await lib.close();
+                }
+            };
+            const first = await backup({});
+            writeFileSync(savedAt(dir, 101), "not the photo any more");
+            writeFileSync(savedAt(dir, 105), "not the photo any more");
+            const second = await backup({ verify: true });
+
+            // Each entry of failures.json without the time it was last
+            // tried, which differs from run to run.
+            const { files } = JSON.parse(
+                readFileSync(join(dir, "failures.json"), "utf-8"),
+            ) as { files: Record<string, { lastTriedAt?: number }> };
+            for (const entry of Object.values(files)) delete entry.lastTriedAt;
+
+            return {
+                first,
+                second,
+                lines: lines.sort(),
+                failures: files,
+                tree: listTree(dir),
+            };
+        };
+
+        const oneAtATime = await twoRuns(
+            new RequestPools({ contentConcurrency: 1 }),
+        );
+        const fiveAtATime = await twoRuns(new RequestPools());
+
+        expect(oneAtATime.first).toMatchObject({
+            totalFiles: 8,
+            downloaded: 7,
+            skipped: 0,
+            failed: 1,
+        });
+        expect(oneAtATime.second).toMatchObject({
+            totalFiles: 8,
+            downloaded: 2,
+            skipped: 5,
+            verified: 5,
+            mismatched: 2,
+            unchecked: 0,
+            failed: 1,
+        });
+        expect(oneAtATime.failures).toEqual({
+            "103": expect.objectContaining({ fileID: 103, attempts: 2 }),
+        });
+        expect(fiveAtATime).toEqual(oneAtATime);
+    });
+});

test/cli/backup-undecryptable.test.ts

@@ -0,0 +1,554 @@
+/**
+ * `quak backup` over an account holding an album or a file quak cannot decrypt
+ * (issue 181).
+ *
+ * The server sends every album and file encrypted, and a refresh decrypts each
+ * one it fetches. A record whose key no key of the account opens, or whose
+ * metadata is not JSON, must not fail the refresh: a refresh that fails commits
+ * nothing, so `quak backup` would store nothing, on every run, until the
+ * server's copy changed. Instead the refresh leaves that record out, and the
+ * library keeps a list of what it left out in its cache. It has to: the next
+ * refresh asks the server only for what changed since the last one, so it will
+ * not see the record again.
+ *
+ * The backup stores every other file. It records each file it could not
+ * decrypt in `failures.json`, prints one line for each album it could not
+ * decrypt, and exits 1 while any remain. Once the server sends a copy that
+ * decrypts, or deletes the record, the next run drops it and can exit 0. The
+ * read commands answer from what did decrypt.
+ *
+ * These tests run the commands with a real saved session over a stand-in Ente
+ * server, which serves albums, files and originals encrypted with libsodium as
+ * the real server does. A record is broken the way a corrupt one arrives: its
+ * key replaced by bytes no key opens, or its metadata text that is not JSON.
+ */
+
+import {
+    existsSync,
+    mkdtempSync,
+    readdirSync,
+    readFileSync,
+    rmSync,
+} from "node:fs";
+import { join } from "node:path";
+import { tmpdir } from "node:os";
+import sodium from "libsodium-wrappers-sumo";
+import { afterEach, beforeAll, beforeEach, describe, expect, it } from "vitest";
+
+import {
+    backupCommand,
+    collectionsCommand,
+    filesCommand,
+    saveSession,
+    type CliContext,
+} from "../../src/cli-commands.js";
+import { loadSession } from "../../src/cli-session.js";
+import { Client, type ClientSnapshot } from "../../src/client.js";
+import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
+import { Library } from "../../src/library/index.js";
+
+const USER_ID = 42;
+
+// Every file is taken at noon local time on 2026-03-01, in microseconds as Ente
+// stores times, so its original is saved in the folder `DAY` whatever the time
+// zone of the machine running the tests.
+const TAKEN = new Date(2026, 2, 1, 12).getTime() * 1000;
+const DAY = join("2026", "2026-03", "2026-03-01");
+
+type Row = { id: number; updationTime: number } & Record<string, unknown>;
+
+// `plaintext` sealed under `key` with a fresh nonce, both base64, as Ente
+// sends a key or an album name.
+const secretbox = (
+    plaintext: Uint8Array,
+    key: Uint8Array,
+): { ciphertext: string; nonce: string } => {
+    const nonce = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
+    return {
+        ciphertext: toBase64(
+            sodium.crypto_secretbox_easy(plaintext, nonce, key),
+        ),
+        nonce: toBase64(nonce),
+    };
+};
+
+// What a corrupt row carries in place of its sealed key: random bytes of the
+// same length, which no key opens.
+const corruptKey = (): string =>
+    toBase64(
+        sodium.randombytes_buf(
+            sodium.crypto_secretbox_KEYBYTES + sodium.crypto_secretbox_MACBYTES,
+        ),
+    );
+
+const json = (body: unknown): Response =>
+    new Response(JSON.stringify(body), {
+        status: 200,
+        headers: { "content-type": "application/json" },
+    });
+
+// A stand-in Ente server for one account. It holds each album, and each file in
+// an album, as the row `/collections/v2` and `/collections/v2/diff` serve, and
+// answers a request with the rows whose `updationTime` is past the request's
+// `sinceTime`, as the real server does. A change to a file also moves its
+// album's `updationTime`, which is what tells a client to list the album's
+// files again. Originals are served by file ID, as the files CDN serves them.
+class FakeEnte {
+    private readonly masterKey = sodium.crypto_secretbox_keygen();
+    private readonly keyPair = sodium.crypto_box_keypair();
+    private readonly albums = new Map<number, Row>();
+    private readonly albumKeys = new Map<number, Uint8Array>();
+    // Each album's file rows, by album ID and then file ID.
+    private readonly files = new Map<number, Map<number, Row>>();
+    // Each file's key and its original as served, by file ID. A file in two
+    // albums is one original under one key, sealed under each album's key.
+    private readonly originals = new Map<
+        number,
+        { key: Uint8Array; header: string; body: Uint8Array }
+    >();
+
+    // The session `quak login` would have saved for the account.
+    snapshot(): ClientSnapshot {
+        return {
+            email: "user@example.com",
+            userID: USER_ID,
+            token: "token",
+            masterKey: toBase64(this.masterKey),
+            secretKey: toBase64(this.keyPair.privateKey),
+            publicKey: toBase64(this.keyPair.publicKey),
+        };
+    }
+
+    // Album `id`, named `name`, as of time `t`. With `corrupt`, its key cannot
+    // be decrypted. An album keeps its key from one call to the next, so its
+    // files' rows still decrypt once it is put back without `corrupt`.
+    putAlbum(id: number, name: string, t: number, corrupt = false): void {
+        const key = this.albumKeys.get(id) ?? sodium.crypto_secretbox_keygen();
+        this.albumKeys.set(id, key);
+        const sealedKey = secretbox(key, this.masterKey);
+        const sealedName = secretbox(new TextEncoder().encode(name), key);
+        this.albums.set(id, {
+            id,
+            owner: { id: USER_ID },
+            encryptedKey: corrupt ? corruptKey() : sealedKey.ciphertext,
+            keyDecryptionNonce: sealedKey.nonce,
+            encryptedName: sealedName.ciphertext,
+            nameDecryptionNonce: sealedName.nonce,
+            type: "album",
+            updationTime: t,
+        });
+        if (!this.files.has(id)) this.files.set(id, new Map());
+    }
+
+    // File `id`, titled `photo-<id>.jpg`, in album `albumID` as of time `t`.
+    // With `corruptKey`, its key cannot be decrypted. With `metadata`, its
+    // metadata decrypts to that text instead of the JSON a client uploads.
+    putFile(
+        albumID: number,
+        id: number,
+        t: number,
+        opts: { corruptKey?: boolean; metadata?: string } = {},
+    ): void {
+        const original = this.originalOf(id);
+        const sealedKey = secretbox(original.key, this.albumKeys.get(albumID)!);
+        const metadata =
+            opts.metadata ??
+            JSON.stringify({
+                title: `photo-${id}.jpg`,
+                fileType: 0,
+                creationTime: TAKEN,
+                modificationTime: TAKEN,
+            });
+        const sealedMetadata = encryptBlob(
+            new TextEncoder().encode(metadata),
+            original.key,
+        );
+        this.files.get(albumID)!.set(id, {
+            id,
+            collectionID: albumID,
+            ownerID: USER_ID,
+            encryptedKey: opts.corruptKey ? corruptKey() : sealedKey.ciphertext,
+            keyDecryptionNonce: sealedKey.nonce,
+            metadata: {
+                encryptedData: toBase64(sealedMetadata.ciphertext),
+                decryptionHeader: toBase64(sealedMetadata.header),
+            },
+            file: { decryptionHeader: original.header },
+            thumbnail: { decryptionHeader: original.header },
+            updationTime: t,
+        });
+        this.moveAlbum(albumID, t);
+    }
+
+    // A deleted record stays on the server as a tombstone holding only its
+    // ID, its time and `isDeleted`.
+    deleteFile(albumID: number, id: number, t: number): void {
+        this.files
+            .get(albumID)!
+            .set(id, { id, updationTime: t, isDeleted: true });
+        this.moveAlbum(albumID, t);
+    }
+
+    deleteAlbum(id: number, t: number): void {
+        this.albums.set(id, { id, updationTime: t, isDeleted: true });
+    }
+
+    // The `fetch` the saved session's client is given.
+    readonly fetch = (async (input: string | URL | Request) => {
+        const url = new URL(input instanceof Request ? input.url : input);
+        const since = Number(url.searchParams.get("sinceTime"));
+        const changed = (rows: Iterable<Row>): Row[] =>
+            [...rows].filter((row) => row.updationTime > since);
+        switch (url.pathname) {
+            case "/collections/v2":
+                return json({ collections: changed(this.albums.values()) });
+            case "/collections/v2/diff": {
+                const albumID = Number(url.searchParams.get("collectionID"));
+                const rows = this.files.get(albumID)?.values() ?? [];
+                return json({ diff: changed(rows), hasMore: false });
+            }
+            case "/files/data/fetch":
+                // No file has ML data.
+                return json({ data: [] });
+            default: {
+                const fileID = Number(url.searchParams.get("fileID"));
+                const original = this.originals.get(fileID);
+                return original
+                    ? new Response(original.body)
+                    : new Response("not found", { status: 404 });
+            }
+        }
+    }) as typeof globalThis.fetch;
+
+    private originalOf(id: number): {
+        key: Uint8Array;
+        header: string;
+        body: Uint8Array;
+    } {
+        const known = this.originals.get(id);
+        if (known) return known;
+        const key = sodium.crypto_secretbox_keygen();
+        const { header, ciphertext } = encryptBlob(
+            new TextEncoder().encode(`original of ${id}`),
+            key,
+        );
+        const original = { key, header: toBase64(header), body: ciphertext };
+        this.originals.set(id, original);
+        return original;
+    }
+
+    private moveAlbum(albumID: number, t: number): void {
+        this.albums.set(albumID, {
+            ...this.albums.get(albumID)!,
+            updationTime: t,
+        });
+    }
+}
+
+// Album 1, "Trip", holds file 10, which decrypts, file 11, whose key does not,
+// and file 12, whose metadata is not JSON. Album 2, "Family", holds file 20,
+// but the album's own key does not decrypt. All of it as of time 1.
+const brokenAccount = (): FakeEnte => {
+    const ente = new FakeEnte();
+    ente.putAlbum(1, "Trip", 1);
+    ente.putFile(1, 10, 1);
+    ente.putFile(1, 11, 1, { corruptKey: true });
+    ente.putFile(1, 12, 1, { metadata: "not json" });
+    ente.putAlbum(2, "Family", 1, true);
+    ente.putFile(2, 20, 1);
+    return ente;
+};
+
+// Album 1, "Trip", holds file 10, and album 2, "Family", holds file 20. Both
+// files decrypt, but album 2's own key does not, so the album is the only
+// record left out. All of it as of time 1.
+const brokenAlbumAccount = (): FakeEnte => {
+    const ente = new FakeEnte();
+    ente.putAlbum(1, "Trip", 1);
+    ente.putFile(1, 10, 1);
+    ente.putAlbum(2, "Family", 1, true);
+    ente.putFile(2, 20, 1);
+    return ente;
+};
+
+// Collects everything written to it.
+class Output {
+    text = "";
+    write(text: string): void {
+        this.text += text;
+    }
+}
+
+let root: string;
+let stdout: Output;
+let stderr: Output;
+
+beforeAll(async () => {
+    await init();
+    await sodium.ready;
+});
+
+beforeEach(() => {
+    root = mkdtempSync(join(tmpdir(), "quak-undecryptable-"));
+});
+
+afterEach(() => {
+    rmSync(root, { recursive: true, force: true });
+});
+
+// A context for one command, with fresh output, whose saved session talks to
+// `ente`. Every run shares one cache directory, as runs of `quak` do.
+const context = (ente: FakeEnte): CliContext => {
+    stdout = new Output();
+    stderr = new Output();
+    saveSession(join(root, "session"), ente.snapshot());
+    return {
+        stdout,
+        stderr,
+        sessionDir: join(root, "session"),
+        cacheDir: join(root, "cache"),
+        loadSession: (path) =>
+            loadSession(path, { fetch: ente.fetch, retry: { attempts: 1 } }),
+        login: async () => {
+            throw new Error("login not expected");
+        },
+        prompt: async () => {
+            throw new Error("prompt not expected");
+        },
+        promptSecret: async () => {
+            throw new Error("prompt not expected");
+        },
+    };
+};
+
+const backupDir = (): string => join(root, "backup");
+
+// Runs `quak backup` into the backup directory and returns its exit code.
+const backup = (ente: FakeEnte, opts: { json?: boolean } = {}) =>
+    backupCommand(context(ente), backupDir(), opts);
+
+// The backup's `failures.json` entries by file ID, or {} when it has none.
+const failures = (): Record<string, { error: string; attempts: number }> => {
+    const path = join(backupDir(), "failures.json");
+    return existsSync(path)
+        ? JSON.parse(readFileSync(path, "utf-8")).files
+        : {};
+};
+
+// Whether file `id`'s original is at its save path in the backup.
+const stored = (id: number): boolean =>
+    existsSync(join(backupDir(), DAY, `2026-03-01.${id}.jpg`));
+
+// The IDs of the files that album `name`'s JSON lists.
+const albumFiles = (name: string): number[] =>
+    JSON.parse(
+        readFileSync(join(backupDir(), "collections", `${name}.json`), "utf-8"),
+    ).files.map((f: { id: number }) => f.id);
+
+describe("quak backup with records it cannot decrypt", () => {
+    it("backs up every other file, records each file it cannot decrypt in failures.json, names each such album, and exits 1", async () => {
+        expect(await backup(brokenAccount())).toBe(1);
+
+        expect(stored(10)).toBe(true);
+        // Without its album's key, file 20 cannot even be listed.
+        expect(stored(20)).toBe(false);
+        expect(failures()).toEqual({
+            "11": expect.objectContaining({
+                fileID: 11,
+                error: expect.stringMatching(
+                    /^could not be decrypted: .*wrong secret key/,
+                ),
+            }),
+            "12": expect.objectContaining({
+                fileID: 12,
+                error: expect.stringMatching(/^could not be decrypted: .*JSON/),
+            }),
+        });
+        expect(stderr.text).toMatch(
+            /^FAILED album 2: could not be decrypted: .*wrong secret key/m,
+        );
+        expect(stderr.text).toContain("  Failed:      2\n");
+    });
+
+    it("names each album it cannot decrypt in the --json result", async () => {
+        expect(await backup(brokenAccount(), { json: true })).toBe(1);
+
+        expect(JSON.parse(stdout.text)).toMatchObject({
+            downloaded: 1,
+            failed: 2,
+            undecryptableAlbums: [2],
+        });
+    });
+
+    it("still reports them on the next run, though the server does not send them again", async () => {
+        const ente = brokenAccount();
+        await backup(ente);
+
+        // Nothing changed on the server, so this run's refresh gets no rows at
+        // all. What it reports, it remembered from the first run.
+        expect(await backup(ente)).toBe(1);
+
+        expect(Object.keys(failures())).toEqual(["11", "12"]);
+        expect(failures()["11"]!.attempts).toBe(2);
+        expect(stderr.text).toMatch(/^FAILED album 2: could not be decrypted/m);
+    });
+
+    it("drops each once the server sends a copy that decrypts, and then exits 0", async () => {
+        const ente = brokenAccount();
+        await backup(ente);
+
+        // At time 2 files 11 and 12 and album 2 change on the server, and now
+        // each decrypts. Album 2 brings file 20 with it.
+        ente.putFile(1, 11, 2);
+        ente.putFile(1, 12, 2);
+        ente.putAlbum(2, "Family", 2);
+        expect(await backup(ente)).toBe(0);
+
+        expect([11, 12, 20].map(stored)).toEqual([true, true, true]);
+        expect(existsSync(join(backupDir(), "failures.json"))).toBe(false);
+        expect(stderr.text).not.toContain("FAILED");
+    });
+
+    it("drops each once the server deletes it, and then exits 0", async () => {
+        const ente = brokenAccount();
+        await backup(ente);
+
+        ente.deleteFile(1, 11, 2);
+        ente.deleteFile(1, 12, 2);
+        ente.deleteAlbum(2, 2);
+        expect(await backup(ente)).toBe(0);
+
+        expect(existsSync(join(backupDir(), "failures.json"))).toBe(false);
+        expect(stderr.text).not.toContain("FAILED");
+    });
+
+    it("exits 1 when the only record it cannot decrypt is an album, and 0 once the album decrypts", async () => {
+        const ente = brokenAlbumAccount();
+
+        // No file is in `failures.json`: the album alone keeps the exit code
+        // at 1.
+        expect(await backup(ente)).toBe(1);
+        expect(failures()).toEqual({});
+
+        ente.putAlbum(2, "Family", 2);
+        expect(await backup(ente)).toBe(0);
+        expect(stored(20)).toBe(true);
+    });
+
+    it("exits 1 when the only record it cannot decrypt is an album, and 0 once the album is deleted", async () => {
+        const ente = brokenAlbumAccount();
+
+        expect(await backup(ente)).toBe(1);
+        expect(failures()).toEqual({});
+
+        ente.deleteAlbum(2, 2);
+        expect(await backup(ente)).toBe(0);
+    });
+
+    it("removes what an earlier run linked for an album or a file that no longer decrypts, and keeps the originals", async () => {
+        // At time 1 everything decrypts: the first run links files 10 and 11
+        // in Trip and file 20 in Family.
+        const ente = new FakeEnte();
+        ente.putAlbum(1, "Trip", 1);
+        ente.putFile(1, 10, 1);
+        ente.putFile(1, 11, 1);
+        ente.putAlbum(2, "Family", 1);
+        ente.putFile(2, 20, 1);
+        expect(await backup(ente)).toBe(0);
+        const collections = join(backupDir(), "collections");
+        expect(readdirSync(join(collections, "Trip")).sort()).toEqual([
+            "photo-10.jpg",
+            "photo-11.jpg",
+        ]);
+        expect(readdirSync(join(collections, "Family"))).toEqual([
+            "photo-20.jpg",
+        ]);
+
+        // At time 2 the server's copy of file 11 in Trip, and album 2 itself,
+        // no longer decrypt. A run builds the album folders, their JSON and
+        // their links from what the library holds, so it removes them as if
+        // the album and the file had been deleted: Family's folder and JSON
+        // go, and Trip neither links nor lists file 11. The originals stay at
+        // their save paths, and the run exits 1.
+        ente.putFile(1, 11, 2, { corruptKey: true });
+        ente.putAlbum(2, "Family", 2, true);
+        expect(await backup(ente)).toBe(1);
+
+        expect(readdirSync(collections).sort()).toEqual(["Trip", "Trip.json"]);
+        expect(readdirSync(join(collections, "Trip"))).toEqual([
+            "photo-10.jpg",
+        ]);
+        expect(albumFiles("Trip")).toEqual([10]);
+        expect([10, 11, 20].map(stored)).toEqual([true, true, true]);
+        expect(Object.keys(failures())).toEqual(["11"]);
+    });
+
+    it("backs up a file that does not decrypt in one album but does in another, and does not report it", async () => {
+        // File 30 is in albums 1 and 3, each holding its key sealed under the
+        // album's own key. In album 1 that is corrupt; in album 3 it is not,
+        // and one copy that decrypts is all the backup needs.
+        const ente = new FakeEnte();
+        ente.putAlbum(1, "Trip", 1);
+        ente.putFile(1, 30, 1, { corruptKey: true });
+        ente.putAlbum(3, "Favorites", 1);
+        ente.putFile(3, 30, 1);
+
+        expect(await backup(ente)).toBe(0);
+
+        expect(stored(30)).toBe(true);
+        expect(failures()).toEqual({});
+        // Album 1 has no copy of file 30 that decrypts, so it neither links
+        // the original nor lists the file; album 3 lists it.
+        expect(readdirSync(join(backupDir(), "collections", "Trip"))).toEqual(
+            [],
+        );
+        expect(albumFiles("Trip")).toEqual([]);
+        expect(albumFiles("Favorites")).toEqual([30]);
+    });
+
+    it("backs up that file in a run limited to the album where it does not decrypt", async () => {
+        // The same account, backed up by `lib.backup()` limited to album 1,
+        // "Trip", as `quak backup` cannot be. File 30 is in Trip, so the run
+        // must store it, from its copy in album 3 that the run otherwise
+        // skips.
+        const ente = new FakeEnte();
+        ente.putAlbum(1, "Trip", 1);
+        ente.putFile(1, 30, 1, { corruptKey: true });
+        ente.putAlbum(3, "Favorites", 1);
+        ente.putFile(3, 30, 1);
+        const lib = await Library.open({
+            client: Client.fromJSON(ente.snapshot(), {
+                fetch: ente.fetch,
+                retry: { attempts: 1 },
+            }),
+            downloadDirectory: backupDir(),
+            cacheDirectory: join(root, "cache"),
+            precacheThumbnails: false,
+            precacheOriginals: false,
+        });
+        try {
+            const result = await lib.backup({
+                downloadDirectory: backupDir(),
+                onlyAlbumNames: ["Trip"],
+            });
+            expect(result).toMatchObject({ downloaded: 1, failed: 0 });
+        } finally {
+            await lib.close();
+        }
+
+        expect(stored(30)).toBe(true);
+        expect(failures()).toEqual({});
+    });
+});
+
+describe("the read commands with records quak cannot decrypt", () => {
+    it("quak collections and quak files answer from what decrypts", async () => {
+        const ente = brokenAccount();
+
+        expect(await collectionsCommand(context(ente), {})).toBe(0);
+        expect(stdout.text).toBe("1\talbum\tTrip\n");
+
+        expect(await filesCommand(context(ente), { collection: "1" })).toBe(0);
+        expect(stdout.text).toBe("10\timage\tphoto-10.jpg\n");
+    });
+});

test/cli/backup.test.ts

@@ -909,6 +909,27 @@ describe("the backup lock", () => {
         expect(existsSync(lockPath(outDir))).toBe(false);
         await lib.close();
     });
+
+    // `runBackup` takes a library object from its caller. With a
+    // `contentConcurrency` below 1, or one that is not a number, it would
+    // back up no original and report no error. It rejects any that is not a
+    // positive integer, as `BoundedPool` does, before it takes the lock,
+    // which would create the download directory, and before it asks the
+    // library for anything: the object here has nothing else.
+    it("rejects a library whose contentConcurrency is not a positive integer, before taking the lock", async () => {
+        const outDir = join(root, "backup");
+        for (const contentConcurrency of [undefined, 0, -1, 1.5, NaN]) {
+            const lib = { contentConcurrency } as unknown as BackupLibrary;
+
+            await expect(
+                runBackup(lib, { downloadDirectory: outDir }),
+            ).rejects.toThrow(
+                `contentConcurrency must be a positive integer, got ${contentConcurrency}`,
+            );
+
+            expect(existsSync(outDir)).toBe(false);
+        }
+    });
 });
 
 // Also serves ML data: file 100 has `ML_PAYLOAD`, the other two have none.
@@ -1482,6 +1503,7 @@ describe("backup album folders", () => {
         listCollections: () => albums.map((a) => a.collection),
         listFiles: (id) =>
             albums.find((a) => a.collection.id === id)?.files ?? [],
+        undecryptable: () => [],
         original: async (fileID) => {
             const path = join(root, `source-${fileID}`);
             writeFileSync(path, `original ${fileID}`);
@@ -1490,6 +1512,7 @@ describe("backup album folders", () => {
         thumbnail: async () => {
             throw new Error("no thumbnails in this stand-in");
         },
+        contentConcurrency: 5,
         fetchMLData: async () => {},
         mlData: async () => undefined,
         whoami: () => ({ email: "backup@example.com", userID: USER_ID }),

test/client/enumeration.test.ts

@@ -10,7 +10,9 @@
  * The return shape keeps live records and tombstones apart — `collections` /
  * `files` are decrypted live records, `deleted` is a plain list of the ids the
  * server tombstoned. A tombstone carries no decryptable key or metadata, so it
- * is a bare id rather than a hollowed-out `Collection` / `EnteFile`.
+ * is a bare id rather than a hollowed-out `Collection` / `EnteFile`. A live row
+ * that does not decrypt, or whose metadata is not JSON, is listed under
+ * `undecryptable` with the reason, so one bad row does not fail the call.
  *
  * All tests inject a fake `fetch` and drive a real `Client` (built with
  * `Client.fromJSON`) so the decryption path runs for real. Live rows are built
@@ -91,27 +93,38 @@ const ownedCollectionRow = (
     };
 };
 
-/** A live file row inside a collection, keyed under that collection's key. */
+/**
+ * A live file row inside a collection, keyed under that collection's key. Its
+ * metadata decrypts to `metadata` when given, otherwise to the JSON a client
+ * uploads for `title`.
+ */
 const fileRow = (
     collectionKey: Uint8Array,
-    opts: { id: number; title: string; updationTime: number },
+    opts: {
+        id: number;
+        title: string;
+        updationTime: number;
+        metadata?: string;
+    },
 ): Record<string, unknown> => {
     const fileKey = sodium.crypto_secretbox_keygen();
     const { ciphertext: encFileKey, nonce: fileKeyNonce } = secretboxEncrypt(
         fileKey,
         collectionKey,
     );
-    const metadata = {
-        title: opts.title,
-        fileType: 0,
-        creationTime: opts.updationTime,
-        modificationTime: opts.updationTime,
-    };
+    const metadata =
+        opts.metadata ??
+        JSON.stringify({
+            title: opts.title,
+            fileType: 0,
+            creationTime: opts.updationTime,
+            modificationTime: opts.updationTime,
+        });
     const push =
         sodium.crypto_secretstream_xchacha20poly1305_init_push(fileKey);
     const encMeta = sodium.crypto_secretstream_xchacha20poly1305_push(
         push.state,
-        new TextEncoder().encode(JSON.stringify(metadata)),
+        new TextEncoder().encode(metadata),
         null,
         sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
     );
@@ -131,6 +144,21 @@ const fileRow = (
     };
 };
 
+/**
+ * `row` with its sealed key replaced by random bytes of the same length, as a
+ * corrupt row arrives: no key opens it, so the row cannot be decrypted.
+ */
+const withCorruptKey = (
+    row: Record<string, unknown>,
+): Record<string, unknown> => ({
+    ...row,
+    encryptedKey: toBase64(
+        sodium.randombytes_buf(
+            sodium.crypto_secretbox_KEYBYTES + sodium.crypto_secretbox_MACBYTES,
+        ),
+    ),
+});
+
 /** A tombstone row. Never decrypted, so only these fields are ever read. */
 const tombstoneRow = (
     id: number,
@@ -278,6 +306,58 @@ describe("Client.filesSince", () => {
         // It gave up after the first page rather than looping.
         expect(sinceTimes).toEqual(["50"]);
     });
+
+    it("leaves out each file it cannot decrypt or parse, with the reason, and decrypts the rest", async () => {
+        const keys = buildKeys();
+        const collectionKey = sodium.crypto_secretbox_keygen();
+
+        // File 1002's key is sealed under no key quak holds, and file 1003's
+        // metadata decrypts to text that is not JSON. Neither may cost the
+        // caller file 1001: each is left out of `files` and reported under
+        // `undecryptable` with the reason instead of failing the call.
+        const { fetch } = recordingFetch(
+            jsonResponse({
+                diff: [
+                    fileRow(collectionKey, {
+                        id: 1001,
+                        title: "fine.jpg",
+                        updationTime: 100,
+                    }),
+                    withCorruptKey(
+                        fileRow(collectionKey, {
+                            id: 1002,
+                            title: "bad-key.jpg",
+                            updationTime: 200,
+                        }),
+                    ),
+                    fileRow(collectionKey, {
+                        id: 1003,
+                        title: "bad-metadata.jpg",
+                        updationTime: 300,
+                        metadata: "not json",
+                    }),
+                ],
+                hasMore: false,
+            }),
+        );
+        const client = Client.fromJSON(snapshotFor(keys), { fetch });
+
+        const { files, undecryptable, cursor } = await client.filesSince({
+            collectionID: 1,
+            collectionKey,
+            sinceTime: 0,
+        });
+
+        expect(files.map((f) => f.id)).toEqual([1001]);
+        expect(undecryptable).toEqual([
+            { id: 1002, error: expect.stringContaining("wrong secret key") },
+            { id: 1003, error: expect.stringContaining("JSON") },
+        ]);
+        // The cursor moves past the two left out, as past any other row, so
+        // the server does not send them again until its copy changes. The
+        // caller has to keep what it was told here.
+        expect(cursor).toBe(300);
+    });
 });
 
 describe("Client.collectionsSince", () => {
@@ -331,6 +411,42 @@ describe("Client.collectionsSince", () => {
         expect(result.cursor).toBe(150);
         expect(sinceTimes).toEqual(["150"]);
     });
+
+    it("leaves out an album it cannot decrypt, with the reason, and decrypts the rest", async () => {
+        const keys = buildKeys();
+
+        // Album 2's key is sealed under no key quak holds. Without its key
+        // neither its name nor its files can be read, so it is left out and
+        // reported under `undecryptable`; album 1 is unaffected.
+        const { fetch } = recordingFetch(
+            jsonResponse({
+                collections: [
+                    ownedCollectionRow(keys.masterKey, {
+                        id: 1,
+                        name: "Vacation",
+                        updationTime: 100,
+                    }),
+                    withCorruptKey(
+                        ownedCollectionRow(keys.masterKey, {
+                            id: 2,
+                            name: "Broken",
+                            updationTime: 200,
+                        }),
+                    ),
+                ],
+            }),
+        );
+        const client = Client.fromJSON(snapshotFor(keys), { fetch });
+
+        const { collections, undecryptable, cursor } =
+            await client.collectionsSince({ sinceTime: 0 });
+
+        expect(collections.map((c) => c.name)).toEqual(["Vacation"]);
+        expect(undecryptable).toEqual([
+            { id: 2, error: expect.stringContaining("wrong secret key") },
+        ]);
+        expect(cursor).toBe(200);
+    });
 });
 
 describe("Client list wrappers still hide deletions", () => {

test/library/store.test.ts

@@ -172,6 +172,29 @@ describe("MetadataStore", () => {
         expect(store.listCollections()).toEqual([]);
     });
 
+    it("loads an empty store when the list of what the refresh could not decrypt is damaged", async () => {
+        // A backup reads that list on every run, so a damaged one that loaded
+        // would fail every backup until the cache was deleted. A string is
+        // damaged too, though JavaScript can loop over its characters.
+        mkdirSync(join(dir, "cache"), { recursive: true });
+        for (const undecryptable of [{}, "not a list", [1]]) {
+            writeFileSync(
+                path,
+                JSON.stringify({
+                    schemaVersion: METADATA_SCHEMA_VERSION,
+                    userID: 42,
+                    collectionsSinceTime: 0,
+                    collections: [],
+                    files: [],
+                    undecryptable,
+                }),
+            );
+            const store = await MetadataStore.load(path);
+            expect(store.userID).toBe(0);
+            expect(store.listUndecryptable()).toEqual([]);
+        }
+    });
+
     it("creates the directory 0700 and the file 0600", async () => {
         const store = await MetadataStore.load(path);
         store.putCollection(sampleCollection());