4 Commits
Author SHA1 Message Date
clawbot aeb275e137 README: the close() sentence split in two (closes #231)
check / check (push) Waiting to run
The sentence in "Opening a library" had a relative clause with its
"that" left out, so "returns resolves" read as two verbs in a row. It
is now two sentences; the behaviour described is unchanged and matches
`close()` in `src/library/index.ts`.

Model: fable-5-1 (change); opus-5-5 (landing message)
2026-10-26 19:32:03 +01:00
clawbot 67794bf5f8 README: four sentences fixed for the documents gate (closes #229)
check / check (push) Waiting to run
The documents gate on `next` failed five README sentences
(#219 (comment)). Four
are rewritten after a check against the code:

- the keywords score sentence is split in two;
- `mlData` drops the whole `clip` block, as `src/backup.ts` and
  `src/metadata-backup.ts` do;
- a file name gets its ID past 255 bytes and a collection name past
  250, and either is then cut short, as `uniqueNames` does;
- `lastTriedAt` is taken once per run, after its refresh.

The IDs sentence keeps `-1`: the pinned option parser passes a negative
number to a command with no subcommands as an argument, so
`quak get -1` reaches the ID check. Nothing else changes; `make fmt`
reflowed the touched paragraphs.

Model: fable-5-1 (change); opus-5-5 (landing message)
2026-10-26 18:59:28 +01:00
clawbot b5fcbf4400 README.md rewritten sentence by sentence (closes #219)
check / check (push) Waiting to run
The old text served as the list of what to cover. The new text leads
with what quak is, what it is for and how to use it. Every command,
option, exit code and file layout follows in reference sections, then the
library reference, then design, entrypoints and development. Every
statement was checked against the code on next; the pull request lists
each disagreement. Sentences stay under about 30 words, tables are lists,
and no issue links or project history remain in text for users. Two code
comments now name the new README sections, and the one in
`src/cli-log.ts` matches "What quak prints" on stderr, stdout and exit
status; a third no longer cites the README.

Model: fable-5-1
2026-10-26 17:48:25 +01:00
clawbot a1ea47f9a8 quak helper fix-missing-thumbnails --file on an empty cache stops fetching collections once it finds every file (closes #223)
check / check (push) Waiting to run
`helper fix-missing-thumbnails --file` used to wait for a refresh, which
on an empty cache fetched the files of every collection. It now finds the
named files through the new `freshFilesByID` in `src/cli-read.ts`, by the
rule `freshFile` follows. On an empty cache it reads the collection list,
then one collection's files at a time, and stores none of the collections
or file records it reads. It searches the most recently updated
collection first. So a file in several albums is still named after the
first of them that `Library.albums.list()` lists. `fixMissingThumbnails`
now takes the records the command found, and fetches each original
through `lib.originalOf()`.

Model: opus-5-5
2026-10-26 11:25:10 +01:00
11 changed files with 2034 additions and 1301 deletions
+1718 -1238
View File
File diff suppressed because it is too large Load Diff
+9
View File
@@ -26,6 +26,15 @@ declares one.
# Completed Steps # Completed Steps
- 2026-10-26: On an empty cache, `quak helper fix-missing-thumbnails --file`
stops fetching collections' files once it finds every file it names
(https://git.eeqj.de/sneak/quak/issues/223). It used to wait for a refresh,
which on an empty cache fetched the files of every collection. Now it reads
the collection list, then one collection's files at a time, the most recently
updated first. It stores none of the collections or file records it reads. It
fetches each original through `lib.originalOf()`, which keeps it in the cache.
Once the cache holds the account's collections, it still waits for a refresh.
- 2026-10-26: On an empty cache, `quak get` and `quak get-thumb` stop fetching - 2026-10-26: On an empty cache, `quak get` and `quak get-thumb` stop fetching
collections' files once they find the file collections' files once they find the file
(https://git.eeqj.de/sneak/quak/issues/222). They used to wait for a refresh, (https://git.eeqj.de/sneak/quak/issues/222). They used to wait for a refresh,
+19 -6
View File
@@ -37,9 +37,16 @@ import {
thumbnailName, thumbnailName,
} from "./cli-output.js"; } from "./cli-output.js";
import { type Log, logRefresh } from "./cli-log.js"; import { type Log, logRefresh } from "./cli-log.js";
import { freshCollections, freshFiles, freshFile } from "./cli-read.js"; import {
cachedFilesByID,
freshCollections,
freshFiles,
freshFile,
freshFilesByID,
} from "./cli-read.js";
import { withExtension } from "./filename.js"; import { withExtension } from "./filename.js";
import { runMetadataBackup } from "./metadata-backup.js"; import { runMetadataBackup } from "./metadata-backup.js";
import type { Collection, EnteFile } from "./model/types.js";
import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js"; import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js";
export interface CliContext { export interface CliContext {
@@ -133,8 +140,9 @@ const readLibraryClient = (client: Client): LibraryClient => ({
// not start downloading the whole account. No refresh starts as the library // not start downloading the whole account. No refresh starts as the library
// opens or on a timer. Each command asks for the refresh it needs. // opens or on a timer. Each command asks for the refresh it needs.
// `collections` and `files` with a collection ID need none (issues #210 and // `collections` and `files` with a collection ID need none (issues #210 and
// #217). Nor do `get` and `get-thumb` on an empty cache (issue #222). While a // #217). Nor do `get` and `get-thumb` on an empty cache (issue #222). Nor does
// refresh runs, each collection whose files it has fetched is logged, so a // `helper fix-missing-thumbnails --file` on an empty cache (issue #223). While
// a refresh runs, each collection whose files it has fetched is logged, so a
// large account shows progress rather than nothing. // large account shows progress rather than nothing.
const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> => const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
Library.open({ Library.open({
@@ -565,10 +573,13 @@ export const fixMissingThumbnailsCommand = async (
} }
const lib = await openReadLibrary(ctx, client); const lib = await openReadLibrary(ctx, client);
try { try {
// Refresh first so files added since the cache was written are found; let found: Map<number, { file: EnteFile; collection: Collection }>;
// a failed refresh throws. if (fileIDs.length > 0) {
found = await freshFilesByID(lib, fileIDs);
} else {
// Refresh first so files added since the cache was written are
// checked; a failed refresh throws.
await lib.fresh(); await lib.fresh();
if (fileIDs.length === 0) {
ctx.log("info", "Scanning for missing thumbnails..."); ctx.log("info", "Scanning for missing thumbnails...");
const missing = await listMissingThumbnails( const missing = await listMissingThumbnails(
lib, lib,
@@ -581,12 +592,14 @@ export const fixMissingThumbnailsCommand = async (
return 0; return 0;
} }
ctx.log("info", `Found ${fileIDs.length} file(s) to fix.`); ctx.log("info", `Found ${fileIDs.length} file(s) to fix.`);
found = cachedFilesByID(lib, fileIDs);
} }
const results = await fixMissingThumbnails( const results = await fixMissingThumbnails(
lib, lib,
client, client,
fileIDs, fileIDs,
found,
(msg, level) => ctx.log(level ?? "info", msg), (msg, level) => ctx.log(level ?? "info", msg),
); );
+10 -7
View File
@@ -1,15 +1,18 @@
// The CLI's log level, and the one function the commands write stderr lines // The CLI's log level, and the one function the commands write stderr lines
// through. // through.
// //
// Every line quak prints on stderr is a log line at one of six levels, from // Every line quak logs goes to stderr, at one of six levels, from the
// the quietest: fatal, error, warning, info, debug, trace. A run prints the // quietest: fatal, error, warning, info, debug, trace. A run prints the lines
// lines at its own level and at every quieter one. Its level is info unless // at its own level and at every quieter one. Its level is info unless `-v` or
// `-v` or `-q` asks for another (see "CLI surface" in the README). Every // `-q` asks for another (see "What quak prints" in the README). A command that
// command that exits non-zero logs a fatal line saying why, so even `-q`, which // runs and exits non-zero logs a fatal line saying why, so even `-q`, which
// prints only fatal lines, never leaves a non-zero exit unexplained. An error // prints only fatal lines, never leaves such an exit unexplained. An error
// line reports something that failed, such as a file or an album, and a // line reports something that failed, such as a file or an album, and a
// warning something that did not fail but needs the user's attention. stdout // warning something that did not fail but needs the user's attention. stdout
// carries only a command's output and is not a log. // carries a command's output and the prompts of `login`, and is not a log.
// Two things on stderr are not log lines: the `error:` message for a
// command line that quak cannot parse, and the usage printed when no command
// is given.
// //
// The library itself writes no lines: its layers report through callbacks, and // The library itself writes no lines: its layers report through callbacks, and
// the CLI logs what they report. `logRequests` gives the API client options // the CLI logs what they report. `logRequests` gives the API client options
+58
View File
@@ -12,6 +12,8 @@
// list too, then that one collection's files through `Library.freshFiles()` // list too, then that one collection's files through `Library.freshFiles()`
// (issue #217). On an empty cache, `get` and `get-thumb` read that list, then // (issue #217). On an empty cache, `get` and `get-thumb` read that list, then
// one collection's files at a time, until they find the file (issue #222). // one collection's files at a time, until they find the file (issue #222).
// `helper fix-missing-thumbnails --file` does the same until it finds every
// file it names (issue #223).
// //
// `collections` and `files` with a collection ID list in the server's order. // `collections` and `files` with a collection ID list in the server's order.
// `files` without one lists in the library's own enumeration order // `files` without one lists in the library's own enumeration order
@@ -90,3 +92,59 @@ export const freshFile = async (
} }
return undefined; return undefined;
}; };
// The most recently updated collection first, and the higher ID first on a
// tie. `Library.albums.list()` lists albums in this order.
const byNewestCollection = (a: Collection, b: Collection): number =>
b.updationTime - a.updationTime || b.id - a.id;
// Each file named by `fileIDs` that the cache holds, mapped to its record and
// the collection it was found in. Collections are searched in
// `byNewestCollection` order, and a file is found in the first that holds it.
// So `helper fix-missing-thumbnails` names a file after the same album as
// `helper list-missing-thumbnails`. It reads the cache alone, so the caller
// refreshes first.
export const cachedFilesByID = (
lib: FreshReadLibrary,
fileIDs: number[],
): Map<number, { file: EnteFile; collection: Collection }> => {
const wanted = new Set(fileIDs);
const found = new Map<number, { file: EnteFile; collection: Collection }>();
const collections = [...lib.listCollections()].sort(byNewestCollection);
for (const collection of collections) {
for (const file of lib.listFiles(collection.id)) {
if (wanted.has(file.id) && !found.has(file.id)) {
found.set(file.id, { file, collection });
}
}
}
return found;
};
// Each file named by `fileIDs` as the server holds it now, mapped to its record
// and collection as `cachedFilesByID` maps it. When the cache holds the
// account's collections, it refreshes and looks the files up in the cache. On
// an empty cache, it reads the collection list, then one collection's files at
// a time, and stops once it has found every file. That search stores nothing.
export const freshFilesByID = async (
lib: FreshReadLibrary,
fileIDs: number[],
): Promise<Map<number, { file: EnteFile; collection: Collection }>> => {
// Only a refresh stores collections, so a cache holding any holds them all.
if (lib.listCollections().length > 0) {
await lib.fresh();
return cachedFilesByID(lib, fileIDs);
}
const wanted = new Set(fileIDs);
const found = new Map<number, { file: EnteFile; collection: Collection }>();
const collections = await lib.freshCollections();
for (const collection of [...collections].sort(byNewestCollection)) {
if (found.size === wanted.size) break;
for (const file of await lib.freshFiles(collection)) {
if (wanted.has(file.id) && !found.has(file.id)) {
found.set(file.id, { file, collection });
}
}
}
return found;
};
+11 -36
View File
@@ -5,7 +5,7 @@ import type { Client } from "./client.js";
import type { Library } from "./library/index.js"; import type { Library } from "./library/index.js";
import { ApiError } from "./api/client.js"; import { ApiError } from "./api/client.js";
import { encryptBlob, toBase64 } from "./crypto/index.js"; import { encryptBlob, toBase64 } from "./crypto/index.js";
import type { EnteFile } from "./model/types.js"; import type { Collection, EnteFile } from "./model/types.js";
// The server refuses a thumbnail larger than the one it already records for the // The server refuses a thumbnail larger than the one it already records for the
// file (`thumbnail.size`, the encrypted size), so these encodings are tried // file (`thumbnail.size`, the encrypted size), so these encodings are tried
@@ -226,16 +226,18 @@ const encryptThumbnailWithin = (
return undefined; return undefined;
}; };
// Regenerate and upload a thumbnail for each requested file. Originals are read // Regenerate and upload a thumbnail for each requested file. `found` holds the
// through the library's content cache (`photo.original()`); the generated // record of each requested file that a collection holds, and the collection it
// thumbnail is JPEG-encoded, encrypted under the file's own key, and registered // was found in. Each original is read through the library's content cache, by
// with the server — the encrypt-and-upload path is unchanged. Each file is // `lib.originalOf(file)`. The new thumbnail is a JPEG, encrypted under the
// file's own key. It is uploaded, then registered with the server. Each file is
// resolved to one outcome (fixed / skipped / failed) and a failure on one file // resolved to one outcome (fixed / skipped / failed) and a failure on one file
// never stops the others. // never stops the others.
export const fixMissingThumbnails = async ( export const fixMissingThumbnails = async (
lib: Library, lib: Library,
client: Client, client: Client,
fileIDs: number[], fileIDs: number[],
found: Map<number, { file: EnteFile; collection: Collection }>,
onProgress?: ProgressCallback, onProgress?: ProgressCallback,
): Promise<ThumbnailFixResult[]> => { ): Promise<ThumbnailFixResult[]> => {
const log = onProgress ?? (() => {}); const log = onProgress ?? (() => {});
@@ -243,31 +245,8 @@ export const fixMissingThumbnails = async (
const api = client.getApiClient(); const api = client.getApiClient();
const { userID } = client.whoami(); const { userID } = client.whoami();
// Resolve each requested fileID to its file record and owning album by
// enumerating the library, each file taken from the first album that holds
// it. The raw `EnteFile` carries the per-file key the thumbnail is
// encrypted under, which the projected records deliberately do not.
const wanted = new Set(fileIDs);
const fileMap = new Map<
number,
{ file: EnteFile; collectionName: string }
>();
for (const album of lib.albums.list()) {
for (const photo of album.photos.list()) {
if (!wanted.has(photo.fileID) || fileMap.has(photo.fileID))
continue;
const file = lib.getFile(album.collectionID, photo.fileID);
if (file) {
fileMap.set(photo.fileID, {
file,
collectionName: album.name,
});
}
}
}
for (const fileID of fileIDs) { for (const fileID of fileIDs) {
const entry = fileMap.get(fileID); const entry = found.get(fileID);
if (!entry) { if (!entry) {
results.push({ results.push({
fileID, fileID,
@@ -279,7 +258,8 @@ export const fixMissingThumbnails = async (
continue; continue;
} }
const { file, collectionName } = entry; const { file } = entry;
const collectionName = entry.collection.name;
const title = file.metadata.title; const title = file.metadata.title;
const skipReason = reasonToSkip(file, userID); const skipReason = reasonToSkip(file, userID);
@@ -297,15 +277,10 @@ export const fixMissingThumbnails = async (
const maxSize = file.thumbnail.size!; const maxSize = file.thumbnail.size!;
try { try {
const photo = lib.photos.byID({ fileID });
if (!photo) {
throw new Error("file not present in the library cache");
}
log( log(
`[${collectionName}] Downloading ${title} for thumbnail generation...`, `[${collectionName}] Downloading ${title} for thumbnail generation...`,
); );
const { path } = await photo.original(); const { path } = await lib.originalOf(file);
const fileBytes = new Uint8Array(readFileSync(path)); const fileBytes = new Uint8Array(readFileSync(path));
if (!isJpeg(fileBytes)) { if (!isJpeg(fileBytes)) {
+1 -1
View File
@@ -418,7 +418,7 @@ describe("the first backup of the account", () => {
expect(downloads.sort()).toEqual([100, 200, 300]); expect(downloads.sort()).toEqual([100, 200, 300]);
}); });
// The tree "Backup layout" in the README describes. Each original is at // The tree under "The backup directory" in the README. Each original is at
// its save path, `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, with // its save path, `YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>`, with
// the extension of its title; the live photo is its image and its video, // the extension of its title; the live photo is its image and its video,
// each with the extension it had inside the ZIP, and a `.livephoto.json` // each with the extension it had inside the ZIP, and a `.livephoto.json`
+101 -6
View File
@@ -1353,6 +1353,23 @@ describe("helper fix-missing-thumbnails", () => {
).toBe(1); ).toBe(1);
expect(stderr.text).toBe(last); expect(stderr.text).toBe(last);
}); });
// beach.jpg (100) is in Vacation (1) and Work (2), both updated at the same
// time. The fixer names a file after the first album holding it in the
// order of `Library.albums.list()`. On a tie, the album with the higher ID
// comes first, so beach.jpg is named after Work.
it("names a photo in two albums after the same album on an empty cache as on a cache that holds the account", async () => {
const ctx = context(fakeClient({ beachInWork: true }));
const opts = { file: ["100"], json: true };
const inWork = [{ fileID: 100, collection: "Work", status: "skipped" }];
expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
expect(JSON.parse(stdout.text)).toMatchObject(inWork);
expect(await filesCommand(ctx, undefined, {})).toBe(0);
stdout.text = "";
expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
expect(JSON.parse(stdout.text)).toMatchObject(inWork);
});
}); });
describe("backup-metadata --exif", () => { describe("backup-metadata --exif", () => {
@@ -1572,11 +1589,11 @@ describe("an ID given on the command line", () => {
// An account with many albums (issues #210, #217 and #222). On an empty cache // An account with many albums (issues #210, #217 and #222). On an empty cache
// `collections`, `files` with a collection ID and `get` used to wait for a // `collections`, `files` with a collection ID and `get` used to wait for a
// refresh that fetched and decrypted the files of every album, one album after // refresh that fetched and decrypted the files of every album, one album after
// another. On such an account they seemed to hang. The account here has 40 // another. So did `helper fix-missing-thumbnails --file` (issue #223). On such
// albums of one photo each, encrypted as Ente's clients encrypt them, on the // an account they seemed to hang. The account here has 40 albums of one photo
// fake Ente server of `test/ente-server.ts`. The tests record the path and query // each, encrypted as Ente's clients encrypt them, on the fake Ente server of
// of each request quak sends. The session is the one logging in to that server // `test/ente-server.ts`. The tests record the path and query of each request
// saves. // quak sends. The session is the one logging in to that server saves.
describe("an account with many albums", () => { describe("an account with many albums", () => {
const ALBUMS = 40; const ALBUMS = 40;
const PASSWORD = "many-albums"; const PASSWORD = "many-albums";
@@ -1599,7 +1616,9 @@ describe("an account with many albums", () => {
// A photo in album `collectionID` as the server sends it, and its original // A photo in album `collectionID` as the server sends it, and its original
// encrypted as the server stores it. The original's bytes are the title. // encrypted as the server stores it. The original's bytes are the title.
// These tests fetch no thumbnail. // These tests fetch no thumbnail. The server records a thumbnail size, so
// `helper fix-missing-thumbnails` downloads the original before it skips
// the photo as not a JPEG.
const photoRow = (photo: { const photoRow = (photo: {
id: number; id: number;
collectionID: number; collectionID: number;
@@ -1638,6 +1657,7 @@ describe("an account with many albums", () => {
}, },
file: { decryptionHeader: toBase64(original.header) }, file: { decryptionHeader: toBase64(original.header) },
thumbnail: { decryptionHeader: "" }, thumbnail: { decryptionHeader: "" },
info: { thumbSize: 1000 },
updationTime: photo.updationTime, updationTime: photo.updationTime,
}; };
return { row, original: original.ciphertext }; return { row, original: original.ciphertext };
@@ -1855,6 +1875,81 @@ describe("an account with many albums", () => {
expect(requests).toEqual(["/collections/v2?sinceTime=1"]); expect(requests).toEqual(["/collections/v2?sinceTime=1"]);
}); });
// The helper searches the most recently updated album first. All 40 were
// updated at the same time, and on a tie the higher ID comes first. Photo
// 3900 is in album 39, so the helper asks for no file of album 38 or
// earlier. The last request downloads the photo's original.
it("fix-missing-thumbnails --file on an empty cache asks for the albums' files only until one holds the photo", async () => {
const opts = { file: ["3900"], json: true };
expect(await fixMissingThumbnailsCommand(accountContext(), opts)).toBe(
0,
);
const reason =
"unsupported image format (only baseline JPEG can be regenerated)";
expect(JSON.parse(stdout.text)).toEqual([
{
fileID: 3900,
title: "photo-39.jpg",
collection: "Album 39",
status: "skipped",
reason,
},
]);
expect(stderr.text).toBe(
"[Album 39] Downloading photo-39.jpg for thumbnail generation...\n" +
`[Album 39] Skipping photo-39.jpg: ${reason}\n`,
);
expect(requests).toEqual([
"/collections/v2?sinceTime=0",
"/collections/v2/diff?collectionID=40&sinceTime=0",
"/collections/v2/diff?collectionID=39&sinceTime=0",
"/?fileID=3900",
]);
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
});
// `files` first fills the cache, so the helper waits for a refresh. No
// album has changed since, so the refresh asks for no album's files.
it("fix-missing-thumbnails --file on a cache that holds the account asks for no unchanged album's files, also for a photo in no album", async () => {
const ctx = accountContext();
expect(await filesCommand(ctx, undefined, {})).toBe(0);
requests.length = 0;
stdout.text = "";
stderr.text = "";
const opts = { file: ["3900"], json: true };
expect(await fixMissingThumbnailsCommand(ctx, opts)).toBe(0);
const reason =
"unsupported image format (only baseline JPEG can be regenerated)";
expect(JSON.parse(stdout.text)).toEqual([
{
fileID: 3900,
title: "photo-39.jpg",
collection: "Album 39",
status: "skipped",
reason,
},
]);
expect(stderr.text).toBe(
"[Album 39] Downloading photo-39.jpg for thumbnail generation...\n" +
`[Album 39] Skipping photo-39.jpg: ${reason}\n`,
);
expect(requests).toEqual([
"/collections/v2?sinceTime=1",
"/?fileID=3900",
]);
requests.length = 0;
stdout.text = "";
const notFound = { file: ["999"], json: true };
expect(await fixMissingThumbnailsCommand(ctx, notFound)).toBe(1);
expect(JSON.parse(stdout.text)).toMatchObject([
{ fileID: 999, reason: "file not found in any collection" },
]);
expect(requests).toEqual(["/collections/v2?sinceTime=1"]);
});
it("files counts on stderr each album whose files its refresh fetched", async () => { it("files counts on stderr each album whose files its refresh fetched", async () => {
expect(await filesCommand(accountContext(), undefined, {})).toBe(0); expect(await filesCommand(accountContext(), undefined, {})).toBe(0);
+4 -4
View File
@@ -234,9 +234,9 @@ describe("-q", () => {
}); });
}); });
// No album holds file 999, so fixing it fails. The refresh's progress and // No album holds file 999, so fixing it fails. The summary from
// the list of failed files are left out. The line saying why the command // `--- Done ---` to the end of the `Failed files:` list is left out. The
// failed is the fatal one, and it is printed. // line saying why the command failed is the fatal one, and it is printed.
it("prints only the line a failing command exits with", async () => { it("prints only the line a failing command exits with", async () => {
const fix = ["helper", "fix-missing-thumbnails", "--file", "999"]; const fix = ["helper", "fix-missing-thumbnails", "--file", "999"];
expect(await quak("-q", ...fix)).toEqual({ expect(await quak("-q", ...fix)).toEqual({
@@ -328,7 +328,7 @@ describe("what the library reports", () => {
]); ]);
}); });
// Each operation is named in plain words, as the README names it. // Each operation is named in plain words.
it("logs a refresh's progress at info, and its start and end at debug", () => { it("logs a refresh's progress at info, and its start and end at debug", () => {
const { log, lines } = collect(); const { log, lines } = collect();
const onProgress = logRefresh(log); const onProgress = logRefresh(log);
+70
View File
@@ -13,6 +13,9 @@
* #217). Neither may wait for a refresh, which would fetch the files of every * #217). Neither may wait for a refresh, which would fetch the files of every
* collection. On an empty cache, `get` and `get-thumb` ask for that list, then * collection. On an empty cache, `get` and `get-thumb` ask for that list, then
* for one collection's files at a time, until they find the file (issue #222). * for one collection's files at a time, until they find the file (issue #222).
* `helper fix-missing-thumbnails --file` does the same until it finds every
* file it names, reading the most recently updated collection first (issue
* #223).
* *
* `collections` and `files` also list in the order the library gives them, not * `collections` and `files` also list in the order the library gives them, not
* the albums/photos projection's newest-first order. The fixtures are seeded in * the albums/photos projection's newest-first order. The fixtures are seeded in
@@ -27,6 +30,7 @@ import {
freshCollections, freshCollections,
freshFiles, freshFiles,
freshFile, freshFile,
freshFilesByID,
type FreshReadLibrary, type FreshReadLibrary,
} from "../../src/cli-read.js"; } from "../../src/cli-read.js";
import { fileListRow } from "../../src/cli-output.js"; import { fileListRow } from "../../src/cli-output.js";
@@ -253,4 +257,70 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
expect(lib.freshFilesCalls).toEqual([1, 2, 3]); expect(lib.freshFilesCalls).toEqual([1, 2, 3]);
expect(found).toBeUndefined(); expect(found).toBeUndefined();
}); });
// The server lists collections 1, 2 and 3, and collection 3 was updated
// most recently. File 11 is in collections 1 and 2.
const fileInTwoCollections = (): FakeLibrary =>
new FakeLibrary(
[collection(1, 100), collection(2, 200), collection(3, 300)],
[
file(10, 1, 1_700_000_000_000_000),
file(11, 1, 1_700_000_000_000_001),
file(11, 2, 1_700_000_000_000_001),
file(20, 2, 1_700_000_000_000_002),
file(30, 3, 1_700_000_000_000_003),
],
);
it("freshFilesByID on an empty cache reads the most recently updated collection's files first, until it has found every file", async () => {
const lib = fileInTwoCollections();
const found = await freshFilesByID(lib, [11, 20]);
expect(lib.freshCalls).toBe(0);
expect(lib.freshCollectionsCalls).toBe(1);
expect(lib.freshFilesCalls).toEqual([3, 2]);
expect(found.get(11)?.collection.id).toBe(2);
expect(found.get(11)?.file.collectionID).toBe(2);
expect(found.get(20)?.collection.id).toBe(2);
});
// Collection 1 was updated last, though collections 2 and 3 have higher
// IDs. File 11 is in collections 1 and 2.
it("freshFilesByID on an empty cache reads the most recently updated collection first, though its ID is the lowest", async () => {
const lib = new FakeLibrary(
[collection(1, 400), collection(2, 200), collection(3, 300)],
[
file(11, 1, 1_700_000_000_000_001),
file(11, 2, 1_700_000_000_000_001),
],
);
const found = await freshFilesByID(lib, [11]);
expect(lib.freshFilesCalls).toEqual([1]);
expect(found.get(11)?.collection.id).toBe(1);
expect(found.get(11)?.file.collectionID).toBe(1);
});
// The stand-in's cache holds what the server holds once `fresh()` has run.
it("freshFilesByID on a cache that holds the collections refreshes, then finds each file in the most recently updated collection that holds it", async () => {
const lib = fileInTwoCollections();
await lib.fresh();
const found = await freshFilesByID(lib, [11, 404]);
expect(lib.freshCalls).toBe(2);
expect(lib.freshCollectionsCalls).toBe(0);
expect(lib.freshFilesCalls).toEqual([]);
expect(found.get(11)?.collection.id).toBe(2);
expect(found.get(11)?.file.collectionID).toBe(2);
expect(found.has(404)).toBe(false);
});
it("freshFilesByID on an empty cache reads every collection's files when a file is in none of them", async () => {
const lib = fileInTwoCollections();
const found = await freshFilesByID(lib, [10, 404]);
expect(lib.freshFilesCalls).toEqual([3, 2, 1]);
expect(found.get(10)?.collection.id).toBe(1);
expect(found.has(404)).toBe(false);
});
}); });
+39 -9
View File
@@ -6,10 +6,12 @@
* working thumbnails, others return 404 or empty bodies. The tests * working thumbnails, others return 404 or empty bodies. The tests
* verify that the detection and repair logic handles each case correctly. * verify that the detection and repair logic handles each case correctly.
* *
* As of issue #52 both helpers take an open `Library` for enumeration and the * `listMissingThumbnails` takes an open `Library` to enumerate the files. Both
* `Client` for the API operations that stay unchanged (the thumbnail existence * helpers take the `Client` for the API operations: the thumbnail existence
* check, and the encrypt-and-upload path). `fixMissingThumbnails` reads each * check, and the encrypt-and-upload path. `fixMissingThumbnails` takes the
* original through the library's content cache (`photo.original()`). * record of each file, which these tests look up with `cachedFilesByID`. It
* reads each original through the library's content cache, by
* `lib.originalOf(file)`.
* *
* `fixMissingThumbnails` is the most complex function in quak: it * `fixMissingThumbnails` is the most complex function in quak: it
* downloads the original file, generates a JPEG thumbnail with jpeg-js, * downloads the original file, generates a JPEG thumbnail with jpeg-js,
@@ -41,6 +43,7 @@ import {
import { SRP, SrpServer } from "fast-srp-hap"; import { SRP, SrpServer } from "fast-srp-hap";
import { Client } from "../../src/client.js"; import { Client } from "../../src/client.js";
import { Library } from "../../src/library/index.js"; import { Library } from "../../src/library/index.js";
import { cachedFilesByID } from "../../src/cli-read.js";
import { import {
listMissingThumbnails, listMissingThumbnails,
fixMissingThumbnails, fixMissingThumbnails,
@@ -656,7 +659,12 @@ describe("fixMissingThumbnails", () => {
const client = await login(buildThumbFetch(fixMock)); const client = await login(buildThumbFetch(fixMock));
const lib = await openLib(client); const lib = await openLib(client);
const results = await fixMissingThumbnails(lib, client, [101]); const results = await fixMissingThumbnails(
lib,
client,
[101],
cachedFilesByID(lib, [101]),
);
lib.close(); lib.close();
expect(results.length).toBe(1); expect(results.length).toBe(1);
@@ -693,7 +701,12 @@ describe("fixMissingThumbnails", () => {
const client = await login(buildThumbFetch(fixMock)); const client = await login(buildThumbFetch(fixMock));
const lib = await openLib(client); const lib = await openLib(client);
const results = await fixMissingThumbnails(lib, client, [999]); const results = await fixMissingThumbnails(
lib,
client,
[999],
cachedFilesByID(lib, [999]),
);
lib.close(); lib.close();
expect(results.length).toBe(1); expect(results.length).toBe(1);
@@ -715,6 +728,7 @@ describe("fixMissingThumbnails", () => {
lib, lib,
client, client,
[101, 102], [101, 102],
cachedFilesByID(lib, [101, 102]),
(msg, level) => { (msg, level) => {
if (level === "error") errors.push(msg); if (level === "error") errors.push(msg);
}, },
@@ -745,6 +759,7 @@ describe("fixMissingThumbnails", () => {
lib, lib,
client, client,
[101, 103, 104], [101, 103, 104],
cachedFilesByID(lib, [101, 103, 104]),
); );
lib.close(); lib.close();
@@ -779,7 +794,12 @@ describe("fixMissingThumbnails", () => {
const client = await login(counted.fetch); const client = await login(counted.fetch);
const lib = await openLib(client); const lib = await openLib(client);
const results = await fixMissingThumbnails(lib, client, [101]); const results = await fixMissingThumbnails(
lib,
client,
[101],
cachedFilesByID(lib, [101]),
);
lib.close(); lib.close();
expect(results[0]!.status).toBe("skipped"); expect(results[0]!.status).toBe("skipped");
@@ -800,7 +820,12 @@ describe("fixMissingThumbnails", () => {
const client = await login(counted.fetch); const client = await login(counted.fetch);
const lib = await openLib(client); const lib = await openLib(client);
const results = await fixMissingThumbnails(lib, client, [101]); const results = await fixMissingThumbnails(
lib,
client,
[101],
cachedFilesByID(lib, [101]),
);
lib.close(); lib.close();
expect(results[0]!.status).toBe("skipped"); expect(results[0]!.status).toBe("skipped");
@@ -841,7 +866,12 @@ describe("fixMissingThumbnails", () => {
const client = await login(buildThumbFetch(fixMock)); const client = await login(buildThumbFetch(fixMock));
const lib = await openLib(client); const lib = await openLib(client);
const results = await fixMissingThumbnails(lib, client, [101]); const results = await fixMissingThumbnails(
lib,
client,
[101],
cachedFilesByID(lib, [101]),
);
lib.close(); lib.close();
expect(results[0]!.status).toBe("fixed"); expect(results[0]!.status).toBe("fixed");