quak files COLLECTION fetches that collection's files alone (closes #217)
check / check (push) Waiting to run
check / check (push) Waiting to run
`quak files COLLECTION` used to wait for a full refresh, which on an empty cache fetched the files of every collection before it listed one. It now reads the collection list, then calls the new `Library.refreshCollection()`, which brings that one collection up to date in the store as a refresh would, and saves it. It sends no request for the files when the stored copy is current, and it keeps a newer copy that a refresh stores while it fetches. It leaves the refresh cursor alone, so the next refresh still fetches every other collection that changed. `quak files` with no argument still waits for a refresh. Model: opus-5-5
This commit is contained in:
@@ -685,17 +685,20 @@ library. The read commands — `collections`, `files`, `get`, `get-thumb`,
|
||||
`helper fix-missing-thumbnails` — force a fresh server round-trip before they
|
||||
answer, so they report current account state rather than whatever the cache last
|
||||
held. `collections` asks the server for the collection list alone, in one
|
||||
request, and neither fetches any collection's files nor changes the cache. The
|
||||
others wait for a refresh. With an empty cache, as on a first run, that refresh
|
||||
fetches the files of every collection, one collection after another, and stores
|
||||
nothing until it has them all, which on a large account takes a while. Meanwhile
|
||||
the command logs a line on stderr for each collection whose files it has
|
||||
fetched, such as `Fetched the files of 3 of 120 collections`, with `--json` too;
|
||||
`backup` does the same while the refresh it starts with runs. stdout holds only
|
||||
the command's output. If that round-trip fails, the command prints the error on
|
||||
one line and exits 1, or 3 when the server no longer accepts the saved session
|
||||
(see "Session handling"). An album or file quak cannot decrypt does not fail it;
|
||||
the command answers from what did decrypt (see "Default reads vs. fresh reads").
|
||||
request, and neither fetches any collection's files nor changes the cache.
|
||||
`files` with a collection ID asks for the collection list the same way, then for
|
||||
the files of that collection alone, and stores them in the cache. It asks for
|
||||
none of them when the cache's copy of that collection is current. The others
|
||||
wait for a refresh. With an empty cache, as on a first run, that refresh fetches
|
||||
the files of every collection, one collection after another, and stores nothing
|
||||
until it has them all, which on a large account takes a while. Meanwhile the
|
||||
command logs a line on stderr for each collection whose files it has fetched,
|
||||
such as `Fetched the files of 3 of 120 collections`, with `--json` too; `backup`
|
||||
does the same while the refresh it starts with runs. stdout holds only the
|
||||
command's output. If that round-trip fails, the command prints the error on one
|
||||
line and exits 1, or 3 when the server no longer accepts the saved session (see
|
||||
"Session handling"). An album or file quak cannot decrypt does not fail it; the
|
||||
command answers from what did decrypt (see "Default reads vs. fresh reads").
|
||||
`--cache-dir` overrides where the cache lives; without it each account gets its
|
||||
own directory under the per-user cache path.
|
||||
|
||||
@@ -1180,6 +1183,15 @@ since the stored copy, so a collection stored without its files would never get
|
||||
them. `quak collections` uses it (issue
|
||||
https://git.eeqj.de/sneak/quak/issues/210).
|
||||
|
||||
`await lib.refreshCollection(collection)` brings one collection up to date in
|
||||
the cache, given its `Collection` from `freshCollections()`. It does for that
|
||||
collection what a refresh does for a changed one, and fetches no other
|
||||
collection's files. It sends no request when the cache's copy has the same
|
||||
`updationTime`. Otherwise it fetches the collection's files changed since that
|
||||
copy, or all of them when there is none, then stores and saves them. The refresh
|
||||
cursor stays where it was, so the next refresh still fetches the files of every
|
||||
other collection that changed. `quak files` with a collection ID uses it.
|
||||
|
||||
An album or file in the server's answer that quak cannot decrypt, or whose
|
||||
metadata is not JSON, does not fail the refresh. The refresh leaves it out, an
|
||||
album with all its files, and drops any older copy the cache held, so reads
|
||||
|
||||
@@ -26,6 +26,15 @@ declares one.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-25: `quak files COLLECTION` fetches the files of that collection alone
|
||||
(https://git.eeqj.de/sneak/quak/issues/217). It used to wait for a refresh,
|
||||
which on an empty cache fetched the files of every collection first. It now
|
||||
reads the collection list through `lib.freshCollections()`, then brings that
|
||||
one collection up to date through the new `lib.refreshCollection()`. That
|
||||
stores and saves its files as a refresh would, and leaves the refresh cursor
|
||||
where it was. When the stored copy is current, it asks for none of the files.
|
||||
`quak files` with no argument still waits for a refresh.
|
||||
|
||||
- 2026-10-22: The JSON `quak backup` and `quak backup-metadata` write for a file
|
||||
holds `keywords` in place of the CLIP embedding in its ML data
|
||||
(https://git.eeqj.de/sneak/quak/issues/218). The keywords are the titles of
|
||||
|
||||
+4
-3
@@ -131,9 +131,10 @@ const readLibraryClient = (client: Client): LibraryClient => ({
|
||||
// Open a library for a single point command: the aggressive background precache
|
||||
// (issue #48) is off — a one-shot `collections` or `get` must not start
|
||||
// downloading the whole account — and no refresh starts as it opens or on a
|
||||
// timer: each command asks for the one it needs, and `collections` needs none
|
||||
// (issue #210). While a refresh runs, each collection whose files it has
|
||||
// fetched is logged, so a large account shows progress rather than nothing.
|
||||
// timer: each command asks for the one it needs, and `collections` and
|
||||
// `files COLLECTION` need none (issues #210 and #217). While a refresh runs,
|
||||
// each collection whose files it has fetched is logged, so a large account
|
||||
// shows progress rather than nothing.
|
||||
const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
|
||||
Library.open({
|
||||
client: readLibraryClient(client),
|
||||
|
||||
+25
-18
@@ -2,13 +2,17 @@
|
||||
//
|
||||
// `collections`, `files`, `get`, and `get-thumb` must answer for
|
||||
// the account's state at the moment the command runs, not for whatever the
|
||||
// local cache last happened to hold (owner amendment, issue #36). `files`,
|
||||
// `get` and `get-thumb` therefore force a server round-trip through
|
||||
// `Library.fresh()` and only then read — so a collection, file, or metadata
|
||||
// change made elsewhere is visible. `collections` asks the library for the
|
||||
// collection list alone through `Library.freshCollections()` (issue #210): a
|
||||
// refresh would also fetch the files of every collection, which on a large
|
||||
// account takes long and which `collections` does not print.
|
||||
// local cache last happened to hold (owner amendment, issue #36). `files`
|
||||
// without a collection ID, `get` and `get-thumb` therefore force a server
|
||||
// round-trip through `Library.fresh()` and only then read — so a collection,
|
||||
// file, or metadata change made elsewhere is visible. `collections` asks the
|
||||
// library for the collection list alone through `Library.freshCollections()`
|
||||
// (issue #210): a refresh would also fetch the files of every collection, which
|
||||
// on a large account takes long and which `collections` does not print. For the
|
||||
// same reason, `files COLLECTION` reads the collection list the same way and
|
||||
// then brings that one collection up to date through
|
||||
// `Library.refreshCollection()` (issue #217), fetching no other collection's
|
||||
// files.
|
||||
//
|
||||
// `collections` lists in the server's order and `files` in the library's own
|
||||
// enumeration order (`listFiles()`), the orders the pre-library CLI printed,
|
||||
@@ -24,12 +28,12 @@ import type { Photo, PhotosAPI } from "./library/index.js";
|
||||
|
||||
// The slice of `Library` these helpers read. `Library` satisfies it
|
||||
// structurally; a test can drive them with a stand-in that records the
|
||||
// `fresh()` and `freshCollections()` calls and serves records in a known
|
||||
// enumeration order.
|
||||
// `fresh()`, `freshCollections()` and `refreshCollection()` calls and serves
|
||||
// records in a known enumeration order.
|
||||
export interface FreshReadLibrary {
|
||||
fresh(): Promise<unknown>;
|
||||
freshCollections(): Promise<Collection[]>;
|
||||
getCollection(id: number): Collection | undefined;
|
||||
refreshCollection(collection: Collection): Promise<void>;
|
||||
listCollections(): Collection[];
|
||||
listFiles(collectionID: number): EnteFile[];
|
||||
getFileByID(fileID: number): EnteFile | undefined;
|
||||
@@ -42,21 +46,24 @@ export const freshCollections = (
|
||||
lib: FreshReadLibrary,
|
||||
): Promise<Collection[]> => lib.freshCollections();
|
||||
|
||||
// The files of one collection, current as of a forced refresh, in enumeration
|
||||
// order. `undefined` (not an empty list) when the collection does not exist, so
|
||||
// The files of one collection, in enumeration order, current as of one
|
||||
// collections request and the requests that bring that collection alone up to
|
||||
// date. `undefined` (not an empty list) when the collection does not exist, so
|
||||
// the caller can tell "no such collection" from "an empty collection". Without
|
||||
// a collection ID, every file in the account, each once, as held by the first
|
||||
// collection that holds it.
|
||||
// a collection ID, every file in the account, current as of a forced refresh,
|
||||
// each once, as held by the first collection that holds it.
|
||||
export const freshFiles = async (
|
||||
lib: FreshReadLibrary,
|
||||
collectionID?: number,
|
||||
): Promise<EnteFile[] | undefined> => {
|
||||
await lib.fresh();
|
||||
if (collectionID !== undefined) {
|
||||
return lib.getCollection(collectionID)
|
||||
? lib.listFiles(collectionID)
|
||||
: undefined;
|
||||
const collections = await lib.freshCollections();
|
||||
const collection = collections.find((c) => c.id === collectionID);
|
||||
if (!collection) return undefined;
|
||||
await lib.refreshCollection(collection);
|
||||
return lib.listFiles(collectionID);
|
||||
}
|
||||
await lib.fresh();
|
||||
const listed = new Set<number>();
|
||||
const files: EnteFile[] = [];
|
||||
for (const collection of lib.listCollections()) {
|
||||
|
||||
+51
-7
@@ -20,9 +20,12 @@
|
||||
//
|
||||
// `freshCollections()` is the fresh read of the collection list alone (issue
|
||||
// #210): one collections request, no collection's files, and nothing stored,
|
||||
// for a caller such as `quak collections` that needs no more. A caller that
|
||||
// asks for each refresh it needs can also open the library with
|
||||
// `refreshOnOpen: false`, so that `open()` starts none.
|
||||
// for a caller such as `quak collections` that needs no more.
|
||||
// `refreshCollection()` brings one collection from that list up to date in the
|
||||
// store (issue #217): it fetches that collection's files alone, as a refresh
|
||||
// would, for a caller such as `quak files COLLECTION`. A caller that asks for
|
||||
// each refresh it needs can also open the library with `refreshOnOpen: false`,
|
||||
// so that `open()` starts none.
|
||||
//
|
||||
// A refresh reports through `onProgress` each collection whose files it has
|
||||
// fetched, so a large account, whose first refresh fetches the files of every
|
||||
@@ -625,6 +628,39 @@ export class Library {
|
||||
return collections;
|
||||
}
|
||||
|
||||
// Bring one collection up to date in the store (issue #217), given its
|
||||
// record from `freshCollections()`. It does for that collection what a
|
||||
// refresh does for a changed one, and fetches no other collection's files.
|
||||
// It sends no request when the stored copy's `updationTime` is as new.
|
||||
// Otherwise it fetches the files changed since the stored copy, or all of
|
||||
// them when none is stored, then commits and saves them. A refresh already
|
||||
// running is waited for first.
|
||||
async refreshCollection(collection: Collection): Promise<void> {
|
||||
if (this.closed) throw new Error("the library is closed");
|
||||
await this.cycle?.catch(() => {});
|
||||
const known = this.store.getCollection(collection.id);
|
||||
if (known && collection.updationTime <= known.updationTime) return;
|
||||
const filePage = await this.client.filesSince({
|
||||
collectionID: collection.id,
|
||||
collectionKey: collection.key,
|
||||
sinceTime: known ? known.updationTime : 0,
|
||||
});
|
||||
// A refresh that ran during the fetch may have stored a copy as new as
|
||||
// `collection` or newer. Committing this result would replace it.
|
||||
const stored = this.store.getCollection(collection.id);
|
||||
if (stored && collection.updationTime <= stored.updationTime) return;
|
||||
// The stored cursor stays where it is, so the next refresh still
|
||||
// fetches the files of every other collection that changed.
|
||||
await this.commit(
|
||||
{
|
||||
collections: [collection],
|
||||
deleted: [],
|
||||
cursor: this.store.collectionsSinceTime,
|
||||
},
|
||||
[{ collectionID: collection.id, page: filePage }],
|
||||
);
|
||||
}
|
||||
|
||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||
// library's, each original at its save path, with a durable failure
|
||||
// ledger (issue #51). Takes the lock in that directory first, unless the
|
||||
@@ -766,10 +802,7 @@ export class Library {
|
||||
|
||||
// Fetch every change since the stored cursor, then commit. All network
|
||||
// reads happen before any store mutation, so a fetch that throws leaves the
|
||||
// store untouched and the previous snapshot intact. A collection or file
|
||||
// that does not decrypt is left out instead, as if deleted, with the reason
|
||||
// kept in the store: the cursor moves past it, so no later refresh sees it
|
||||
// again until the server's copy changes.
|
||||
// store untouched and the previous snapshot intact.
|
||||
private async refreshOnce(): Promise<void> {
|
||||
const page = await this.client.collectionsSince({
|
||||
sinceTime: this.store.collectionsSinceTime,
|
||||
@@ -802,6 +835,17 @@ export class Library {
|
||||
});
|
||||
}
|
||||
|
||||
await this.commit(page, filePages);
|
||||
}
|
||||
|
||||
// Commit a collections page and the file pages fetched for its collections
|
||||
// to the store, then save. A collection or file that does not decrypt is
|
||||
// left out, as if deleted, with the reason kept in the store, and no later
|
||||
// refresh sees it again until the server's copy changes.
|
||||
private async commit(
|
||||
page: CollectionsPage,
|
||||
filePages: { collectionID: number; page: FilesPage }[],
|
||||
): Promise<void> {
|
||||
// Network work done; commit to the store and persist only if something
|
||||
// actually changed.
|
||||
let changed = false;
|
||||
|
||||
+52
-26
@@ -535,9 +535,7 @@ describe("files", () => {
|
||||
|
||||
it("exits 1 for an unknown collection", async () => {
|
||||
expect(await filesCommand(context(), "9", {})).toBe(1);
|
||||
expect(stderr.text).toBe(
|
||||
FETCHED_BOTH_ALBUMS + "Collection 9 not found\n",
|
||||
);
|
||||
expect(stderr.text).toBe("Collection 9 not found\n");
|
||||
});
|
||||
|
||||
// Without a collection ID it lists every file in the account once (issue
|
||||
@@ -1389,9 +1387,9 @@ describe("backup-metadata --exif", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// Each test first runs `files` so the cache holds the account as it was, then
|
||||
// changes the server under it. (`collections` would not do: it reads the
|
||||
// collection list alone and stores nothing.)
|
||||
// Each test first runs `files 1` so the cache holds Vacation as it was, then
|
||||
// adds a file to Vacation on the server. (`collections` would not do: it reads
|
||||
// the collection list alone and stores nothing.)
|
||||
describe("backup-metadata and the thumbnail helpers refresh first", () => {
|
||||
beforeEach(async () => {
|
||||
expect(await filesCommand(context(), "1", {})).toBe(0);
|
||||
@@ -1580,10 +1578,11 @@ describe("an ID given on the command line", () => {
|
||||
// An account with many albums (issue #210). On an empty cache `collections`
|
||||
// used to wait for a refresh that fetched and decrypted the files of every
|
||||
// album, one album after another, before it printed anything, so on such an
|
||||
// account it seemed to hang. The account here has 40 albums of one photo each,
|
||||
// encrypted as Ente's clients encrypt them, on the fake Ente server of
|
||||
// `test/ente-server.ts`, which also records the path of each request quak
|
||||
// sends. The session is the one logging in to that server saves.
|
||||
// account it seemed to hang. `files` with an album's ID did the same before it
|
||||
// listed that album (issue #217). The account here has 40 albums of one photo
|
||||
// each, encrypted as Ente's clients encrypt them, on the fake Ente server of
|
||||
// `test/ente-server.ts`, which also records the path and query of each request
|
||||
// quak sends. The session is the one logging in to that server saves.
|
||||
describe("an account with many albums", () => {
|
||||
const ALBUMS = 40;
|
||||
const PASSWORD = "many-albums";
|
||||
@@ -1664,7 +1663,8 @@ describe("an account with many albums", () => {
|
||||
: input instanceof URL
|
||||
? input.href
|
||||
: input.url;
|
||||
paths.push(new URL(url).pathname);
|
||||
const { pathname, search } = new URL(url);
|
||||
paths.push(pathname + search);
|
||||
return ente(input, init);
|
||||
}) as typeof globalThis.fetch;
|
||||
const client = await Client.login({
|
||||
@@ -1698,7 +1698,7 @@ describe("an account with many albums", () => {
|
||||
albums += `${id}\talbum\tAlbum ${id}\n`;
|
||||
expect(stdout.text).toBe(albums);
|
||||
expect(stderr.text).toBe("");
|
||||
expect(paths).toEqual(["/collections/v2"]);
|
||||
expect(paths).toEqual(["/collections/v2?sinceTime=0"]);
|
||||
|
||||
// It stored nothing that would make a refresh skip an album's files:
|
||||
// `files` after it still finds the album's photo.
|
||||
@@ -1707,33 +1707,59 @@ describe("an account with many albums", () => {
|
||||
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
||||
});
|
||||
|
||||
it("files counts on stderr each album whose files its refresh fetched", async () => {
|
||||
it("files fetches the files of the album it lists and of no other", async () => {
|
||||
expect(await filesCommand(accountContext(), "7", {})).toBe(0);
|
||||
|
||||
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
||||
expect(stderr.text).toBe("");
|
||||
expect(paths).toEqual([
|
||||
"/collections/v2?sinceTime=0",
|
||||
"/collections/v2/diff?collectionID=7&sinceTime=0",
|
||||
]);
|
||||
});
|
||||
|
||||
// The first listing stored the album, so the second finds its stored copy
|
||||
// current and asks for no album's files.
|
||||
it("files run again for the same album sends no request for its files", async () => {
|
||||
const ctx = accountContext();
|
||||
expect(await filesCommand(ctx, "7", {})).toBe(0);
|
||||
stdout.text = "";
|
||||
paths.length = 0;
|
||||
|
||||
expect(await filesCommand(ctx, "7", {})).toBe(0);
|
||||
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
||||
expect(paths).toEqual(["/collections/v2?sinceTime=0"]);
|
||||
});
|
||||
|
||||
it("files with no album's ID counts on stderr each album whose files its refresh fetched", async () => {
|
||||
expect(await filesCommand(accountContext(), undefined, {})).toBe(0);
|
||||
|
||||
let photos = "";
|
||||
let progress = "";
|
||||
for (let n = 1; n <= ALBUMS; n++) {
|
||||
photos += `${n * 100}\timage\tphoto-${n}.jpg\n`;
|
||||
progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`;
|
||||
}
|
||||
expect(stdout.text).toBe(photos);
|
||||
expect(stderr.text).toBe(progress);
|
||||
});
|
||||
|
||||
// `--json` shapes stdout only, so the refresh's progress is still counted
|
||||
// on stderr.
|
||||
it("files --json counts on stderr each album whose files its refresh fetched", async () => {
|
||||
expect(await filesCommand(accountContext(), "7", { json: true })).toBe(
|
||||
0,
|
||||
);
|
||||
it("files --json with no album's ID counts on stderr each album whose files its refresh fetched", async () => {
|
||||
expect(
|
||||
await filesCommand(accountContext(), undefined, { json: true }),
|
||||
).toBe(0);
|
||||
|
||||
expect(JSON.parse(stdout.text)).toEqual([
|
||||
{
|
||||
id: 700,
|
||||
title: "photo-7.jpg",
|
||||
fileType: "image",
|
||||
creationTime: 1000,
|
||||
collectionID: 7,
|
||||
},
|
||||
]);
|
||||
const rows = JSON.parse(stdout.text);
|
||||
expect(rows).toHaveLength(ALBUMS);
|
||||
expect(rows[6]).toEqual({
|
||||
id: 700,
|
||||
title: "photo-7.jpg",
|
||||
fileType: "image",
|
||||
creationTime: 1000,
|
||||
collectionID: 7,
|
||||
});
|
||||
let progress = "";
|
||||
for (let n = 1; n <= ALBUMS; n++) {
|
||||
progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`;
|
||||
|
||||
+34
-19
@@ -3,13 +3,16 @@
|
||||
* issue #36, issue #52).
|
||||
*
|
||||
* The `collections`, `files`, `get`, and `get-thumb` commands must answer for
|
||||
* current server state, not the local cache. `files`, `get` and `get-thumb`
|
||||
* force a `Library.fresh()` round-trip before they read; the stand-in library
|
||||
* below serves nothing until `fresh()` has been awaited, so a helper that read
|
||||
* without refreshing would come back empty and fail here. `collections` asks
|
||||
* the library for the collection list alone (`freshCollections()`, issue
|
||||
* #210), and must not wait for a refresh, which would fetch the files of every
|
||||
* collection.
|
||||
* current server state, not the local cache. `files` without a collection ID,
|
||||
* `get` and `get-thumb` force a `Library.fresh()` round-trip before they read;
|
||||
* the stand-in library below serves nothing until `fresh()` has been awaited,
|
||||
* so a helper that read without refreshing would come back empty and fail
|
||||
* here. `collections` asks the library for the collection list alone
|
||||
* (`freshCollections()`, issue #210), and must not wait for a refresh, which
|
||||
* would fetch the files of every collection. `files COLLECTION` reads that list
|
||||
* too and then brings that one collection up to date
|
||||
* (`refreshCollection()`, issue #217); the stand-in serves a collection's files
|
||||
* once either has run.
|
||||
*
|
||||
* `collections` and `files` also list in the library's enumeration order
|
||||
* (`listCollections`/`listFiles`) — the order the pre-library CLI printed — not
|
||||
@@ -64,12 +67,14 @@ const file = (
|
||||
|
||||
// A library that reveals its records only after `fresh()` has been awaited, and
|
||||
// serves them in the enumeration order it was given. `freshCollections()`
|
||||
// answers the collection list in that order without a refresh. `photos.byID`
|
||||
// returns a stand-in `Photo` carrying just the fileID the helper passes
|
||||
// through.
|
||||
// answers the collection list in that order without a refresh.
|
||||
// `refreshCollection()` reveals the files of that one collection.
|
||||
// `photos.byID` returns a stand-in `Photo` carrying just the fileID the helper
|
||||
// passes through.
|
||||
class FakeLibrary implements FreshReadLibrary {
|
||||
freshCalls = 0;
|
||||
freshCollectionsCalls = 0;
|
||||
refreshCollectionIDs: number[] = [];
|
||||
private refreshed = false;
|
||||
|
||||
constructor(
|
||||
@@ -88,9 +93,8 @@ class FakeLibrary implements FreshReadLibrary {
|
||||
return this.collections;
|
||||
}
|
||||
|
||||
getCollection(id: number): Collection | undefined {
|
||||
if (!this.refreshed) return undefined;
|
||||
return this.collections.find((c) => c.id === id);
|
||||
async refreshCollection(collection: Collection): Promise<void> {
|
||||
this.refreshCollectionIDs.push(collection.id);
|
||||
}
|
||||
|
||||
listCollections(): Collection[] {
|
||||
@@ -98,7 +102,9 @@ class FakeLibrary implements FreshReadLibrary {
|
||||
}
|
||||
|
||||
listFiles(collectionID: number): EnteFile[] {
|
||||
return this.refreshed
|
||||
const current =
|
||||
this.refreshed || this.refreshCollectionIDs.includes(collectionID);
|
||||
return current
|
||||
? this.files.filter((f) => f.collectionID === collectionID)
|
||||
: [];
|
||||
}
|
||||
@@ -138,18 +144,25 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
||||
expect(rows.map((c) => c.id)).not.toEqual(newestFirst);
|
||||
});
|
||||
|
||||
it("freshFiles refreshes first, lists in enumeration order, keeps raw fields", async () => {
|
||||
it("freshFiles brings the collection alone up to date, lists in enumeration order, keeps raw fields", async () => {
|
||||
// Enumeration order by id 10, 11, 12; creationTimes ascending, so a
|
||||
// newest-first sort would reverse them.
|
||||
// newest-first sort would reverse them. Collection 2's file is not
|
||||
// listed.
|
||||
const files = [
|
||||
file(10, 1, 1_700_000_000_000_000),
|
||||
file(11, 1, 1_700_000_000_000_001),
|
||||
file(12, 1, 1_700_000_000_000_002),
|
||||
file(20, 2, 1_700_000_000_000_003),
|
||||
];
|
||||
const lib = new FakeLibrary([collection(1, 100)], files);
|
||||
const lib = new FakeLibrary(
|
||||
[collection(1, 100), collection(2, 100)],
|
||||
files,
|
||||
);
|
||||
|
||||
const rows = await freshFiles(lib, 1);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(lib.freshCollectionsCalls).toBe(1);
|
||||
expect(lib.refreshCollectionIDs).toEqual([1]);
|
||||
expect(lib.freshCalls).toBe(0);
|
||||
expect(rows?.map((f) => f.id)).toEqual([10, 11, 12]);
|
||||
|
||||
// Field values come from raw metadata: microsecond creationTime and the
|
||||
@@ -206,7 +219,9 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
||||
it("freshFiles returns undefined for an unknown collection", async () => {
|
||||
const lib = new FakeLibrary([collection(1, 100)], []);
|
||||
const rows = await freshFiles(lib, 999);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(lib.freshCollectionsCalls).toBe(1);
|
||||
expect(lib.refreshCollectionIDs).toEqual([]);
|
||||
expect(lib.freshCalls).toBe(0);
|
||||
expect(rows).toBeUndefined();
|
||||
});
|
||||
|
||||
|
||||
@@ -17,6 +17,10 @@
|
||||
* 4. `freshCollections()` reads the collection list alone, with one request
|
||||
* and no collection's files, and stores nothing, so the next refresh still
|
||||
* fetches the files of every collection (issue #210).
|
||||
* 5. `refreshCollection()` brings one collection up to date in the store and
|
||||
* leaves the stored cursor where it was, so the next refresh still fetches
|
||||
* the files of every other collection that changed (issue #217). A newer
|
||||
* copy that a refresh stores while it fetches the files is kept.
|
||||
*
|
||||
* The client is the same metadata-only mock the background-refresh tests use:
|
||||
* no crypto, no network, scripted pages, and a record of each call. A long
|
||||
@@ -340,3 +344,132 @@ describe("Library.freshCollections", () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("Library.refreshCollection", () => {
|
||||
let dir: string;
|
||||
let cacheDirectory: string;
|
||||
|
||||
beforeEach(() => {
|
||||
dir = mkdtempSync(join(tmpdir(), "quak-refresh-collection-"));
|
||||
cacheDirectory = join(dir, "cache");
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// Collections 1 and 2 are stored at updationTime 100, with a stored cursor
|
||||
// of 100, and then both change on the server.
|
||||
it("fetches the files of one changed collection and leaves the others to the next refresh", async () => {
|
||||
const client = new MockClient();
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 100), collection(2, 100)],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
client.filesFor(1, {
|
||||
files: [file(1001, 1, 90)],
|
||||
deleted: [],
|
||||
cursor: 90,
|
||||
});
|
||||
client.filesFor(2, {
|
||||
files: [file(2001, 2, 90)],
|
||||
deleted: [],
|
||||
cursor: 90,
|
||||
});
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: SLOW_INTERVAL,
|
||||
});
|
||||
try {
|
||||
client.filesCalls.length = 0;
|
||||
client.filesFor(1, {
|
||||
files: [file(1002, 1, 110)],
|
||||
deleted: [],
|
||||
cursor: 110,
|
||||
});
|
||||
client.filesFor(2, {
|
||||
files: [file(2002, 2, 120)],
|
||||
deleted: [],
|
||||
cursor: 120,
|
||||
});
|
||||
|
||||
// Collection 1's files alone, since the stored copy.
|
||||
await lib.refreshCollection(collection(1, 110));
|
||||
expect(client.filesCalls).toEqual([
|
||||
{ collectionID: 1, sinceTime: 100 },
|
||||
]);
|
||||
expect(lib.listFiles(1).map((f) => f.id)).toEqual([1001, 1002]);
|
||||
expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001]);
|
||||
|
||||
// The stored copy is now current, so no request.
|
||||
await lib.refreshCollection(collection(1, 110));
|
||||
expect(client.filesCalls).toHaveLength(1);
|
||||
|
||||
// The next refresh asks from the stored cursor, 100, so it still
|
||||
// sees collection 2 changed and fetches its files.
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 110), collection(2, 120)],
|
||||
deleted: [],
|
||||
cursor: 120,
|
||||
});
|
||||
await lib.fresh();
|
||||
expect(client.collectionsSinceTimes).toEqual([0, 100]);
|
||||
expect(client.filesCalls).toEqual([
|
||||
{ collectionID: 1, sinceTime: 100 },
|
||||
{ collectionID: 2, sinceTime: 100 },
|
||||
]);
|
||||
expect(lib.listFiles(2).map((f) => f.id)).toEqual([2001, 2002]);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// Collection 1 is stored at 100. `refreshCollection()` asks for it at 110,
|
||||
// which adds file 1002. While that files request waits, a refresh stores
|
||||
// it at 120, where 1002 is deleted again and 1003 added.
|
||||
it("keeps a newer copy that a refresh stores while the files are fetched", async () => {
|
||||
const { client, lib } = await openSeeded(cacheDirectory);
|
||||
let release!: () => void;
|
||||
const held = new Promise<void>((r) => (release = r));
|
||||
try {
|
||||
client.filesFor(
|
||||
1,
|
||||
{ files: [file(1002, 1, 110)], deleted: [], cursor: 110 },
|
||||
{ files: [file(1003, 1, 120)], deleted: [1002], cursor: 120 },
|
||||
);
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 120)],
|
||||
deleted: [],
|
||||
cursor: 120,
|
||||
});
|
||||
|
||||
// Hold the next files request, refreshCollection's, until the
|
||||
// test releases it. The refresh's request after it is answered
|
||||
// at once.
|
||||
let fetchStarted!: () => void;
|
||||
const started = new Promise<void>((r) => (fetchStarted = r));
|
||||
const realFilesSince = client.filesSince.bind(client);
|
||||
client.filesSince = async (args) => {
|
||||
client.filesSince = realFilesSince;
|
||||
const page = await realFilesSince(args);
|
||||
fetchStarted();
|
||||
await held;
|
||||
return page;
|
||||
};
|
||||
|
||||
const refreshing = lib.refreshCollection(collection(1, 110));
|
||||
await started;
|
||||
await lib.fresh();
|
||||
release();
|
||||
await refreshing;
|
||||
|
||||
expect(lib.getCollection(1)?.updationTime).toBe(120);
|
||||
expect(lib.listFiles(1).map((f) => f.id)).toEqual([1001, 1003]);
|
||||
} finally {
|
||||
release();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user