quak files COLLECTION fetches that collection's files alone (closes #217)
check / check (push) Waiting to run
check / check (push) Waiting to run
`quak files` with a collection ID used to wait for a full refresh, which on an empty cache fetched the files of every collection in the account. It now reads the collection list, then that one collection's files through the new `Library.freshFiles()`, and stores nothing. The files print in the server's order, and no refresh progress lines appear. Each file prints once, as the latest copy the server sent. `quak files` without a collection ID still waits for a refresh. Some tests ran `quak files` with a collection ID to fill the cache, to show a refresh's progress or to show what `-q` hides. They now run a command that still refreshes. Model: opus-5-5
This commit was merged in pull request #225.
This commit is contained in:
+12
-11
@@ -128,12 +128,13 @@ const readLibraryClient = (client: Client): LibraryClient => ({
|
||||
contentSource: () => client.contentSource(),
|
||||
});
|
||||
|
||||
// Open a library for a single point command: the aggressive background precache
|
||||
// (issue #48) is off — a one-shot `collections` or `get` must not start
|
||||
// downloading the whole account — and no refresh starts as it opens or on a
|
||||
// timer: each command asks for the one it needs, and `collections` needs none
|
||||
// (issue #210). While a refresh runs, each collection whose files it has
|
||||
// fetched is logged, so a large account shows progress rather than nothing.
|
||||
// Open a library for a single point command. The aggressive background
|
||||
// precache (issue #48) is off, because a one-shot `collections` or `get` must
|
||||
// not start downloading the whole account. No refresh starts as the library
|
||||
// opens or on a timer. Each command asks for the refresh it needs.
|
||||
// `collections` and `files` with a collection ID need none (issues #210 and
|
||||
// #217). While a refresh runs, each collection whose files it has fetched is
|
||||
// logged, so a large account shows progress rather than nothing.
|
||||
const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
|
||||
Library.open({
|
||||
client: readLibraryClient(client),
|
||||
@@ -282,11 +283,11 @@ export const filesCommand = async (
|
||||
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52). Each file prints from its own decrypted
|
||||
// metadata (raw title, microsecond creationTime) via cli-output, and in
|
||||
// the pre-library CLI's enumeration order, not the projection's
|
||||
// newest-first order.
|
||||
// Ask the server for current state (issue #36 amendment, issue #52).
|
||||
// Each file prints from its own decrypted metadata (raw title,
|
||||
// microsecond creationTime) via cli-output. The files print in the
|
||||
// order `freshFiles` returns them, not the projection's newest-first
|
||||
// order.
|
||||
const files = await freshFiles(lib, collectionID);
|
||||
if (!files) {
|
||||
ctx.log("fatal", `Collection ${collectionID} not found`);
|
||||
|
||||
+27
-26
@@ -2,34 +2,35 @@
|
||||
//
|
||||
// `collections`, `files`, `get`, and `get-thumb` must answer for
|
||||
// the account's state at the moment the command runs, not for whatever the
|
||||
// local cache last happened to hold (owner amendment, issue #36). `files`,
|
||||
// `get` and `get-thumb` therefore force a server round-trip through
|
||||
// `Library.fresh()` and only then read — so a collection, file, or metadata
|
||||
// change made elsewhere is visible. `collections` asks the library for the
|
||||
// collection list alone through `Library.freshCollections()` (issue #210): a
|
||||
// refresh would also fetch the files of every collection, which on a large
|
||||
// account takes long and which `collections` does not print.
|
||||
// local cache last happened to hold (owner amendment, issue #36). `get`,
|
||||
// `get-thumb` and `files` without a collection ID therefore force a server
|
||||
// round-trip through `Library.fresh()` and only then read, so a change made
|
||||
// elsewhere is visible. `collections` asks the library for the collection list
|
||||
// alone through `Library.freshCollections()` (issue #210): a refresh would also
|
||||
// fetch the files of every collection, which on a large account takes long and
|
||||
// which `collections` does not print. `files` with a collection ID reads that
|
||||
// list too, then that one collection's files through `Library.freshFiles()`
|
||||
// (issue #217).
|
||||
//
|
||||
// `collections` lists in the server's order and `files` in the library's own
|
||||
// enumeration order (`listFiles()`), the orders the pre-library CLI printed,
|
||||
// rather than the `albums`/`photos` projection's newest-first order, which
|
||||
// re-sorts the rows. The field values still come from each record's raw
|
||||
// metadata via `cli-output.ts`. `files` without a collection ID lists every
|
||||
// file in the account once (issue #211): the library keeps a record of a file
|
||||
// for each collection that holds it, and only the first collection's record is
|
||||
// listed.
|
||||
// `collections` and `files` with a collection ID list in the server's order.
|
||||
// `files` without one lists in the library's own enumeration order
|
||||
// (`listFiles()`). Neither command uses the `albums`/`photos` projection's
|
||||
// newest-first order, which re-sorts the rows. The field values still come
|
||||
// from each record's raw metadata via `cli-output.ts`. `files` without a
|
||||
// collection ID lists every file in the account once (issue #211): the library
|
||||
// keeps a record of a file for each collection that holds it, and only the
|
||||
// first collection's record is listed.
|
||||
|
||||
import type { Collection, EnteFile } from "./model/types.js";
|
||||
import type { Photo, PhotosAPI } from "./library/index.js";
|
||||
|
||||
// The slice of `Library` these helpers read. `Library` satisfies it
|
||||
// structurally; a test can drive them with a stand-in that records the
|
||||
// `fresh()` and `freshCollections()` calls and serves records in a known
|
||||
// enumeration order.
|
||||
// structurally; a test can drive them with a stand-in that records each fresh
|
||||
// read and serves records in a known enumeration order.
|
||||
export interface FreshReadLibrary {
|
||||
fresh(): Promise<unknown>;
|
||||
freshCollections(): Promise<Collection[]>;
|
||||
getCollection(id: number): Collection | undefined;
|
||||
freshFiles(collection: Collection): Promise<EnteFile[]>;
|
||||
listCollections(): Collection[];
|
||||
listFiles(collectionID: number): EnteFile[];
|
||||
getFileByID(fileID: number): EnteFile | undefined;
|
||||
@@ -42,21 +43,21 @@ export const freshCollections = (
|
||||
lib: FreshReadLibrary,
|
||||
): Promise<Collection[]> => lib.freshCollections();
|
||||
|
||||
// The files of one collection, current as of a forced refresh, in enumeration
|
||||
// The files of one collection, as the server holds them now, in the server's
|
||||
// order. `undefined` (not an empty list) when the collection does not exist, so
|
||||
// the caller can tell "no such collection" from "an empty collection". Without
|
||||
// a collection ID, every file in the account, each once, as held by the first
|
||||
// collection that holds it.
|
||||
// a collection ID, every file in the account as of a forced refresh, each once,
|
||||
// as held by the first collection that holds it.
|
||||
export const freshFiles = async (
|
||||
lib: FreshReadLibrary,
|
||||
collectionID?: number,
|
||||
): Promise<EnteFile[] | undefined> => {
|
||||
await lib.fresh();
|
||||
if (collectionID !== undefined) {
|
||||
return lib.getCollection(collectionID)
|
||||
? lib.listFiles(collectionID)
|
||||
: undefined;
|
||||
const collections = await lib.freshCollections();
|
||||
const collection = collections.find((c) => c.id === collectionID);
|
||||
return collection ? lib.freshFiles(collection) : undefined;
|
||||
}
|
||||
await lib.fresh();
|
||||
const listed = new Set<number>();
|
||||
const files: EnteFile[] = [];
|
||||
for (const collection of lib.listCollections()) {
|
||||
|
||||
+22
-3
@@ -20,9 +20,10 @@
|
||||
//
|
||||
// `freshCollections()` is the fresh read of the collection list alone (issue
|
||||
// #210): one collections request, no collection's files, and nothing stored,
|
||||
// for a caller such as `quak collections` that needs no more. A caller that
|
||||
// asks for each refresh it needs can also open the library with
|
||||
// `refreshOnOpen: false`, so that `open()` starts none.
|
||||
// for a caller such as `quak collections` that needs no more. `freshFiles()`
|
||||
// is the fresh read of one collection's files, and stores nothing either
|
||||
// (issue #217). A caller that asks for each refresh it needs can also open the
|
||||
// library with `refreshOnOpen: false`, so that `open()` starts none.
|
||||
//
|
||||
// A refresh reports through `onProgress` each collection whose files it has
|
||||
// fetched, so a large account, whose first refresh fetches the files of every
|
||||
@@ -625,6 +626,24 @@ export class Library {
|
||||
return collections;
|
||||
}
|
||||
|
||||
// The fresh read of one collection's files (issue #217): every live file
|
||||
// in it that decrypts, in the server's order. `collection` is a record
|
||||
// `freshCollections()` returned, whose key decrypts the files. It fetches
|
||||
// no other collection's files. Like `freshCollections()`, it stores
|
||||
// nothing. It returns each file once, as the latest copy the server sent,
|
||||
// where the first copy was.
|
||||
async freshFiles(collection: Collection): Promise<EnteFile[]> {
|
||||
if (this.closed) throw new Error("the library is closed");
|
||||
const { files } = await this.client.filesSince({
|
||||
collectionID: collection.id,
|
||||
collectionKey: collection.key,
|
||||
sinceTime: 0,
|
||||
});
|
||||
const byID = new Map<number, EnteFile>();
|
||||
for (const file of files) byID.set(file.id, file);
|
||||
return [...byID.values()];
|
||||
}
|
||||
|
||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||
// library's, each original at its save path, with a durable failure
|
||||
// ledger (issue #51). Takes the lock in that directory first, unless the
|
||||
|
||||
Reference in New Issue
Block a user