quak backup works on up to the content pool's limit of originals at once (closes #182)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
The backup awaited each original before starting the next, so it used one of the content pool's five slots. lib.backup(), and so quak backup, now runs as many workers as the content pool's limit (5 by default, or the contentConcurrency of the pools the library was opened with), each taking the next file from one shared list, so the limit covers the --verify hashing too. Once a worker has thrown or the lock is lost, no worker takes another file; the backup waits for the files under way and checks its lock before the ML data fetch. runBackup rejects a contentConcurrency that is not a positive integer before it takes the lock. Failed files are listed in the order they failed. Model: opus-5-5
This commit was merged in pull request #192.
This commit is contained in:
@@ -909,6 +909,27 @@ describe("the backup lock", () => {
|
||||
expect(existsSync(lockPath(outDir))).toBe(false);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
// `runBackup` takes a library object from its caller. With a
|
||||
// `contentConcurrency` below 1, or one that is not a number, it would
|
||||
// back up no original and report no error. It rejects any that is not a
|
||||
// positive integer, as `BoundedPool` does, before it takes the lock,
|
||||
// which would create the download directory, and before it asks the
|
||||
// library for anything: the object here has nothing else.
|
||||
it("rejects a library whose contentConcurrency is not a positive integer, before taking the lock", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
for (const contentConcurrency of [undefined, 0, -1, 1.5, NaN]) {
|
||||
const lib = { contentConcurrency } as unknown as BackupLibrary;
|
||||
|
||||
await expect(
|
||||
runBackup(lib, { downloadDirectory: outDir }),
|
||||
).rejects.toThrow(
|
||||
`contentConcurrency must be a positive integer, got ${contentConcurrency}`,
|
||||
);
|
||||
|
||||
expect(existsSync(outDir)).toBe(false);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
// Also serves ML data: file 100 has `ML_PAYLOAD`, the other two have none.
|
||||
@@ -1491,6 +1512,7 @@ describe("backup album folders", () => {
|
||||
thumbnail: async () => {
|
||||
throw new Error("no thumbnails in this stand-in");
|
||||
},
|
||||
contentConcurrency: 5,
|
||||
fetchMLData: async () => {},
|
||||
mlData: async () => undefined,
|
||||
whoami: () => ({ email: "backup@example.com", userID: USER_ID }),
|
||||
|
||||
Reference in New Issue
Block a user