quak backup works on up to the content pool's limit of originals at once (closes #182)
check / check (push) Waiting to run

The backup awaited each original before starting the next, so it used
one of the content pool's five slots. lib.backup(), and so quak backup,
now runs as many workers as the content pool's limit (5 by default, or
the contentConcurrency of the pools the library was opened with), each
taking the next file from one shared list, so the limit covers the
--verify hashing too. Once a worker has thrown or the lock is lost, no
worker takes another file; the backup waits for the files under way and
checks its lock before the ML data fetch. runBackup rejects a
contentConcurrency that is not a positive integer before it takes the
lock. Failed files are listed in the order they failed.

Model: opus-5-5
This commit was merged in pull request #192.
This commit is contained in:
2026-10-14 09:06:59 +02:00
parent 22ee2db479
commit 3412e42af5
7 changed files with 763 additions and 83 deletions
+10 -7
View File
@@ -704,10 +704,11 @@ describe("a backup whose lock is lost", () => {
}
};
// beach.jpg, whose download was under way, is stored; no other file is
// started. `quak backup` prints why as one line, with no stack trace, and
// exits 1.
it("stops quak backup before its next file when backup.lock is removed, and exits 1", async () => {
// The backup downloads up to five originals at once, so it has started
// all three when the lock is lost, and beach.jpg is stored. The backup
// then stops before it fetches ML data. `quak backup` prints why as one
// line, with no stack trace, and exits 1.
it("stops quak backup before it fetches ML data when backup.lock is removed, and exits 1", async () => {
const result = backup(dir, {}, fetch);
await heldRequested;
rmSync(lockPath, { recursive: true });
@@ -720,9 +721,11 @@ describe("a backup whose lock is lost", () => {
"Starting backup...\n" +
"Refreshing library...\n" +
"Fetching original beach.jpg (100)...\n" +
"Fetching original IMG_0002.HEIC (300)...\n" +
"Fetching original toast.mp4 (200)...\n" +
`quak: lost the lock on ${dir}: ENOENT: no such file or directory, stat '${lockPath}'\n`,
);
expect(downloads).toEqual([BEACH.id]);
expect(downloads.sort()).toEqual([100, 200, 300]);
expect(readFileSync(join(dir, DAY, "2024-03-01.100.jpg"))).toEqual(
Buffer.from(JPEG_WITH_EXIF),
);
@@ -798,7 +801,7 @@ describe("a backup whose lock is lost", () => {
await expect(result).rejects.toThrow(
`lost the lock on ${dir}: Unable to update lock within the stale threshold`,
);
expect(downloads).toEqual([BEACH.id]);
expect(downloads.sort()).toEqual([100, 200, 300]);
expect(existsSync(lockPath)).toBe(true);
} finally {
await lib.close();
@@ -823,7 +826,7 @@ describe("a backup whose lock is lost", () => {
const lost = `lost the lock on ${dir}: ENOENT: no such file or directory, stat '${lockPath}'`;
await expect(result).rejects.toThrow(lost);
expect(lock.lost?.message).toBe(lost);
expect(downloads).toEqual([BEACH.id]);
expect(downloads.sort()).toEqual([100, 200, 300]);
} finally {
await lib.close();
await lock.release();