quak backup works on up to the content pool's limit of originals at once (closes #182) #192

Merged
clawbot merged 10 commits from issue-182-parallel-backup into next 2026-10-14 09:07:00 +02:00
3 changed files with 21 additions and 10 deletions
Showing only changes of commit 8d85364a01 - Show all commits

No files matched your search

+4 -2
View File
@@ -32,8 +32,10 @@ declares one.
the work for one original, its `--verify` hashing included, so no more the work for one original, its `--verify` hashing included, so no more
originals than that are open at once. `BackupLibrary` says the limit as originals than that are open at once. `BackupLibrary` says the limit as
`contentConcurrency`. A test holds each download at the fake Ente server to `contentConcurrency`. A test holds each download at the fake Ente server to
count those in flight. Each worker checks the backup's lock before it takes count those in flight. Once a worker has thrown, as one does when `onProgress`
the next file, so once the lock is lost no further original starts. throws, or the lock is lost, no worker takes another file; the backup fails
once the originals under way are in place, and a lost lock stops it before it
fetches ML data.
- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's - 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's
review, though that issue closed on 2026-10-03 (issue 194). It now names the review, though that issue closed on 2026-10-03 (issue 194). It now names the
+15 -6
View File
@@ -730,16 +730,24 @@ const runLockedBackup = async (
// Up to the content pool's limit of originals are worked on at once, // Up to the content pool's limit of originals are worked on at once,
// their `verify` hashing included, so however large the account is, no // their `verify` hashing included, so however large the account is, no
// more than that many are open. Each worker takes the next file from the // more than that many are open. Each worker takes the next file from the
// iterator they all share until none is left, and throws instead once the // iterator they all share until none is left. Once a worker has thrown,
// lock is lost. A worker that throws, as one also does when `onProgress` // as one does when `onProgress` throws, or the lock is lost, no worker
// throws, fails the backup only once every worker has stopped, so none // takes another file. The backup fails only once every worker has
// writes into the backup after it has ended. // stopped, so none writes into the backup after it has ended. It checks
// the lock then too, so it does not go on to the thumbnails or the ML
// data fetch without it.
if (includeOriginals) { if (includeOriginals) {
const pending = distinct.values(); const pending = distinct.values();
let workerThrew = false;
const worker = async (): Promise<void> => { const worker = async (): Promise<void> => {
for (const file of pending) { for (const file of pending) {
if (lock.lost) throw lock.lost; if (workerThrew || lock.lost) return;
await storeOriginal(file); try {
await storeOriginal(file);
} catch (err) {
workerThrew = true;
throw err;
}
} }
}; };
const workers = await Promise.allSettled( const workers = await Promise.allSettled(
@@ -748,6 +756,7 @@ const runLockedBackup = async (
for (const w of workers) { for (const w of workers) {
if (w.status === "rejected") throw w.reason; if (w.status === "rejected") throw w.reason;
} }
if (lock.lost) throw lock.lost;
} }
if (includeThumbnails) { if (includeThumbnails) {
+2 -2
View File
@@ -704,8 +704,8 @@ describe("a backup whose lock is lost", () => {
} }
}; };
// The backup downloads up to five originals at once, so all three are // The backup downloads up to five originals at once, so it has started
// under way when the lock is lost, and beach.jpg is stored. The backup // all three when the lock is lost, and beach.jpg is stored. The backup
// then stops before it fetches ML data. `quak backup` prints why as one // then stops before it fetches ML data. `quak backup` prints why as one
// line, with no stack trace, and exits 1. // line, with no stack trace, and exits 1.
it("stops quak backup before it fetches ML data when backup.lock is removed, and exits 1", async () => { it("stops quak backup before it fetches ML data when backup.lock is removed, and exits 1", async () => {