1 Commits
Author SHA1 Message Date
clawbot 3412e42af5 quak backup works on up to the content pool's limit of originals at once (closes #182)
check / check (push) Canceled after 0s
The backup awaited each original before starting the next, so it used
one of the content pool's five slots. lib.backup(), and so quak backup,
now runs as many workers as the content pool's limit (5 by default, or
the contentConcurrency of the pools the library was opened with), each
taking the next file from one shared list, so the limit covers the
--verify hashing too. Once a worker has thrown or the lock is lost, no
worker takes another file; the backup waits for the files under way and
checks its lock before the ML data fetch. runBackup rejects a
contentConcurrency that is not a positive integer before it takes the
lock. Failed files are listed in the order they failed.

Model: opus-5-5
2026-10-14 09:06:59 +02:00
3 changed files with 21 additions and 10 deletions

No files matched your search

+4 -2
View File
@@ -32,8 +32,10 @@ declares one.
the work for one original, its `--verify` hashing included, so no more the work for one original, its `--verify` hashing included, so no more
originals than that are open at once. `BackupLibrary` says the limit as originals than that are open at once. `BackupLibrary` says the limit as
`contentConcurrency`. A test holds each download at the fake Ente server to `contentConcurrency`. A test holds each download at the fake Ente server to
count those in flight. Each worker checks the backup's lock before it takes count those in flight. Once a worker has thrown, as one does when `onProgress`
the next file, so once the lock is lost no further original starts. throws, or the lock is lost, no worker takes another file; the backup fails
once the originals under way are in place, and a lost lock stops it before it
fetches ML data.
- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's - 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's
review, though that issue closed on 2026-10-03 (issue 194). It now names the review, though that issue closed on 2026-10-03 (issue 194). It now names the
+15 -6
View File
@@ -730,16 +730,24 @@ const runLockedBackup = async (
// Up to the content pool's limit of originals are worked on at once, // Up to the content pool's limit of originals are worked on at once,
// their `verify` hashing included, so however large the account is, no // their `verify` hashing included, so however large the account is, no
// more than that many are open. Each worker takes the next file from the // more than that many are open. Each worker takes the next file from the
// iterator they all share until none is left, and throws instead once the // iterator they all share until none is left. Once a worker has thrown,
// lock is lost. A worker that throws, as one also does when `onProgress` // as one does when `onProgress` throws, or the lock is lost, no worker
// throws, fails the backup only once every worker has stopped, so none // takes another file. The backup fails only once every worker has
// writes into the backup after it has ended. // stopped, so none writes into the backup after it has ended. It checks
// the lock then too, so it does not go on to the thumbnails or the ML
// data fetch without it.
if (includeOriginals) { if (includeOriginals) {
const pending = distinct.values(); const pending = distinct.values();
let workerThrew = false;
const worker = async (): Promise<void> => { const worker = async (): Promise<void> => {
for (const file of pending) { for (const file of pending) {
if (lock.lost) throw lock.lost; if (workerThrew || lock.lost) return;
await storeOriginal(file); try {
await storeOriginal(file);
} catch (err) {
workerThrew = true;
throw err;
}
} }
}; };
const workers = await Promise.allSettled( const workers = await Promise.allSettled(
@@ -748,6 +756,7 @@ const runLockedBackup = async (
for (const w of workers) { for (const w of workers) {
if (w.status === "rejected") throw w.reason; if (w.status === "rejected") throw w.reason;
} }
if (lock.lost) throw lock.lost;
} }
if (includeThumbnails) { if (includeThumbnails) {
+2 -2
View File
@@ -704,8 +704,8 @@ describe("a backup whose lock is lost", () => {
} }
}; };
// The backup downloads up to five originals at once, so all three are // The backup downloads up to five originals at once, so it has started
// under way when the lock is lost, and beach.jpg is stored. The backup // all three when the lock is lost, and beach.jpg is stored. The backup
// then stops before it fetches ML data. `quak backup` prints why as one // then stops before it fetches ML data. `quak backup` prints why as one
// line, with no stack trace, and exits 1. // line, with no stack trace, and exits 1.
it("stops quak backup before it fetches ML data when backup.lock is removed, and exits 1", async () => { it("stops quak backup before it fetches ML data when backup.lock is removed, and exits 1", async () => {