quak backup works on up to the content pool's limit of originals at once (closes #182) #192

Merged
clawbot merged 10 commits from issue-182-parallel-backup into next 2026-10-14 09:07:00 +02:00
3 changed files with 13 additions and 8 deletions
Showing only changes of commit d95c49e548 - Show all commits
+3 -3
View File
@@ -779,9 +779,9 @@ A running backup loses its lock when someone removes `backup.lock`, or when
another backup takes it over because the running one left it untouched for 10
seconds, as a stopped or stalled process does. The backup finds this out the
next time it goes to touch the lock, which it does every 5 seconds, and then
stops before it starts another file. The original it is downloading at that
moment is still stored. It leaves `failures.json` and `backup.lock` as they are,
since another backup may be using them. `quak backup` prints
stops before it starts another file. The originals it is downloading at that
moment are still stored. It leaves `failures.json` and `backup.lock` as they
are, since another backup may be using them. `quak backup` prints
`quak: lost the lock on <dir>: <reason>` as one line and exits 1, and
`lib.backup()` rejects with `lost the lock on <dir>: <reason>`.
+2 -1
View File
@@ -32,7 +32,8 @@ declares one.
the work for one original, its `--verify` hashing included, so no more
originals than that are open at once. `BackupLibrary` says the limit as
`contentConcurrency`. A test holds each download at the fake Ente server to
count those in flight.
count those in flight. Each worker checks the backup's lock before it takes
the next file, so once the lock is lost no further original starts.
- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's
review, though that issue closed on 2026-10-03 (issue 194). It now names the
+8 -4
View File
@@ -730,13 +730,17 @@ const runLockedBackup = async (
// Up to the content pool's limit of originals are worked on at once,
// their `verify` hashing included, so however large the account is, no
// more than that many are open. Each worker takes the next file from the
// iterator they all share until none is left. A worker that throws, as
// one does when `onProgress` throws, fails the backup only once every
// worker has stopped, so none writes into the backup after it has ended.
// iterator they all share until none is left, and throws instead once the
// lock is lost. A worker that throws, as one also does when `onProgress`
// throws, fails the backup only once every worker has stopped, so none
// writes into the backup after it has ended.
if (includeOriginals) {
const pending = distinct.values();
const worker = async (): Promise<void> => {
for (const file of pending) await storeOriginal(file);
for (const file of pending) {
if (lock.lost) throw lock.lost;
await storeOriginal(file);
}
};
const workers = await Promise.allSettled(
Array.from({ length: lib.contentConcurrency }, worker),