9 Commits
Author SHA1 Message Date
sneak 767545f4ab Test: lib.backup() takes no new original once onProgress has thrown or its lock is lost
check / check (push) Canceled after 0s
The onProgress test now runs eight photos with a pool of five and holds
only the first five downloads, so a backup that went on after the throw
would download and store the other three. The end-to-end test of a
removed backup.lock expects the backup to stop after its three originals,
before it fetches ML data.

Model: opus-5-5
2026-10-14 06:37:26 +00:00
sneak d95c49e548 lib.backup() starts no further original once its lock is lost
Issue 184 checked the backup's lock before each original, in the loop
this branch replaces with several workers. Each worker now checks the
lock before it takes the next file, so once the lock is lost no further
original starts; the originals under way are finished, and the backup
rejects once every worker has stopped. The README says the originals
under way, not the one, are still stored.

Model: opus-5-5
2026-10-14 06:34:09 +00:00
sneak 7f9bf5eb90 Test: lib.backup() starts no further original once its lock is lost
Issue 184 made the backup check its lock before each original. The
workers that now back up several originals at once take the next file
without checking it. With five downloads waiting, backup.lock is removed
and proper-lockfile's timer finds it gone. The backup must download no
other photo, and reject with "lost the lock on DIR" once those five are
in place. Today its workers go on to the other three photos.

Model: opus-5-5
2026-10-14 06:34:09 +00:00
sneak 65b0de4694 runBackup rejects a contentConcurrency that is not a positive integer
A library object whose contentConcurrency was missing, 0 or negative
started no worker, so the backup stored no original, dropped the earlier
entries of failures.json and reported no error. runBackup now throws a
RangeError for any limit that is not a positive integer, as BoundedPool
does, before it takes the lock.

Model: opus-5-5
2026-10-14 06:34:09 +00:00
sneak 4dc9e9e23c Test: runBackup rejects a contentConcurrency that is not a positive integer
runBackup takes a library object from its caller. One whose
contentConcurrency is missing, 0 or negative starts no worker, so the
backup stores no original and reports no error. The test expects
runBackup to reject any limit that is not a positive integer, as
BoundedPool does, before it takes the lock, which would create the
download directory. Today it takes the lock and fails on the first method
the object lacks.

Model: opus-5-5
2026-10-14 06:34:09 +00:00
sneak b5fb5b1738 lib.backup() fails only once every worker has stopped
A worker that threw, as one does when onProgress throws on a FAILED line,
made lib.backup() reject at once, so the lock was removed while the other
workers still wrote originals into the backup. The backup now waits for
every worker to settle and then throws the first error found. The README
no longer says thumbnails are fetched one at a time, which no test checks.

Model: opus-5-5
2026-10-14 06:34:09 +00:00
sneak 9f6fa1990f Test: lib.backup() rejects only once its workers have stopped when onProgress throws
With a content pool of eight, all eight downloads wait at once. Photo 103's
is answered 404 and the line reporting it throws while the other seven
still wait. When lib.backup() rejects, the other seven originals must be in
place and the lock gone, so nothing is written after the backup has ended.
Today the backup rejects as soon as that one worker throws, before the
others have written anything.

Model: opus-5-5
2026-10-14 06:34:09 +00:00
sneak 8904477f75 quak backup works on up to the content pool's limit of originals at once (closes #182)
lib.backup() ran its originals one at a time, so the content pool's five
slots, or the contentConcurrency of the pools a library was opened with,
went unused. The work for one original, its verify hashing included, is
now storeOriginal, and that many workers take files from one iterator
over the backup's files, so no more originals than the limit are open at
once. BackupLibrary gains contentConcurrency, which the library sets from
its content pool. The README's "Request pools" says so. The tests of a
backup whose lock is lost now expect all three originals downloaded,
since all three are under way at once.

Model: opus-5-5
2026-10-14 06:34:09 +00:00
sneak 36d7cdd5db Test: lib.backup() works on up to the content pool's limit of originals at once
Against the fake Ente server, with each download held until the test lets
it go: a library with the default pools has five downloads in flight and a
pool of two has two; with verify, only the originals whose downloads are
in flight have been hashed; a 404 for one photo does not stop the rest;
and two runs end the same with a pool of one as with the default pools.
Today the backup downloads one original at a time, so the first four fail.

Model: opus-5-5
2026-10-14 06:33:55 +00:00
3 changed files with 10 additions and 21 deletions
+2 -4
View File
@@ -32,10 +32,8 @@ declares one.
the work for one original, its `--verify` hashing included, so no more
originals than that are open at once. `BackupLibrary` says the limit as
`contentConcurrency`. A test holds each download at the fake Ente server to
count those in flight. Once a worker has thrown, as one does when `onProgress`
throws, or the lock is lost, no worker takes another file; the backup fails
once the originals under way are in place, and a lost lock stops it before it
fetches ML data.
count those in flight. Each worker checks the backup's lock before it takes
the next file, so once the lock is lost no further original starts.
- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's
review, though that issue closed on 2026-10-03 (issue 194). It now names the
+6 -15
View File
@@ -730,24 +730,16 @@ const runLockedBackup = async (
// Up to the content pool's limit of originals are worked on at once,
// their `verify` hashing included, so however large the account is, no
// more than that many are open. Each worker takes the next file from the
// iterator they all share until none is left. Once a worker has thrown,
// as one does when `onProgress` throws, or the lock is lost, no worker
// takes another file. The backup fails only once every worker has
// stopped, so none writes into the backup after it has ended. It checks
// the lock then too, so it does not go on to the thumbnails or the ML
// data fetch without it.
// iterator they all share until none is left, and throws instead once the
// lock is lost. A worker that throws, as one also does when `onProgress`
// throws, fails the backup only once every worker has stopped, so none
// writes into the backup after it has ended.
if (includeOriginals) {
const pending = distinct.values();
let workerThrew = false;
const worker = async (): Promise<void> => {
for (const file of pending) {
if (workerThrew || lock.lost) return;
try {
await storeOriginal(file);
} catch (err) {
workerThrew = true;
throw err;
}
if (lock.lost) throw lock.lost;
await storeOriginal(file);
}
};
const workers = await Promise.allSettled(
@@ -756,7 +748,6 @@ const runLockedBackup = async (
for (const w of workers) {
if (w.status === "rejected") throw w.reason;
}
if (lock.lost) throw lock.lost;
}
if (includeThumbnails) {
+2 -2
View File
@@ -704,8 +704,8 @@ describe("a backup whose lock is lost", () => {
}
};
// The backup downloads up to five originals at once, so it has started
// all three when the lock is lost, and beach.jpg is stored. The backup
// The backup downloads up to five originals at once, so all three are
// under way when the lock is lost, and beach.jpg is stored. The backup
// then stops before it fetches ML data. `quak backup` prints why as one
// line, with no stack trace, and exits 1.
it("stops quak backup before it fetches ML data when backup.lock is removed, and exits 1", async () => {