10 Commits
Author SHA1 Message Date
sneak 767545f4ab Test: lib.backup() takes no new original once onProgress has thrown or its lock is lost
check / check (push) Canceled after 0s
The onProgress test now runs eight photos with a pool of five and holds
only the first five downloads, so a backup that went on after the throw
would download and store the other three. The end-to-end test of a
removed backup.lock expects the backup to stop after its three originals,
before it fetches ML data.

Model: opus-5-5
2026-10-14 06:37:26 +00:00
sneak d95c49e548 lib.backup() starts no further original once its lock is lost
Issue 184 checked the backup's lock before each original, in the loop
this branch replaces with several workers. Each worker now checks the
lock before it takes the next file, so once the lock is lost no further
original starts; the originals under way are finished, and the backup
rejects once every worker has stopped. The README says the originals
under way, not the one, are still stored.

Model: opus-5-5
2026-10-14 06:34:09 +00:00
sneak 7f9bf5eb90 Test: lib.backup() starts no further original once its lock is lost
Issue 184 made the backup check its lock before each original. The
workers that now back up several originals at once take the next file
without checking it. With five downloads waiting, backup.lock is removed
and proper-lockfile's timer finds it gone. The backup must download no
other photo, and reject with "lost the lock on DIR" once those five are
in place. Today its workers go on to the other three photos.

Model: opus-5-5
2026-10-14 06:34:09 +00:00
sneak 65b0de4694 runBackup rejects a contentConcurrency that is not a positive integer
A library object whose contentConcurrency was missing, 0 or negative
started no worker, so the backup stored no original, dropped the earlier
entries of failures.json and reported no error. runBackup now throws a
RangeError for any limit that is not a positive integer, as BoundedPool
does, before it takes the lock.

Model: opus-5-5
2026-10-14 06:34:09 +00:00
sneak 4dc9e9e23c Test: runBackup rejects a contentConcurrency that is not a positive integer
runBackup takes a library object from its caller. One whose
contentConcurrency is missing, 0 or negative starts no worker, so the
backup stores no original and reports no error. The test expects
runBackup to reject any limit that is not a positive integer, as
BoundedPool does, before it takes the lock, which would create the
download directory. Today it takes the lock and fails on the first method
the object lacks.

Model: opus-5-5
2026-10-14 06:34:09 +00:00
sneak b5fb5b1738 lib.backup() fails only once every worker has stopped
A worker that threw, as one does when onProgress throws on a FAILED line,
made lib.backup() reject at once, so the lock was removed while the other
workers still wrote originals into the backup. The backup now waits for
every worker to settle and then throws the first error found. The README
no longer says thumbnails are fetched one at a time, which no test checks.

Model: opus-5-5
2026-10-14 06:34:09 +00:00
sneak 9f6fa1990f Test: lib.backup() rejects only once its workers have stopped when onProgress throws
With a content pool of eight, all eight downloads wait at once. Photo 103's
is answered 404 and the line reporting it throws while the other seven
still wait. When lib.backup() rejects, the other seven originals must be in
place and the lock gone, so nothing is written after the backup has ended.
Today the backup rejects as soon as that one worker throws, before the
others have written anything.

Model: opus-5-5
2026-10-14 06:34:09 +00:00
sneak 8904477f75 quak backup works on up to the content pool's limit of originals at once (closes #182)
lib.backup() ran its originals one at a time, so the content pool's five
slots, or the contentConcurrency of the pools a library was opened with,
went unused. The work for one original, its verify hashing included, is
now storeOriginal, and that many workers take files from one iterator
over the backup's files, so no more originals than the limit are open at
once. BackupLibrary gains contentConcurrency, which the library sets from
its content pool. The README's "Request pools" says so. The tests of a
backup whose lock is lost now expect all three originals downloaded,
since all three are under way at once.

Model: opus-5-5
2026-10-14 06:34:09 +00:00
sneak 36d7cdd5db Test: lib.backup() works on up to the content pool's limit of originals at once
Against the fake Ente server, with each download held until the test lets
it go: a library with the default pools has five downloads in flight and a
pool of two has two; with verify, only the originals whose downloads are
in flight have been hashed; a 404 for one photo does not stop the rest;
and two runs end the same with a pool of one as with the default pools.
Today the backup downloads one original at a time, so the first four fail.

Model: opus-5-5
2026-10-14 06:33:55 +00:00
clawbot 22ee2db479 TODO.md's Next Step names the open issues, not a closed one (closes #194)
check / check (push) Canceled after 0s
Next Step said the cache design, issue 36, waits on sneak's review. That
issue closed on 2026-10-03. Next Step now names what is open: the backup
concurrency issue, built in pull request 192 and waiting on review, and
the live-account check that only sneak can run.

Model: opus-5-5
2026-10-14 08:07:48 +02:00
3 changed files with 43 additions and 25 deletions
+7 -2
View File
@@ -17,8 +17,9 @@ pre-1.0
# Next Step
None: no implementation work is open. The cache design,
https://git.eeqj.de/sneak/quak/issues/36, waits on sneak's review.
None: no implementation work is open. The check of quak against a live Ente
account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
has, so it waits on him.
Tagging and releases are decided by sneak alone, and happen only when he
declares one.
@@ -34,6 +35,10 @@ declares one.
count those in flight. Each worker checks the backup's lock before it takes
the next file, so once the lock is lost no further original starts.
- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's
review, though that issue closed on 2026-10-03 (issue 194). It now names the
issues open on the tracker.
- 2026-10-14: A backup whose lock is lost no longer dies with a stack trace
(issue 184). `lockBackupDirectory` gives proper-lockfile an `onCompromised`
that records why the lock was lost instead of throwing from its timer, and
+32 -19
View File
@@ -397,19 +397,33 @@ describe("lib.backup()", () => {
});
// A throw from `onProgress` fails the backup: `lib.backup()` rejects with
// it. It rejects only once it has finished with every original it was
// working on, so nothing is written into the backup after it has ended
// and removed its lock. Here, with a content pool of eight, all eight
// downloads wait at once. Photo 103's is let go first and answered 404,
// and the line reporting that failure throws while the other seven wait.
it("when onProgress throws, rejects only once the originals it was working on are in place", async () => {
// it. From then on the backup starts no further original. It finishes the
// ones it is working on, and rejects only once they are in place, so
// nothing is written into the backup after it has ended and removed its
// lock. Here the first five downloads wait. Photo 103's is let go first
// and answered 404, and the line reporting that failure throws while the
// other four wait. Only those five downloads are held: a backup that went
// on would store the other three photos.
it("when onProgress throws, starts no further original, and rejects once the originals it was working on are in place", async () => {
const dir = mkdtempSync(join(root, "backup-"));
const holding = holdingServer(103);
const lib = await openLibrary(
dir,
holding.fetch,
new RequestPools({ contentConcurrency: 8 }),
);
const downloads: number[] = [];
const held: { id: number; release: () => void }[] = [];
const fetch: typeof globalThis.fetch = async (input, init) => {
const id = downloadOf(input);
if (id !== undefined) {
downloads.push(id);
if (downloads.length <= 5) {
await new Promise<void>((release) =>
held.push({ id, release }),
);
}
if (id === 103) {
return new Response("not found", { status: 404 });
}
}
return server(input, init);
};
const lib = await openLibrary(dir, fetch);
const lines: string[] = [];
try {
const backup = lib.backup({
@@ -432,12 +446,12 @@ describe("lib.backup()", () => {
return err;
});
await vi.waitFor(() => expect(holding.held).toHaveLength(8), {
await vi.waitFor(() => expect(held).toHaveLength(5), {
interval: 5,
timeout: 5000,
});
const at103 = holding.held.findIndex((h) => h.id === 103);
holding.held.splice(at103, 1)[0]!.release();
const at103 = held.findIndex((h) => h.id === 103);
held.splice(at103, 1)[0]!.release();
await vi.waitFor(
() =>
expect(
@@ -445,14 +459,13 @@ describe("lib.backup()", () => {
).toBe(true),
{ interval: 5, timeout: 5000 },
);
for (const download of holding.held) download.release();
for (const download of held) download.release();
expect(await rejection).toEqual(
new Error("the progress output is closed"),
);
expect(storedWhenRejected).toEqual(
PHOTO_IDS.filter((id) => id !== 103),
);
expect(downloads.sort()).toEqual([101, 102, 103, 104, 105]);
expect(storedWhenRejected).toEqual([101, 102, 104, 105]);
expect(lockedWhenRejected).toBe(false);
} finally {
await lib.close();
+4 -4
View File
@@ -705,9 +705,10 @@ describe("a backup whose lock is lost", () => {
};
// The backup downloads up to five originals at once, so all three are
// under way when the lock is lost, and beach.jpg is stored. `quak backup`
// prints why as one line, with no stack trace, and exits 1.
it("stops quak backup before its next file when backup.lock is removed, and exits 1", async () => {
// under way when the lock is lost, and beach.jpg is stored. The backup
// then stops before it fetches ML data. `quak backup` prints why as one
// line, with no stack trace, and exits 1.
it("stops quak backup before it fetches ML data when backup.lock is removed, and exits 1", async () => {
const result = backup(dir, {}, fetch);
await heldRequested;
rmSync(lockPath, { recursive: true });
@@ -722,7 +723,6 @@ describe("a backup whose lock is lost", () => {
"Fetching original beach.jpg (100)...\n" +
"Fetching original IMG_0002.HEIC (300)...\n" +
"Fetching original toast.mp4 (200)...\n" +
"Fetching ML data...\n" +
`quak: lost the lock on ${dir}: ENOENT: no such file or directory, stat '${lockPath}'\n`,
);
expect(downloads.sort()).toEqual([100, 200, 300]);