A backup whose lock is lost stops with one line, not a stack trace (closes #184)

proper-lockfile's default onCompromised throws from a timer, so a backup
whose backup.lock was removed or taken over by another backup died with
a stack trace. lockBackupDirectory now returns the lock, its release
function and why it was lost, and passes a handler that only records
the reason. The backup checks it after the refresh and before each
original, thumbnail, file's JSON and album, and before it writes
failures.json, and then fails with "lost the lock on DIR: REASON",
which quak backup prints as one line, exiting 1. lib.backup() takes a
caller's lock as lock, in place of lockHeld: true. Releasing a lost lock
leaves backup.lock alone, since it may be another backup's.

Model: opus-5-5

test/cli/backup-end-to-end.test.ts

@@ -9,7 +9,8 @@
  * data this file encrypts with libsodium, as Ente's clients encrypt it. So
  * quak's own code fetches each original, decrypts it, checks it against the
  * hash its uploader recorded and unpacks a live photo, and quak's own retries
- * repeat a failed request.
+ * repeat a failed request. Two tests run `lib.backup()` instead, as a program
+ * using quak as a library does.
  *
  * The account, user 42, has three albums:
  *
@@ -24,6 +25,7 @@
 import {
     existsSync,
     lstatSync,
+    mkdirSync,
     mkdtempSync,
     readdirSync,
     readFileSync,
@@ -36,7 +38,16 @@ import { join } from "node:path";
 import { Writable } from "node:stream";
 import { gzipSync } from "node:zlib";
 import sodium from "libsodium-wrappers-sumo";
-import { afterAll, beforeAll, describe, expect, it } from "vitest";
+import {
+    afterAll,
+    afterEach,
+    beforeAll,
+    beforeEach,
+    describe,
+    expect,
+    it,
+    vi,
+} from "vitest";
 
 import {
     backupCommand,
@@ -47,6 +58,7 @@ import { run } from "../../src/cli-run.js";
 import { loadSession } from "../../src/cli-session.js";
 import { Client } from "../../src/client.js";
 import { init, toBase64 } from "../../src/crypto/index.js";
+import { Library, lockBackupDirectory } from "../../src/library/index.js";
 import { UNATTENDED_RETRY_OPTIONS } from "../../src/retry.js";
 import {
     buildAccount,
@@ -633,3 +645,188 @@ describe("quak backup", () => {
         expect(stderr).toContain("  Verified:    2\n  Mismatched:  1\n");
     });
 });
+
+// `backup.lock` is a directory that proper-lockfile touches every 5 seconds,
+// from a timer, while the backup runs. Each time, it first looks at the lock
+// on disk. The lock is lost when it is gone, or when another backup has made
+// its own in its place, as one does with a lock nobody touched for 10
+// seconds. Another backup may then be writing to the directory, so the backup
+// stops before it starts another file.
+//
+// The fake server holds one file's download open until the test lets it go,
+// so the lock can be lost while the backup runs: beach.jpg's, the first file
+// backed up, unless the test names another. Only `setTimeout` and
+// `clearTimeout` are faked, so a test can fire proper-lockfile's timer without
+// waiting 5 seconds.
+describe("a backup whose lock is lost", () => {
+    let dir: string;
+    let lockPath: string;
+    let downloads: number[];
+    let heldFile: number;
+    let heldRequested: Promise<void>;
+    let releaseHeld: () => void;
+    let fetch: typeof globalThis.fetch;
+
+    beforeEach(() => {
+        dir = mkdtempSync(join(root, "backup-"));
+        lockPath = join(dir, "backup.lock");
+        downloads = [];
+        heldFile = BEACH.id;
+        let requested!: () => void;
+        heldRequested = new Promise((resolve) => {
+            requested = resolve;
+        });
+        const released = new Promise<void>((resolve) => {
+            releaseHeld = resolve;
+        });
+        const recording = recordingDownloads(downloads);
+        fetch = async (input, init) => {
+            if (downloadOf(input) === heldFile) {
+                requested();
+                await released;
+            }
+            return recording(input, init);
+        };
+        vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
+    });
+
+    afterEach(() => {
+        vi.useRealTimers();
+    });
+
+    // Fire proper-lockfile's timer, and wait for the look at the lock on disk
+    // that it starts to finish: Node lists a file system call in progress as
+    // an `FSReqCallback`.
+    const fireLockTimer = async (): Promise<void> => {
+        vi.advanceTimersByTime(5_000);
+        while (process.getActiveResourcesInfo().includes("FSReqCallback")) {
+            await new Promise((resolve) => setImmediate(resolve));
+        }
+    };
+
+    // beach.jpg, whose download was under way, is stored; no other file is
+    // started. `quak backup` prints why as one line, with no stack trace, and
+    // exits 1.
+    it("stops quak backup before its next file when backup.lock is removed, and exits 1", async () => {
+        const result = backup(dir, {}, fetch);
+        await heldRequested;
+        rmSync(lockPath, { recursive: true });
+        await fireLockTimer();
+        releaseHeld();
+        const { code, stderr } = await result;
+
+        expect(code).toBe(1);
+        expect(stderr).toBe(
+            "Starting backup...\n" +
+                "Refreshing library...\n" +
+                "Fetching original beach.jpg (100)...\n" +
+                `quak: lost the lock on ${dir}: ENOENT: no such file or directory, stat '${lockPath}'\n`,
+        );
+        expect(downloads).toEqual([BEACH.id]);
+        expect(readFileSync(join(dir, DAY, "2024-03-01.100.jpg"))).toEqual(
+            Buffer.from(JPEG_WITH_EXIF),
+        );
+    });
+
+    // Here the lock is lost while toast.mp4, the last original, downloads.
+    // All three originals are stored, beside what the backup wrote before it
+    // started them, but no file's JSON and no album is written.
+    it("stops quak backup before any file's JSON when the lock is lost during the last original", async () => {
+        heldFile = TOAST_VIDEO.id;
+        const result = backup(dir, {}, fetch);
+        await heldRequested;
+        rmSync(lockPath, { recursive: true });
+        await fireLockTimer();
+        releaseHeld();
+        const { code, stderr } = await result;
+
+        expect(code).toBe(1);
+        expect(stderr).toContain(`quak: lost the lock on ${dir}: `);
+        expect(listTree(dir)).toEqual([
+            "2024",
+            "2024/2024-03",
+            "2024/2024-03/2024-03-01",
+            "2024/2024-03/2024-03-01/2024-03-01.100.jpg",
+            "2024/2024-03/2024-03-01/2024-03-01.200.mp4",
+            "2024/2024-03/2024-03-01/2024-03-01.300.heic",
+            "2024/2024-03/2024-03-01/2024-03-01.300.livephoto.json",
+            "2024/2024-03/2024-03-01/2024-03-01.300.mov",
+            "account.json",
+            "collections",
+        ]);
+    });
+
+    // Nothing touches the lock but quak, which finds it as it left it each
+    // time it looks, so the backup runs to the end and removes its lock.
+    it("lets quak backup run to the end while its lock stays", async () => {
+        const result = backup(dir, {}, fetch);
+        await heldRequested;
+        await fireLockTimer();
+        await fireLockTimer();
+        releaseHeld();
+        const { code } = await result;
+
+        expect(code).toBe(0);
+        expect(downloads.sort()).toEqual([100, 200, 300]);
+        expect(existsSync(lockPath)).toBe(false);
+    });
+
+    // The library a program using quak opens to back `dir` up.
+    const openLibrary = () =>
+        Library.open({
+            client: loadSession(join(sessionDir, "session.json"), { fetch })!,
+            cacheDirectory: `${dir}-cache`,
+            downloadDirectory: dir,
+            refreshIntervalSeconds: Infinity,
+            precacheThumbnails: false,
+            precacheOriginals: false,
+        });
+
+    // A program using quak as a library sees the same: `lib.backup()` rejects
+    // with the reason. Here another backup has made its own lock in place of
+    // quak's, and that lock is left where it is.
+    it("rejects lib.backup() when another backup has taken the lock over", async () => {
+        const lib = await openLibrary();
+        try {
+            const result = lib.backup();
+            await heldRequested;
+            rmSync(lockPath, { recursive: true });
+            mkdirSync(lockPath);
+            await fireLockTimer();
+            releaseHeld();
+
+            await expect(result).rejects.toThrow(
+                `lost the lock on ${dir}: Unable to update lock within the stale threshold`,
+            );
+            expect(downloads).toEqual([BEACH.id]);
+            expect(existsSync(lockPath)).toBe(true);
+        } finally {
+            await lib.close();
+        }
+    });
+
+    // A program can take the lock before it opens its library, as `quak
+    // backup` does: `lockBackupDirectory` returns the lock, which the program
+    // passes to `lib.backup()` and releases once the backup ends. The backup
+    // checks it as it checks a lock it took, and the lock's `lost` says why it
+    // was lost.
+    it("rejects lib.backup() given its caller's lock once that lock is lost", async () => {
+        const lock = await lockBackupDirectory(dir);
+        const lib = await openLibrary();
+        try {
+            const result = lib.backup({ lock });
+            await heldRequested;
+            rmSync(lockPath, { recursive: true });
+            await fireLockTimer();
+            releaseHeld();
+
+            const lost = `lost the lock on ${dir}: ENOENT: no such file or directory, stat '${lockPath}'`;
+            await expect(result).rejects.toThrow(lost);
+            expect(lock.lost?.message).toBe(lost);
+            expect(downloads).toEqual([BEACH.id]);
+        } finally {
+            await lib.close();
+            await lock.release();
+        }
+    });
+});