A backup whose lock is lost stops with one line, not a stack trace (closes #184)
check / check (push) Waiting to run
check / check (push) Waiting to run
proper-lockfile's default onCompromised throws from a timer, so a backup whose backup.lock was removed or taken over by another backup died with a stack trace. lockBackupDirectory now returns the lock, its release function and why it was lost, and passes a handler that only records the reason. The backup checks it after the refresh and before each original, thumbnail, file's JSON and album, and before it writes failures.json, and then fails with "lost the lock on DIR: REASON", which quak backup prints as one line, exiting 1. lib.backup() takes a caller's lock as lock, in place of lockHeld: true. Releasing a lost lock leaves backup.lock alone, since it may be another backup's. Model: opus-5-5
This commit was merged in pull request #193.
This commit is contained in:
@@ -761,9 +761,9 @@ drops it.
|
||||
`backup.lock` keeps two backups of the same directory from running at once, such
|
||||
as a cron run that starts while the previous one is still going. A backup
|
||||
creates `<dir>` if it is missing and takes the lock before its refresh, and
|
||||
removes the lock when it ends, whether it succeeds or fails; `quak backup` takes
|
||||
it before it opens its library, so a refused run sends no request. The lock is a
|
||||
directory that
|
||||
removes the lock when it ends, whether it succeeds or fails, unless it lost the
|
||||
lock (see below); `quak backup` takes it before it opens its library, so a
|
||||
refused run sends no request. The lock is a directory that
|
||||
[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates
|
||||
and keeps touching while the backup runs. A second backup of the directory, from
|
||||
another process or the same one, fails at once: `quak backup` prints
|
||||
@@ -775,6 +775,16 @@ over, so nobody has to remove it. The lock is outside the date folders and
|
||||
`collections/`, so it is never taken for an original, and the removal of old
|
||||
album directories never touches it.
|
||||
|
||||
A running backup loses its lock when someone removes `backup.lock`, or when
|
||||
another backup takes it over because the running one left it untouched for 10
|
||||
seconds, as a stopped or stalled process does. The backup finds this out the
|
||||
next time it goes to touch the lock, which it does every 5 seconds, and then
|
||||
stops before it starts another file. The original it is downloading at that
|
||||
moment is still stored. It leaves `failures.json` and `backup.lock` as they are,
|
||||
since another backup may be using them. `quak backup` prints
|
||||
`quak: lost the lock on <dir>: <reason>` as one line and exits 1, and
|
||||
`lib.backup()` rejects with `lost the lock on <dir>: <reason>`.
|
||||
|
||||
A collection's directory and JSON are named after the collection, and a symlink
|
||||
after the file's title, both with unsafe characters replaced. When two
|
||||
collections would get the same name, or two symlinks in one collection the same
|
||||
@@ -1060,21 +1070,24 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
directory, and fails at once with an error whose `code` is `ELOCKED` while
|
||||
another backup of it runs. A caller can instead take the lock itself, before
|
||||
it opens its library, as `quak backup` does: `await lockBackupDirectory(dir)`
|
||||
takes it, failing the same way, and returns the function that releases it, and
|
||||
the caller passes `lockHeld: true` to the backup. It waits for a refresh as
|
||||
`fresh()` does, puts every in-scope original not already at its save path
|
||||
there as `photo.download()` does (and, with `includeThumbnails`, fetches
|
||||
thumbnails) through the content cache, waits for an ML data fetch, and
|
||||
rebuilds the on-disk backup tree with a durable failure ledger. Each file's
|
||||
JSON holds its ML data and, for an image (for a live photo, its image), its
|
||||
original's EXIF and XMP, with its dimensions for a JPEG only; a video's JSON
|
||||
holds none of these three. A fetched original is written straight to its save
|
||||
path and not into the cache, which then counts it as present; one the cache
|
||||
already held is copied from there. `BackupOptions`: `downloadDirectory` (falls
|
||||
back to the library's), `includeOriginals` (default `true`),
|
||||
`includeThumbnails` (default `false`), `onlyAlbumNames`, `verify` (default
|
||||
`false`), `onProgress`, and `lockHeld` (default `false`). See Backup layout
|
||||
above for the tree it writes.
|
||||
takes it, failing the same way, and returns a `BackupLock`, whose `release()`
|
||||
removes the lock unless it was lost and whose `lost` holds why it was lost.
|
||||
The caller passes it to the backup as `lock`, and releases it once the backup
|
||||
ends. It waits for a refresh as `fresh()` does, puts every in-scope original
|
||||
not already at its save path there as `photo.download()` does (and, with
|
||||
`includeThumbnails`, fetches thumbnails) through the content cache, waits for
|
||||
an ML data fetch, and rebuilds the on-disk backup tree with a durable failure
|
||||
ledger. Each file's JSON holds its ML data and, for an image (for a live
|
||||
photo, its image), its original's EXIF and XMP, with its dimensions for a JPEG
|
||||
only; a video's JSON holds none of these three. A fetched original is written
|
||||
straight to its save path and not into the cache, which then counts it as
|
||||
present; one the cache already held is copied from there. `BackupOptions`:
|
||||
`downloadDirectory` (falls back to the library's), `includeOriginals` (default
|
||||
`true`), `includeThumbnails` (default `false`), `onlyAlbumNames`, `verify`
|
||||
(default `false`), `onProgress`, and `lock`. See Backup layout above for the
|
||||
tree it writes. Whether the backup took the lock or its caller did, once the
|
||||
lock is lost the backup rejects with `lost the lock on <dir>: <reason>` before
|
||||
it starts another file.
|
||||
|
||||
### Request pools
|
||||
|
||||
@@ -1134,7 +1147,7 @@ from a very old client, is stored unchecked.
|
||||
- `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult`
|
||||
- `src/exif.ts`: `ExifTags`, `PhotoExif`
|
||||
- `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool`
|
||||
- `src/backup.ts`: `BackupOptions`, `BackupResult`, `BackupError`
|
||||
- `src/backup.ts`: `BackupLock`, `BackupOptions`, `BackupResult`, `BackupError`
|
||||
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
|
||||
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `StreamOptions`
|
||||
- `src/errors.ts`: `ApiError`, `TruncatedStreamError`
|
||||
|
||||
@@ -25,6 +25,16 @@ declares one.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-14: A backup whose lock is lost no longer dies with a stack trace
|
||||
(issue 184). `lockBackupDirectory` gives proper-lockfile an `onCompromised`
|
||||
that records why the lock was lost instead of throwing from its timer, and
|
||||
returns the lock, which `quak backup` passes to `lib.backup()` as `lock`. The
|
||||
backup then stops before it starts another file with
|
||||
`lost the lock on DIR: REASON`, which `quak backup` prints as one line before
|
||||
it exits 1. Releasing a lost lock leaves `backup.lock` alone. Tests in
|
||||
`test/cli/backup-end-to-end.test.ts` remove the lock, or put another backup's
|
||||
in its place, while a download is held open.
|
||||
|
||||
- 2026-10-14: An album or file quak cannot decrypt, or whose metadata is not
|
||||
JSON, no longer fails the refresh, so one bad record no longer stops
|
||||
`quak backup` from storing anything (issue 181). The refresh leaves it out, an
|
||||
|
||||
+61
-21
@@ -1,8 +1,9 @@
|
||||
// The backup command, rebuilt on the library API (issue #51).
|
||||
//
|
||||
// `lib.backup()` takes the lock in `downloadDirectory` (unless its caller
|
||||
// already holds it), and fails at once when another backup of it holds the
|
||||
// lock. It waits for a completed refresh of the library (a failed one fails the
|
||||
// passes one in), and fails at once when another backup of it holds the
|
||||
// lock; when the lock is lost while it runs, it fails before it starts another
|
||||
// file. It waits for a completed refresh of the library (a failed one fails the
|
||||
// backup before any file is touched), then, for every file in scope, puts its
|
||||
// original at its save path under `downloadDirectory`, as `Photo.download()`
|
||||
// does, waits for an ML data fetch, and rebuilds the derived views (per-file
|
||||
@@ -96,6 +97,14 @@ import type { Collection, EnteFile, FileMetadata } from "./model/types.js";
|
||||
|
||||
export type ProgressCallback = (message: string) => void;
|
||||
|
||||
// The lock in a backup directory, as `lockBackupDirectory` returns it.
|
||||
export interface BackupLock {
|
||||
// Removes `backup.lock`, unless the lock was lost.
|
||||
release(): Promise<void>;
|
||||
// Why the lock was lost, once it is.
|
||||
lost?: Error;
|
||||
}
|
||||
|
||||
export interface BackupOptions {
|
||||
// Where the backup tree lives. `lib.backup()` defaults it to the library's
|
||||
// download directory; `runBackup` with none throws before any network
|
||||
@@ -112,10 +121,11 @@ export interface BackupOptions {
|
||||
// bytes do not match the content hash its metadata records. Default false.
|
||||
verify?: boolean;
|
||||
onProgress?: ProgressCallback;
|
||||
// The caller already holds the lock in `downloadDirectory`, taken with
|
||||
// `lockBackupDirectory`, and releases it itself, so the backup does not
|
||||
// take it. `quak backup` takes it before it opens its library.
|
||||
lockHeld?: boolean;
|
||||
// The lock in `downloadDirectory`, when the caller took it with
|
||||
// `lockBackupDirectory` and releases it itself. The backup then does not
|
||||
// take the lock, and fails once this one is lost. `quak backup` takes it
|
||||
// before it opens its library.
|
||||
lock?: BackupLock;
|
||||
}
|
||||
|
||||
export interface BackupError {
|
||||
@@ -510,11 +520,14 @@ const writeAlbumJSON = (
|
||||
writeFileSync(path, JSON.stringify(album, null, 2));
|
||||
};
|
||||
|
||||
// The backup itself, which `runBackup` below runs while the lock is held.
|
||||
// The backup itself, which `runBackup` below runs while `lock` is held. Once
|
||||
// the lock is lost, another backup may be writing to `downloadDirectory`, so
|
||||
// this one fails before it starts another file.
|
||||
const runLockedBackup = async (
|
||||
lib: BackupLibrary,
|
||||
opts: BackupOptions,
|
||||
downloadDirectory: string,
|
||||
lock: BackupLock,
|
||||
): Promise<BackupResult> => {
|
||||
const includeOriginals = opts.includeOriginals ?? true;
|
||||
const includeThumbnails = opts.includeThumbnails ?? false;
|
||||
@@ -524,6 +537,7 @@ const runLockedBackup = async (
|
||||
|
||||
log("Refreshing library...");
|
||||
await lib.refresh();
|
||||
if (lock.lost) throw lock.lost;
|
||||
|
||||
const collectionsDir = join(downloadDirectory, "collections");
|
||||
const thumbnailsDir = join(downloadDirectory, "thumbnails");
|
||||
@@ -642,6 +656,7 @@ const runLockedBackup = async (
|
||||
// that cannot be read is recorded as failed and left where it is.
|
||||
if (includeOriginals) {
|
||||
for (const [fileID, file] of distinct) {
|
||||
if (lock.lost) throw lock.lost;
|
||||
let stored = storedAtSavePath(downloadDirectory, file);
|
||||
let mismatch = false;
|
||||
if (stored !== undefined && verify) {
|
||||
@@ -721,6 +736,7 @@ const runLockedBackup = async (
|
||||
|
||||
if (includeThumbnails) {
|
||||
for (const [fileID, file] of distinct) {
|
||||
if (lock.lost) throw lock.lost;
|
||||
const dest = join(thumbnailsDir, `${fileID}.jpg`);
|
||||
if (isPresent(dest)) continue;
|
||||
try {
|
||||
@@ -753,6 +769,7 @@ const runLockedBackup = async (
|
||||
log(`FAILED ML data: ${mlDataError}`);
|
||||
}
|
||||
for (const file of distinct.values()) {
|
||||
if (lock.lost) throw lock.lost;
|
||||
const stored = storedAtSavePath(downloadDirectory, file);
|
||||
if (stored === undefined) continue;
|
||||
const path = withExtension(
|
||||
@@ -796,6 +813,7 @@ const runLockedBackup = async (
|
||||
const albumDirNames = new Map(
|
||||
allCollections.map((c, i) => [c.id, dirNames[i]!]),
|
||||
);
|
||||
if (lock.lost) throw lock.lost;
|
||||
try {
|
||||
removeStaleAlbumDirs(
|
||||
collectionsDir,
|
||||
@@ -807,6 +825,7 @@ const runLockedBackup = async (
|
||||
}
|
||||
|
||||
for (const c of collections) {
|
||||
if (lock.lost) throw lock.lost;
|
||||
const colDirName = albumDirNames.get(c.id)!;
|
||||
const colDir = join(collectionsDir, colDirName);
|
||||
mkdirSync(colDir, { recursive: true });
|
||||
@@ -864,7 +883,9 @@ const runLockedBackup = async (
|
||||
// its failure resolved; a file gone from the library (deleted) or outside
|
||||
// this run's scope is not something this run can resolve, so keeping its
|
||||
// stale entry would keep the exit code non-zero forever — a single
|
||||
// since-deleted photo would fail every future scheduled backup.
|
||||
// since-deleted photo would fail every future scheduled backup. A backup
|
||||
// whose lock was lost leaves the ledger as it is.
|
||||
if (lock.lost) throw lock.lost;
|
||||
for (const fileID of [...ledger.keys()]) {
|
||||
if (!failedThisRun.has(fileID)) ledger.delete(fileID);
|
||||
}
|
||||
@@ -885,18 +906,36 @@ const runLockedBackup = async (
|
||||
|
||||
// Only one backup of a directory runs at a time, in this process or another.
|
||||
// Creates `downloadDirectory` if it is missing, takes the lock in it and
|
||||
// returns the function that releases it; while another backup holds the lock,
|
||||
// fails at once with an error whose `code` is `ELOCKED`. The lock is the
|
||||
// directory `backup.lock`, whose modification time proper-lockfile keeps
|
||||
// current while it is held. One it has not touched for 10 seconds was left by
|
||||
// a run that could not remove it, and is taken over.
|
||||
// returns it; while another backup holds the lock, fails at once with an error
|
||||
// whose `code` is `ELOCKED`. The lock is the directory `backup.lock`, whose
|
||||
// modification time proper-lockfile keeps current while it is held. One it has
|
||||
// not touched for 10 seconds was left by a run that could not remove it, and
|
||||
// is taken over.
|
||||
//
|
||||
// proper-lockfile finds the lock lost when `backup.lock` is gone or is another
|
||||
// backup's, and calls `onCompromised` from a timer, where throwing would end
|
||||
// the process. The `onCompromised` given here sets the lock's `lost` instead,
|
||||
// and the backup holding the lock stops before it starts another file (see
|
||||
// `runLockedBackup`). Releasing a lost lock leaves `backup.lock` alone, since
|
||||
// it may be another backup's.
|
||||
export const lockBackupDirectory = async (
|
||||
downloadDirectory: string,
|
||||
): Promise<() => Promise<void>> => {
|
||||
): Promise<BackupLock> => {
|
||||
mkdirSync(downloadDirectory, { recursive: true });
|
||||
let release: () => Promise<void>;
|
||||
const lock: BackupLock = {
|
||||
release: async () => {
|
||||
if (lock.lost === undefined) await release();
|
||||
},
|
||||
};
|
||||
try {
|
||||
return await lockfile.lock(downloadDirectory, {
|
||||
release = await lockfile.lock(downloadDirectory, {
|
||||
lockfilePath: join(downloadDirectory, "backup.lock"),
|
||||
onCompromised: (err) => {
|
||||
lock.lost = new Error(
|
||||
`lost the lock on ${downloadDirectory}: ${err.message}`,
|
||||
);
|
||||
},
|
||||
});
|
||||
} catch (err) {
|
||||
if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;
|
||||
@@ -905,10 +944,11 @@ export const lockBackupDirectory = async (
|
||||
{ code: "ELOCKED" },
|
||||
);
|
||||
}
|
||||
return lock;
|
||||
};
|
||||
|
||||
// Runs the backup holding the lock, which it takes and releases itself unless
|
||||
// the caller already holds it (`opts.lockHeld`).
|
||||
// the caller passes the lock it took (`opts.lock`).
|
||||
export const runBackup = async (
|
||||
lib: BackupLibrary,
|
||||
opts: BackupOptions,
|
||||
@@ -920,13 +960,13 @@ export const runBackup = async (
|
||||
"open the library with one)",
|
||||
);
|
||||
}
|
||||
if (opts.lockHeld) {
|
||||
return runLockedBackup(lib, opts, downloadDirectory);
|
||||
if (opts.lock) {
|
||||
return runLockedBackup(lib, opts, downloadDirectory, opts.lock);
|
||||
}
|
||||
const release = await lockBackupDirectory(downloadDirectory);
|
||||
const lock = await lockBackupDirectory(downloadDirectory);
|
||||
try {
|
||||
return await runLockedBackup(lib, opts, downloadDirectory);
|
||||
return await runLockedBackup(lib, opts, downloadDirectory, lock);
|
||||
} finally {
|
||||
await release();
|
||||
await lock.release();
|
||||
}
|
||||
};
|
||||
|
||||
+5
-5
@@ -20,7 +20,7 @@ import {
|
||||
type ClientSnapshot,
|
||||
type LoginOptions,
|
||||
} from "./client.js";
|
||||
import { lockBackupDirectory } from "./backup.js";
|
||||
import { lockBackupDirectory, type BackupLock } from "./backup.js";
|
||||
import { init } from "./crypto/index.js";
|
||||
import {
|
||||
defaultCacheDirectory,
|
||||
@@ -416,9 +416,9 @@ export const backupCommand = async (
|
||||
ctx.stderr.write("Starting backup...\n");
|
||||
// The lock is taken before the library opens and starts its refresh, so a
|
||||
// run refused while another backup of `dir` runs sends no request.
|
||||
let release: () => Promise<void>;
|
||||
let lock: BackupLock;
|
||||
try {
|
||||
release = await lockBackupDirectory(dir);
|
||||
lock = await lockBackupDirectory(dir);
|
||||
} catch (err) {
|
||||
if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;
|
||||
ctx.stderr.write(`quak: ${(err as Error).message}\n`);
|
||||
@@ -440,7 +440,7 @@ export const backupCommand = async (
|
||||
try {
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: dir,
|
||||
lockHeld: true,
|
||||
lock,
|
||||
verify: opts.verify,
|
||||
onProgress: (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
@@ -477,7 +477,7 @@ export const backupCommand = async (
|
||||
await lib.close();
|
||||
}
|
||||
} finally {
|
||||
await release();
|
||||
await lock.release();
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@@ -69,6 +69,7 @@ export {
|
||||
type EnsureEvent,
|
||||
lockBackupDirectory,
|
||||
runBackup,
|
||||
type BackupLock,
|
||||
type BackupOptions,
|
||||
type BackupResult,
|
||||
type BackupError,
|
||||
|
||||
@@ -96,6 +96,7 @@ import { runBackup, type BackupOptions, type BackupResult } from "../backup.js";
|
||||
export {
|
||||
lockBackupDirectory,
|
||||
runBackup,
|
||||
type BackupLock,
|
||||
type BackupOptions,
|
||||
type BackupResult,
|
||||
type BackupError,
|
||||
@@ -570,9 +571,10 @@ export class Library {
|
||||
|
||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||
// library's, each original at its save path, with a durable failure
|
||||
// ledger (issue #51). Takes the lock in that directory first, unless
|
||||
// `opts.lockHeld` says the caller holds it, failing at once while another
|
||||
// backup of it runs (see `runBackup`). Waits for a
|
||||
// ledger (issue #51). Takes the lock in that directory first, unless the
|
||||
// caller passes the lock it took as `opts.lock`, failing at once while
|
||||
// another backup of it runs, and before it starts another file once the
|
||||
// lock is lost (see `lockBackupDirectory`). Waits for a
|
||||
// completed refresh, as `fresh()` does, joining one already running, and
|
||||
// rejects before touching any file but the lock when it fails. Then puts
|
||||
// pending originals at their save paths as `Photo.download()` does (and
|
||||
|
||||
@@ -9,7 +9,8 @@
|
||||
* data this file encrypts with libsodium, as Ente's clients encrypt it. So
|
||||
* quak's own code fetches each original, decrypts it, checks it against the
|
||||
* hash its uploader recorded and unpacks a live photo, and quak's own retries
|
||||
* repeat a failed request.
|
||||
* repeat a failed request. Two tests run `lib.backup()` instead, as a program
|
||||
* using quak as a library does.
|
||||
*
|
||||
* The account, user 42, has three albums:
|
||||
*
|
||||
@@ -24,6 +25,7 @@
|
||||
import {
|
||||
existsSync,
|
||||
lstatSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
readdirSync,
|
||||
readFileSync,
|
||||
@@ -36,7 +38,16 @@ import { join } from "node:path";
|
||||
import { Writable } from "node:stream";
|
||||
import { gzipSync } from "node:zlib";
|
||||
import sodium from "libsodium-wrappers-sumo";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
afterAll,
|
||||
afterEach,
|
||||
beforeAll,
|
||||
beforeEach,
|
||||
describe,
|
||||
expect,
|
||||
it,
|
||||
vi,
|
||||
} from "vitest";
|
||||
|
||||
import {
|
||||
backupCommand,
|
||||
@@ -47,6 +58,7 @@ import { run } from "../../src/cli-run.js";
|
||||
import { loadSession } from "../../src/cli-session.js";
|
||||
import { Client } from "../../src/client.js";
|
||||
import { init, toBase64 } from "../../src/crypto/index.js";
|
||||
import { Library, lockBackupDirectory } from "../../src/library/index.js";
|
||||
import { UNATTENDED_RETRY_OPTIONS } from "../../src/retry.js";
|
||||
import {
|
||||
buildAccount,
|
||||
@@ -633,3 +645,188 @@ describe("quak backup", () => {
|
||||
expect(stderr).toContain(" Verified: 2\n Mismatched: 1\n");
|
||||
});
|
||||
});
|
||||
|
||||
// `backup.lock` is a directory that proper-lockfile touches every 5 seconds,
|
||||
// from a timer, while the backup runs. Each time, it first looks at the lock
|
||||
// on disk. The lock is lost when it is gone, or when another backup has made
|
||||
// its own in its place, as one does with a lock nobody touched for 10
|
||||
// seconds. Another backup may then be writing to the directory, so the backup
|
||||
// stops before it starts another file.
|
||||
//
|
||||
// The fake server holds one file's download open until the test lets it go,
|
||||
// so the lock can be lost while the backup runs: beach.jpg's, the first file
|
||||
// backed up, unless the test names another. Only `setTimeout` and
|
||||
// `clearTimeout` are faked, so a test can fire proper-lockfile's timer without
|
||||
// waiting 5 seconds.
|
||||
describe("a backup whose lock is lost", () => {
|
||||
let dir: string;
|
||||
let lockPath: string;
|
||||
let downloads: number[];
|
||||
let heldFile: number;
|
||||
let heldRequested: Promise<void>;
|
||||
let releaseHeld: () => void;
|
||||
let fetch: typeof globalThis.fetch;
|
||||
|
||||
beforeEach(() => {
|
||||
dir = mkdtempSync(join(root, "backup-"));
|
||||
lockPath = join(dir, "backup.lock");
|
||||
downloads = [];
|
||||
heldFile = BEACH.id;
|
||||
let requested!: () => void;
|
||||
heldRequested = new Promise((resolve) => {
|
||||
requested = resolve;
|
||||
});
|
||||
const released = new Promise<void>((resolve) => {
|
||||
releaseHeld = resolve;
|
||||
});
|
||||
const recording = recordingDownloads(downloads);
|
||||
fetch = async (input, init) => {
|
||||
if (downloadOf(input) === heldFile) {
|
||||
requested();
|
||||
await released;
|
||||
}
|
||||
return recording(input, init);
|
||||
};
|
||||
vi.useFakeTimers({ toFake: ["setTimeout", "clearTimeout"] });
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
vi.useRealTimers();
|
||||
});
|
||||
|
||||
// Fire proper-lockfile's timer, and wait for the look at the lock on disk
|
||||
// that it starts to finish: Node lists a file system call in progress as
|
||||
// an `FSReqCallback`.
|
||||
const fireLockTimer = async (): Promise<void> => {
|
||||
vi.advanceTimersByTime(5_000);
|
||||
while (process.getActiveResourcesInfo().includes("FSReqCallback")) {
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
}
|
||||
};
|
||||
|
||||
// beach.jpg, whose download was under way, is stored; no other file is
|
||||
// started. `quak backup` prints why as one line, with no stack trace, and
|
||||
// exits 1.
|
||||
it("stops quak backup before its next file when backup.lock is removed, and exits 1", async () => {
|
||||
const result = backup(dir, {}, fetch);
|
||||
await heldRequested;
|
||||
rmSync(lockPath, { recursive: true });
|
||||
await fireLockTimer();
|
||||
releaseHeld();
|
||||
const { code, stderr } = await result;
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toBe(
|
||||
"Starting backup...\n" +
|
||||
"Refreshing library...\n" +
|
||||
"Fetching original beach.jpg (100)...\n" +
|
||||
`quak: lost the lock on ${dir}: ENOENT: no such file or directory, stat '${lockPath}'\n`,
|
||||
);
|
||||
expect(downloads).toEqual([BEACH.id]);
|
||||
expect(readFileSync(join(dir, DAY, "2024-03-01.100.jpg"))).toEqual(
|
||||
Buffer.from(JPEG_WITH_EXIF),
|
||||
);
|
||||
});
|
||||
|
||||
// Here the lock is lost while toast.mp4, the last original, downloads.
|
||||
// All three originals are stored, beside what the backup wrote before it
|
||||
// started them, but no file's JSON and no album is written.
|
||||
it("stops quak backup before any file's JSON when the lock is lost during the last original", async () => {
|
||||
heldFile = TOAST_VIDEO.id;
|
||||
const result = backup(dir, {}, fetch);
|
||||
await heldRequested;
|
||||
rmSync(lockPath, { recursive: true });
|
||||
await fireLockTimer();
|
||||
releaseHeld();
|
||||
const { code, stderr } = await result;
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toContain(`quak: lost the lock on ${dir}: `);
|
||||
expect(listTree(dir)).toEqual([
|
||||
"2024",
|
||||
"2024/2024-03",
|
||||
"2024/2024-03/2024-03-01",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.100.jpg",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.200.mp4",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.300.heic",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.300.livephoto.json",
|
||||
"2024/2024-03/2024-03-01/2024-03-01.300.mov",
|
||||
"account.json",
|
||||
"collections",
|
||||
]);
|
||||
});
|
||||
|
||||
// Nothing touches the lock but quak, which finds it as it left it each
|
||||
// time it looks, so the backup runs to the end and removes its lock.
|
||||
it("lets quak backup run to the end while its lock stays", async () => {
|
||||
const result = backup(dir, {}, fetch);
|
||||
await heldRequested;
|
||||
await fireLockTimer();
|
||||
await fireLockTimer();
|
||||
releaseHeld();
|
||||
const { code } = await result;
|
||||
|
||||
expect(code).toBe(0);
|
||||
expect(downloads.sort()).toEqual([100, 200, 300]);
|
||||
expect(existsSync(lockPath)).toBe(false);
|
||||
});
|
||||
|
||||
// The library a program using quak opens to back `dir` up.
|
||||
const openLibrary = () =>
|
||||
Library.open({
|
||||
client: loadSession(join(sessionDir, "session.json"), { fetch })!,
|
||||
cacheDirectory: `${dir}-cache`,
|
||||
downloadDirectory: dir,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
|
||||
// A program using quak as a library sees the same: `lib.backup()` rejects
|
||||
// with the reason. Here another backup has made its own lock in place of
|
||||
// quak's, and that lock is left where it is.
|
||||
it("rejects lib.backup() when another backup has taken the lock over", async () => {
|
||||
const lib = await openLibrary();
|
||||
try {
|
||||
const result = lib.backup();
|
||||
await heldRequested;
|
||||
rmSync(lockPath, { recursive: true });
|
||||
mkdirSync(lockPath);
|
||||
await fireLockTimer();
|
||||
releaseHeld();
|
||||
|
||||
await expect(result).rejects.toThrow(
|
||||
`lost the lock on ${dir}: Unable to update lock within the stale threshold`,
|
||||
);
|
||||
expect(downloads).toEqual([BEACH.id]);
|
||||
expect(existsSync(lockPath)).toBe(true);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// A program can take the lock before it opens its library, as `quak
|
||||
// backup` does: `lockBackupDirectory` returns the lock, which the program
|
||||
// passes to `lib.backup()` and releases once the backup ends. The backup
|
||||
// checks it as it checks a lock it took, and the lock's `lost` says why it
|
||||
// was lost.
|
||||
it("rejects lib.backup() given its caller's lock once that lock is lost", async () => {
|
||||
const lock = await lockBackupDirectory(dir);
|
||||
const lib = await openLibrary();
|
||||
try {
|
||||
const result = lib.backup({ lock });
|
||||
await heldRequested;
|
||||
rmSync(lockPath, { recursive: true });
|
||||
await fireLockTimer();
|
||||
releaseHeld();
|
||||
|
||||
const lost = `lost the lock on ${dir}: ENOENT: no such file or directory, stat '${lockPath}'`;
|
||||
await expect(result).rejects.toThrow(lost);
|
||||
expect(lock.lost?.message).toBe(lost);
|
||||
expect(downloads).toEqual([BEACH.id]);
|
||||
} finally {
|
||||
await lib.close();
|
||||
await lock.release();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user