quak backup refuses to run while another backup of the same directory runs, exit 2 (closes #169)
check / check (push) Successful in 3m10s
check / check (push) Successful in 3m10s
lib.backup() takes a lock, backup.lock in its download directory, made with proper-lockfile, before its refresh, and removes it when it ends. A second backup of the directory fails at once with an error naming it. quak backup takes the lock itself before it opens its library and passes lockHeld to the backup, so a refused run sends no request; it prints the error as one line and exits 2. A lock untouched for 10 seconds, left by a run that could not remove it, is taken over. Deviation: yarn.lock was regenerated by yarn add in the pinned node image. Judgement call: a run failing at its refresh leaves the directory, empty. Judgement call: a lock removed mid-run stops that run with an uncaught error, the library's default. Model: opus-5-5
This commit was merged in pull request #178.
This commit is contained in:
+65
-14
@@ -1,11 +1,13 @@
|
||||
// The backup command, rebuilt on the library API (issue #51).
|
||||
//
|
||||
// `lib.backup()` waits for a completed refresh of the library (a failed one
|
||||
// fails the backup before any file is touched), then, for every file in scope,
|
||||
// puts its original at its save path under `downloadDirectory`, as
|
||||
// `Photo.download()` does, waits for an ML data fetch, and rebuilds the derived
|
||||
// views (per-file sidecars, per-collection symlink trees, per-collection JSON)
|
||||
// from the model. The on-disk layout:
|
||||
// `lib.backup()` takes the lock in `downloadDirectory` (unless its caller
|
||||
// already holds it), and fails at once when another backup of it holds the
|
||||
// lock. It waits for a completed refresh of the library (a failed one fails the
|
||||
// backup before any file is touched), then, for every file in scope, puts its
|
||||
// original at its save path under `downloadDirectory`, as `Photo.download()`
|
||||
// does, waits for an ML data fetch, and rebuilds the derived views (per-file
|
||||
// sidecars, per-collection symlink trees, per-collection JSON) from the model.
|
||||
// The on-disk layout:
|
||||
//
|
||||
// <downloadDirectory>/
|
||||
// YYYY/YYYY-MM/YYYY-MM-DD/
|
||||
@@ -17,6 +19,7 @@
|
||||
// collections/<name>/<title> symlink to the original
|
||||
// collections/<name>.json per-collection metadata
|
||||
// account.json the account's email and user ID
|
||||
// backup.lock the lock, while a backup runs
|
||||
// failures.json durable ledger of unresolved failures
|
||||
//
|
||||
// A live photo's original is its image and its video, each with its own
|
||||
@@ -60,6 +63,7 @@ import {
|
||||
} from "node:fs";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { dirname, extname, join, relative, resolve } from "node:path";
|
||||
import lockfile from "proper-lockfile";
|
||||
|
||||
import { removeLeftoverTempFiles } from "./download/index.js";
|
||||
import { sanitizeFileName, withExtension } from "./filename.js";
|
||||
@@ -89,6 +93,10 @@ export interface BackupOptions {
|
||||
// Restrict the backup to albums with these names; others are left untouched.
|
||||
onlyAlbumNames?: string[];
|
||||
onProgress?: ProgressCallback;
|
||||
// The caller already holds the lock in `downloadDirectory`, taken with
|
||||
// `lockBackupDirectory`, and releases it itself, so the backup does not
|
||||
// take it. `quak backup` takes it before it opens its library.
|
||||
lockHeld?: boolean;
|
||||
}
|
||||
|
||||
export interface BackupError {
|
||||
@@ -450,17 +458,12 @@ const writeAlbumJSON = (
|
||||
writeFileSync(path, JSON.stringify(album, null, 2));
|
||||
};
|
||||
|
||||
export const runBackup = async (
|
||||
// The backup itself, which `runBackup` below runs while the lock is held.
|
||||
const runLockedBackup = async (
|
||||
lib: BackupLibrary,
|
||||
opts: BackupOptions,
|
||||
downloadDirectory: string,
|
||||
): Promise<BackupResult> => {
|
||||
const downloadDirectory = opts.downloadDirectory;
|
||||
if (!downloadDirectory) {
|
||||
throw new Error(
|
||||
"backup requires a downloadDirectory (pass one to backup() or " +
|
||||
"open the library with one)",
|
||||
);
|
||||
}
|
||||
const includeOriginals = opts.includeOriginals ?? true;
|
||||
const includeThumbnails = opts.includeThumbnails ?? false;
|
||||
const log = opts.onProgress ?? (() => {});
|
||||
@@ -736,3 +739,51 @@ export const runBackup = async (
|
||||
errors,
|
||||
};
|
||||
};
|
||||
|
||||
// Only one backup of a directory runs at a time, in this process or another.
|
||||
// Creates `downloadDirectory` if it is missing, takes the lock in it and
|
||||
// returns the function that releases it; while another backup holds the lock,
|
||||
// fails at once with an error whose `code` is `ELOCKED`. The lock is the
|
||||
// directory `backup.lock`, whose modification time proper-lockfile keeps
|
||||
// current while it is held. One it has not touched for 10 seconds was left by
|
||||
// a run that could not remove it, and is taken over.
|
||||
export const lockBackupDirectory = async (
|
||||
downloadDirectory: string,
|
||||
): Promise<() => Promise<void>> => {
|
||||
mkdirSync(downloadDirectory, { recursive: true });
|
||||
try {
|
||||
return await lockfile.lock(downloadDirectory, {
|
||||
lockfilePath: join(downloadDirectory, "backup.lock"),
|
||||
});
|
||||
} catch (err) {
|
||||
if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;
|
||||
throw Object.assign(
|
||||
new Error(`another backup of ${downloadDirectory} is running`),
|
||||
{ code: "ELOCKED" },
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
// Runs the backup holding the lock, which it takes and releases itself unless
|
||||
// the caller already holds it (`opts.lockHeld`).
|
||||
export const runBackup = async (
|
||||
lib: BackupLibrary,
|
||||
opts: BackupOptions,
|
||||
): Promise<BackupResult> => {
|
||||
const downloadDirectory = opts.downloadDirectory;
|
||||
if (!downloadDirectory) {
|
||||
throw new Error(
|
||||
"backup requires a downloadDirectory (pass one to backup() or " +
|
||||
"open the library with one)",
|
||||
);
|
||||
}
|
||||
if (opts.lockHeld) {
|
||||
return runLockedBackup(lib, opts, downloadDirectory);
|
||||
}
|
||||
const release = await lockBackupDirectory(downloadDirectory);
|
||||
try {
|
||||
return await runLockedBackup(lib, opts, downloadDirectory);
|
||||
} finally {
|
||||
await release();
|
||||
}
|
||||
};
|
||||
|
||||
Reference in New Issue
Block a user