Compare commits
3
Commits
73846b32fc
..
next
| Author | SHA1 | Date | |
|---|---|---|---|
 clawbot
|
2402ec97f7 | ||
|
clawbot
|
ddf58af3cc | ||
|
clawbot
|
bd77422965 |
@@ -10,9 +10,12 @@ account into a deduplicated local directory tree, skipping files that already
|
||||
exist on disk and continuing past individual download failures instead of
|
||||
crashing. For each file it persists the basic metadata fields quak keeps (title,
|
||||
file type, creation and modification time, latitude, longitude, content hash),
|
||||
and the private and public magic metadata in full. A helper subcommand can
|
||||
detect and regenerate missing thumbnails, encrypting and uploading them back to
|
||||
the server.
|
||||
its update time, the private and public magic metadata in full, Ente's ML data
|
||||
for it when there is any, and, for an image (for a live photo, its image), its
|
||||
original's EXIF and XMP, with its dimensions for a JPEG only; for a video it
|
||||
keeps none of these three. It runs unattended from cron (see "Running the backup
|
||||
from cron"). A helper subcommand can detect and regenerate missing thumbnails,
|
||||
encrypting and uploading them back to the server.
|
||||
|
||||
## Getting Started
|
||||
|
||||
@@ -80,6 +83,64 @@ await lib.close();
|
||||
The lower-level `Client` (login, session serialization, and the raw
|
||||
enumeration/download calls) is exported too and documented under Design below.
|
||||
|
||||
## Running the backup from cron
|
||||
|
||||
`quak backup` never prompts, so cron can run it. `make install` builds quak as a
|
||||
single binary and copies it to `~/bin/quak`. Log in once with it, as the user
|
||||
the cron job will run as; the session is saved in that user's data directory
|
||||
(see "Session handling"):
|
||||
|
||||
```bash
|
||||
~/bin/quak login
|
||||
```
|
||||
|
||||
Then add the backup to that user's crontab with `crontab -e`. These two lines
|
||||
back up the account to `~/photos-backup` at 03:30 every night, with `--verify`
|
||||
on Sundays, and append all output to `~/quak-backup.log`:
|
||||
|
||||
```
|
||||
30 3 * * 1-6 $HOME/bin/quak backup $HOME/photos-backup >> $HOME/quak-backup.log 2>&1
|
||||
30 3 * * 0 $HOME/bin/quak backup --verify $HOME/photos-backup >> $HOME/quak-backup.log 2>&1
|
||||
```
|
||||
|
||||
A run with `--verify` does all a plain run does, and also checks each original
|
||||
already in the backup against the content hash Ente records for it, and replaces
|
||||
any that do not match (see "Backup layout"). Sunday's run is the `--verify` one,
|
||||
not a second job that night, because a backup that starts while another backup
|
||||
of the same directory is running exits 2 without backing anything up.
|
||||
|
||||
Cron runs the job with a short `PATH`, usually `/usr/bin:/bin`, so the crontab
|
||||
names quak by its full path. To find the saved session, quak needs the same
|
||||
`HOME` as when you logged in, which cron sets from the password file, and on
|
||||
Linux the same `XDG_DATA_HOME`: quak looks for the session in
|
||||
`$XDG_DATA_HOME/quak`, or in `~/.local/share/quak` when that is not set. Cron
|
||||
does not set `XDG_DATA_HOME`, so if your login shell does, set it at the top of
|
||||
the crontab too. Cron does not expand variables in such a line, so give the full
|
||||
path:
|
||||
|
||||
```
|
||||
XDG_DATA_HOME=/home/you/.data
|
||||
```
|
||||
|
||||
On macOS the session is in `~/Library/Application Support/quak`, and only `HOME`
|
||||
matters.
|
||||
|
||||
### Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
| ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `0` | The backup is complete: every original is at its save path, and no file failed. |
|
||||
| `1` | The run finished with files in `failures.json`, or it stopped on another error, printed as one line `quak: <message>`. The next run tries again. |
|
||||
| `2` | Another backup of the same directory is running. This run sent no request and changed nothing. |
|
||||
| `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. |
|
||||
|
||||
After a `1`, the next run fetches each missing original again, fetches the ML
|
||||
data that is not cached, and rebuilds the album links. An original that a
|
||||
`--verify` run could not read, or put back with bytes that still do not match,
|
||||
stays at its save path. Only a later `--verify` run checks it again: a run
|
||||
without `--verify` leaves it as it is and takes the file out of `failures.json`,
|
||||
so that run can exit 0.
|
||||
|
||||
## Examples
|
||||
|
||||
`examples/download-albums.ts` downloads every album's photos and their metadata
|
||||
@@ -524,10 +585,10 @@ stack trace. All three exit with status 3, which means the user must run
|
||||
`quak login` again. `quak logout` is the exception: with no file it says there
|
||||
is no session and exits 0, and it handles a corrupt file or a failed server call
|
||||
as described below. `quak backup` meets an expired session on the refresh that
|
||||
starts every run, before it touches any file. A session that stops working
|
||||
partway through a backup instead fails each remaining file into `failures.json`,
|
||||
so that run exits 1 and the next one stops at its refresh with status 3. No
|
||||
command but `quak login` ever prompts.
|
||||
starts every run, before it touches any file but its lock (see "Backup layout").
|
||||
A session that stops working partway through a backup instead fails each
|
||||
remaining file into `failures.json`, so that run exits 1 and the next one stops
|
||||
at its refresh with status 3. No command but `quak login` ever prompts.
|
||||
|
||||
`quak logout` ends the session on the server, so the token in `session.json`
|
||||
stops working even in a copy of the file, and then deletes the file. If the
|
||||
@@ -582,8 +643,9 @@ reads no tag from is recorded, base64, as `exifRaw`, with the reason in
|
||||
`exifError`. `collections`, `files`, `backup`, `helper list-missing-thumbnails`
|
||||
and `helper fix-missing-thumbnails` take `--json` for machine-readable output.
|
||||
|
||||
`backup --verify` also hashes the originals already in the backup and downloads
|
||||
again any that do not match the content hash Ente records (see "Backup layout").
|
||||
`backup --verify` also hashes the originals already in the backup and replaces
|
||||
any that do not match the content hash Ente records; one it cannot replace goes
|
||||
into `failures.json` (see "Backup layout").
|
||||
|
||||
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
||||
fails, the error is logged, each of its files is written with the reason in an
|
||||
@@ -614,8 +676,11 @@ the smallest does not.
|
||||
below)
|
||||
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
|
||||
keeps, its update time, its private and
|
||||
public magic metadata, its ML data, and
|
||||
its original's EXIF, XMP and dimensions
|
||||
public magic metadata, its ML data, and,
|
||||
for an image (for a live photo, its
|
||||
image), its original's EXIF and XMP, with
|
||||
dimensions for a JPEG only; none of these
|
||||
three for a video
|
||||
YYYY-MM-DD.<fileID>.livephoto.json
|
||||
which of a live photo's two files is which
|
||||
collections/
|
||||
@@ -624,6 +689,7 @@ the smallest does not.
|
||||
(symlink)
|
||||
<name>.json collection metadata + file list
|
||||
account.json the account's email and user ID
|
||||
backup.lock the lock a running backup holds (see below)
|
||||
failures.json files that failed and have not yet succeeded
|
||||
```
|
||||
|
||||
@@ -668,6 +734,23 @@ succeeds, or once it is no longer in the library or in the backup's scope. The
|
||||
library's `lib.backup({ includeThumbnails: true })` also writes
|
||||
`thumbnails/<fileID>.jpg` beside `collections/`; `quak backup` does not.
|
||||
|
||||
`backup.lock` keeps two backups of the same directory from running at once, such
|
||||
as a cron run that starts while the previous one is still going. A backup
|
||||
creates `<dir>` if it is missing and takes the lock before its refresh, and
|
||||
removes the lock when it ends, whether it succeeds or fails; `quak backup` takes
|
||||
it before it opens its library, so a refused run sends no request. The lock is a
|
||||
directory that
|
||||
[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates
|
||||
and keeps touching while the backup runs. A second backup of the directory, from
|
||||
another process or the same one, fails at once: `quak backup` prints
|
||||
`quak: another backup of <dir> is running` and exits with status 2. A run
|
||||
stopped with Ctrl-C or `kill` removes the lock as it exits. Only a run that
|
||||
cannot, such as one killed with SIGKILL or cut off by a crash or power loss,
|
||||
leaves it behind; once it has gone 10 seconds untouched, the next run takes it
|
||||
over, so nobody has to remove it. The lock is outside the date folders and
|
||||
`collections/`, so it is never taken for an original, and the removal of old
|
||||
album directories never touches it.
|
||||
|
||||
A collection's directory and JSON are named after the collection, and a symlink
|
||||
after the file's title, both with unsafe characters replaced. When two
|
||||
collections would get the same name, or two symlinks in one collection the same
|
||||
@@ -705,15 +788,19 @@ original already at its save path the way a download is checked (see "On-disk
|
||||
cache layout" below): its bytes, read in chunks, or a live photo's image and
|
||||
video, joined as `<imageHash>:<videoHash>`. An original that matches the content
|
||||
hash its metadata records is left as it is. One that does not is logged on one
|
||||
line naming the file, deleted (a live photo's image and video both), and
|
||||
downloaded again in the same run like a missing one; if that download fails, the
|
||||
file goes into `failures.json`. A file whose metadata records no hash is left as
|
||||
it is and counted as unchecked. A stored original that cannot be read is left as
|
||||
it is and counts as failed. The summary and `--json` add the counts `verified`,
|
||||
line naming the file, deleted (a live photo's image and video both), and put
|
||||
back in the same run like a missing one: downloaded, or copied from the cache if
|
||||
the cache holds it. What is put back is hashed too, because a copy from the
|
||||
cache is not checked as a download is. If it still does not match, it stays at
|
||||
its save path and the file goes into `failures.json`, as it does when the
|
||||
download fails. A file whose metadata records no hash is left as it is and
|
||||
counted as unchecked. A stored original that cannot be read is left as it is and
|
||||
counts as failed. The summary and `--json` add the counts `verified`,
|
||||
`mismatched` and `unchecked`, all of originals that were already stored; one
|
||||
first downloaded in this run is in none of them. A mismatch that was downloaded
|
||||
again does not make the exit code non-zero. Without `--verify` nothing is
|
||||
hashed, the summary is unchanged, and the three counts are 0 in `--json`.
|
||||
first downloaded in this run is in none of them. A mismatch that was put back
|
||||
with matching bytes does not make the exit code non-zero. Without `--verify`
|
||||
nothing is hashed, the summary is unchanged, and the three counts are 0 in
|
||||
`--json`.
|
||||
|
||||
Each original is written to a temporary file in the same directory, synced to
|
||||
disk, and renamed into place, so an original is either complete or absent, even
|
||||
@@ -930,18 +1017,25 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
`SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
|
||||
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
|
||||
a query vector the caller produced elsewhere.
|
||||
- `await lib.backup(opts?)` → `BackupResult`. It waits for a refresh as
|
||||
- `await lib.backup(opts?)` → `BackupResult`. It takes the lock in the download
|
||||
directory, and fails at once with an error whose `code` is `ELOCKED` while
|
||||
another backup of it runs. A caller can instead take the lock itself, before
|
||||
it opens its library, as `quak backup` does: `await lockBackupDirectory(dir)`
|
||||
takes it, failing the same way, and returns the function that releases it, and
|
||||
the caller passes `lockHeld: true` to the backup. It waits for a refresh as
|
||||
`fresh()` does, puts every in-scope original not already at its save path
|
||||
there as `photo.download()` does (and, with `includeThumbnails`, fetches
|
||||
thumbnails) through the content cache, waits for an ML data fetch, and
|
||||
rebuilds the on-disk backup tree, each file's JSON with its ML data and its
|
||||
original's EXIF, XMP and dimensions, with a durable failure ledger. A fetched
|
||||
original is written straight to its save path and not into the cache, which
|
||||
then counts it as present; one the cache already held is copied from there.
|
||||
`BackupOptions`: `downloadDirectory` (falls back to the library's),
|
||||
`includeOriginals` (default `true`), `includeThumbnails` (default `false`),
|
||||
`onlyAlbumNames`, and `onProgress`. See Backup layout above for the tree it
|
||||
writes.
|
||||
rebuilds the on-disk backup tree with a durable failure ledger. Each file's
|
||||
JSON holds its ML data and, for an image (for a live photo, its image), its
|
||||
original's EXIF and XMP, with its dimensions for a JPEG only; a video's JSON
|
||||
holds none of these three. A fetched original is written straight to its save
|
||||
path and not into the cache, which then counts it as present; one the cache
|
||||
already held is copied from there. `BackupOptions`: `downloadDirectory` (falls
|
||||
back to the library's), `includeOriginals` (default `true`),
|
||||
`includeThumbnails` (default `false`), `onlyAlbumNames`, `verify` (default
|
||||
`false`), `onProgress`, and `lockHeld` (default `false`). See Backup layout
|
||||
above for the tree it writes.
|
||||
|
||||
### Request pools
|
||||
|
||||
|
||||
@@ -25,14 +25,34 @@ declares one.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-06: The README says how to run `quak backup` from cron (issue 170):
|
||||
log in once as the job's user, a crontab with a backup every night and
|
||||
`--verify` on Sundays appending to a log file, and the `HOME` and
|
||||
`XDG_DATA_HOME` the job needs to find the saved session. A table gives each
|
||||
exit code: 0, the backup is complete; 1, files are in `failures.json` or
|
||||
another error stopped the run; 2, another backup of the directory is running;
|
||||
3, there is no usable session. The introduction lists everything the backup
|
||||
keeps for each file.
|
||||
|
||||
- 2026-10-06: `quak backup --verify` and `lib.backup({ verify: true })` hash
|
||||
each original already at its save path as the download check does, streamed, a
|
||||
live photo as `<imageHash>:<videoHash>` (issue 168). One that does not match
|
||||
the content hash its metadata records is logged, removed (both files of a live
|
||||
photo) and downloaded again in the same run; a failed download goes into
|
||||
`failures.json`. One with no recorded hash is left alone. The result, `--json`
|
||||
and the summary gain `verified`, `mismatched` and `unchecked`. Without
|
||||
`--verify` nothing is hashed.
|
||||
photo) and put back in the same run, downloaded or copied from the cache, and
|
||||
what is put back is hashed too. One that still does not match, or whose
|
||||
download fails, goes into `failures.json`. One with no recorded hash is left
|
||||
alone. The result, `--json` and the summary gain `verified`, `mismatched` and
|
||||
`unchecked`. Without `--verify` nothing is hashed.
|
||||
|
||||
- 2026-10-06: Two backups of the same directory never run at once (issue 169).
|
||||
`lib.backup()` takes a lock, `backup.lock` in its download directory, made
|
||||
with `proper-lockfile`, before its refresh, and removes it when it ends,
|
||||
whether it succeeds or fails. A second backup of the directory, from another
|
||||
process or the same one, fails at once with an error naming the directory.
|
||||
`quak backup` takes the lock before it opens its library, so a refused run
|
||||
sends no request; it prints the error as one line and exits 2. A lock that has
|
||||
gone 10 seconds untouched, left by a run that could not remove it, is taken
|
||||
over by the next run.
|
||||
|
||||
- 2026-10-06: `quak backup` writes each original's EXIF, XMP and dimensions into
|
||||
the file's JSON as `imageMetadata`, what `backup-metadata --exif` records
|
||||
|
||||
+3
-1
@@ -36,6 +36,7 @@
|
||||
"devDependencies": {
|
||||
"@eslint/js": "9.38.0",
|
||||
"@types/node": "22.18.13",
|
||||
"@types/proper-lockfile": "4.1.4",
|
||||
"eslint": "9.38.0",
|
||||
"prettier": "3.8.1",
|
||||
"typescript": "5.9.3",
|
||||
@@ -50,6 +51,7 @@
|
||||
"fast-srp-hap": "2.0.4",
|
||||
"fflate": "0.8.3",
|
||||
"jpeg-js": "0.4.4",
|
||||
"libsodium-wrappers-sumo": "0.8.4"
|
||||
"libsodium-wrappers-sumo": "0.8.4",
|
||||
"proper-lockfile": "4.1.2"
|
||||
}
|
||||
}
|
||||
|
||||
+85
-17
@@ -1,11 +1,13 @@
|
||||
// The backup command, rebuilt on the library API (issue #51).
|
||||
//
|
||||
// `lib.backup()` waits for a completed refresh of the library (a failed one
|
||||
// fails the backup before any file is touched), then, for every file in scope,
|
||||
// puts its original at its save path under `downloadDirectory`, as
|
||||
// `Photo.download()` does, waits for an ML data fetch, and rebuilds the derived
|
||||
// views (per-file sidecars, per-collection symlink trees, per-collection JSON)
|
||||
// from the model. The on-disk layout:
|
||||
// `lib.backup()` takes the lock in `downloadDirectory` (unless its caller
|
||||
// already holds it), and fails at once when another backup of it holds the
|
||||
// lock. It waits for a completed refresh of the library (a failed one fails the
|
||||
// backup before any file is touched), then, for every file in scope, puts its
|
||||
// original at its save path under `downloadDirectory`, as `Photo.download()`
|
||||
// does, waits for an ML data fetch, and rebuilds the derived views (per-file
|
||||
// sidecars, per-collection symlink trees, per-collection JSON) from the model.
|
||||
// The on-disk layout:
|
||||
//
|
||||
// <downloadDirectory>/
|
||||
// YYYY/YYYY-MM/YYYY-MM-DD/
|
||||
@@ -17,6 +19,7 @@
|
||||
// collections/<name>/<title> symlink to the original
|
||||
// collections/<name>.json per-collection metadata
|
||||
// account.json the account's email and user ID
|
||||
// backup.lock the lock, while a backup runs
|
||||
// failures.json durable ledger of unresolved failures
|
||||
//
|
||||
// A live photo's original is its image and its video, each with its own
|
||||
@@ -37,7 +40,8 @@
|
||||
//
|
||||
// With `verify`, each original already at its save path is hashed as the
|
||||
// download check hashes it, and one that does not match the content hash its
|
||||
// metadata records is removed and fetched again in the same run.
|
||||
// metadata records is removed and fetched again in the same run. What is put
|
||||
// back is hashed too, and recorded as failed if it still does not match.
|
||||
//
|
||||
// Resilience (issue #8): no per-file condition aborts the run. A failed
|
||||
// download, a failed symlink, or ML data missing because the ML data fetch
|
||||
@@ -65,6 +69,7 @@ import {
|
||||
} from "node:fs";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { dirname, extname, join, relative, resolve } from "node:path";
|
||||
import lockfile from "proper-lockfile";
|
||||
|
||||
import {
|
||||
chunkHashFinal,
|
||||
@@ -103,6 +108,10 @@ export interface BackupOptions {
|
||||
// bytes do not match the content hash its metadata records. Default false.
|
||||
verify?: boolean;
|
||||
onProgress?: ProgressCallback;
|
||||
// The caller already holds the lock in `downloadDirectory`, taken with
|
||||
// `lockBackupDirectory`, and releases it itself, so the backup does not
|
||||
// take it. `quak backup` takes it before it opens its library.
|
||||
lockHeld?: boolean;
|
||||
}
|
||||
|
||||
export interface BackupError {
|
||||
@@ -491,17 +500,12 @@ const writeAlbumJSON = (
|
||||
writeFileSync(path, JSON.stringify(album, null, 2));
|
||||
};
|
||||
|
||||
export const runBackup = async (
|
||||
// The backup itself, which `runBackup` below runs while the lock is held.
|
||||
const runLockedBackup = async (
|
||||
lib: BackupLibrary,
|
||||
opts: BackupOptions,
|
||||
downloadDirectory: string,
|
||||
): Promise<BackupResult> => {
|
||||
const downloadDirectory = opts.downloadDirectory;
|
||||
if (!downloadDirectory) {
|
||||
throw new Error(
|
||||
"backup requires a downloadDirectory (pass one to backup() or " +
|
||||
"open the library with one)",
|
||||
);
|
||||
}
|
||||
const includeOriginals = opts.includeOriginals ?? true;
|
||||
const includeThumbnails = opts.includeThumbnails ?? false;
|
||||
const log = opts.onProgress ?? (() => {});
|
||||
@@ -606,6 +610,7 @@ export const runBackup = async (
|
||||
if (includeOriginals) {
|
||||
for (const [fileID, file] of distinct) {
|
||||
let stored = storedAtSavePath(downloadDirectory, file);
|
||||
let mismatch = false;
|
||||
if (stored !== undefined && verify) {
|
||||
try {
|
||||
if (file.metadata.hash === undefined) {
|
||||
@@ -619,6 +624,7 @@ export const runBackup = async (
|
||||
`MISMATCH original ${file.metadata.title} (${fileID}): its bytes do not match its content hash`,
|
||||
);
|
||||
mismatched++;
|
||||
mismatch = true;
|
||||
rmSync(stored.path);
|
||||
if (stored.videoPath !== undefined) {
|
||||
rmSync(stored.videoPath);
|
||||
@@ -646,10 +652,24 @@ export const runBackup = async (
|
||||
// A fetched original is written straight to its save path (a
|
||||
// live photo beside it); only one that was already cached
|
||||
// elsewhere is copied.
|
||||
await placeOriginal(downloadDirectory, file, (dest) =>
|
||||
lib.original(fileID, dest),
|
||||
const placed = await placeOriginal(
|
||||
downloadDirectory,
|
||||
file,
|
||||
(dest) => lib.original(fileID, dest),
|
||||
);
|
||||
storedThisRun.add(fileID);
|
||||
// A copy from the cache is not checked as a download is and can
|
||||
// hold the same bad bytes, so what is put back after a
|
||||
// mismatch is hashed too. A bad copy stays where it is and the
|
||||
// file fails.
|
||||
if (
|
||||
mismatch &&
|
||||
(await storedHash(placed)) !== file.metadata.hash
|
||||
) {
|
||||
throw new Error(
|
||||
"the original put back does not match its content hash either",
|
||||
);
|
||||
}
|
||||
downloaded++;
|
||||
} catch (err) {
|
||||
log(
|
||||
@@ -819,3 +839,51 @@ export const runBackup = async (
|
||||
errors,
|
||||
};
|
||||
};
|
||||
|
||||
// Only one backup of a directory runs at a time, in this process or another.
|
||||
// Creates `downloadDirectory` if it is missing, takes the lock in it and
|
||||
// returns the function that releases it; while another backup holds the lock,
|
||||
// fails at once with an error whose `code` is `ELOCKED`. The lock is the
|
||||
// directory `backup.lock`, whose modification time proper-lockfile keeps
|
||||
// current while it is held. One it has not touched for 10 seconds was left by
|
||||
// a run that could not remove it, and is taken over.
|
||||
export const lockBackupDirectory = async (
|
||||
downloadDirectory: string,
|
||||
): Promise<() => Promise<void>> => {
|
||||
mkdirSync(downloadDirectory, { recursive: true });
|
||||
try {
|
||||
return await lockfile.lock(downloadDirectory, {
|
||||
lockfilePath: join(downloadDirectory, "backup.lock"),
|
||||
});
|
||||
} catch (err) {
|
||||
if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;
|
||||
throw Object.assign(
|
||||
new Error(`another backup of ${downloadDirectory} is running`),
|
||||
{ code: "ELOCKED" },
|
||||
);
|
||||
}
|
||||
};
|
||||
|
||||
// Runs the backup holding the lock, which it takes and releases itself unless
|
||||
// the caller already holds it (`opts.lockHeld`).
|
||||
export const runBackup = async (
|
||||
lib: BackupLibrary,
|
||||
opts: BackupOptions,
|
||||
): Promise<BackupResult> => {
|
||||
const downloadDirectory = opts.downloadDirectory;
|
||||
if (!downloadDirectory) {
|
||||
throw new Error(
|
||||
"backup requires a downloadDirectory (pass one to backup() or " +
|
||||
"open the library with one)",
|
||||
);
|
||||
}
|
||||
if (opts.lockHeld) {
|
||||
return runLockedBackup(lib, opts, downloadDirectory);
|
||||
}
|
||||
const release = await lockBackupDirectory(downloadDirectory);
|
||||
try {
|
||||
return await runLockedBackup(lib, opts, downloadDirectory);
|
||||
} finally {
|
||||
await release();
|
||||
}
|
||||
};
|
||||
|
||||
+52
-36
@@ -20,6 +20,7 @@ import {
|
||||
type ClientSnapshot,
|
||||
type LoginOptions,
|
||||
} from "./client.js";
|
||||
import { lockBackupDirectory } from "./backup.js";
|
||||
import { init } from "./crypto/index.js";
|
||||
import {
|
||||
defaultCacheDirectory,
|
||||
@@ -408,50 +409,65 @@ export const backupCommand = async (
|
||||
if (!client) return 3;
|
||||
|
||||
ctx.stderr.write("Starting backup...\n");
|
||||
// The precache is off: the backup fetches what it needs, and must not
|
||||
// also fill the cache with every thumbnail and the recent originals.
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
downloadDirectory: dir,
|
||||
cacheDirectory: ctx.cacheDir,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
// The lock is taken before the library opens and starts its refresh, so a
|
||||
// run refused while another backup of `dir` runs sends no request.
|
||||
let release: () => Promise<void>;
|
||||
try {
|
||||
const result = await lib.backup({
|
||||
release = await lockBackupDirectory(dir);
|
||||
} catch (err) {
|
||||
if ((err as NodeJS.ErrnoException).code !== "ELOCKED") throw err;
|
||||
ctx.stderr.write(`quak: ${(err as Error).message}\n`);
|
||||
return 2;
|
||||
}
|
||||
try {
|
||||
// The precache is off: the backup fetches what it needs, and must not
|
||||
// also fill the cache with every thumbnail and the recent originals.
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
downloadDirectory: dir,
|
||||
verify: opts.verify,
|
||||
onProgress: (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
},
|
||||
cacheDirectory: ctx.cacheDir,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
try {
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: dir,
|
||||
lockHeld: true,
|
||||
verify: opts.verify,
|
||||
onProgress: (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
},
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(JSON.stringify(result, null, 2) + "\n");
|
||||
} else {
|
||||
ctx.stderr.write("\n--- Backup complete ---\n");
|
||||
ctx.stderr.write(` Total files: ${result.totalFiles}\n`);
|
||||
ctx.stderr.write(` Downloaded: ${result.downloaded}\n`);
|
||||
ctx.stderr.write(` Skipped: ${result.skipped}\n`);
|
||||
if (opts.verify) {
|
||||
ctx.stderr.write(` Verified: ${result.verified}\n`);
|
||||
ctx.stderr.write(` Mismatched: ${result.mismatched}\n`);
|
||||
ctx.stderr.write(` Unchecked: ${result.unchecked}\n`);
|
||||
}
|
||||
ctx.stderr.write(` Failed: ${result.failed}\n`);
|
||||
if (result.errors.length > 0) {
|
||||
ctx.stderr.write("\nFailed files:\n");
|
||||
for (const e of result.errors) {
|
||||
ctx.stderr.write(
|
||||
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
|
||||
);
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(JSON.stringify(result, null, 2) + "\n");
|
||||
} else {
|
||||
ctx.stderr.write("\n--- Backup complete ---\n");
|
||||
ctx.stderr.write(` Total files: ${result.totalFiles}\n`);
|
||||
ctx.stderr.write(` Downloaded: ${result.downloaded}\n`);
|
||||
ctx.stderr.write(` Skipped: ${result.skipped}\n`);
|
||||
if (opts.verify) {
|
||||
ctx.stderr.write(` Verified: ${result.verified}\n`);
|
||||
ctx.stderr.write(` Mismatched: ${result.mismatched}\n`);
|
||||
ctx.stderr.write(` Unchecked: ${result.unchecked}\n`);
|
||||
}
|
||||
ctx.stderr.write(` Failed: ${result.failed}\n`);
|
||||
if (result.errors.length > 0) {
|
||||
ctx.stderr.write("\nFailed files:\n");
|
||||
for (const e of result.errors) {
|
||||
ctx.stderr.write(
|
||||
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return result.failed > 0 ? 1 : 0;
|
||||
return result.failed > 0 ? 1 : 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
} finally {
|
||||
await lib.close();
|
||||
await release();
|
||||
}
|
||||
};
|
||||
|
||||
|
||||
@@ -67,6 +67,7 @@ export {
|
||||
type EnsureOptions,
|
||||
type EnsureResult,
|
||||
type EnsureEvent,
|
||||
lockBackupDirectory,
|
||||
runBackup,
|
||||
type BackupOptions,
|
||||
type BackupResult,
|
||||
|
||||
+11
-7
@@ -94,6 +94,7 @@ import type { Collection, EnteFile } from "../model/types.js";
|
||||
import { runBackup, type BackupOptions, type BackupResult } from "../backup.js";
|
||||
|
||||
export {
|
||||
lockBackupDirectory,
|
||||
runBackup,
|
||||
type BackupOptions,
|
||||
type BackupResult,
|
||||
@@ -566,13 +567,16 @@ export class Library {
|
||||
|
||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||
// library's, each original at its save path, with a durable failure
|
||||
// ledger (issue #51). Waits for a completed refresh first, as `fresh()`
|
||||
// does, joining one already running, and rejects before touching any file
|
||||
// when it fails. Then puts pending originals at their save paths as
|
||||
// `Photo.download()` does (and optional thumbnails) through the content
|
||||
// cache and pools, waits for an ML data fetch, and rebuilds the derived
|
||||
// symlink/JSON views from the model, each file's JSON with its ML data,
|
||||
// beside an `account.json` with the account's email and user ID.
|
||||
// ledger (issue #51). Takes the lock in that directory first, unless
|
||||
// `opts.lockHeld` says the caller holds it, failing at once while another
|
||||
// backup of it runs (see `runBackup`). Waits for a
|
||||
// completed refresh, as `fresh()` does, joining one already running, and
|
||||
// rejects before touching any file but the lock when it fails. Then puts
|
||||
// pending originals at their save paths as `Photo.download()` does (and
|
||||
// optional thumbnails) through the content cache and pools, waits for an
|
||||
// ML data fetch, and rebuilds the derived symlink/JSON views from the
|
||||
// model, each file's JSON with its ML data, beside an `account.json` with
|
||||
// the account's email and user ID.
|
||||
// Throws before any network work when no content cache backs the
|
||||
// originals it must fetch.
|
||||
backup(opts?: BackupOptions): Promise<BackupResult> {
|
||||
|
||||
+103
-1
@@ -43,6 +43,7 @@ import {
|
||||
readlinkSync,
|
||||
rmSync,
|
||||
symlinkSync,
|
||||
utimesSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { spawnSync } from "node:child_process";
|
||||
@@ -831,7 +832,81 @@ describe("the refresh before a backup", () => {
|
||||
);
|
||||
|
||||
expect(source.originalCalls).toBe(0);
|
||||
expect(existsSync(outDir)).toBe(false);
|
||||
// The backup made the directory for its lock, and removed the lock.
|
||||
expect(readdirSync(outDir)).toEqual([]);
|
||||
await lib.close();
|
||||
});
|
||||
});
|
||||
|
||||
describe("the backup lock", () => {
|
||||
const lockPath = (outDir: string): string => join(outDir, "backup.lock");
|
||||
|
||||
it("refuses a second backup of the directory while one runs", async () => {
|
||||
await fillCache();
|
||||
const client = new HeldClient();
|
||||
const lib = await openLibrary(stubSource(), client);
|
||||
const outDir = join(root, "backup");
|
||||
// The first backup holds the lock once it starts its refresh, which
|
||||
// `HeldClient` keeps from finishing.
|
||||
let refreshing!: () => void;
|
||||
const started = new Promise<void>((resolve) => {
|
||||
refreshing = resolve;
|
||||
});
|
||||
const first = lib.backup({
|
||||
downloadDirectory: outDir,
|
||||
onProgress: (msg) => {
|
||||
if (msg === "Refreshing library...") refreshing();
|
||||
},
|
||||
});
|
||||
await started;
|
||||
|
||||
await expect(lib.backup({ downloadDirectory: outDir })).rejects.toThrow(
|
||||
`another backup of ${outDir} is running`,
|
||||
);
|
||||
|
||||
client.release();
|
||||
expect((await first).failed).toBe(0);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it("releases the lock after a backup succeeds", async () => {
|
||||
const lib = await openLibrary(stubSource());
|
||||
const outDir = join(root, "backup");
|
||||
|
||||
await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
expect(existsSync(lockPath(outDir))).toBe(false);
|
||||
// So the next backup of the directory runs.
|
||||
expect((await lib.backup({ downloadDirectory: outDir })).skipped).toBe(
|
||||
3,
|
||||
);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it("releases the lock after a backup fails", async () => {
|
||||
const lib = await openLibrary(stubSource(), new FailingClient());
|
||||
const outDir = join(root, "backup");
|
||||
|
||||
await expect(lib.backup({ downloadDirectory: outDir })).rejects.toThrow(
|
||||
"HTTP 401 from server",
|
||||
);
|
||||
|
||||
expect(existsSync(lockPath(outDir))).toBe(false);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it("takes over a lock left by a run that was killed", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
// A killed run's lock, last touched a minute ago.
|
||||
mkdirSync(lockPath(outDir), { recursive: true });
|
||||
const minuteAgo = new Date(Date.now() - 60_000);
|
||||
utimesSync(lockPath(outDir), minuteAgo, minuteAgo);
|
||||
const lib = await openLibrary(stubSource());
|
||||
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
expect(result.downloaded).toBe(3);
|
||||
expect(existsSync(lockPath(outDir))).toBe(false);
|
||||
await lib.close();
|
||||
});
|
||||
});
|
||||
@@ -1308,6 +1383,33 @@ describe("backup with verify", () => {
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it("records in failures.json an original the cache puts back with the same bad bytes", async () => {
|
||||
const lib = await openLibrary(stubSource(), new HashedClient());
|
||||
const outDir = join(root, "backup");
|
||||
// The cache holds a bad copy, and a backup copies an original the
|
||||
// cache holds to its save path.
|
||||
const cached = await lib.photos.byID({ fileID: 100 })!.original();
|
||||
writeFileSync(cached.path, corrupt);
|
||||
await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: outDir,
|
||||
verify: true,
|
||||
});
|
||||
|
||||
expect(result).toMatchObject({
|
||||
downloaded: 0,
|
||||
mismatched: 1,
|
||||
failed: 1,
|
||||
});
|
||||
expect(result.errors.map((e) => e.error)).toEqual([
|
||||
"the original put back does not match its content hash either",
|
||||
]);
|
||||
expect(Object.keys(readLedger(outDir).files)).toEqual(["100"]);
|
||||
expect(readFileSync(saved(outDir, "100.jpg"))).toEqual(corrupt);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it("hashes nothing without verify", async () => {
|
||||
const { lib, outDir } = await backedUp();
|
||||
writeFileSync(saved(outDir, "100.jpg"), corrupt);
|
||||
|
||||
@@ -18,6 +18,7 @@ import {
|
||||
readFileSync,
|
||||
rmSync,
|
||||
statSync,
|
||||
utimesSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { join } from "node:path";
|
||||
@@ -704,6 +705,7 @@ describe("backup", () => {
|
||||
" Failed: 0\n",
|
||||
);
|
||||
expect(stdout.text).toBe("");
|
||||
expect(existsSync(join(dir, "backup.lock"))).toBe(false);
|
||||
});
|
||||
|
||||
// The backup opens its library with the precache off: it fetches the
|
||||
@@ -848,7 +850,7 @@ describe("backup", () => {
|
||||
expect(code).toBe(1);
|
||||
expect(runText).toBe("quak: HTTP 503 from server\n");
|
||||
expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n");
|
||||
expect(existsSync(dir)).toBe(false);
|
||||
expect(readdirSync(dir)).toEqual([]);
|
||||
});
|
||||
|
||||
// A real saved session, read back by `loadSession`, whose server answers
|
||||
@@ -879,7 +881,38 @@ describe("backup", () => {
|
||||
`quak: the saved session is no longer valid; run "quak login"\n`,
|
||||
);
|
||||
expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n");
|
||||
expect(existsSync(dir)).toBe(false);
|
||||
expect(readdirSync(dir)).toEqual([]);
|
||||
});
|
||||
|
||||
it("exits 2 with one line naming the directory, sending no request, while another backup of it runs", async () => {
|
||||
const dir = join(root, "backup");
|
||||
// The lock another backup holds. Its modification time is set an hour
|
||||
// ahead, so it stays current however long this test takes.
|
||||
const lock = join(dir, "backup.lock");
|
||||
mkdirSync(lock, { recursive: true });
|
||||
const hourAhead = new Date(Date.now() + 3_600_000);
|
||||
utimesSync(lock, hourAhead, hourAhead);
|
||||
// A client whose refresh never finishes, so a run that started one
|
||||
// before it exits would never return.
|
||||
let requests = 0;
|
||||
const never = (): Promise<never> => {
|
||||
requests++;
|
||||
return new Promise(() => {});
|
||||
};
|
||||
const client = {
|
||||
...fakeClient(),
|
||||
collectionsSince: never,
|
||||
filesSince: never,
|
||||
} as unknown as Client;
|
||||
|
||||
expect(await backupCommand(context(client), dir, {})).toBe(2);
|
||||
expect(requests).toBe(0);
|
||||
expect(stderr.text).toBe(
|
||||
`Starting backup...\nquak: another backup of ${dir} is running\n`,
|
||||
);
|
||||
expect(readdirSync(dir)).toEqual(["backup.lock"]);
|
||||
// Opening the library would have made its cache directory.
|
||||
expect(existsSync(join(root, "cache"))).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -535,6 +535,18 @@
|
||||
dependencies:
|
||||
undici-types "~6.21.0"
|
||||
|
||||
"@types/proper-lockfile@4.1.4":
|
||||
version "4.1.4"
|
||||
resolved "https://registry.yarnpkg.com/@types/proper-lockfile/-/proper-lockfile-4.1.4.tgz#cd9fab92bdb04730c1ada542c356f03620f84008"
|
||||
integrity sha512-uo2ABllncSqg9F1D4nugVl9v93RmjxF6LJzQLMLDdPaXCUIDPeOJ21Gbqi43xNKzBi/WQ0Q0dICqufzQbMjipQ==
|
||||
dependencies:
|
||||
"@types/retry" "*"
|
||||
|
||||
"@types/retry@*":
|
||||
version "0.12.5"
|
||||
resolved "https://registry.yarnpkg.com/@types/retry/-/retry-0.12.5.tgz#f090ff4bd8d2e5b940ff270ab39fd5ca1834a07e"
|
||||
integrity sha512-3xSjTp3v03X/lSQLkczaN9UIEwJMoMCA1+Nb5HfbJEQWogdeQIyVtTvxPXDQjZ5zws8rFQfVfRdz03ARihPJgw==
|
||||
|
||||
"@typescript-eslint/eslint-plugin@8.46.2":
|
||||
version "8.46.2"
|
||||
resolved "https://registry.yarnpkg.com/@typescript-eslint/eslint-plugin/-/eslint-plugin-8.46.2.tgz#dc4ab93ee3d7e6c8e38820a0d6c7c93c7183e2dc"
|
||||
@@ -1140,6 +1152,11 @@ globals@^14.0.0:
|
||||
resolved "https://registry.yarnpkg.com/globals/-/globals-14.0.0.tgz#898d7413c29babcf6bafe56fcadded858ada724e"
|
||||
integrity sha512-oahGvuMGQlPw/ivIYBjVSrWAfWLBeku5tpPE2fOPLi+WHffIWbuh2tCjhyQhTBPMf5E9jDEH4FOmTYgYwbKwtQ==
|
||||
|
||||
graceful-fs@^4.2.4:
|
||||
version "4.2.11"
|
||||
resolved "https://registry.yarnpkg.com/graceful-fs/-/graceful-fs-4.2.11.tgz#4183e4e8bf08bb6e05bbb2f7d2e0c8f712ca40e3"
|
||||
integrity sha512-RbJ5/jmFcNNCcDV5o9eTnBLJ/HszWV0P73bc+Ff4nS/rJj+YaS6IGyiOL0VoBYX+l1Wrl3k63h/KrH+nhJ0XvQ==
|
||||
|
||||
graphemer@^1.4.0:
|
||||
version "1.4.0"
|
||||
resolved "https://registry.yarnpkg.com/graphemer/-/graphemer-1.4.0.tgz#fb2f1d55e0e3a1849aeffc90c4fa0dd53a0e66c6"
|
||||
@@ -1414,6 +1431,15 @@ prettier@3.8.1:
|
||||
resolved "https://registry.yarnpkg.com/prettier/-/prettier-3.8.1.tgz#edf48977cf991558f4fcbd8a3ba6015ba2a3a173"
|
||||
integrity sha512-UOnG6LftzbdaHZcKoPFtOcCKztrQ57WkHDeRD9t/PTQtmT0NHSeWWepj6pS0z/N7+08BHFDQVUrfmfMRcZwbMg==
|
||||
|
||||
proper-lockfile@4.1.2:
|
||||
version "4.1.2"
|
||||
resolved "https://registry.yarnpkg.com/proper-lockfile/-/proper-lockfile-4.1.2.tgz#c8b9de2af6b2f1601067f98e01ac66baa223141f"
|
||||
integrity sha512-TjNPblN4BwAWMXU8s9AEz4JmQxnD1NNL7bNOY/AKUzyamc379FWASUhc/K1pL2noVb+XmZKLL68cjzLsiOAMaA==
|
||||
dependencies:
|
||||
graceful-fs "^4.2.4"
|
||||
retry "^0.12.0"
|
||||
signal-exit "^3.0.2"
|
||||
|
||||
punycode@^2.1.0:
|
||||
version "2.3.1"
|
||||
resolved "https://registry.yarnpkg.com/punycode/-/punycode-2.3.1.tgz#027422e2faec0b25e1549c3e1bd8309b9133b6e5"
|
||||
@@ -1429,6 +1455,11 @@ resolve-from@^4.0.0:
|
||||
resolved "https://registry.yarnpkg.com/resolve-from/-/resolve-from-4.0.0.tgz#4abcd852ad32dd7baabfe9b40e00a36db5f392e6"
|
||||
integrity sha512-pb/MYmXstAkysRFx8piNI1tGFNQIFA3vkE3Gq4EuA1dF6gHp/+vgZqsCGJapvy8N3Q+4o7FwvquPJcnZ7RYy4g==
|
||||
|
||||
retry@^0.12.0:
|
||||
version "0.12.0"
|
||||
resolved "https://registry.yarnpkg.com/retry/-/retry-0.12.0.tgz#1b42a6266a21f07421d1b0b54b7dc167b01c013b"
|
||||
integrity sha512-9LkiTwjUh6rT555DtE9rTX+BKByPfrMzEAtnlEtdEwr3Nkffwiihqe2bWADg+OQRjt9gl6ICdmB/ZFDCGAtSow==
|
||||
|
||||
reusify@^1.0.4:
|
||||
version "1.1.0"
|
||||
resolved "https://registry.yarnpkg.com/reusify/-/reusify-1.1.0.tgz#0fe13b9522e1473f51b558ee796e08f11f9b489f"
|
||||
@@ -1502,6 +1533,11 @@ siginfo@^2.0.0:
|
||||
resolved "https://registry.yarnpkg.com/siginfo/-/siginfo-2.0.0.tgz#32e76c70b79724e3bb567cb9d543eb858ccfaf30"
|
||||
integrity sha512-ybx0WO1/8bSBLEWXZvEd7gMW3Sn3JFlW3TvX1nREbDLRNQNaeNN8WK0meBwPdAaOI7TtRRRJn/Es1zhrrCHu7g==
|
||||
|
||||
signal-exit@^3.0.2:
|
||||
version "3.0.7"
|
||||
resolved "https://registry.yarnpkg.com/signal-exit/-/signal-exit-3.0.7.tgz#a9a1767f8af84155114eaabd73f99273c8f59ad9"
|
||||
integrity sha512-wnD2ZE+l+SPC/uoS0vXeE9L1+0wuaMqKlfz9AMUo38JsyLSBWSFcHR1Rri62LZc12vLr1gb3jl7iwQhgwpAbGQ==
|
||||
|
||||
signal-exit@^4.1.0:
|
||||
version "4.1.0"
|
||||
resolved "https://registry.yarnpkg.com/signal-exit/-/signal-exit-4.1.0.tgz#952188c1cbd546070e2dd20d0f41c0ae0530cb04"
|
||||
|
||||
Reference in New Issue
Block a user