Compare commits
2
Commits
1283048ce2
...
6967117cee
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6967117cee | ||
|
|
ab4d5d2cc2 |
@@ -516,8 +516,17 @@ The CLI stores the snapshot at the platform-appropriate data directory via
|
||||
`0600`. The key material is stored in cleartext in the JSON; treat this file as
|
||||
you would treat the password itself. A missing file is reported as "not logged
|
||||
in"; a file that exists but is corrupt is reported as such, naming the bad
|
||||
field. Both exit with status 1, except that `quak logout` with no file says
|
||||
there is no session and exits 0.
|
||||
field. When the refresh a command starts with gets HTTP 401 from the server,
|
||||
because it no longer accepts the saved session's token, the command prints one
|
||||
line, `quak: the saved session is no longer valid; run "quak login"`, with no
|
||||
stack trace. All three exit with status 3, which means the user must run
|
||||
`quak login` again. `quak logout` is the exception: with no file it says there
|
||||
is no session and exits 0, and it handles a corrupt file or a failed server call
|
||||
as described below. `quak backup` meets an expired session on the refresh that
|
||||
starts every run, before it touches any file. A session that stops working
|
||||
partway through a backup instead fails each remaining file into `failures.json`,
|
||||
so that run exits 1 and the next one stops at its refresh with status 3. No
|
||||
command but `quak login` ever prompts.
|
||||
|
||||
`quak logout` ends the session on the server, so the token in `session.json`
|
||||
stops working even in a copy of the file, and then deletes the file. If the
|
||||
@@ -551,8 +560,9 @@ library. The read commands — `collections`, `files`, `get`, `get-thumb`,
|
||||
`helper fix-missing-thumbnails` — force a fresh server round-trip before they
|
||||
answer, so they report current account state rather than whatever the cache last
|
||||
held. If that round-trip fails, the command prints the error on one line and
|
||||
exits 1. `--cache-dir` overrides where the cache lives; without it each account
|
||||
gets its own directory under the per-user cache path.
|
||||
exits 1, or 3 when the server no longer accepts the saved session (see "Session
|
||||
handling"). `--cache-dir` overrides where the cache lives; without it each
|
||||
account gets its own directory under the per-user cache path.
|
||||
|
||||
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
|
||||
accepted for backward compatibility but ignored. For a live photo, `get` writes
|
||||
@@ -599,8 +609,8 @@ the smallest does not.
|
||||
per unique file, two for a live photo: see
|
||||
below)
|
||||
YYYY-MM-DD.<fileID>.json the file's basic metadata fields quak
|
||||
keeps, and its private and public magic
|
||||
metadata
|
||||
keeps, its private and public magic
|
||||
metadata, and its ML data
|
||||
YYYY-MM-DD.<fileID>.livephoto.json
|
||||
which of a live photo's two files is which
|
||||
collections/
|
||||
@@ -619,6 +629,14 @@ name as uploaded, case kept, or `.bin` when it has none or it holds anything but
|
||||
letters and digits. When the date or the time zone changes, the next run saves
|
||||
the original at its new path and leaves the old copy where it is.
|
||||
|
||||
A file's JSON holds Ente's ML data for it (its faces and its CLIP embedding) as
|
||||
`mlData`, the same payload `backup-metadata` writes; a file Ente has no ML data
|
||||
for has no `mlData`. The backup waits for the library's ML data fetch to finish
|
||||
before it writes the JSON files. If that fetch fails, each file whose ML data is
|
||||
not in the cache gets the reason in `mlDataError` instead and counts as failed,
|
||||
and the next run fetches it again. The JSON files are rewritten on every run, so
|
||||
ML data that arrived since the last run appears.
|
||||
|
||||
`failures.json` records each failed file with the kind of failure, how many
|
||||
times it has been tried and when it was last tried. A file leaves it once it
|
||||
succeeds, or once it is no longer in the library or in the backup's scope. The
|
||||
@@ -655,8 +673,7 @@ subsequent runs, existing originals are skipped. If a download fails, the error
|
||||
is logged and the backup continues with the next file. The exit code is non-zero
|
||||
if any files failed. `quak backup` opens its library with the thumbnail and
|
||||
originals precache off, so the only file content it fetches is the originals the
|
||||
backup stores. The library's ML data fetch still runs and fills the cache's
|
||||
`mldata/`.
|
||||
backup stores.
|
||||
|
||||
Each original is written to a temporary file in the same directory, synced to
|
||||
disk, and renamed into place, so an original is either complete or absent, even
|
||||
@@ -876,11 +893,12 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
- `await lib.backup(opts?)` → `BackupResult`. It waits for a refresh as
|
||||
`fresh()` does, puts every in-scope original not already at its save path
|
||||
there as `photo.download()` does (and, with `includeThumbnails`, fetches
|
||||
thumbnails) through the content cache, and rebuilds the on-disk backup tree
|
||||
with a durable failure ledger. A fetched original is written straight to its
|
||||
save path and not into the cache, which then counts it as present; one the
|
||||
cache already held is copied from there. `BackupOptions`: `downloadDirectory`
|
||||
(falls back to the library's), `includeOriginals` (default `true`),
|
||||
thumbnails) through the content cache, waits for an ML data fetch, and
|
||||
rebuilds the on-disk backup tree, each file's JSON with its ML data, with a
|
||||
durable failure ledger. A fetched original is written straight to its save
|
||||
path and not into the cache, which then counts it as present; one the cache
|
||||
already held is copied from there. `BackupOptions`: `downloadDirectory` (falls
|
||||
back to the library's), `includeOriginals` (default `true`),
|
||||
`includeThumbnails` (default `false`), `onlyAlbumNames`, and `onProgress`. See
|
||||
Backup layout above for the tree it writes.
|
||||
|
||||
|
||||
@@ -25,6 +25,22 @@ declares one.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-05: When the refresh a command starts with gets HTTP 401, because the
|
||||
server no longer accepts the saved session's token, the command prints one
|
||||
line, `quak: the saved session is no longer valid; run "quak login"`, and
|
||||
exits 3 (issue 164). A missing or corrupt session file keeps its message and
|
||||
now also exits 3, so a cron job can tell that the user must log in again.
|
||||
`run` in `src/cli-run.ts` recognises the 401, which reaches it unchanged from
|
||||
the refresh, so `quak backup` stops there before it touches any file. Every
|
||||
other error still exits 1.
|
||||
|
||||
- 2026-10-05: `quak backup` writes each file's ML data (its faces and CLIP
|
||||
embedding) into the file's JSON as `mlData`, the payload
|
||||
`lib.mldata.forFile()` returns (issue 163). `lib.backup()` waits for an ML
|
||||
data fetch before it writes the JSON files. When that fetch fails, each file
|
||||
whose ML data is not cached gets the reason in `mlDataError` and counts as
|
||||
failed, and the next run fetches it again.
|
||||
|
||||
- 2026-10-03: The download-albums example test no longer fails on a disk with
|
||||
under 50 GiB free (issue 160). It opens its libraries with
|
||||
`freeBelowBytes: 0`, so the free space of the disk it runs on cannot shrink
|
||||
|
||||
+60
-16
@@ -3,14 +3,15 @@
|
||||
// `lib.backup()` waits for a completed refresh of the library (a failed one
|
||||
// fails the backup before any file is touched), then, for every file in scope,
|
||||
// puts its original at its save path under `downloadDirectory`, as
|
||||
// `Photo.download()` does, and rebuilds the derived views (per-file sidecars,
|
||||
// per-collection symlink trees, per-collection JSON) from the model. The
|
||||
// on-disk layout:
|
||||
// `Photo.download()` does, waits for an ML data fetch, and rebuilds the derived
|
||||
// views (per-file sidecars, per-collection symlink trees, per-collection JSON)
|
||||
// from the model. The on-disk layout:
|
||||
//
|
||||
// <downloadDirectory>/
|
||||
// YYYY/YYYY-MM/YYYY-MM-DD/
|
||||
// YYYY-MM-DD.<fileID>.<ext> the decrypted bytes (the save path)
|
||||
// YYYY-MM-DD.<fileID>.json per-file metadata sidecar
|
||||
// YYYY-MM-DD.<fileID>.json per-file metadata sidecar, with
|
||||
// the file's ML data
|
||||
// collections/<name>/<title> symlink to the original
|
||||
// collections/<name>.json per-collection metadata
|
||||
// failures.json durable ledger of unresolved failures
|
||||
@@ -29,13 +30,15 @@
|
||||
// directories of albums that no longer exist.
|
||||
//
|
||||
// Resilience (issue #8): no per-file condition aborts the run. A failed
|
||||
// download or a failed symlink is caught, recorded in `failures.json` with a
|
||||
// classification, a running attempt count, and the last-tried time, and the run
|
||||
// continues. `result.failed` — and thus the CLI's exit code — stays non-zero
|
||||
// while any failure remains unresolved and clears once every one succeeds. Each
|
||||
// run reconciles the ledger against the files it attempted, so an entry for a
|
||||
// file that has since left the library (deleted) or this run's scope is dropped
|
||||
// rather than counted forever, which would poison a scheduled backup's exit code.
|
||||
// download, a failed symlink, or ML data missing because the ML data fetch
|
||||
// failed is caught, recorded in `failures.json` with a classification, a
|
||||
// running attempt count, and the last-tried time, and the run continues.
|
||||
// `result.failed` — and thus the CLI's exit code — stays non-zero while any
|
||||
// failure remains unresolved and clears once every one succeeds. Each run
|
||||
// reconciles the ledger against the files it attempted, so an entry for a file
|
||||
// that has since left the library (deleted) or this run's scope is dropped
|
||||
// rather than counted forever, which would poison a scheduled backup's exit
|
||||
// code.
|
||||
|
||||
import {
|
||||
lstatSync,
|
||||
@@ -60,6 +63,7 @@ import {
|
||||
storedAtSavePath,
|
||||
} from "./library/content.js";
|
||||
import { representative } from "./library/records.js";
|
||||
import type { MLData } from "./mldata-fetch.js";
|
||||
import type { Collection, EnteFile } from "./model/types.js";
|
||||
|
||||
export type ProgressCallback = (message: string) => void;
|
||||
@@ -117,6 +121,13 @@ export interface BackupLibrary {
|
||||
destination: string,
|
||||
): Promise<{ path: string; videoPath?: string }>;
|
||||
thumbnail(fileID: number): Promise<{ path: string }>;
|
||||
// Wait for an ML data fetch to complete, joining one already running or
|
||||
// starting one. Rejects with the reason when it fails; resolves at once
|
||||
// when the library cannot fetch ML data.
|
||||
fetchMLData(): Promise<void>;
|
||||
// A file's cached ML data, as `lib.mldata.forFile()` returns it, or
|
||||
// undefined when none is cached.
|
||||
mlData(fileID: number): Promise<MLData | undefined>;
|
||||
}
|
||||
|
||||
type FailureClass = "transient" | "permanent" | "unknown";
|
||||
@@ -340,7 +351,13 @@ const saveLedger = (path: string, ledger: Map<number, FailureEntry>): void => {
|
||||
);
|
||||
};
|
||||
|
||||
const writeSidecar = (path: string, file: EnteFile): void => {
|
||||
// The file's JSON: its basic fields, its magic metadata, and its ML data, or
|
||||
// the reason the ML data is missing.
|
||||
const writeSidecar = (
|
||||
path: string,
|
||||
file: EnteFile,
|
||||
ml: { mlData?: MLData; mlDataError?: string },
|
||||
): void => {
|
||||
const meta: Record<string, unknown> = {
|
||||
id: file.id,
|
||||
collectionID: file.collectionID,
|
||||
@@ -349,6 +366,8 @@ const writeSidecar = (path: string, file: EnteFile): void => {
|
||||
};
|
||||
if (file.magicMetadata) meta.magicMetadata = file.magicMetadata;
|
||||
if (file.pubMagicMetadata) meta.pubMagicMetadata = file.pubMagicMetadata;
|
||||
if (ml.mlData) meta.mlData = ml.mlData;
|
||||
if (ml.mlDataError) meta.mlDataError = ml.mlDataError;
|
||||
writeFileSync(path, JSON.stringify(meta, null, 2));
|
||||
};
|
||||
|
||||
@@ -497,12 +516,37 @@ export const runBackup = async (
|
||||
}
|
||||
|
||||
// Phase 2: rebuild the derived views from the model. Sidecars first, for
|
||||
// every present original (this repairs stale ones).
|
||||
// every present original (this repairs stale ones), each with the file's
|
||||
// ML data once an ML data fetch has completed. When the fetch fails, a
|
||||
// file with no cached ML data gets the reason instead and is recorded as
|
||||
// failed. The next run fetches its ML data again because none is cached.
|
||||
if (includeOriginals) {
|
||||
let mlDataError: string | undefined;
|
||||
try {
|
||||
log("Fetching ML data...");
|
||||
await lib.fetchMLData();
|
||||
} catch (err) {
|
||||
mlDataError = errorMessage(err);
|
||||
log(`FAILED ML data: ${mlDataError}`);
|
||||
}
|
||||
for (const file of distinct.values()) {
|
||||
if (storedAtSavePath(downloadDirectory, file) !== undefined) {
|
||||
const path = savePath(downloadDirectory, file);
|
||||
writeSidecar(withExtension(path, ".json"), file);
|
||||
if (storedAtSavePath(downloadDirectory, file) === undefined) {
|
||||
continue;
|
||||
}
|
||||
const path = withExtension(
|
||||
savePath(downloadDirectory, file),
|
||||
".json",
|
||||
);
|
||||
const mlData = await lib.mlData(file.id);
|
||||
if (mlData === undefined && mlDataError !== undefined) {
|
||||
writeSidecar(path, file, { mlDataError });
|
||||
recordFailure(
|
||||
file,
|
||||
collectionName.get(file.collectionID) ?? "",
|
||||
new Error(`ML data: ${mlDataError}`),
|
||||
);
|
||||
} else {
|
||||
writeSidecar(path, file, { mlData });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
+12
-10
@@ -73,7 +73,9 @@ export const saveSession = (
|
||||
);
|
||||
};
|
||||
|
||||
// The saved client, or undefined after telling the user why there is none.
|
||||
// The saved client, or undefined after telling the user why there is none. The
|
||||
// command then exits 3, the code for "log in again", as `run` in `cli-run.ts`
|
||||
// does when the server no longer accepts the saved session.
|
||||
const requireSession = (ctx: CliContext): Client | undefined => {
|
||||
let client: Client | null;
|
||||
try {
|
||||
@@ -150,7 +152,7 @@ export const loginCommand = async (ctx: CliContext): Promise<number> => {
|
||||
export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
if (!client) return 3;
|
||||
const info = client.whoami();
|
||||
ctx.stdout.write(JSON.stringify(info) + "\n");
|
||||
return 0;
|
||||
@@ -201,7 +203,7 @@ export const collectionsCommand = async (
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
if (!client) return 3;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
@@ -243,7 +245,7 @@ export const filesCommand = async (
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
if (!client) return 3;
|
||||
const collectionID = Number(opts.collection);
|
||||
if (!Number.isFinite(collectionID)) {
|
||||
ctx.stderr.write("Invalid collection ID\n");
|
||||
@@ -285,7 +287,7 @@ export const getCommand = async (
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
if (!client) return 3;
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
ctx.stderr.write("Invalid file ID\n");
|
||||
@@ -343,7 +345,7 @@ export const getThumbCommand = async (
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
if (!client) return 3;
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
ctx.stderr.write("Invalid file ID\n");
|
||||
@@ -380,7 +382,7 @@ export const backupMetadataCommand = async (
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
if (!client) return 3;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Refresh first so the dump holds current account state, not what the
|
||||
@@ -403,7 +405,7 @@ export const backupCommand = async (
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
if (!client) return 3;
|
||||
|
||||
ctx.stderr.write("Starting backup...\n");
|
||||
// The precache is off: the backup fetches what it needs, and must not
|
||||
@@ -453,7 +455,7 @@ export const listMissingThumbnailsCommand = async (
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
if (!client) return 3;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Refresh first so files added since the cache was written are
|
||||
@@ -491,7 +493,7 @@ export const fixMissingThumbnailsCommand = async (
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
if (!client) return 3;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Refresh first so files added since the cache was written are found;
|
||||
|
||||
+11
-1
@@ -1,12 +1,15 @@
|
||||
// Runs one CLI command for `bin/quak.ts` and exits with its code.
|
||||
|
||||
import type { Writable } from "node:stream";
|
||||
import { ApiError } from "./api/client.js";
|
||||
|
||||
// Run a command and exit with its code once stdout/stderr have drained.
|
||||
// Exiting before the drain can truncate piped output, and the library can keep
|
||||
// the event loop alive after a command returns, so a plain return could hang.
|
||||
// An error the command throws is printed as one `quak: MESSAGE` line, without
|
||||
// the stack trace, and exits 1.
|
||||
// the stack trace, and exits 1. A 401 from the server means it no longer
|
||||
// accepts the saved session: that prints one line saying to log in again and
|
||||
// exits 3, as a missing or corrupt session file does.
|
||||
export const run = async (
|
||||
command: Promise<number>,
|
||||
stdout: Writable,
|
||||
@@ -17,11 +20,18 @@ export const run = async (
|
||||
try {
|
||||
code = await command;
|
||||
} catch (err) {
|
||||
if (err instanceof ApiError && err.status === 401) {
|
||||
stderr.write(
|
||||
`quak: the saved session is no longer valid; run "quak login"\n`,
|
||||
);
|
||||
code = 3;
|
||||
} else {
|
||||
stderr.write(
|
||||
`quak: ${err instanceof Error ? err.message : String(err)}\n`,
|
||||
);
|
||||
code = 1;
|
||||
}
|
||||
}
|
||||
const pending = [stdout, stderr].filter((s) => s.writableLength > 0);
|
||||
if (pending.length === 0) {
|
||||
exit(code);
|
||||
|
||||
+23
-10
@@ -279,7 +279,8 @@ export class Library {
|
||||
private cycle?: Promise<void>;
|
||||
// Guards the ML fetch pass so a slow backfill never runs twice at once; a
|
||||
// refresh whose pass is still running kicks nothing new. Holds the running
|
||||
// pass, so `close()` can wait for it.
|
||||
// pass, so `close()` and `backup()` can wait for it. It rejects when the
|
||||
// pass fails.
|
||||
private mlFetch?: Promise<void>;
|
||||
private closed = false;
|
||||
private lastRefreshAt?: number;
|
||||
@@ -569,8 +570,9 @@ export class Library {
|
||||
// does, joining one already running, and rejects before touching any file
|
||||
// when it fails. Then puts pending originals at their save paths as
|
||||
// `Photo.download()` does (and optional thumbnails) through the content
|
||||
// cache and pools, and rebuilds the derived symlink/JSON views from the
|
||||
// model. Throws before any network work when no content cache backs the
|
||||
// cache and pools, waits for an ML data fetch, and rebuilds the derived
|
||||
// symlink/JSON views from the model, each file's JSON with its ML data.
|
||||
// Throws before any network work when no content cache backs the
|
||||
// originals it must fetch.
|
||||
backup(opts?: BackupOptions): Promise<BackupResult> {
|
||||
const downloadDirectory =
|
||||
@@ -593,6 +595,8 @@ export class Library {
|
||||
original: (fileID, destination) =>
|
||||
cache!.backupOriginal(fileID, destination),
|
||||
thumbnail: (fileID) => cache!.thumbnail(fileID),
|
||||
fetchMLData: () => this.fetchMLDataNow(),
|
||||
mlData: (fileID) => this.mldata.forFile({ fileID }),
|
||||
},
|
||||
{ ...opts, downloadDirectory },
|
||||
);
|
||||
@@ -612,7 +616,7 @@ export class Library {
|
||||
this.timer = undefined;
|
||||
}
|
||||
await this.cycle?.catch(() => {});
|
||||
await this.mlFetch;
|
||||
await this.mlFetch?.catch(() => {});
|
||||
await precacheClosed;
|
||||
}
|
||||
|
||||
@@ -676,10 +680,9 @@ export class Library {
|
||||
// Backfill ML data for the files this refresh knows about. It runs
|
||||
// outside the refresh's success/failure so a fetch or disk problem
|
||||
// there never marks the metadata refresh failed, and it is not
|
||||
// awaited so it never stalls the refresh interval.
|
||||
this.mlFetch ??= this.runMLFetch().finally(() => {
|
||||
this.mlFetch = undefined;
|
||||
});
|
||||
// awaited so it never stalls the refresh interval. Its failure is
|
||||
// reported through `status()` and `onProgress`.
|
||||
void this.fetchMLDataNow().catch(() => {});
|
||||
} catch (err) {
|
||||
const error = err instanceof Error ? err.message : String(err);
|
||||
this.lastError = error;
|
||||
@@ -784,10 +787,19 @@ export class Library {
|
||||
}
|
||||
}
|
||||
|
||||
// Join the running ML fetch pass, or start one when none runs. Resolves at
|
||||
// once when the client cannot fetch ML data; rejects when the pass fails.
|
||||
private fetchMLDataNow(): Promise<void> {
|
||||
this.mlFetch ??= this.runMLFetch().finally(() => {
|
||||
this.mlFetch = undefined;
|
||||
});
|
||||
return this.mlFetch;
|
||||
}
|
||||
|
||||
// One ML fetch pass: fetch, decrypt and store the ML data for every file
|
||||
// the store knows about that is not cached (or whose `updationTime` has
|
||||
// advanced), through the metadata pool, and update the CLIP index. Guarded
|
||||
// so passes never overlap; a failure is reported, not thrown.
|
||||
// advanced), through the metadata pool, and update the CLIP index. A
|
||||
// failure is reported through `status()` and `onProgress`, then thrown.
|
||||
private async runMLFetch(): Promise<void> {
|
||||
const mldata = this.mlStore;
|
||||
// Bind so the call keeps the client as its receiver when invoked
|
||||
@@ -831,6 +843,7 @@ export class Library {
|
||||
const error = err instanceof Error ? err.message : String(err);
|
||||
this.lastMLError = error;
|
||||
this.emit({ operation: "fetchMLData", status: "failed", error });
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -52,6 +52,7 @@ import { runBackup, type BackupLibrary } from "../../src/backup.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import type { ContentSource } from "../../src/library/content.js";
|
||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||
import type { MLData } from "../../src/mldata-fetch.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
import {
|
||||
asLivePhoto,
|
||||
@@ -828,6 +829,94 @@ describe("the refresh before a backup", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// Also serves ML data: file 100 has `ML_PAYLOAD`, the other two have none.
|
||||
// While `mlError` is set, every ML data request fails with it.
|
||||
const ML_PAYLOAD = { face: { faces: [] }, clip: { embedding: [0.5, 0.25] } };
|
||||
|
||||
class MLClient extends MockClient {
|
||||
mlError?: string;
|
||||
async fetchMLData(args: {
|
||||
fileIDs: number[];
|
||||
}): Promise<Map<number, MLData>> {
|
||||
if (this.mlError) throw new Error(this.mlError);
|
||||
const result = new Map<number, MLData>();
|
||||
if (args.fileIDs.includes(100)) result.set(100, ML_PAYLOAD);
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
// The JSON the backup in `outDir` wrote beside the original of `fileID`.
|
||||
const fileJSON = (outDir: string, fileID: number): Record<string, unknown> =>
|
||||
JSON.parse(readFileSync(saved(outDir, `${fileID}.json`), "utf-8"));
|
||||
|
||||
describe("ML data in each file's JSON", () => {
|
||||
it("writes a file's ML data, and no ML field for a file that has none", async () => {
|
||||
const lib = await openLibrary(stubSource(), new MLClient());
|
||||
const outDir = join(root, "backup");
|
||||
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
|
||||
expect(fileJSON(outDir, 101)).not.toHaveProperty("mlData");
|
||||
expect(fileJSON(outDir, 101)).not.toHaveProperty("mlDataError");
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it("gives each file with no cached ML data the reason when the fetch fails, and fails the run", async () => {
|
||||
const client = new MLClient();
|
||||
const outDir = join(root, "backup");
|
||||
const first = await openLibrary(stubSource(), client);
|
||||
await first.backup({ downloadDirectory: outDir });
|
||||
await first.close();
|
||||
|
||||
client.mlError = "HTTP 503 from server";
|
||||
const lib = await openLibrary(stubSource(), client);
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
// File 100's ML data was cached by the first run and is kept.
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
|
||||
expect(fileJSON(outDir, 100)).not.toHaveProperty("mlDataError");
|
||||
for (const fileID of [101, 200]) {
|
||||
expect(fileJSON(outDir, fileID).mlDataError).toBe(
|
||||
"HTTP 503 from server",
|
||||
);
|
||||
}
|
||||
expect(result.failed).toBe(2);
|
||||
expect(result.errors.map((e) => [e.fileID, e.error])).toEqual([
|
||||
[101, "ML data: HTTP 503 from server"],
|
||||
[200, "ML data: HTTP 503 from server"],
|
||||
]);
|
||||
expect(Object.keys(readLedger(outDir).files)).toEqual(["101", "200"]);
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it("writes the ML data on the run after a failed fetch", async () => {
|
||||
const client = new MLClient();
|
||||
client.mlError = "HTTP 503 from server";
|
||||
const outDir = join(root, "backup");
|
||||
const first = await openLibrary(stubSource(), client);
|
||||
expect((await first.backup({ downloadDirectory: outDir })).failed).toBe(
|
||||
3,
|
||||
);
|
||||
expect(fileJSON(outDir, 100)).not.toHaveProperty("mlData");
|
||||
await first.close();
|
||||
|
||||
client.mlError = undefined;
|
||||
const lib = await openLibrary(stubSource(), client);
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(result.skipped).toBe(3);
|
||||
expect(fileJSON(outDir, 100).mlData).toEqual(ML_PAYLOAD);
|
||||
for (const fileID of [100, 101, 200]) {
|
||||
expect(fileJSON(outDir, fileID)).not.toHaveProperty("mlDataError");
|
||||
}
|
||||
expect(existsSync(join(outDir, "failures.json"))).toBe(false);
|
||||
await lib.close();
|
||||
});
|
||||
});
|
||||
|
||||
// Every entry under collections/, one level of directories deep, with each
|
||||
// symlink's target.
|
||||
const tree = (outDir: string): string[] => {
|
||||
@@ -871,6 +960,8 @@ describe("backup album folders", () => {
|
||||
thumbnail: async () => {
|
||||
throw new Error("no thumbnails in this stand-in");
|
||||
},
|
||||
fetchMLData: async () => {},
|
||||
mlData: async () => undefined,
|
||||
});
|
||||
|
||||
const albumID = (outDir: string, jsonName: string): number =>
|
||||
|
||||
+80
-14
@@ -230,20 +230,30 @@ describe("session file", () => {
|
||||
expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot);
|
||||
});
|
||||
|
||||
it("a missing session exits 1 with 'Not logged in'", async () => {
|
||||
it("a missing session exits 3 with 'Not logged in' from every command that needs one", async () => {
|
||||
const ctx = { ...context(), loadSession };
|
||||
expect(await whoamiCommand(ctx)).toBe(1);
|
||||
expect(stderr.text).toBe(
|
||||
const dir = join(root, "backup");
|
||||
expect(await whoamiCommand(ctx)).toBe(3);
|
||||
expect(await collectionsCommand(ctx, {})).toBe(3);
|
||||
expect(await filesCommand(ctx, { collection: "1" })).toBe(3);
|
||||
expect(await getCommand(ctx, "100", {})).toBe(3);
|
||||
expect(await getThumbCommand(ctx, "100", {})).toBe(3);
|
||||
expect(await backupMetadataCommand(ctx, dir, {})).toBe(3);
|
||||
expect(await backupCommand(ctx, dir, {})).toBe(3);
|
||||
expect(await listMissingThumbnailsCommand(ctx, {})).toBe(3);
|
||||
expect(await fixMissingThumbnailsCommand(ctx, {})).toBe(3);
|
||||
const notLoggedIn =
|
||||
`Not logged in. Run "quak login" first.\n` +
|
||||
`Session file: ${join(ctx.sessionDir, "session.json")}\n`,
|
||||
);
|
||||
`Session file: ${join(ctx.sessionDir, "session.json")}\n`;
|
||||
expect(stderr.text).toBe(notLoggedIn.repeat(9));
|
||||
expect(stdout.text).toBe("");
|
||||
expect(existsSync(dir)).toBe(false);
|
||||
});
|
||||
|
||||
it("a corrupt session exits 1 and says it is corrupt", async () => {
|
||||
it("a corrupt session exits 3 and says it is corrupt", async () => {
|
||||
const ctx = { ...context(), loadSession };
|
||||
saveSession(ctx.sessionDir, snapshot);
|
||||
expect(await collectionsCommand(ctx, {})).toBe(1);
|
||||
expect(await collectionsCommand(ctx, {})).toBe(3);
|
||||
expect(stderr.text).toContain("is corrupt");
|
||||
expect(stderr.text).toContain(
|
||||
`Run "quak logout" and then "quak login" to replace it.\n`,
|
||||
@@ -732,15 +742,28 @@ describe("backup", () => {
|
||||
expect(stderr.text).toBe("Starting backup...\n");
|
||||
});
|
||||
|
||||
it("exits 1 with the error on one line when the refresh fails", async () => {
|
||||
it("exits 1 and lists each file when the ML data fetch fails", async () => {
|
||||
const client = {
|
||||
...fakeClient(),
|
||||
collectionsSince: async () => {
|
||||
throw new Error("HTTP 401 from server");
|
||||
fetchMLData: async () => {
|
||||
throw new Error("HTTP 503 from server");
|
||||
},
|
||||
} as unknown as Client;
|
||||
const dir = join(root, "backup");
|
||||
// Through `run`, as `bin/quak.ts` does, which prints a thrown error.
|
||||
expect(
|
||||
await backupCommand(context(client), join(root, "backup"), {}),
|
||||
).toBe(1);
|
||||
expect(stderr.text).toContain(" Failed: 3\n");
|
||||
expect(stderr.text).toContain(
|
||||
" [Vacation] beach.jpg (id 100): ML data: HTTP 503 from server\n",
|
||||
);
|
||||
});
|
||||
|
||||
// Runs `backup` through `run`, as `bin/quak.ts` does, which prints a thrown
|
||||
// error; returns the exit code and what `run` printed.
|
||||
const backupThroughRun = async (
|
||||
ctx: CliContext,
|
||||
dir: string,
|
||||
): Promise<{ code: number; runText: string }> => {
|
||||
const runStderr = new PassThrough();
|
||||
let runText = "";
|
||||
runStderr.on("data", (chunk: Buffer) => {
|
||||
@@ -748,14 +771,57 @@ describe("backup", () => {
|
||||
});
|
||||
const code = await new Promise<number>((resolve) => {
|
||||
void run(
|
||||
backupCommand(context(client), dir, {}),
|
||||
backupCommand(ctx, dir, {}),
|
||||
new PassThrough(),
|
||||
runStderr,
|
||||
resolve,
|
||||
);
|
||||
});
|
||||
return { code, runText };
|
||||
};
|
||||
|
||||
it("exits 1 with the error on one line when the refresh fails", async () => {
|
||||
const client = {
|
||||
...fakeClient(),
|
||||
collectionsSince: async () => {
|
||||
throw new Error("HTTP 503 from server");
|
||||
},
|
||||
} as unknown as Client;
|
||||
const dir = join(root, "backup");
|
||||
const { code, runText } = await backupThroughRun(context(client), dir);
|
||||
expect(code).toBe(1);
|
||||
expect(runText).toBe("quak: HTTP 401 from server\n");
|
||||
expect(runText).toBe("quak: HTTP 503 from server\n");
|
||||
expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n");
|
||||
expect(existsSync(dir)).toBe(false);
|
||||
});
|
||||
|
||||
// A real saved session, read back by `loadSession`, whose server answers
|
||||
// every request with 401, as it does once it no longer accepts the token.
|
||||
// The context's prompts throw, so a prompt would end the run with another
|
||||
// line and exit 1.
|
||||
it("exits 3 with one line saying to log in again when the server answers 401", async () => {
|
||||
const key = toBase64(new Uint8Array(32));
|
||||
saveSession(join(root, "session"), {
|
||||
email: "cli@example.com",
|
||||
userID: USER_ID,
|
||||
token: "expired",
|
||||
masterKey: key,
|
||||
secretKey: key,
|
||||
publicKey: key,
|
||||
});
|
||||
const unauthorized = async (): Promise<Response> =>
|
||||
new Response(null, { status: 401 });
|
||||
const ctx = {
|
||||
...context(),
|
||||
loadSession: (path: string) =>
|
||||
loadSession(path, { fetch: unauthorized }),
|
||||
};
|
||||
const dir = join(root, "backup");
|
||||
const { code, runText } = await backupThroughRun(ctx, dir);
|
||||
expect(code).toBe(3);
|
||||
expect(runText).toBe(
|
||||
`quak: the saved session is no longer valid; run "quak login"\n`,
|
||||
);
|
||||
expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n");
|
||||
expect(existsSync(dir)).toBe(false);
|
||||
});
|
||||
|
||||
@@ -5,6 +5,7 @@
|
||||
import { PassThrough } from "node:stream";
|
||||
import { describe, it, expect } from "vitest";
|
||||
|
||||
import { ApiError } from "../../src/api/client.js";
|
||||
import { run } from "../../src/cli-run.js";
|
||||
|
||||
// A stream whose written text is kept in `text`; writes finish at once, so
|
||||
@@ -55,4 +56,26 @@ describe("run", () => {
|
||||
stderr: "quak: offline\n",
|
||||
});
|
||||
});
|
||||
|
||||
it("on a 401 from the server says to run quak login, on one line, and exits 3", async () => {
|
||||
const result = await runToExit(
|
||||
Promise.reject(new ApiError("unauthorized", 401)),
|
||||
);
|
||||
expect(result).toEqual({
|
||||
code: 3,
|
||||
stdout: "",
|
||||
stderr: `quak: the saved session is no longer valid; run "quak login"\n`,
|
||||
});
|
||||
});
|
||||
|
||||
it("prints another HTTP error as it is and exits 1", async () => {
|
||||
const result = await runToExit(
|
||||
Promise.reject(new ApiError("forbidden", 403)),
|
||||
);
|
||||
expect(result).toEqual({
|
||||
code: 1,
|
||||
stdout: "",
|
||||
stderr: "quak: forbidden\n",
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user