Compare commits

...
3 Commits
Author SHA1 Message Date
sneak 581e73f0bf Build every ApiClient request URL in one place (closes #18)
check / check (push) Successful in 30s
getJSON built its URL with new URL and then overwrote the host and
path, which dropped a base path in a self-hosted apiOrigin; postJSON,
putJSON and the file and thumbnail download URLs joined strings. All of
them now go through one function next to ApiClient that accepts a path
with or without a leading slash and an origin with or without a
trailing slash or base path, and percent-encodes query parameters. A
path containing "?" or "#" is rejected.

Model: opus-5-5
2026-09-23 00:54:25 +00:00
clawbot d07692897b Move CLI commands into testable functions and test them (closes #12)
check / check (push) Successful in 27s
The command bodies in bin/quak.ts become functions in
src/cli-commands.ts that take their options and a context (output
streams, session directory, cache directory, session loader) and return
an exit code. bin/quak.ts wires them to commander and exits with that
code once stdout and stderr have drained; nothing below it calls
process.exit. test/cli/commands.test.ts drives the commands with a fake
client and temp directories. Output is unchanged.

Model: opus-5-5
2026-09-23 02:47:47 +02:00
clawbot ed535be1da Harden the backup tree's atomic copy (closes #22)
check / check (push) Successful in 27s
The backup copy now fsyncs its temp file before the rename and the
directory after it, using the download writer's new fsyncPath helper.
Each backup run deletes .quak-backup-*.tmp files whose process is no
longer running, leaving those of a concurrent backup alone. The rename
sites and the README backup layout state that a symlink at the
destination is replaced and the new file takes the temp file's
permissions, and the README names the temp files. Adds tests for a
missing and an unwritable destination directory for downloadFile and
downloadThumbnail.

Model: opus-5-5
2026-09-23 02:44:46 +02:00
11 changed files with 1292 additions and 416 deletions
+10
View File
@@ -494,6 +494,16 @@ appears in. On subsequent runs, existing originals are skipped. If a download
fails, the error is logged and the backup continues with the next file. The exit fails, the error is logged and the backup continues with the next file. The exit
code is non-zero if any files failed. code is non-zero if any files failed.
Each original is copied to a temporary file named
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp` in the same directory, synced
to disk, and renamed into place, so an original is either complete or absent,
even after a power cut. A run that is killed can leave one of these temporary
files behind; the next backup deletes those whose process is no longer running.
Downloads and the content cache use the same scheme with `.quak-<random>.tmp`
names. The rename replaces whatever was at the destination rather than writing
through it: a symlink there is replaced, not followed, and the new file has the
temporary file's permissions, not those of the file it replaced.
## TODO ## TODO
- [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network - [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
+21
View File
@@ -18,6 +18,27 @@ Tag v1.0.0.
# Completed Steps # Completed Steps
- 2026-09-23: Every `ApiClient` request URL is now built by one function next to
the class (issue 18), so a self-hosted `apiOrigin` with a base path keeps it
on every request, a path works with or without a leading slash, and query
parameters are percent-encoded. A path containing `?` or `#` is rejected with
an error instead of being silently cut.
- 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies
moved from `bin/quak.ts` into `src/cli-commands.ts` as functions that take
their options and a context (output streams, session directory, cache
directory, session loader) and return an exit code; `bin/quak.ts` only wires
them to commander and exits with the code once stdout and stderr have drained,
so nothing below it calls `process.exit`. `test/cli/commands.test.ts` drives
them with a fake client: session file modes, logout, the missing and corrupt
session paths, and the output and exit code of `whoami`, `collections`,
`files`, `get`, `get-thumb`, `backup` and `helper list-missing-thumbnails`.
- 2026-09-23: Hardened the backup tree's atomic copy (issue 22). `copyAtomic`
fsyncs its temp file before the rename and the directory after it, through the
download writer's `fsyncPath`; each backup run deletes `.quak-backup-*.tmp`
files whose process is no longer running. The README backup layout names the
temp files and states that the rename replaces a symlink and takes the temp
file's permissions. Added tests for a missing and an unwritable destination
directory for `downloadFile` and `downloadThumbnail`.
- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue - 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue
11). Every segment length is checked against the remaining bytes and lengths 11). Every segment length is checked against the remaining bytes and lengths
under 2 stop the scan, so a truncated or corrupt original can neither throw under 2 stop the scan, so a truncated or corrupt original can neither throw
+53 -380
View File
@@ -1,62 +1,26 @@
#!/usr/bin/env node #!/usr/bin/env node
import { input, password as passwordPrompt } from "@inquirer/prompts";
import { stdout, stderr } from "node:process"; import { stdout, stderr } from "node:process";
import { copyFileSync, existsSync, mkdirSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { Command } from "commander"; import { Command } from "commander";
import envPaths from "env-paths"; import envPaths from "env-paths";
import { Client, type ClientSnapshot } from "../src/client.js";
import { init } from "../src/crypto/index.js"; import { init } from "../src/crypto/index.js";
import { Library, type LibraryClient } from "../src/library/index.js";
import { import {
fileListRow, type CliContext,
fileListLine, loginCommand,
originalName, whoamiCommand,
thumbnailName, logoutCommand,
} from "../src/cli-output.js"; collectionsCommand,
import { freshCollections, freshFiles, freshFile } from "../src/cli-read.js"; filesCommand,
getCommand,
getThumbCommand,
backupMetadataCommand,
backupCommand,
listMissingThumbnailsCommand,
fixMissingThumbnailsCommand,
} from "../src/cli-commands.js";
import { loadSession } from "../src/cli-session.js"; import { loadSession } from "../src/cli-session.js";
import { runMetadataBackup } from "../src/metadata-backup.js";
import {
listMissingThumbnails,
fixMissingThumbnails,
} from "../src/thumbnails.js";
const paths = envPaths("quak", { suffix: "" }); const paths = envPaths("quak", { suffix: "" });
const sessionPath = join(paths.data, "session.json");
const saveSession = (snapshot: ClientSnapshot): void => {
mkdirSync(paths.data, { recursive: true, mode: 0o700 });
writeFileSync(sessionPath, JSON.stringify(snapshot, null, 2), {
mode: 0o600,
});
};
const requireSession = (): Client => {
let client: Client | null;
try {
client = loadSession(sessionPath);
} catch (err) {
stderr.write(
`${err instanceof Error ? err.message : err}\n` +
`Run "quak logout" and then "quak login" to replace it.\n`,
);
process.exit(1);
}
if (!client) {
stderr.write(
`Not logged in. Run "quak login" first.\nSession file: ${sessionPath}\n`,
);
process.exit(1);
}
return client;
};
const prompt = async (message: string): Promise<string> => input({ message });
const promptSecret = async (message: string): Promise<string> =>
passwordPrompt({ message, mask: true });
const program = new Command(); const program = new Command();
@@ -70,51 +34,28 @@ program
"(default: the per-user cache directory)", "(default: the per-user cache directory)",
); );
// The `--cache-dir` global, or undefined to let the library pick its per-user const context = (): CliContext => ({
// default keyed by the account id. stdout,
const cacheDirOption = (): string | undefined => stderr,
program.opts<{ cacheDir?: string }>().cacheDir; sessionDir: paths.data,
cacheDir: program.opts<{ cacheDir?: string }>().cacheDir,
// A library client that omits `fetchMLData`, so the point commands below do not loadSession,
// kick the library's background ML backfill: they read metadata, or fetch one
// file's content, and exit. `backup` and `backup-metadata` handle ML on their
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
// originals through the on-disk cache.
const readLibraryClient = (client: Client): LibraryClient => ({
whoami: () => client.whoami(),
collectionsSince: (args) => client.collectionsSince(args),
filesSince: (args) => client.filesSince(args),
contentSource: () => client.contentSource(),
}); });
// Open a library for a single point command: the aggressive background precache // Run a command and exit with its code once stdout/stderr have drained.
// (issue #48) is off — a one-shot `collections` or `get` must not start // Exiting before the drain can truncate piped output, and the library can keep
// downloading the whole account — and the refresh interval is long so no second // the event loop alive after a command returns, so a plain return could hang.
// refresh fires mid-command. const run = async (command: Promise<number>): Promise<void> => {
const openReadLibrary = (client: Client): Promise<Library> => process.exitCode = await command;
Library.open({
client: readLibraryClient(client),
cacheDirectory: cacheDirOption(),
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
});
// Close the library and exit once stdout/stderr have drained. `process.exit`
// alone can truncate buffered piped output, and the library keeps the event
// loop alive with a background refresh, so a plain return could hang; this does
// neither.
const finish = (lib: Library | undefined, code: number): void => {
lib?.close();
const pending = [stdout, stderr].filter((s) => s.writableLength > 0); const pending = [stdout, stderr].filter((s) => s.writableLength > 0);
if (pending.length === 0) { if (pending.length === 0) {
process.exit(code); process.exit();
return; return;
} }
let remaining = pending.length; let remaining = pending.length;
for (const s of pending) { for (const s of pending) {
s.once("drain", () => { s.once("drain", () => {
if (--remaining === 0) process.exit(code); if (--remaining === 0) process.exit();
}); });
} }
}; };
@@ -122,93 +63,25 @@ const finish = (lib: Library | undefined, code: number): void => {
program program
.command("login") .command("login")
.description("Log in to an Ente account and save the session") .description("Log in to an Ente account and save the session")
.action(async () => { .action(() => run(loginCommand(context())));
await init();
const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
const password =
process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
stderr.write("Authenticating...\n");
try {
const client = await Client.login({
email,
password,
totp: async () => prompt("TOTP code: "),
emailOTP: async () => prompt("Email verification code: "),
});
saveSession(client.toJSON());
const info = client.whoami();
stderr.write(`Logged in as ${info.email} (user ${info.userID})\n`);
stderr.write(`Session saved to ${sessionPath}\n`);
} catch (err) {
stderr.write(
`Login failed: ${err instanceof Error ? err.message : err}\n`,
);
process.exit(1);
}
});
program program
.command("whoami") .command("whoami")
.description("Print the logged-in account") .description("Print the logged-in account")
.action(async () => { .action(() => run(whoamiCommand(context())));
await init();
const client = requireSession();
const info = client.whoami();
stdout.write(JSON.stringify(info) + "\n");
});
program program
.command("logout") .command("logout")
.description("Delete the saved session") .description("Delete the saved session")
.action(async () => { .action(() => run(logoutCommand(context())));
if (existsSync(sessionPath)) {
const { unlinkSync } = await import("node:fs");
unlinkSync(sessionPath);
stderr.write("Session deleted.\n");
} else {
stderr.write("No session found.\n");
}
});
program program
.command("collections") .command("collections")
.description("List all collections (albums)") .description("List all collections (albums)")
.option("--json", "Output as JSON array") .option("--json", "Output as JSON array")
.action(async (opts: { json?: boolean }) => { .action((opts: { json?: boolean }) =>
await init(); run(collectionsCommand(context(), opts)),
const client = requireSession(); );
const lib = await openReadLibrary(client);
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52): the pre-library CLI printed current state in
// this order, not the albums projection's newest-first order.
const collections = await freshCollections(lib);
if (opts.json) {
stdout.write(
JSON.stringify(
collections.map((c) => ({
id: c.id,
name: c.name,
type: c.type,
ownerID: c.ownerID,
isShared: c.isShared,
updationTime: c.updationTime,
})),
null,
2,
) + "\n",
);
} else {
for (const c of collections) {
stdout.write(
`${c.id}\t${c.type}\t${c.name}${c.isShared ? " (shared)" : ""}\n`,
);
}
}
finish(lib, 0);
});
program program
.command("files") .command("files")
@@ -218,39 +91,9 @@ program
"Collection ID (from `quak collections`)", "Collection ID (from `quak collections`)",
) )
.option("--json", "Output as JSON array") .option("--json", "Output as JSON array")
.action(async (opts: { collection: string; json?: boolean }) => { .action((opts: { collection: string; json?: boolean }) =>
await init(); run(filesCommand(context(), opts)),
const client = requireSession(); );
const collectionID = Number(opts.collection);
if (!Number.isFinite(collectionID)) {
stderr.write("Invalid collection ID\n");
process.exit(1);
}
const lib = await openReadLibrary(client);
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52). Each file prints from its own decrypted
// metadata (raw title, microsecond creationTime) via cli-output, and in
// the pre-library CLI's enumeration order, not the projection's
// newest-first order.
const files = await freshFiles(lib, collectionID);
if (!files) {
stderr.write(`Collection ${collectionID} not found\n`);
finish(lib, 1);
return;
}
if (opts.json) {
stdout.write(
JSON.stringify(files.map(fileListRow), null, 2) + "\n",
);
} else {
for (const file of files) {
stdout.write(fileListLine(file) + "\n");
}
}
finish(lib, 0);
});
program program
.command("get") .command("get")
@@ -258,34 +101,9 @@ program
.argument("<fileID>", "File ID (from `quak files`)") .argument("<fileID>", "File ID (from `quak files`)")
.option("--out <path>", "Output file path") .option("--out <path>", "Output file path")
.option("--collection <id>", "Accepted for compatibility; ignored") .option("--collection <id>", "Accepted for compatibility; ignored")
.action(async (fileIDStr: string, opts: { out?: string }) => { .action((fileID: string, opts: { out?: string }) =>
await init(); run(getCommand(context(), fileID, opts)),
const client = requireSession(); );
const fileID = Number(fileIDStr);
if (!Number.isFinite(fileID)) {
stderr.write("Invalid file ID\n");
process.exit(1);
}
const lib = await openReadLibrary(client);
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
stderr.write(`File ${fileID} not found\n`);
finish(lib, 1);
return;
}
const { photo, file } = resolved;
const result = await photo.original();
// Default name is the file's own title, as the pre-library CLI used
// (not the editedName-preferring projection title) (issue #52).
const outPath = opts.out ?? originalName(file);
copyFileSync(result.path, outPath);
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
finish(lib, 0);
});
program program
.command("get-thumb") .command("get-thumb")
@@ -293,34 +111,9 @@ program
.argument("<fileID>", "File ID (from `quak files`)") .argument("<fileID>", "File ID (from `quak files`)")
.option("--out <path>", "Output file path") .option("--out <path>", "Output file path")
.option("--collection <id>", "Accepted for compatibility; ignored") .option("--collection <id>", "Accepted for compatibility; ignored")
.action(async (fileIDStr: string, opts: { out?: string }) => { .action((fileID: string, opts: { out?: string }) =>
await init(); run(getThumbCommand(context(), fileID, opts)),
const client = requireSession(); );
const fileID = Number(fileIDStr);
if (!Number.isFinite(fileID)) {
stderr.write("Invalid file ID\n");
process.exit(1);
}
const lib = await openReadLibrary(client);
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
stderr.write(`File ${fileID} not found\n`);
finish(lib, 1);
return;
}
const { photo, file } = resolved;
const result = await photo.thumbnail();
// Default name is thumb_<file's own title>, as the pre-library CLI
// used (not the projection title) (issue #52).
const outPath = opts.out ?? thumbnailName(file);
copyFileSync(result.path, outPath);
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
finish(lib, 0);
});
program program
.command("backup-metadata") .command("backup-metadata")
@@ -333,16 +126,9 @@ program
"Download each file and extract full EXIF/IPTC/XMP metadata (slow)", "Download each file and extract full EXIF/IPTC/XMP metadata (slow)",
) )
.option("--all", "Alias for --exif") .option("--all", "Alias for --exif")
.action(async (dir: string, opts: { exif?: boolean; all?: boolean }) => { .action((dir: string, opts: { exif?: boolean; all?: boolean }) =>
await init(); run(backupMetadataCommand(context(), dir, opts)),
const client = requireSession(); );
const lib = await openReadLibrary(client);
await runMetadataBackup(lib, client, dir, {
exif: opts.exif || opts.all,
onProgress: (msg) => stderr.write(msg + "\n"),
});
finish(lib, 0);
});
program program
.command("backup") .command("backup")
@@ -351,43 +137,9 @@ program
) )
.argument("<dir>", "Output directory") .argument("<dir>", "Output directory")
.option("--json", "Print result as JSON instead of human-readable summary") .option("--json", "Print result as JSON instead of human-readable summary")
.action(async (dir: string, opts: { json?: boolean }) => { .action((dir: string, opts: { json?: boolean }) =>
await init(); run(backupCommand(context(), dir, opts)),
const client = requireSession(); );
stderr.write("Starting backup...\n");
const lib = await Library.open({
client,
downloadDirectory: dir,
cacheDirectory: cacheDirOption(),
});
const result = await lib.backup({
downloadDirectory: dir,
onProgress: (msg) => {
if (!opts.json) stderr.write(msg + "\n");
},
});
if (opts.json) {
stdout.write(JSON.stringify(result, null, 2) + "\n");
} else {
stderr.write("\n--- Backup complete ---\n");
stderr.write(` Total files: ${result.totalFiles}\n`);
stderr.write(` Downloaded: ${result.downloaded}\n`);
stderr.write(` Skipped: ${result.skipped}\n`);
stderr.write(` Failed: ${result.failed}\n`);
if (result.errors.length > 0) {
stderr.write("\nFailed files:\n");
for (const e of result.errors) {
stderr.write(
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
);
}
}
}
finish(lib, result.failed > 0 ? 1 : 0);
});
const helper = program const helper = program
.command("helper") .command("helper")
@@ -397,32 +149,9 @@ helper
.command("list-missing-thumbnails") .command("list-missing-thumbnails")
.description("List files whose thumbnails are missing or empty") .description("List files whose thumbnails are missing or empty")
.option("--json", "Output as JSON array") .option("--json", "Output as JSON array")
.action(async (opts: { json?: boolean }) => { .action((opts: { json?: boolean }) =>
await init(); run(listMissingThumbnailsCommand(context(), opts)),
const client = requireSession(); );
const lib = await openReadLibrary(client);
const missing = await listMissingThumbnails(lib, client, (msg) => {
if (!opts.json) stderr.write(msg + "\n");
});
if (opts.json) {
stdout.write(JSON.stringify(missing, null, 2) + "\n");
} else {
if (missing.length === 0) {
stderr.write("No missing thumbnails found.\n");
} else {
stderr.write(
`\n${missing.length} file(s) with missing thumbnails:\n`,
);
for (const m of missing) {
stdout.write(
`${m.fileID}\t${m.title}\t${m.collection}\t${m.reason}\n`,
);
}
}
}
finish(lib, 0);
});
helper helper
.command("fix-missing-thumbnails") .command("fix-missing-thumbnails")
@@ -434,65 +163,9 @@ helper
"Specific file IDs to fix (default: fix all missing)", "Specific file IDs to fix (default: fix all missing)",
) )
.option("--json", "Output as JSON") .option("--json", "Output as JSON")
.action(async (opts: { file?: string[]; json?: boolean }) => { .action((opts: { file?: string[]; json?: boolean }) =>
await init(); run(fixMissingThumbnailsCommand(context(), opts)),
const client = requireSession(); );
const lib = await openReadLibrary(client);
let fileIDs: number[];
if (opts.file && opts.file.length > 0) {
fileIDs = opts.file.map(Number).filter(Number.isFinite);
} else {
stderr.write("Scanning for missing thumbnails...\n");
const missing = await listMissingThumbnails(lib, client, (msg) => {
if (!opts.json) stderr.write(msg + "\n");
});
fileIDs = missing.map((m) => m.fileID);
if (fileIDs.length === 0) {
stderr.write("No missing thumbnails found.\n");
finish(lib, 0);
return;
}
stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
}
const results = await fixMissingThumbnails(
lib,
client,
fileIDs,
(msg) => {
if (!opts.json) stderr.write(msg + "\n");
},
);
if (opts.json) {
stdout.write(JSON.stringify(results, null, 2) + "\n");
} else {
const fixed = results.filter((r) => r.status === "fixed").length;
const skipped = results.filter(
(r) => r.status === "skipped",
).length;
const failed = results.filter((r) => r.status === "failed").length;
stderr.write(`\n--- Done ---\n`);
stderr.write(` Fixed: ${fixed}\n`);
stderr.write(` Skipped: ${skipped}\n`);
stderr.write(` Failed: ${failed}\n`);
if (skipped > 0) {
stderr.write("\nSkipped (unsupported format):\n");
for (const r of results.filter((r) => r.status === "skipped")) {
stderr.write(` ${r.fileID}\t${r.title}\t${r.reason}\n`);
}
}
if (failed > 0) {
stderr.write("\nFailed files:\n");
for (const r of results.filter((r) => r.status === "failed")) {
stderr.write(` ${r.fileID}\t${r.title}\t${r.reason}\n`);
}
}
}
finish(lib, results.some((r) => r.status === "failed") ? 1 : 0);
});
await init(); await init();
program.parse(); program.parse();
+32 -20
View File
@@ -98,6 +98,30 @@ const deadlineStream = (
}); });
}; };
// The one place a request URL is built. `origin` may carry a base path (a
// self-hosted server behind a prefix) and may end in a slash; `path` may or
// may not start with one. Query parameters go only through `query`, which
// percent-encodes them: a `?` or `#` in `path` is an error, because
// `new URL` would otherwise quietly treat what follows as something else.
const buildURL = (
origin: string,
path: string,
query?: Record<string, string | number | undefined>,
): string => {
if (path.includes("?") || path.includes("#")) {
throw new Error(
`request path must not contain "?" or "#"; pass query parameters separately: ${path}`,
);
}
const url = new URL(
`${origin.replace(/\/+$/, "")}/${path.replace(/^\/+/, "")}`,
);
for (const [k, v] of Object.entries(query ?? {})) {
if (v !== undefined) url.searchParams.set(k, String(v));
}
return url.href;
};
export class ApiClient { export class ApiClient {
private readonly apiOrigin: string; private readonly apiOrigin: string;
private readonly isCustomOrigin: boolean; private readonly isCustomOrigin: boolean;
@@ -202,22 +226,10 @@ export class ApiClient {
path: string, path: string,
query?: Record<string, string | number | undefined>, query?: Record<string, string | number | undefined>,
): Promise<T> { ): Promise<T> {
const url = new URL(path, this.apiOrigin + "/"); const url = buildURL(this.apiOrigin, path, query);
// new URL with a base resolves relative paths; ensure we keep the
// origin from apiOrigin even when path starts with /
url.protocol = new URL(this.apiOrigin).protocol;
url.host = new URL(this.apiOrigin).host;
url.pathname = path;
if (query) {
for (const [k, v] of Object.entries(query)) {
if (v !== undefined) {
url.searchParams.set(k, String(v));
}
}
}
// A GET changes nothing, so it is retried under the full policy. // A GET changes nothing, so it is retried under the full policy.
return withRetry(async () => { return withRetry(async () => {
const resp = await this._fetch(url.href, { const resp = await this._fetch(url, {
method: "GET", method: "GET",
headers: this.headers(), headers: this.headers(),
signal: AbortSignal.timeout(this.requestTimeoutMs), signal: AbortSignal.timeout(this.requestTimeoutMs),
@@ -228,7 +240,7 @@ export class ApiClient {
} }
async postJSON<T>(path: string, body: unknown): Promise<T> { async postJSON<T>(path: string, body: unknown): Promise<T> {
const url = `${this.apiOrigin}${path}`; const url = buildURL(this.apiOrigin, path);
// Not idempotent: a POST is replayed only when `isSafeToReplay` // Not idempotent: a POST is replayed only when `isSafeToReplay`
// says no request byte can have reached the server. The endpoints // says no request byte can have reached the server. The endpoints
// this covers are listed in the README under "Endpoints used". // this covers are listed in the README under "Endpoints used".
@@ -261,8 +273,8 @@ export class ApiClient {
opts?: StreamOptions, opts?: StreamOptions,
): Promise<ReadableStream<Uint8Array>> { ): Promise<ReadableStream<Uint8Array>> {
const url = this.isCustomOrigin const url = this.isCustomOrigin
? `${this.apiOrigin}/files/download/${fileID}` ? buildURL(this.apiOrigin, `/files/download/${fileID}`)
: `${this.filesOrigin}/?fileID=${fileID}`; : buildURL(this.filesOrigin, "/", { fileID });
return this.streamRequest(url, opts); return this.streamRequest(url, opts);
} }
@@ -304,7 +316,7 @@ export class ApiClient {
} }
async putJSON<T>(path: string, body: unknown): Promise<T> { async putJSON<T>(path: string, body: unknown): Promise<T> {
const url = `${this.apiOrigin}${path}`; const url = buildURL(this.apiOrigin, path);
// Same replay and redirect rules as `postJSON`, for the same reasons. // Same replay and redirect rules as `postJSON`, for the same reasons.
return withRetry( return withRetry(
async () => { async () => {
@@ -340,8 +352,8 @@ export class ApiClient {
opts?: StreamOptions, opts?: StreamOptions,
): Promise<ReadableStream<Uint8Array>> { ): Promise<ReadableStream<Uint8Array>> {
const url = this.isCustomOrigin const url = this.isCustomOrigin
? `${this.apiOrigin}/files/preview/${fileID}` ? buildURL(this.apiOrigin, `/files/preview/${fileID}`)
: `${this.thumbsOrigin}/?fileID=${fileID}`; : buildURL(this.thumbsOrigin, "/", { fileID });
return this.streamRequest(url, opts); return this.streamRequest(url, opts);
} }
+51 -9
View File
@@ -29,19 +29,20 @@
// rather than counted forever, which would poison a scheduled backup's exit code. // rather than counted forever, which would poison a scheduled backup's exit code.
import { import {
copyFileSync,
lstatSync, lstatSync,
mkdirSync, mkdirSync,
readdirSync,
readFileSync, readFileSync,
readlinkSync, readlinkSync,
renameSync,
rmSync, rmSync,
statSync, statSync,
symlinkSync, symlinkSync,
writeFileSync, writeFileSync,
} from "node:fs"; } from "node:fs";
import { copyFile, rename, rm } from "node:fs/promises";
import { basename, dirname, join, relative } from "node:path"; import { basename, dirname, join, relative } from "node:path";
import { fsyncPath } from "./download/index.js";
import { safeExtension, sanitizeFileName } from "./filename.js"; import { safeExtension, sanitizeFileName } from "./filename.js";
import type { Collection, EnteFile } from "./model/types.js"; import type { Collection, EnteFile } from "./model/types.js";
@@ -154,8 +155,12 @@ const errorMessage = (err: unknown): string =>
err instanceof Error ? err.message : String(err); err instanceof Error ? err.message : String(err);
// Copy bytes into `dest` via a temp file in the same directory plus rename, so // Copy bytes into `dest` via a temp file in the same directory plus rename, so
// `dest` appears only once it is whole ("present means complete"). // `dest` appears only once it is whole ("present means complete"). As in the
const copyAtomic = (src: string, dest: string): void => { // download writer, the temp file is fsynced before the rename and the directory
// after it, so a power cut cannot leave a correctly named but short original.
// The temp name carries this process's ID so a later run can tell a leftover
// from a copy still in progress (see `removeLeftoverTempFiles`).
const copyAtomic = async (src: string, dest: string): Promise<void> => {
if (src === dest) return; if (src === dest) return;
const tmp = join( const tmp = join(
dirname(dest), dirname(dest),
@@ -164,10 +169,45 @@ const copyAtomic = (src: string, dest: string): void => {
.slice(2)}.tmp`, .slice(2)}.tmp`,
); );
try { try {
copyFileSync(src, tmp); await copyFile(src, tmp);
renameSync(tmp, dest); await fsyncPath(tmp);
// `rename` replaces the destination's directory entry: an existing
// symlink at `dest` is replaced, not followed, and the new file has
// the temp file's permissions (copied from `src`).
await rename(tmp, dest);
await fsyncPath(dirname(dest));
} finally { } finally {
rmSync(tmp, { force: true }); await rm(tmp, { force: true });
}
};
// A process-ID check: signal 0 delivers nothing and only reports whether the
// process exists. EPERM means it exists but belongs to another user.
const isRunning = (pid: number): boolean => {
try {
process.kill(pid, 0);
return true;
} catch (err) {
return (err as NodeJS.ErrnoException).code === "EPERM";
}
};
// Delete the temp files `copyAtomic` leaves behind when a backup is killed
// before its rename. Only files whose process is no longer running are
// removed, so a backup running at the same time keeps its own. A reused
// process ID can only keep a leftover a while longer, never remove a live one.
const removeLeftoverTempFiles = (dir: string): void => {
let names: string[];
try {
names = readdirSync(dir);
} catch {
return;
}
for (const name of names) {
const match = /^\.quak-backup-.*-(\d+)-[0-9a-z]*\.tmp$/.exec(name);
if (match && !isRunning(Number(match[1]))) {
rmSync(join(dir, name), { force: true });
}
} }
}; };
@@ -254,6 +294,8 @@ export const runBackup = async (
mkdirSync(originalsDir, { recursive: true }); mkdirSync(originalsDir, { recursive: true });
mkdirSync(collectionsDir, { recursive: true }); mkdirSync(collectionsDir, { recursive: true });
if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true }); if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true });
removeLeftoverTempFiles(originalsDir);
removeLeftoverTempFiles(thumbnailsDir);
const ledgerPath = join(downloadDirectory, "failures.json"); const ledgerPath = join(downloadDirectory, "failures.json");
const ledger = loadLedger(ledgerPath); const ledger = loadLedger(ledgerPath);
@@ -321,7 +363,7 @@ export const runBackup = async (
try { try {
log(`Fetching original ${file.metadata.title} (${fileID})...`); log(`Fetching original ${file.metadata.title} (${fileID})...`);
const { path } = await lib.original(fileID); const { path } = await lib.original(fileID);
copyAtomic(path, dest); await copyAtomic(path, dest);
downloaded++; downloaded++;
} catch (err) { } catch (err) {
log( log(
@@ -342,7 +384,7 @@ export const runBackup = async (
if (isPresent(dest)) continue; if (isPresent(dest)) continue;
try { try {
const { path } = await lib.thumbnail(fileID); const { path } = await lib.thumbnail(fileID);
copyAtomic(path, dest); await copyAtomic(path, dest);
} catch (err) { } catch (err) {
recordFailure( recordFailure(
file, file,
+486
View File
@@ -0,0 +1,486 @@
// The CLI's commands as plain functions.
//
// Each command takes its options and a `CliContext` and resolves to the exit
// code; a thrown error is left to the caller. Nothing here calls
// `process.exit`: `bin/quak.ts` wires these to the command line and exits with
// the returned code once output has drained. Output must stay byte-identical
// (see `cli-output.ts`).
import { input, password as passwordPrompt } from "@inquirer/prompts";
import {
copyFileSync,
existsSync,
mkdirSync,
unlinkSync,
writeFileSync,
} from "node:fs";
import { join } from "node:path";
import { Client, type ClientSnapshot } from "./client.js";
import { init } from "./crypto/index.js";
import { Library, type LibraryClient } from "./library/index.js";
import {
fileListRow,
fileListLine,
originalName,
thumbnailName,
} from "./cli-output.js";
import { freshCollections, freshFiles, freshFile } from "./cli-read.js";
import { runMetadataBackup } from "./metadata-backup.js";
import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js";
export interface CliContext {
stdout: { write(text: string): unknown };
stderr: { write(text: string): unknown };
// Directory holding `session.json`.
sessionDir: string;
// The `--cache-dir` global, or undefined to let the library pick its
// per-user default keyed by the account id.
cacheDir?: string;
// Reads the session file into a client, or null when there is none. The
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
loadSession: (path: string) => Client | null;
}
const sessionPath = (ctx: CliContext): string =>
join(ctx.sessionDir, "session.json");
// Write the session readable by its owner only, in a directory only its owner
// can enter.
export const saveSession = (
sessionDir: string,
snapshot: ClientSnapshot,
): void => {
mkdirSync(sessionDir, { recursive: true, mode: 0o700 });
writeFileSync(
join(sessionDir, "session.json"),
JSON.stringify(snapshot, null, 2),
{ mode: 0o600 },
);
};
// The saved client, or undefined after telling the user why there is none.
const requireSession = (ctx: CliContext): Client | undefined => {
let client: Client | null;
try {
client = ctx.loadSession(sessionPath(ctx));
} catch (err) {
ctx.stderr.write(
`${err instanceof Error ? err.message : err}\n` +
`Run "quak logout" and then "quak login" to replace it.\n`,
);
return undefined;
}
if (!client) {
ctx.stderr.write(
`Not logged in. Run "quak login" first.\nSession file: ${sessionPath(ctx)}\n`,
);
return undefined;
}
return client;
};
// A library client that omits `fetchMLData`, so the point commands below do not
// kick the library's background ML backfill: they read metadata, or fetch one
// file's content, and exit. `backup` and `backup-metadata` handle ML on their
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
// originals through the on-disk cache.
const readLibraryClient = (client: Client): LibraryClient => ({
whoami: () => client.whoami(),
collectionsSince: (args) => client.collectionsSince(args),
filesSince: (args) => client.filesSince(args),
contentSource: () => client.contentSource(),
});
// Open a library for a single point command: the aggressive background precache
// (issue #48) is off — a one-shot `collections` or `get` must not start
// downloading the whole account — and the refresh interval is long so no second
// refresh fires mid-command.
const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
Library.open({
client: readLibraryClient(client),
cacheDirectory: ctx.cacheDir,
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
});
const prompt = async (message: string): Promise<string> => input({ message });
const promptSecret = async (message: string): Promise<string> =>
passwordPrompt({ message, mask: true });
export const loginCommand = async (ctx: CliContext): Promise<number> => {
await init();
const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
const password =
process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
ctx.stderr.write("Authenticating...\n");
try {
const client = await Client.login({
email,
password,
totp: async () => prompt("TOTP code: "),
emailOTP: async () => prompt("Email verification code: "),
});
saveSession(ctx.sessionDir, client.toJSON());
const info = client.whoami();
ctx.stderr.write(`Logged in as ${info.email} (user ${info.userID})\n`);
ctx.stderr.write(`Session saved to ${sessionPath(ctx)}\n`);
} catch (err) {
ctx.stderr.write(
`Login failed: ${err instanceof Error ? err.message : err}\n`,
);
return 1;
}
return 0;
};
export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const info = client.whoami();
ctx.stdout.write(JSON.stringify(info) + "\n");
return 0;
};
export const logoutCommand = async (ctx: CliContext): Promise<number> => {
if (existsSync(sessionPath(ctx))) {
unlinkSync(sessionPath(ctx));
ctx.stderr.write("Session deleted.\n");
} else {
ctx.stderr.write("No session found.\n");
}
return 0;
};
export const collectionsCommand = async (
ctx: CliContext,
opts: { json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52): the pre-library CLI printed current state in
// this order, not the albums projection's newest-first order.
const collections = await freshCollections(lib);
if (opts.json) {
ctx.stdout.write(
JSON.stringify(
collections.map((c) => ({
id: c.id,
name: c.name,
type: c.type,
ownerID: c.ownerID,
isShared: c.isShared,
updationTime: c.updationTime,
})),
null,
2,
) + "\n",
);
} else {
for (const c of collections) {
ctx.stdout.write(
`${c.id}\t${c.type}\t${c.name}${c.isShared ? " (shared)" : ""}\n`,
);
}
}
return 0;
} finally {
lib.close();
}
};
export const filesCommand = async (
ctx: CliContext,
opts: { collection: string; json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const collectionID = Number(opts.collection);
if (!Number.isFinite(collectionID)) {
ctx.stderr.write("Invalid collection ID\n");
return 1;
}
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52). Each file prints from its own decrypted
// metadata (raw title, microsecond creationTime) via cli-output, and in
// the pre-library CLI's enumeration order, not the projection's
// newest-first order.
const files = await freshFiles(lib, collectionID);
if (!files) {
ctx.stderr.write(`Collection ${collectionID} not found\n`);
return 1;
}
if (opts.json) {
ctx.stdout.write(
JSON.stringify(files.map(fileListRow), null, 2) + "\n",
);
} else {
for (const file of files) {
ctx.stdout.write(fileListLine(file) + "\n");
}
}
return 0;
} finally {
lib.close();
}
};
export const getCommand = async (
ctx: CliContext,
fileIDStr: string,
opts: { out?: string },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const fileID = Number(fileIDStr);
if (!Number.isFinite(fileID)) {
ctx.stderr.write("Invalid file ID\n");
return 1;
}
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
ctx.stderr.write(`File ${fileID} not found\n`);
return 1;
}
const { photo, file } = resolved;
const result = await photo.original();
// Default name is the file's own title, as the pre-library CLI used
// (not the editedName-preferring projection title) (issue #52).
const outPath = opts.out ?? originalName(file);
copyFileSync(result.path, outPath);
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
return 0;
} finally {
lib.close();
}
};
export const getThumbCommand = async (
ctx: CliContext,
fileIDStr: string,
opts: { out?: string },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const fileID = Number(fileIDStr);
if (!Number.isFinite(fileID)) {
ctx.stderr.write("Invalid file ID\n");
return 1;
}
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
ctx.stderr.write(`File ${fileID} not found\n`);
return 1;
}
const { photo, file } = resolved;
const result = await photo.thumbnail();
// Default name is thumb_<file's own title>, as the pre-library CLI
// used (not the projection title) (issue #52).
const outPath = opts.out ?? thumbnailName(file);
copyFileSync(result.path, outPath);
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
return 0;
} finally {
lib.close();
}
};
export const backupMetadataCommand = async (
ctx: CliContext,
dir: string,
opts: { exif?: boolean; all?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
await runMetadataBackup(lib, client, dir, {
exif: opts.exif || opts.all,
onProgress: (msg) => ctx.stderr.write(msg + "\n"),
});
return 0;
} finally {
lib.close();
}
};
export const backupCommand = async (
ctx: CliContext,
dir: string,
opts: { json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
ctx.stderr.write("Starting backup...\n");
const lib = await Library.open({
client,
downloadDirectory: dir,
cacheDirectory: ctx.cacheDir,
});
try {
const result = await lib.backup({
downloadDirectory: dir,
onProgress: (msg) => {
if (!opts.json) ctx.stderr.write(msg + "\n");
},
});
if (opts.json) {
ctx.stdout.write(JSON.stringify(result, null, 2) + "\n");
} else {
ctx.stderr.write("\n--- Backup complete ---\n");
ctx.stderr.write(` Total files: ${result.totalFiles}\n`);
ctx.stderr.write(` Downloaded: ${result.downloaded}\n`);
ctx.stderr.write(` Skipped: ${result.skipped}\n`);
ctx.stderr.write(` Failed: ${result.failed}\n`);
if (result.errors.length > 0) {
ctx.stderr.write("\nFailed files:\n");
for (const e of result.errors) {
ctx.stderr.write(
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
);
}
}
}
return result.failed > 0 ? 1 : 0;
} finally {
lib.close();
}
};
export const listMissingThumbnailsCommand = async (
ctx: CliContext,
opts: { json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
const missing = await listMissingThumbnails(lib, client, (msg) => {
if (!opts.json) ctx.stderr.write(msg + "\n");
});
if (opts.json) {
ctx.stdout.write(JSON.stringify(missing, null, 2) + "\n");
} else {
if (missing.length === 0) {
ctx.stderr.write("No missing thumbnails found.\n");
} else {
ctx.stderr.write(
`\n${missing.length} file(s) with missing thumbnails:\n`,
);
for (const m of missing) {
ctx.stdout.write(
`${m.fileID}\t${m.title}\t${m.collection}\t${m.reason}\n`,
);
}
}
}
return 0;
} finally {
lib.close();
}
};
export const fixMissingThumbnailsCommand = async (
ctx: CliContext,
opts: { file?: string[]; json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
let fileIDs: number[];
if (opts.file && opts.file.length > 0) {
fileIDs = opts.file.map(Number).filter(Number.isFinite);
} else {
ctx.stderr.write("Scanning for missing thumbnails...\n");
const missing = await listMissingThumbnails(lib, client, (msg) => {
if (!opts.json) ctx.stderr.write(msg + "\n");
});
fileIDs = missing.map((m) => m.fileID);
if (fileIDs.length === 0) {
ctx.stderr.write("No missing thumbnails found.\n");
return 0;
}
ctx.stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
}
const results = await fixMissingThumbnails(
lib,
client,
fileIDs,
(msg) => {
if (!opts.json) ctx.stderr.write(msg + "\n");
},
);
if (opts.json) {
ctx.stdout.write(JSON.stringify(results, null, 2) + "\n");
} else {
const fixed = results.filter((r) => r.status === "fixed").length;
const skipped = results.filter(
(r) => r.status === "skipped",
).length;
const failed = results.filter((r) => r.status === "failed").length;
ctx.stderr.write(`\n--- Done ---\n`);
ctx.stderr.write(` Fixed: ${fixed}\n`);
ctx.stderr.write(` Skipped: ${skipped}\n`);
ctx.stderr.write(` Failed: ${failed}\n`);
if (skipped > 0) {
ctx.stderr.write("\nSkipped (unsupported format):\n");
for (const r of results.filter((r) => r.status === "skipped")) {
ctx.stderr.write(
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
);
}
}
if (failed > 0) {
ctx.stderr.write("\nFailed files:\n");
for (const r of results.filter((r) => r.status === "failed")) {
ctx.stderr.write(
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
);
}
}
}
return results.some((r) => r.status === "failed") ? 1 : 0;
} finally {
lib.close();
}
};
+17 -6
View File
@@ -158,6 +158,18 @@ const streamDecrypt = async (
return totalPlain; return totalPlain;
}; };
// Fsync a file or a directory, so its contents (for a directory, its entries)
// are on stable storage. Exported for the backup tree's copy, which needs the
// same durability as the writer below.
export const fsyncPath = async (path: string): Promise<void> => {
const handle = await open(path, "r");
try {
await handle.sync();
} finally {
await handle.close();
}
};
// Stage a write to `destination` atomically and durably, then rename it into // Stage a write to `destination` atomically and durably, then rename it into
// place. `fill` writes the contents into the open temp file handle — either the // place. `fill` writes the contents into the open temp file handle — either the
// whole buffer at once (`writeAtomic`) or chunk by chunk as they decrypt // whole buffer at once (`writeAtomic`) or chunk by chunk as they decrypt
@@ -193,16 +205,15 @@ const stageAtomic = async (
} finally { } finally {
await handle.close(); await handle.close();
} }
// `rename` replaces the destination's directory entry rather than
// writing through it: an existing symlink at `destination` is
// replaced, not followed, and the new file has the temp file's
// permissions, not those of the file it replaced.
await rename(tmpPath, destination); await rename(tmpPath, destination);
// Fsync the directory so the rename itself survives a crash: renaming // Fsync the directory so the rename itself survives a crash: renaming
// over a synced temp file still leaves the new directory entry in the // over a synced temp file still leaves the new directory entry in the
// page cache until the directory is synced. // page cache until the directory is synced.
const dirHandle = await open(dir, "r"); await fsyncPath(dir);
try {
await dirHandle.sync();
} finally {
await dirHandle.close();
}
} catch (err) { } catch (err) {
// Best-effort cleanup. A failure to remove the temporary file must // Best-effort cleanup. A failure to remove the temporary file must
// never replace the error that actually explains what went wrong. // never replace the error that actually explains what went wrong.
+87
View File
@@ -385,6 +385,93 @@ describe("ApiClient custom origins", () => {
}); });
}); });
describe("ApiClient request URLs", () => {
it("accepts a path with or without a leading slash", async () => {
const { fetch, calls } = recordingFetch(
jsonResponse({}),
jsonResponse({}),
jsonResponse({}),
);
const client = new ApiClient({ fetch });
await client.getJSON("health");
await client.postJSON("users/ott", {});
await client.putJSON("/files/thumbnail", {});
expect(calls.map((c) => c.url)).toEqual([
"https://api.ente.io/health",
"https://api.ente.io/users/ott",
"https://api.ente.io/files/thumbnail",
]);
});
it("accepts an apiOrigin with a trailing slash", async () => {
const { fetch, calls } = recordingFetch(jsonResponse({}));
const client = new ApiClient({
fetch,
apiOrigin: "https://my-ente.example.com/",
});
await client.getJSON("/health");
expect(calls[0]!.url).toBe("https://my-ente.example.com/health");
});
it("keeps a base path in a self-hosted apiOrigin for every request", async () => {
const body = new Uint8Array([1]);
const { fetch, calls } = recordingFetch(
jsonResponse({}),
jsonResponse({}),
jsonResponse({}),
streamResponse(body),
streamResponse(body),
);
const client = new ApiClient({
fetch,
apiOrigin: "https://example.com/ente/",
});
await client.getJSON("/collections/v2", { sinceTime: 0 });
await client.postJSON("/users/ott", {});
await client.putJSON("/files/thumbnail", {});
await client.getFileStream(99);
await client.getThumbnailStream(77);
expect(calls.map((c) => c.url)).toEqual([
"https://example.com/ente/collections/v2?sinceTime=0",
"https://example.com/ente/users/ott",
"https://example.com/ente/files/thumbnail",
"https://example.com/ente/files/download/99",
"https://example.com/ente/files/preview/77",
]);
});
it("percent-encodes query parameters and skips undefined ones", async () => {
const { fetch, calls } = recordingFetch(jsonResponse({}));
const client = new ApiClient({ fetch });
await client.getJSON("/search", {
q: "a&b=c/d é",
limit: 5,
cursor: undefined,
});
const url = new URL(calls[0]!.url);
expect(url.pathname).toBe("/search");
expect(url.search).toBe("?q=a%26b%3Dc%2Fd+%C3%A9&limit=5");
expect(url.searchParams.get("q")).toBe("a&b=c/d é");
});
it("rejects a path that carries its own query string", async () => {
const { fetch, calls } = recordingFetch();
const client = new ApiClient({ fetch });
await expect(client.getJSON("/diff?sinceTime=0")).rejects.toThrow(
/must not contain "\?"/,
);
await expect(client.postJSON("/users/ott?x=1", {})).rejects.toThrow(
/must not contain "\?"/,
);
expect(calls).toHaveLength(0);
});
});
describe("ApiError", () => { describe("ApiError", () => {
it("throws ApiError on 4xx with status, code, requestID", async () => { it("throws ApiError on 4xx with status, code, requestID", async () => {
const { fetch } = recordingFetch( const { fetch } = recordingFetch(
+92 -1
View File
@@ -42,15 +42,44 @@ import {
rmSync, rmSync,
writeFileSync, writeFileSync,
} from "node:fs"; } from "node:fs";
import { spawnSync } from "node:child_process";
import { join } from "node:path"; import { join } from "node:path";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { describe, it, expect, beforeEach, afterEach } from "vitest"; import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import { Library } from "../../src/library/index.js"; import { Library } from "../../src/library/index.js";
import type { ContentSource } from "../../src/library/content.js"; import type { ContentSource } from "../../src/library/content.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js"; import type { CollectionsPage, FilesPage } from "../../src/client.js";
import type { Collection, EnteFile } from "../../src/model/types.js"; import type { Collection, EnteFile } from "../../src/model/types.js";
// `open` and `rename` are wrapped to record, in order, every fsync and rename,
// so a test can pin the sequence "fsync the temp file, rename, fsync the
// directory" that makes a copied original survive a power cut. `vi.hoisted`
// because `vi.mock` factories run before module-level constants exist.
const fsEvents = vi.hoisted(() => [] as string[]);
vi.mock("node:fs/promises", async (importOriginal) => {
const actual = await importOriginal<typeof import("node:fs/promises")>();
return {
...actual,
open: async (
...args: Parameters<typeof actual.open>
): Promise<Awaited<ReturnType<typeof actual.open>>> => {
const handle = await actual.open(...args);
const realSync = handle.sync.bind(handle);
handle.sync = async (): Promise<void> => {
fsEvents.push(`sync:${String(args[0])}`);
await realSync();
};
return handle;
},
rename: async (from: string, to: string): Promise<void> => {
fsEvents.push(`rename:${to}`);
await actual.rename(from, to);
},
};
});
const USER_ID = 42; const USER_ID = 42;
// Decrypted-byte length each stub original writes, keyed by fileID. // Decrypted-byte length each stub original writes, keyed by fileID.
@@ -530,4 +559,66 @@ describe("lib.backup", () => {
expect(readLedger(outDir).files["101"]!.attempts).toBe(1); expect(readLedger(outDir).files["101"]!.attempts).toBe(1);
lib.close(); lib.close();
}); });
it("fsyncs a copied original before the rename and its directory after", async () => {
const lib = await openLibrary(stubSource());
const outDir = join(root, "backup");
const originals = join(outDir, "originals");
const dest = join(originals, "100.jpg");
fsEvents.length = 0;
await lib.backup({ downloadDirectory: outDir });
const at = fsEvents.indexOf(`rename:${dest}`);
expect(at).toBeGreaterThan(0);
expect(fsEvents[at - 1]).toMatch(
/^sync:.*\/\.quak-backup-100\.jpg-\d+-[0-9a-z]*\.tmp$/,
);
expect(fsEvents[at + 1]).toBe(`sync:${originals}`);
lib.close();
});
it("removes temp files left by a killed backup but not those of one still running", async () => {
const outDir = join(root, "backup");
const originals = join(outDir, "originals");
mkdirSync(originals, { recursive: true });
// A child that has already exited: its process ID is not running.
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
const leftover = `.quak-backup-100.jpg-${exitedPID}-abc123.tmp`;
// This test's own process stands in for a backup running at the same
// time.
const inProgress = `.quak-backup-101.jpg-${process.pid}-def456.tmp`;
writeFileSync(join(originals, leftover), "partial");
writeFileSync(join(originals, inProgress), "partial");
const lib = await openLibrary(stubSource());
await lib.backup({ downloadDirectory: outDir });
const names = readdirSync(originals);
expect(names).not.toContain(leftover);
expect(names).toContain(inProgress);
lib.close();
});
it("removes leftover temp files in thumbnails/ but not those of a backup still running", async () => {
const outDir = join(root, "backup");
const thumbnails = join(outDir, "thumbnails");
mkdirSync(thumbnails, { recursive: true });
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
const leftover = `.quak-backup-100.jpg-${exitedPID}-abc123.tmp`;
const inProgress = `.quak-backup-101.jpg-${process.pid}-def456.tmp`;
writeFileSync(join(thumbnails, leftover), "partial");
writeFileSync(join(thumbnails, inProgress), "partial");
const lib = await openLibrary(stubSource());
await lib.backup({
downloadDirectory: outDir,
includeThumbnails: true,
});
const names = readdirSync(thumbnails);
expect(names).not.toContain(leftover);
expect(names).toContain(inProgress);
lib.close();
});
}); });
+397
View File
@@ -0,0 +1,397 @@
/**
* Tests for the CLI commands (`src/cli-commands.ts`, issue #12).
*
* Each command is called directly with a context whose output streams collect
* text, whose session directory is a fresh temp directory, and whose session
* loader hands back a fake client. The fake serves two albums and three files
* from memory, writes stand-in bytes for originals and thumbnails, and makes no
* network calls. The helpers the commands call (`cli-read`, `cli-output`,
* backup, thumbnails) have their own tests; these check what each command
* prints and the exit code it returns.
*/
import {
existsSync,
mkdtempSync,
readFileSync,
rmSync,
statSync,
writeFileSync,
} from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { describe, it, expect, beforeAll, beforeEach, afterEach } from "vitest";
import {
type CliContext,
saveSession,
whoamiCommand,
logoutCommand,
collectionsCommand,
filesCommand,
getCommand,
getThumbCommand,
backupCommand,
listMissingThumbnailsCommand,
} from "../../src/cli-commands.js";
import { loadSession } from "../../src/cli-session.js";
import type { Client, ClientSnapshot } from "../../src/client.js";
import type { ContentSource } from "../../src/library/content.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
import { init } from "../../src/crypto/index.js";
const USER_ID = 42;
const collection = (
id: number,
name: string,
isShared = false,
): Collection => ({
id,
ownerID: USER_ID,
key: new Uint8Array([id]),
name,
type: "album",
updationTime: 1,
isShared,
});
const file = (id: number, collectionID: number, title: string): EnteFile => ({
id,
collectionID,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
metadata: {
title,
fileType: "image",
creationTime: 1000,
modificationTime: 1000,
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime: 1,
});
const COLLECTIONS = [collection(1, "Vacation"), collection(2, "Work", true)];
const FILES: Record<number, EnteFile[]> = {
1: [file(100, 1, "beach.jpg"), file(101, 1, "sunset.jpg")],
2: [file(200, 2, "diagram.png")],
};
// An original is 7 bytes and a thumbnail 3. `failID` makes that file's
// original fail; `emptyThumbID` makes the server report that file's
// thumbnail as empty.
const fakeClient = (opts: { failID?: number; emptyThumbID?: number } = {}) => {
const source: ContentSource = {
original: async ({ file: f, destination }) => {
if (f.id === opts.failID) throw new Error("HTTP 500 from server");
writeFileSync(destination, Buffer.alloc(7, f.id & 0xff));
return { bytesWritten: 7 };
},
thumbnail: async ({ file: f, destination }) => {
writeFileSync(destination, Buffer.alloc(3, f.id & 0xff));
return { bytesWritten: 3 };
},
};
const fake = {
whoami: () => ({ email: "cli@example.com", userID: USER_ID }),
collectionsSince: async () => ({
collections: COLLECTIONS,
deleted: [],
cursor: 1,
}),
filesSince: async (args: { collectionID: number }) => ({
files: FILES[args.collectionID] ?? [],
deleted: [],
cursor: 1,
}),
contentSource: () => source,
getApiClient: () => ({
getThumbnailStream: async (fileID: number) =>
new ReadableStream<Uint8Array>({
start(controller) {
if (fileID !== opts.emptyThumbID) {
controller.enqueue(new Uint8Array(3));
}
controller.close();
},
}),
}),
};
// The commands only call the methods above.
return fake as unknown as Client;
};
// Collects everything written to it.
class Output {
text = "";
write(text: string): void {
this.text += text;
}
}
let root: string;
let stdout: Output;
let stderr: Output;
const context = (client: Client | null = fakeClient()): CliContext => ({
stdout,
stderr,
sessionDir: join(root, "session"),
cacheDir: join(root, "cache"),
loadSession: () => client,
});
beforeAll(async () => {
await init();
});
beforeEach(() => {
root = mkdtempSync(join(tmpdir(), "quak-cli-test-"));
stdout = new Output();
stderr = new Output();
});
afterEach(() => {
rmSync(root, { recursive: true, force: true });
});
describe("session file", () => {
const snapshot: ClientSnapshot = {
email: "cli@example.com",
userID: USER_ID,
token: "token",
masterKey: "a",
secretKey: "b",
publicKey: "c",
};
it("is written with mode 0600 in a directory with mode 0700", () => {
const dir = join(root, "new", "session");
saveSession(dir, snapshot);
expect(statSync(dir).mode & 0o777).toBe(0o700);
const path = join(dir, "session.json");
expect(statSync(path).mode & 0o777).toBe(0o600);
expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot);
});
it("is removed by logout", async () => {
const ctx = context();
saveSession(ctx.sessionDir, snapshot);
expect(await logoutCommand(ctx)).toBe(0);
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
expect(stderr.text).toBe("Session deleted.\n");
});
it("logout without a session says so and exits 0", async () => {
expect(await logoutCommand(context())).toBe(0);
expect(stderr.text).toBe("No session found.\n");
});
it("a missing session exits 1 with 'Not logged in'", async () => {
const ctx = { ...context(), loadSession };
expect(await whoamiCommand(ctx)).toBe(1);
expect(stderr.text).toBe(
`Not logged in. Run "quak login" first.\n` +
`Session file: ${join(ctx.sessionDir, "session.json")}\n`,
);
expect(stdout.text).toBe("");
});
it("a corrupt session exits 1 and says it is corrupt", async () => {
const ctx = { ...context(), loadSession };
saveSession(ctx.sessionDir, snapshot);
expect(await collectionsCommand(ctx, {})).toBe(1);
expect(stderr.text).toContain("is corrupt");
expect(stderr.text).toContain(
`Run "quak logout" and then "quak login" to replace it.\n`,
);
expect(stdout.text).toBe("");
});
});
describe("whoami", () => {
it("prints the account as one line of JSON", async () => {
expect(await whoamiCommand(context())).toBe(0);
expect(stdout.text).toBe(
`{"email":"cli@example.com","userID":${USER_ID}}\n`,
);
});
});
describe("collections", () => {
it("prints one tab-separated line per album", async () => {
expect(await collectionsCommand(context(), {})).toBe(0);
expect(stdout.text).toBe(
"1\talbum\tVacation\n" + "2\talbum\tWork (shared)\n",
);
});
it("prints a JSON array with --json", async () => {
expect(await collectionsCommand(context(), { json: true })).toBe(0);
expect(JSON.parse(stdout.text)).toEqual([
{
id: 1,
name: "Vacation",
type: "album",
ownerID: USER_ID,
isShared: false,
updationTime: 1,
},
{
id: 2,
name: "Work",
type: "album",
ownerID: USER_ID,
isShared: true,
updationTime: 1,
},
]);
});
});
describe("files", () => {
it("prints one tab-separated line per file", async () => {
expect(await filesCommand(context(), { collection: "1" })).toBe(0);
expect(stdout.text).toBe(
"100\timage\tbeach.jpg\n" + "101\timage\tsunset.jpg\n",
);
});
it("prints a JSON array with --json", async () => {
const code = await filesCommand(context(), {
collection: "2",
json: true,
});
expect(code).toBe(0);
expect(JSON.parse(stdout.text)).toEqual([
{
id: 200,
title: "diagram.png",
fileType: "image",
creationTime: 1000,
collectionID: 2,
},
]);
});
it("exits 1 for an unknown collection", async () => {
expect(await filesCommand(context(), { collection: "9" })).toBe(1);
expect(stderr.text).toBe("Collection 9 not found\n");
});
it("exits 1 for a collection ID that is not a number", async () => {
expect(await filesCommand(context(), { collection: "abc" })).toBe(1);
expect(stderr.text).toBe("Invalid collection ID\n");
});
});
describe("get and get-thumb", () => {
it("get finds a file in any album without --collection", async () => {
const out = join(root, "diagram.png");
expect(await getCommand(context(), "200", { out })).toBe(0);
expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
});
it("get-thumb finds a file in any album without --collection", async () => {
const out = join(root, "thumb.jpg");
expect(await getThumbCommand(context(), "200", { out })).toBe(0);
expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
expect(stderr.text).toBe(`3 bytes -> ${out}\n`);
});
it("get exits 1 when no album has the file", async () => {
const out = join(root, "x");
expect(await getCommand(context(), "999", { out })).toBe(1);
expect(stderr.text).toBe("File 999 not found\n");
expect(existsSync(out)).toBe(false);
});
it("get-thumb exits 1 when no album has the file", async () => {
const out = join(root, "x");
expect(await getThumbCommand(context(), "999", { out })).toBe(1);
expect(stderr.text).toBe("File 999 not found\n");
expect(existsSync(out)).toBe(false);
});
it("both exit 1 for a file ID that is not a number", async () => {
expect(await getCommand(context(), "abc", {})).toBe(1);
expect(await getThumbCommand(context(), "abc", {})).toBe(1);
expect(stderr.text).toBe("Invalid file ID\nInvalid file ID\n");
});
});
describe("backup", () => {
it("exits 0 and prints a summary when every file is saved", async () => {
const dir = join(root, "backup");
expect(await backupCommand(context(), dir, {})).toBe(0);
expect(stderr.text).toContain(
"\n--- Backup complete ---\n" +
" Total files: 3\n" +
" Downloaded: 3\n" +
" Skipped: 0\n" +
" Failed: 0\n",
);
expect(stdout.text).toBe("");
});
it("exits 1 and lists the file when one download fails", async () => {
const ctx = context(fakeClient({ failID: 101 }));
expect(await backupCommand(ctx, join(root, "backup"), {})).toBe(1);
expect(stderr.text).toContain(" Failed: 1\n");
expect(stderr.text).toContain(
"\nFailed files:\n" +
" [Vacation] sunset.jpg (id 101): HTTP 500 from server\n",
);
});
it("prints the result as JSON with --json, still exiting 1 on a failure", async () => {
const ctx = context(fakeClient({ failID: 101 }));
const code = await backupCommand(ctx, join(root, "backup"), {
json: true,
});
expect(code).toBe(1);
const result = JSON.parse(stdout.text);
expect(result).toMatchObject({
totalFiles: 3,
downloaded: 2,
skipped: 0,
failed: 1,
});
expect(result.errors[0].fileID).toBe(101);
expect(stderr.text).toBe("Starting backup...\n");
});
});
describe("helper list-missing-thumbnails", () => {
it("prints one line per file with an empty thumbnail", async () => {
const ctx = context(fakeClient({ emptyThumbID: 200 }));
expect(await listMissingThumbnailsCommand(ctx, {})).toBe(0);
expect(stdout.text).toBe(
"200\tdiagram.png\tWork\tempty thumbnail (0 bytes)\n",
);
expect(stderr.text).toContain("\n1 file(s) with missing thumbnails:\n");
});
it("says so when nothing is missing", async () => {
expect(await listMissingThumbnailsCommand(context(), {})).toBe(0);
expect(stdout.text).toBe("");
expect(stderr.text).toContain("No missing thumbnails found.\n");
});
it("prints a JSON array with --json and no progress", async () => {
const ctx = context(fakeClient({ emptyThumbID: 200 }));
expect(await listMissingThumbnailsCommand(ctx, { json: true })).toBe(0);
expect(JSON.parse(stdout.text)).toEqual([
{
fileID: 200,
title: "diagram.png",
collection: "Work",
reason: "empty thumbnail (0 bytes)",
},
]);
expect(stderr.text).toBe("");
});
});
+46
View File
@@ -48,6 +48,7 @@
*/ */
import { import {
chmodSync,
existsSync, existsSync,
mkdirSync, mkdirSync,
readdirSync, readdirSync,
@@ -989,6 +990,51 @@ describe.each(entryPoints)(
expect(readFileSync(outPath)).toEqual(Buffer.from(existing)); expect(readFileSync(outPath)).toEqual(Buffer.from(existing));
expect(readdirSync(dir)).toEqual(["rename-fails.bin"]); expect(readdirSync(dir)).toEqual(["rename-fails.bin"]);
}); });
it("fails without creating anything when the destination directory does not exist", async () => {
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(
patternBytes(64, 33),
key,
);
const { api, file } = fixtureFor(key, header, ciphertext);
const dir = freshDir();
const outPath = join(dir, "missing", "never.bin");
await expect(download(api, file, outPath)).rejects.toMatchObject({
code: "ENOENT",
});
// The missing directory is not created on the caller's behalf.
expect(readdirSync(dir)).toEqual([]);
});
// Root ignores directory permissions, so this cannot fail as root
// (the Docker test image runs as root).
it.skipIf(process.getuid?.() === 0)(
"fails without creating anything when the destination directory is not writable",
async () => {
const key =
sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(
patternBytes(64, 34),
key,
);
const { api, file } = fixtureFor(key, header, ciphertext);
const dir = freshDir();
const outPath = join(dir, "never.bin");
chmodSync(dir, 0o500);
try {
await expect(
download(api, file, outPath),
).rejects.toMatchObject({ code: "EACCES" });
} finally {
chmodSync(dir, 0o700);
}
expect(readdirSync(dir)).toEqual([]);
},
);
}, },
); );