Compare commits
4
Commits
c7cf3b77b5
...
99a536286d
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
99a536286d | ||
|
|
ed535be1da | ||
|
|
d545dcd8b1 | ||
|
|
52f58f5d2b |
@@ -323,6 +323,7 @@ Endpoints used:
|
||||
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a
|
||||
collection; paginate while `hasMore` is true.
|
||||
- `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes.
|
||||
- `POST /files/data/fetch`: fetch encrypted ML data for a batch of files.
|
||||
- `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair).
|
||||
- `PUT /files/thumbnail`: register an uploaded thumbnail's object key.
|
||||
|
||||
@@ -377,20 +378,23 @@ headers — `getFileStream` returns as soon as headers arrive, so a deadline tha
|
||||
only guarded the initial request would leave the same hang one layer down.
|
||||
|
||||
**Non-idempotent requests are not blindly replayed.** `postJSON` and `putJSON`
|
||||
reach `/users/srp/create-session`, `/users/two-factor/verify` — which consumes
|
||||
one of a small number of second-factor attempts — and `/files/thumbnail`. They
|
||||
are retried only on the three failures that establish no TCP connection to the
|
||||
server ever existed, so no request byte can have been transmitted: `ENOTFOUND`
|
||||
and `EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the
|
||||
peer refused the connection). A 5xx, a mid-flight reset and a deadline are all
|
||||
left to the caller, because each of them can happen after the server has already
|
||||
acted. The routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are
|
||||
excluded for the same reason, despite looking like connect-time failures: on
|
||||
Linux an ICMP unreachable arriving mid-flight, or a local interface going down
|
||||
after the request was written, delivers them on an already-established socket.
|
||||
They stay retryable for the idempotent calls. `putFile` is exempt: a presigned
|
||||
PUT stores one whole object at one key in one request, so replaying it has no
|
||||
partial state to damage.
|
||||
send every `POST` and `PUT` in the endpoint list above; some of them change
|
||||
server state, and `/users/two-factor/verify` consumes one of a small number of
|
||||
second-factor attempts. They are retried only when every errno in the error's
|
||||
`cause` chain is one of the three that establish no TCP connection to the server
|
||||
ever existed, so no request byte can have been transmitted: `ENOTFOUND` and
|
||||
`EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer
|
||||
refused the connection). A 5xx, a mid-flight reset and a deadline are all left
|
||||
to the caller, because each of them can happen after the server has already
|
||||
acted. These two do not follow redirects either: a redirect means the server
|
||||
already received the request, so it is reported as an error and not retried. The
|
||||
routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the
|
||||
same reason, despite looking like connect-time failures: on Linux an ICMP
|
||||
unreachable arriving mid-flight, or a local interface going down after the
|
||||
request was written, delivers them on an already-established socket. They stay
|
||||
retryable for the idempotent calls. `putFile` is exempt: a presigned PUT stores
|
||||
one whole object at one key in one request, so replaying it has no partial state
|
||||
to damage.
|
||||
|
||||
A download is retried as a whole — request, stream consumption, and decryption —
|
||||
because a socket reset after the response headers have arrived surfaces in the
|
||||
@@ -490,6 +494,16 @@ appears in. On subsequent runs, existing originals are skipped. If a download
|
||||
fails, the error is logged and the backup continues with the next file. The exit
|
||||
code is non-zero if any files failed.
|
||||
|
||||
Each original is copied to a temporary file named
|
||||
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp` in the same directory, synced
|
||||
to disk, and renamed into place, so an original is either complete or absent,
|
||||
even after a power cut. A run that is killed can leave one of these temporary
|
||||
files behind; the next backup deletes those whose process is no longer running.
|
||||
Downloads and the content cache use the same scheme with `.quak-<random>.tmp`
|
||||
names. The rename replaces whatever was at the destination rather than writing
|
||||
through it: a symlink there is replaced, not followed, and the new file has the
|
||||
temporary file's permissions, not those of the file it replaced.
|
||||
|
||||
## TODO
|
||||
|
||||
- [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
|
||||
|
||||
@@ -18,6 +18,36 @@ Tag v1.0.0.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies
|
||||
moved from `bin/quak.ts` into `src/cli-commands.ts` as functions that take
|
||||
their options and a context (output streams, session directory, cache
|
||||
directory, session loader) and return an exit code; `bin/quak.ts` only wires
|
||||
them to commander and exits with the code once stdout and stderr have drained,
|
||||
so nothing below it calls `process.exit`. `test/cli/commands.test.ts` drives
|
||||
them with a fake client: session file modes, logout, the missing and corrupt
|
||||
session paths, and the output and exit code of `whoami`, `collections`,
|
||||
`files`, `get`, `get-thumb`, `backup` and `helper list-missing-thumbnails`.
|
||||
- 2026-09-23: Hardened the backup tree's atomic copy (issue 22). `copyAtomic`
|
||||
fsyncs its temp file before the rename and the directory after it, through the
|
||||
download writer's `fsyncPath`; each backup run deletes `.quak-backup-*.tmp`
|
||||
files whose process is no longer running. The README backup layout names the
|
||||
temp files and states that the rename replaces a symlink and takes the temp
|
||||
file's permissions. Added tests for a missing and an unwritable destination
|
||||
directory for `downloadFile` and `downloadThumbnail`.
|
||||
- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue
|
||||
11). Every segment length is checked against the remaining bytes and lengths
|
||||
under 2 stop the scan, so a truncated or corrupt original can neither throw
|
||||
nor loop. A malformed or unparseable EXIF segment is recorded as
|
||||
`imageMetadata.exifError`, and a failure to read the original as
|
||||
`imageMetadataError` in the per-file JSON, instead of the field being left
|
||||
out.
|
||||
- 2026-09-22: Hardened the retry classifier (issue 80). A `POST` or `PUT` is
|
||||
replayed only when every errno in the cause chain is a connect errno, and it
|
||||
no longer follows redirects. `getRetryOptions()` returns a copy. Tests pin
|
||||
every errno the classifier names, the cause-chain depth limit, cycle
|
||||
termination, and a fresh deadline per attempt for every retrying entry point.
|
||||
The README's endpoint list is the one place that names the requests the replay
|
||||
rule covers.
|
||||
- 2026-09-22: Stopped `make test` collecting tests from checkouts nested under
|
||||
`.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a
|
||||
nested checkout ran the whole suite again; `vitest.config.ts` now adds
|
||||
|
||||
+48
-375
@@ -1,62 +1,26 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { input, password as passwordPrompt } from "@inquirer/prompts";
|
||||
import { stdout, stderr } from "node:process";
|
||||
import { copyFileSync, existsSync, mkdirSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { Command } from "commander";
|
||||
import envPaths from "env-paths";
|
||||
import { Client, type ClientSnapshot } from "../src/client.js";
|
||||
import { init } from "../src/crypto/index.js";
|
||||
import { Library, type LibraryClient } from "../src/library/index.js";
|
||||
import {
|
||||
fileListRow,
|
||||
fileListLine,
|
||||
originalName,
|
||||
thumbnailName,
|
||||
} from "../src/cli-output.js";
|
||||
import { freshCollections, freshFiles, freshFile } from "../src/cli-read.js";
|
||||
type CliContext,
|
||||
loginCommand,
|
||||
whoamiCommand,
|
||||
logoutCommand,
|
||||
collectionsCommand,
|
||||
filesCommand,
|
||||
getCommand,
|
||||
getThumbCommand,
|
||||
backupMetadataCommand,
|
||||
backupCommand,
|
||||
listMissingThumbnailsCommand,
|
||||
fixMissingThumbnailsCommand,
|
||||
} from "../src/cli-commands.js";
|
||||
import { loadSession } from "../src/cli-session.js";
|
||||
import { runMetadataBackup } from "../src/metadata-backup.js";
|
||||
import {
|
||||
listMissingThumbnails,
|
||||
fixMissingThumbnails,
|
||||
} from "../src/thumbnails.js";
|
||||
|
||||
const paths = envPaths("quak", { suffix: "" });
|
||||
const sessionPath = join(paths.data, "session.json");
|
||||
|
||||
const saveSession = (snapshot: ClientSnapshot): void => {
|
||||
mkdirSync(paths.data, { recursive: true, mode: 0o700 });
|
||||
writeFileSync(sessionPath, JSON.stringify(snapshot, null, 2), {
|
||||
mode: 0o600,
|
||||
});
|
||||
};
|
||||
|
||||
const requireSession = (): Client => {
|
||||
let client: Client | null;
|
||||
try {
|
||||
client = loadSession(sessionPath);
|
||||
} catch (err) {
|
||||
stderr.write(
|
||||
`${err instanceof Error ? err.message : err}\n` +
|
||||
`Run "quak logout" and then "quak login" to replace it.\n`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
if (!client) {
|
||||
stderr.write(
|
||||
`Not logged in. Run "quak login" first.\nSession file: ${sessionPath}\n`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
return client;
|
||||
};
|
||||
|
||||
const prompt = async (message: string): Promise<string> => input({ message });
|
||||
|
||||
const promptSecret = async (message: string): Promise<string> =>
|
||||
passwordPrompt({ message, mask: true });
|
||||
|
||||
const program = new Command();
|
||||
|
||||
@@ -70,51 +34,28 @@ program
|
||||
"(default: the per-user cache directory)",
|
||||
);
|
||||
|
||||
// The `--cache-dir` global, or undefined to let the library pick its per-user
|
||||
// default keyed by the account id.
|
||||
const cacheDirOption = (): string | undefined =>
|
||||
program.opts<{ cacheDir?: string }>().cacheDir;
|
||||
|
||||
// A library client that omits `fetchMLData`, so the point commands below do not
|
||||
// kick the library's background ML backfill: they read metadata, or fetch one
|
||||
// file's content, and exit. `backup` and `backup-metadata` handle ML on their
|
||||
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
|
||||
// originals through the on-disk cache.
|
||||
const readLibraryClient = (client: Client): LibraryClient => ({
|
||||
whoami: () => client.whoami(),
|
||||
collectionsSince: (args) => client.collectionsSince(args),
|
||||
filesSince: (args) => client.filesSince(args),
|
||||
contentSource: () => client.contentSource(),
|
||||
const context = (): CliContext => ({
|
||||
stdout,
|
||||
stderr,
|
||||
sessionDir: paths.data,
|
||||
cacheDir: program.opts<{ cacheDir?: string }>().cacheDir,
|
||||
loadSession,
|
||||
});
|
||||
|
||||
// Open a library for a single point command: the aggressive background precache
|
||||
// (issue #48) is off — a one-shot `collections` or `get` must not start
|
||||
// downloading the whole account — and the refresh interval is long so no second
|
||||
// refresh fires mid-command.
|
||||
const openReadLibrary = (client: Client): Promise<Library> =>
|
||||
Library.open({
|
||||
client: readLibraryClient(client),
|
||||
cacheDirectory: cacheDirOption(),
|
||||
refreshIntervalSeconds: 3600,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
|
||||
// Close the library and exit once stdout/stderr have drained. `process.exit`
|
||||
// alone can truncate buffered piped output, and the library keeps the event
|
||||
// loop alive with a background refresh, so a plain return could hang; this does
|
||||
// neither.
|
||||
const finish = (lib: Library | undefined, code: number): void => {
|
||||
lib?.close();
|
||||
// Run a command and exit with its code once stdout/stderr have drained.
|
||||
// Exiting before the drain can truncate piped output, and the library can keep
|
||||
// the event loop alive after a command returns, so a plain return could hang.
|
||||
const run = async (command: Promise<number>): Promise<void> => {
|
||||
process.exitCode = await command;
|
||||
const pending = [stdout, stderr].filter((s) => s.writableLength > 0);
|
||||
if (pending.length === 0) {
|
||||
process.exit(code);
|
||||
process.exit();
|
||||
return;
|
||||
}
|
||||
let remaining = pending.length;
|
||||
for (const s of pending) {
|
||||
s.once("drain", () => {
|
||||
if (--remaining === 0) process.exit(code);
|
||||
if (--remaining === 0) process.exit();
|
||||
});
|
||||
}
|
||||
};
|
||||
@@ -122,93 +63,25 @@ const finish = (lib: Library | undefined, code: number): void => {
|
||||
program
|
||||
.command("login")
|
||||
.description("Log in to an Ente account and save the session")
|
||||
.action(async () => {
|
||||
await init();
|
||||
const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
|
||||
const password =
|
||||
process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
|
||||
|
||||
stderr.write("Authenticating...\n");
|
||||
try {
|
||||
const client = await Client.login({
|
||||
email,
|
||||
password,
|
||||
totp: async () => prompt("TOTP code: "),
|
||||
emailOTP: async () => prompt("Email verification code: "),
|
||||
});
|
||||
|
||||
saveSession(client.toJSON());
|
||||
const info = client.whoami();
|
||||
stderr.write(`Logged in as ${info.email} (user ${info.userID})\n`);
|
||||
stderr.write(`Session saved to ${sessionPath}\n`);
|
||||
} catch (err) {
|
||||
stderr.write(
|
||||
`Login failed: ${err instanceof Error ? err.message : err}\n`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
});
|
||||
.action(() => run(loginCommand(context())));
|
||||
|
||||
program
|
||||
.command("whoami")
|
||||
.description("Print the logged-in account")
|
||||
.action(async () => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const info = client.whoami();
|
||||
stdout.write(JSON.stringify(info) + "\n");
|
||||
});
|
||||
.action(() => run(whoamiCommand(context())));
|
||||
|
||||
program
|
||||
.command("logout")
|
||||
.description("Delete the saved session")
|
||||
.action(async () => {
|
||||
if (existsSync(sessionPath)) {
|
||||
const { unlinkSync } = await import("node:fs");
|
||||
unlinkSync(sessionPath);
|
||||
stderr.write("Session deleted.\n");
|
||||
} else {
|
||||
stderr.write("No session found.\n");
|
||||
}
|
||||
});
|
||||
.action(() => run(logoutCommand(context())));
|
||||
|
||||
program
|
||||
.command("collections")
|
||||
.description("List all collections (albums)")
|
||||
.option("--json", "Output as JSON array")
|
||||
.action(async (opts: { json?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const lib = await openReadLibrary(client);
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52): the pre-library CLI printed current state in
|
||||
// this order, not the albums projection's newest-first order.
|
||||
const collections = await freshCollections(lib);
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(
|
||||
JSON.stringify(
|
||||
collections.map((c) => ({
|
||||
id: c.id,
|
||||
name: c.name,
|
||||
type: c.type,
|
||||
ownerID: c.ownerID,
|
||||
isShared: c.isShared,
|
||||
updationTime: c.updationTime,
|
||||
})),
|
||||
null,
|
||||
2,
|
||||
) + "\n",
|
||||
.action((opts: { json?: boolean }) =>
|
||||
run(collectionsCommand(context(), opts)),
|
||||
);
|
||||
} else {
|
||||
for (const c of collections) {
|
||||
stdout.write(
|
||||
`${c.id}\t${c.type}\t${c.name}${c.isShared ? " (shared)" : ""}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
finish(lib, 0);
|
||||
});
|
||||
|
||||
program
|
||||
.command("files")
|
||||
@@ -218,39 +91,9 @@ program
|
||||
"Collection ID (from `quak collections`)",
|
||||
)
|
||||
.option("--json", "Output as JSON array")
|
||||
.action(async (opts: { collection: string; json?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const collectionID = Number(opts.collection);
|
||||
if (!Number.isFinite(collectionID)) {
|
||||
stderr.write("Invalid collection ID\n");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(client);
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52). Each file prints from its own decrypted
|
||||
// metadata (raw title, microsecond creationTime) via cli-output, and in
|
||||
// the pre-library CLI's enumeration order, not the projection's
|
||||
// newest-first order.
|
||||
const files = await freshFiles(lib, collectionID);
|
||||
if (!files) {
|
||||
stderr.write(`Collection ${collectionID} not found\n`);
|
||||
finish(lib, 1);
|
||||
return;
|
||||
}
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(
|
||||
JSON.stringify(files.map(fileListRow), null, 2) + "\n",
|
||||
.action((opts: { collection: string; json?: boolean }) =>
|
||||
run(filesCommand(context(), opts)),
|
||||
);
|
||||
} else {
|
||||
for (const file of files) {
|
||||
stdout.write(fileListLine(file) + "\n");
|
||||
}
|
||||
}
|
||||
finish(lib, 0);
|
||||
});
|
||||
|
||||
program
|
||||
.command("get")
|
||||
@@ -258,34 +101,9 @@ program
|
||||
.argument("<fileID>", "File ID (from `quak files`)")
|
||||
.option("--out <path>", "Output file path")
|
||||
.option("--collection <id>", "Accepted for compatibility; ignored")
|
||||
.action(async (fileIDStr: string, opts: { out?: string }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
stderr.write("Invalid file ID\n");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(client);
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
stderr.write(`File ${fileID} not found\n`);
|
||||
finish(lib, 1);
|
||||
return;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await photo.original();
|
||||
// Default name is the file's own title, as the pre-library CLI used
|
||||
// (not the editedName-preferring projection title) (issue #52).
|
||||
const outPath = opts.out ?? originalName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
finish(lib, 0);
|
||||
});
|
||||
.action((fileID: string, opts: { out?: string }) =>
|
||||
run(getCommand(context(), fileID, opts)),
|
||||
);
|
||||
|
||||
program
|
||||
.command("get-thumb")
|
||||
@@ -293,34 +111,9 @@ program
|
||||
.argument("<fileID>", "File ID (from `quak files`)")
|
||||
.option("--out <path>", "Output file path")
|
||||
.option("--collection <id>", "Accepted for compatibility; ignored")
|
||||
.action(async (fileIDStr: string, opts: { out?: string }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
stderr.write("Invalid file ID\n");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(client);
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
stderr.write(`File ${fileID} not found\n`);
|
||||
finish(lib, 1);
|
||||
return;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await photo.thumbnail();
|
||||
// Default name is thumb_<file's own title>, as the pre-library CLI
|
||||
// used (not the projection title) (issue #52).
|
||||
const outPath = opts.out ?? thumbnailName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
finish(lib, 0);
|
||||
});
|
||||
.action((fileID: string, opts: { out?: string }) =>
|
||||
run(getThumbCommand(context(), fileID, opts)),
|
||||
);
|
||||
|
||||
program
|
||||
.command("backup-metadata")
|
||||
@@ -333,16 +126,9 @@ program
|
||||
"Download each file and extract full EXIF/IPTC/XMP metadata (slow)",
|
||||
)
|
||||
.option("--all", "Alias for --exif")
|
||||
.action(async (dir: string, opts: { exif?: boolean; all?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const lib = await openReadLibrary(client);
|
||||
await runMetadataBackup(lib, client, dir, {
|
||||
exif: opts.exif || opts.all,
|
||||
onProgress: (msg) => stderr.write(msg + "\n"),
|
||||
});
|
||||
finish(lib, 0);
|
||||
});
|
||||
.action((dir: string, opts: { exif?: boolean; all?: boolean }) =>
|
||||
run(backupMetadataCommand(context(), dir, opts)),
|
||||
);
|
||||
|
||||
program
|
||||
.command("backup")
|
||||
@@ -351,43 +137,9 @@ program
|
||||
)
|
||||
.argument("<dir>", "Output directory")
|
||||
.option("--json", "Print result as JSON instead of human-readable summary")
|
||||
.action(async (dir: string, opts: { json?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
|
||||
stderr.write("Starting backup...\n");
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
downloadDirectory: dir,
|
||||
cacheDirectory: cacheDirOption(),
|
||||
});
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: dir,
|
||||
onProgress: (msg) => {
|
||||
if (!opts.json) stderr.write(msg + "\n");
|
||||
},
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(JSON.stringify(result, null, 2) + "\n");
|
||||
} else {
|
||||
stderr.write("\n--- Backup complete ---\n");
|
||||
stderr.write(` Total files: ${result.totalFiles}\n`);
|
||||
stderr.write(` Downloaded: ${result.downloaded}\n`);
|
||||
stderr.write(` Skipped: ${result.skipped}\n`);
|
||||
stderr.write(` Failed: ${result.failed}\n`);
|
||||
if (result.errors.length > 0) {
|
||||
stderr.write("\nFailed files:\n");
|
||||
for (const e of result.errors) {
|
||||
stderr.write(
|
||||
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
|
||||
.action((dir: string, opts: { json?: boolean }) =>
|
||||
run(backupCommand(context(), dir, opts)),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
finish(lib, result.failed > 0 ? 1 : 0);
|
||||
});
|
||||
|
||||
const helper = program
|
||||
.command("helper")
|
||||
@@ -397,32 +149,9 @@ helper
|
||||
.command("list-missing-thumbnails")
|
||||
.description("List files whose thumbnails are missing or empty")
|
||||
.option("--json", "Output as JSON array")
|
||||
.action(async (opts: { json?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const lib = await openReadLibrary(client);
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
if (!opts.json) stderr.write(msg + "\n");
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(JSON.stringify(missing, null, 2) + "\n");
|
||||
} else {
|
||||
if (missing.length === 0) {
|
||||
stderr.write("No missing thumbnails found.\n");
|
||||
} else {
|
||||
stderr.write(
|
||||
`\n${missing.length} file(s) with missing thumbnails:\n`,
|
||||
.action((opts: { json?: boolean }) =>
|
||||
run(listMissingThumbnailsCommand(context(), opts)),
|
||||
);
|
||||
for (const m of missing) {
|
||||
stdout.write(
|
||||
`${m.fileID}\t${m.title}\t${m.collection}\t${m.reason}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
finish(lib, 0);
|
||||
});
|
||||
|
||||
helper
|
||||
.command("fix-missing-thumbnails")
|
||||
@@ -434,65 +163,9 @@ helper
|
||||
"Specific file IDs to fix (default: fix all missing)",
|
||||
)
|
||||
.option("--json", "Output as JSON")
|
||||
.action(async (opts: { file?: string[]; json?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const lib = await openReadLibrary(client);
|
||||
|
||||
let fileIDs: number[];
|
||||
if (opts.file && opts.file.length > 0) {
|
||||
fileIDs = opts.file.map(Number).filter(Number.isFinite);
|
||||
} else {
|
||||
stderr.write("Scanning for missing thumbnails...\n");
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
if (!opts.json) stderr.write(msg + "\n");
|
||||
});
|
||||
fileIDs = missing.map((m) => m.fileID);
|
||||
if (fileIDs.length === 0) {
|
||||
stderr.write("No missing thumbnails found.\n");
|
||||
finish(lib, 0);
|
||||
return;
|
||||
}
|
||||
stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
|
||||
}
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
fileIDs,
|
||||
(msg) => {
|
||||
if (!opts.json) stderr.write(msg + "\n");
|
||||
},
|
||||
.action((opts: { file?: string[]; json?: boolean }) =>
|
||||
run(fixMissingThumbnailsCommand(context(), opts)),
|
||||
);
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(JSON.stringify(results, null, 2) + "\n");
|
||||
} else {
|
||||
const fixed = results.filter((r) => r.status === "fixed").length;
|
||||
const skipped = results.filter(
|
||||
(r) => r.status === "skipped",
|
||||
).length;
|
||||
const failed = results.filter((r) => r.status === "failed").length;
|
||||
stderr.write(`\n--- Done ---\n`);
|
||||
stderr.write(` Fixed: ${fixed}\n`);
|
||||
stderr.write(` Skipped: ${skipped}\n`);
|
||||
stderr.write(` Failed: ${failed}\n`);
|
||||
if (skipped > 0) {
|
||||
stderr.write("\nSkipped (unsupported format):\n");
|
||||
for (const r of results.filter((r) => r.status === "skipped")) {
|
||||
stderr.write(` ${r.fileID}\t${r.title}\t${r.reason}\n`);
|
||||
}
|
||||
}
|
||||
if (failed > 0) {
|
||||
stderr.write("\nFailed files:\n");
|
||||
for (const r of results.filter((r) => r.status === "failed")) {
|
||||
stderr.write(` ${r.fileID}\t${r.title}\t${r.reason}\n`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
finish(lib, results.some((r) => r.status === "failed") ? 1 : 0);
|
||||
});
|
||||
|
||||
await init();
|
||||
program.parse();
|
||||
|
||||
+14
-13
@@ -146,8 +146,9 @@ export class ApiClient {
|
||||
// The policy this client was configured with, so that a caller wrapping a
|
||||
// whole operation in its own `withRetry` — the download layer — runs under
|
||||
// the same settings rather than under the library defaults.
|
||||
// A copy, so the caller cannot change this client's settings through it.
|
||||
getRetryOptions(): ResolvedRetryOptions {
|
||||
return this.retry;
|
||||
return { ...this.retry };
|
||||
}
|
||||
|
||||
private headers(extra?: Record<string, string>): Record<string, string> {
|
||||
@@ -228,15 +229,15 @@ export class ApiClient {
|
||||
|
||||
async postJSON<T>(path: string, body: unknown): Promise<T> {
|
||||
const url = `${this.apiOrigin}${path}`;
|
||||
// Idempotency: this reaches `/users/srp/create-session`,
|
||||
// `/users/two-factor/verify` and `/users/ott`, all of which change
|
||||
// server state — verifying a second factor consumes one of a small
|
||||
// number of attempts. So a POST is replayed only on a failure that
|
||||
// establishes no TCP connection to the server ever existed: DNS
|
||||
// produced no address, or the peer refused the connection. A 5xx, a
|
||||
// mid-flight reset, a routing errno (which Linux also delivers on an
|
||||
// established socket) and a timeout are all left to the caller,
|
||||
// because each of them can occur after the server has already acted.
|
||||
// Not idempotent: a POST is replayed only when `isSafeToReplay`
|
||||
// says no request byte can have reached the server. The endpoints
|
||||
// this covers are listed in the README under "Endpoints used".
|
||||
//
|
||||
// Redirects are not followed. The origin has already received the
|
||||
// request when it answers with one, so a connection refused by the
|
||||
// redirect target would look replay-safe when it is not. The API has
|
||||
// no legitimate redirect, so one surfaces as an `ApiError` with its
|
||||
// 3xx status, which is not retried.
|
||||
return withRetry(
|
||||
async () => {
|
||||
const resp = await this._fetch(url, {
|
||||
@@ -245,6 +246,7 @@ export class ApiClient {
|
||||
"Content-Type": "application/json",
|
||||
}),
|
||||
body: JSON.stringify(body),
|
||||
redirect: "manual",
|
||||
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
||||
});
|
||||
await this.throwIfError(resp);
|
||||
@@ -303,9 +305,7 @@ export class ApiClient {
|
||||
|
||||
async putJSON<T>(path: string, body: unknown): Promise<T> {
|
||||
const url = `${this.apiOrigin}${path}`;
|
||||
// Same idempotency rule as `postJSON`, for the same reason: this
|
||||
// reaches `/files/thumbnail`, which registers an uploaded thumbnail
|
||||
// against a file.
|
||||
// Same replay and redirect rules as `postJSON`, for the same reasons.
|
||||
return withRetry(
|
||||
async () => {
|
||||
const resp = await this._fetch(url, {
|
||||
@@ -314,6 +314,7 @@ export class ApiClient {
|
||||
"Content-Type": "application/json",
|
||||
}),
|
||||
body: JSON.stringify(body),
|
||||
redirect: "manual",
|
||||
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
||||
});
|
||||
await this.throwIfError(resp);
|
||||
|
||||
+51
-9
@@ -29,19 +29,20 @@
|
||||
// rather than counted forever, which would poison a scheduled backup's exit code.
|
||||
|
||||
import {
|
||||
copyFileSync,
|
||||
lstatSync,
|
||||
mkdirSync,
|
||||
readdirSync,
|
||||
readFileSync,
|
||||
readlinkSync,
|
||||
renameSync,
|
||||
rmSync,
|
||||
statSync,
|
||||
symlinkSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { copyFile, rename, rm } from "node:fs/promises";
|
||||
import { basename, dirname, join, relative } from "node:path";
|
||||
|
||||
import { fsyncPath } from "./download/index.js";
|
||||
import { safeExtension, sanitizeFileName } from "./filename.js";
|
||||
import type { Collection, EnteFile } from "./model/types.js";
|
||||
|
||||
@@ -154,8 +155,12 @@ const errorMessage = (err: unknown): string =>
|
||||
err instanceof Error ? err.message : String(err);
|
||||
|
||||
// Copy bytes into `dest` via a temp file in the same directory plus rename, so
|
||||
// `dest` appears only once it is whole ("present means complete").
|
||||
const copyAtomic = (src: string, dest: string): void => {
|
||||
// `dest` appears only once it is whole ("present means complete"). As in the
|
||||
// download writer, the temp file is fsynced before the rename and the directory
|
||||
// after it, so a power cut cannot leave a correctly named but short original.
|
||||
// The temp name carries this process's ID so a later run can tell a leftover
|
||||
// from a copy still in progress (see `removeLeftoverTempFiles`).
|
||||
const copyAtomic = async (src: string, dest: string): Promise<void> => {
|
||||
if (src === dest) return;
|
||||
const tmp = join(
|
||||
dirname(dest),
|
||||
@@ -164,10 +169,45 @@ const copyAtomic = (src: string, dest: string): void => {
|
||||
.slice(2)}.tmp`,
|
||||
);
|
||||
try {
|
||||
copyFileSync(src, tmp);
|
||||
renameSync(tmp, dest);
|
||||
await copyFile(src, tmp);
|
||||
await fsyncPath(tmp);
|
||||
// `rename` replaces the destination's directory entry: an existing
|
||||
// symlink at `dest` is replaced, not followed, and the new file has
|
||||
// the temp file's permissions (copied from `src`).
|
||||
await rename(tmp, dest);
|
||||
await fsyncPath(dirname(dest));
|
||||
} finally {
|
||||
rmSync(tmp, { force: true });
|
||||
await rm(tmp, { force: true });
|
||||
}
|
||||
};
|
||||
|
||||
// A process-ID check: signal 0 delivers nothing and only reports whether the
|
||||
// process exists. EPERM means it exists but belongs to another user.
|
||||
const isRunning = (pid: number): boolean => {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch (err) {
|
||||
return (err as NodeJS.ErrnoException).code === "EPERM";
|
||||
}
|
||||
};
|
||||
|
||||
// Delete the temp files `copyAtomic` leaves behind when a backup is killed
|
||||
// before its rename. Only files whose process is no longer running are
|
||||
// removed, so a backup running at the same time keeps its own. A reused
|
||||
// process ID can only keep a leftover a while longer, never remove a live one.
|
||||
const removeLeftoverTempFiles = (dir: string): void => {
|
||||
let names: string[];
|
||||
try {
|
||||
names = readdirSync(dir);
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
for (const name of names) {
|
||||
const match = /^\.quak-backup-.*-(\d+)-[0-9a-z]*\.tmp$/.exec(name);
|
||||
if (match && !isRunning(Number(match[1]))) {
|
||||
rmSync(join(dir, name), { force: true });
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
@@ -254,6 +294,8 @@ export const runBackup = async (
|
||||
mkdirSync(originalsDir, { recursive: true });
|
||||
mkdirSync(collectionsDir, { recursive: true });
|
||||
if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true });
|
||||
removeLeftoverTempFiles(originalsDir);
|
||||
removeLeftoverTempFiles(thumbnailsDir);
|
||||
|
||||
const ledgerPath = join(downloadDirectory, "failures.json");
|
||||
const ledger = loadLedger(ledgerPath);
|
||||
@@ -321,7 +363,7 @@ export const runBackup = async (
|
||||
try {
|
||||
log(`Fetching original ${file.metadata.title} (${fileID})...`);
|
||||
const { path } = await lib.original(fileID);
|
||||
copyAtomic(path, dest);
|
||||
await copyAtomic(path, dest);
|
||||
downloaded++;
|
||||
} catch (err) {
|
||||
log(
|
||||
@@ -342,7 +384,7 @@ export const runBackup = async (
|
||||
if (isPresent(dest)) continue;
|
||||
try {
|
||||
const { path } = await lib.thumbnail(fileID);
|
||||
copyAtomic(path, dest);
|
||||
await copyAtomic(path, dest);
|
||||
} catch (err) {
|
||||
recordFailure(
|
||||
file,
|
||||
|
||||
@@ -0,0 +1,486 @@
|
||||
// The CLI's commands as plain functions.
|
||||
//
|
||||
// Each command takes its options and a `CliContext` and resolves to the exit
|
||||
// code; a thrown error is left to the caller. Nothing here calls
|
||||
// `process.exit`: `bin/quak.ts` wires these to the command line and exits with
|
||||
// the returned code once output has drained. Output must stay byte-identical
|
||||
// (see `cli-output.ts`).
|
||||
|
||||
import { input, password as passwordPrompt } from "@inquirer/prompts";
|
||||
import {
|
||||
copyFileSync,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
unlinkSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { Client, type ClientSnapshot } from "./client.js";
|
||||
import { init } from "./crypto/index.js";
|
||||
import { Library, type LibraryClient } from "./library/index.js";
|
||||
import {
|
||||
fileListRow,
|
||||
fileListLine,
|
||||
originalName,
|
||||
thumbnailName,
|
||||
} from "./cli-output.js";
|
||||
import { freshCollections, freshFiles, freshFile } from "./cli-read.js";
|
||||
import { runMetadataBackup } from "./metadata-backup.js";
|
||||
import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js";
|
||||
|
||||
export interface CliContext {
|
||||
stdout: { write(text: string): unknown };
|
||||
stderr: { write(text: string): unknown };
|
||||
// Directory holding `session.json`.
|
||||
sessionDir: string;
|
||||
// The `--cache-dir` global, or undefined to let the library pick its
|
||||
// per-user default keyed by the account id.
|
||||
cacheDir?: string;
|
||||
// Reads the session file into a client, or null when there is none. The
|
||||
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
|
||||
loadSession: (path: string) => Client | null;
|
||||
}
|
||||
|
||||
const sessionPath = (ctx: CliContext): string =>
|
||||
join(ctx.sessionDir, "session.json");
|
||||
|
||||
// Write the session readable by its owner only, in a directory only its owner
|
||||
// can enter.
|
||||
export const saveSession = (
|
||||
sessionDir: string,
|
||||
snapshot: ClientSnapshot,
|
||||
): void => {
|
||||
mkdirSync(sessionDir, { recursive: true, mode: 0o700 });
|
||||
writeFileSync(
|
||||
join(sessionDir, "session.json"),
|
||||
JSON.stringify(snapshot, null, 2),
|
||||
{ mode: 0o600 },
|
||||
);
|
||||
};
|
||||
|
||||
// The saved client, or undefined after telling the user why there is none.
|
||||
const requireSession = (ctx: CliContext): Client | undefined => {
|
||||
let client: Client | null;
|
||||
try {
|
||||
client = ctx.loadSession(sessionPath(ctx));
|
||||
} catch (err) {
|
||||
ctx.stderr.write(
|
||||
`${err instanceof Error ? err.message : err}\n` +
|
||||
`Run "quak logout" and then "quak login" to replace it.\n`,
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
if (!client) {
|
||||
ctx.stderr.write(
|
||||
`Not logged in. Run "quak login" first.\nSession file: ${sessionPath(ctx)}\n`,
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
return client;
|
||||
};
|
||||
|
||||
// A library client that omits `fetchMLData`, so the point commands below do not
|
||||
// kick the library's background ML backfill: they read metadata, or fetch one
|
||||
// file's content, and exit. `backup` and `backup-metadata` handle ML on their
|
||||
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
|
||||
// originals through the on-disk cache.
|
||||
const readLibraryClient = (client: Client): LibraryClient => ({
|
||||
whoami: () => client.whoami(),
|
||||
collectionsSince: (args) => client.collectionsSince(args),
|
||||
filesSince: (args) => client.filesSince(args),
|
||||
contentSource: () => client.contentSource(),
|
||||
});
|
||||
|
||||
// Open a library for a single point command: the aggressive background precache
|
||||
// (issue #48) is off — a one-shot `collections` or `get` must not start
|
||||
// downloading the whole account — and the refresh interval is long so no second
|
||||
// refresh fires mid-command.
|
||||
const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
|
||||
Library.open({
|
||||
client: readLibraryClient(client),
|
||||
cacheDirectory: ctx.cacheDir,
|
||||
refreshIntervalSeconds: 3600,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
|
||||
const prompt = async (message: string): Promise<string> => input({ message });
|
||||
|
||||
const promptSecret = async (message: string): Promise<string> =>
|
||||
passwordPrompt({ message, mask: true });
|
||||
|
||||
export const loginCommand = async (ctx: CliContext): Promise<number> => {
|
||||
await init();
|
||||
const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
|
||||
const password =
|
||||
process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
|
||||
|
||||
ctx.stderr.write("Authenticating...\n");
|
||||
try {
|
||||
const client = await Client.login({
|
||||
email,
|
||||
password,
|
||||
totp: async () => prompt("TOTP code: "),
|
||||
emailOTP: async () => prompt("Email verification code: "),
|
||||
});
|
||||
|
||||
saveSession(ctx.sessionDir, client.toJSON());
|
||||
const info = client.whoami();
|
||||
ctx.stderr.write(`Logged in as ${info.email} (user ${info.userID})\n`);
|
||||
ctx.stderr.write(`Session saved to ${sessionPath(ctx)}\n`);
|
||||
} catch (err) {
|
||||
ctx.stderr.write(
|
||||
`Login failed: ${err instanceof Error ? err.message : err}\n`,
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
};
|
||||
|
||||
export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const info = client.whoami();
|
||||
ctx.stdout.write(JSON.stringify(info) + "\n");
|
||||
return 0;
|
||||
};
|
||||
|
||||
export const logoutCommand = async (ctx: CliContext): Promise<number> => {
|
||||
if (existsSync(sessionPath(ctx))) {
|
||||
unlinkSync(sessionPath(ctx));
|
||||
ctx.stderr.write("Session deleted.\n");
|
||||
} else {
|
||||
ctx.stderr.write("No session found.\n");
|
||||
}
|
||||
return 0;
|
||||
};
|
||||
|
||||
export const collectionsCommand = async (
|
||||
ctx: CliContext,
|
||||
opts: { json?: boolean },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52): the pre-library CLI printed current state in
|
||||
// this order, not the albums projection's newest-first order.
|
||||
const collections = await freshCollections(lib);
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(
|
||||
JSON.stringify(
|
||||
collections.map((c) => ({
|
||||
id: c.id,
|
||||
name: c.name,
|
||||
type: c.type,
|
||||
ownerID: c.ownerID,
|
||||
isShared: c.isShared,
|
||||
updationTime: c.updationTime,
|
||||
})),
|
||||
null,
|
||||
2,
|
||||
) + "\n",
|
||||
);
|
||||
} else {
|
||||
for (const c of collections) {
|
||||
ctx.stdout.write(
|
||||
`${c.id}\t${c.type}\t${c.name}${c.isShared ? " (shared)" : ""}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
export const filesCommand = async (
|
||||
ctx: CliContext,
|
||||
opts: { collection: string; json?: boolean },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const collectionID = Number(opts.collection);
|
||||
if (!Number.isFinite(collectionID)) {
|
||||
ctx.stderr.write("Invalid collection ID\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52). Each file prints from its own decrypted
|
||||
// metadata (raw title, microsecond creationTime) via cli-output, and in
|
||||
// the pre-library CLI's enumeration order, not the projection's
|
||||
// newest-first order.
|
||||
const files = await freshFiles(lib, collectionID);
|
||||
if (!files) {
|
||||
ctx.stderr.write(`Collection ${collectionID} not found\n`);
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(
|
||||
JSON.stringify(files.map(fileListRow), null, 2) + "\n",
|
||||
);
|
||||
} else {
|
||||
for (const file of files) {
|
||||
ctx.stdout.write(fileListLine(file) + "\n");
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
export const getCommand = async (
|
||||
ctx: CliContext,
|
||||
fileIDStr: string,
|
||||
opts: { out?: string },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
ctx.stderr.write("Invalid file ID\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
ctx.stderr.write(`File ${fileID} not found\n`);
|
||||
return 1;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await photo.original();
|
||||
// Default name is the file's own title, as the pre-library CLI used
|
||||
// (not the editedName-preferring projection title) (issue #52).
|
||||
const outPath = opts.out ?? originalName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
return 0;
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
export const getThumbCommand = async (
|
||||
ctx: CliContext,
|
||||
fileIDStr: string,
|
||||
opts: { out?: string },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
ctx.stderr.write("Invalid file ID\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
ctx.stderr.write(`File ${fileID} not found\n`);
|
||||
return 1;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await photo.thumbnail();
|
||||
// Default name is thumb_<file's own title>, as the pre-library CLI
|
||||
// used (not the projection title) (issue #52).
|
||||
const outPath = opts.out ?? thumbnailName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
return 0;
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
export const backupMetadataCommand = async (
|
||||
ctx: CliContext,
|
||||
dir: string,
|
||||
opts: { exif?: boolean; all?: boolean },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
await runMetadataBackup(lib, client, dir, {
|
||||
exif: opts.exif || opts.all,
|
||||
onProgress: (msg) => ctx.stderr.write(msg + "\n"),
|
||||
});
|
||||
return 0;
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
export const backupCommand = async (
|
||||
ctx: CliContext,
|
||||
dir: string,
|
||||
opts: { json?: boolean },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
|
||||
ctx.stderr.write("Starting backup...\n");
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
downloadDirectory: dir,
|
||||
cacheDirectory: ctx.cacheDir,
|
||||
});
|
||||
try {
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: dir,
|
||||
onProgress: (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
},
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(JSON.stringify(result, null, 2) + "\n");
|
||||
} else {
|
||||
ctx.stderr.write("\n--- Backup complete ---\n");
|
||||
ctx.stderr.write(` Total files: ${result.totalFiles}\n`);
|
||||
ctx.stderr.write(` Downloaded: ${result.downloaded}\n`);
|
||||
ctx.stderr.write(` Skipped: ${result.skipped}\n`);
|
||||
ctx.stderr.write(` Failed: ${result.failed}\n`);
|
||||
if (result.errors.length > 0) {
|
||||
ctx.stderr.write("\nFailed files:\n");
|
||||
for (const e of result.errors) {
|
||||
ctx.stderr.write(
|
||||
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return result.failed > 0 ? 1 : 0;
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
export const listMissingThumbnailsCommand = async (
|
||||
ctx: CliContext,
|
||||
opts: { json?: boolean },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(JSON.stringify(missing, null, 2) + "\n");
|
||||
} else {
|
||||
if (missing.length === 0) {
|
||||
ctx.stderr.write("No missing thumbnails found.\n");
|
||||
} else {
|
||||
ctx.stderr.write(
|
||||
`\n${missing.length} file(s) with missing thumbnails:\n`,
|
||||
);
|
||||
for (const m of missing) {
|
||||
ctx.stdout.write(
|
||||
`${m.fileID}\t${m.title}\t${m.collection}\t${m.reason}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
export const fixMissingThumbnailsCommand = async (
|
||||
ctx: CliContext,
|
||||
opts: { file?: string[]; json?: boolean },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
let fileIDs: number[];
|
||||
if (opts.file && opts.file.length > 0) {
|
||||
fileIDs = opts.file.map(Number).filter(Number.isFinite);
|
||||
} else {
|
||||
ctx.stderr.write("Scanning for missing thumbnails...\n");
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
});
|
||||
fileIDs = missing.map((m) => m.fileID);
|
||||
if (fileIDs.length === 0) {
|
||||
ctx.stderr.write("No missing thumbnails found.\n");
|
||||
return 0;
|
||||
}
|
||||
ctx.stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
|
||||
}
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
fileIDs,
|
||||
(msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
},
|
||||
);
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(JSON.stringify(results, null, 2) + "\n");
|
||||
} else {
|
||||
const fixed = results.filter((r) => r.status === "fixed").length;
|
||||
const skipped = results.filter(
|
||||
(r) => r.status === "skipped",
|
||||
).length;
|
||||
const failed = results.filter((r) => r.status === "failed").length;
|
||||
ctx.stderr.write(`\n--- Done ---\n`);
|
||||
ctx.stderr.write(` Fixed: ${fixed}\n`);
|
||||
ctx.stderr.write(` Skipped: ${skipped}\n`);
|
||||
ctx.stderr.write(` Failed: ${failed}\n`);
|
||||
if (skipped > 0) {
|
||||
ctx.stderr.write("\nSkipped (unsupported format):\n");
|
||||
for (const r of results.filter((r) => r.status === "skipped")) {
|
||||
ctx.stderr.write(
|
||||
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
if (failed > 0) {
|
||||
ctx.stderr.write("\nFailed files:\n");
|
||||
for (const r of results.filter((r) => r.status === "failed")) {
|
||||
ctx.stderr.write(
|
||||
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return results.some((r) => r.status === "failed") ? 1 : 0;
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
};
|
||||
+17
-6
@@ -158,6 +158,18 @@ const streamDecrypt = async (
|
||||
return totalPlain;
|
||||
};
|
||||
|
||||
// Fsync a file or a directory, so its contents (for a directory, its entries)
|
||||
// are on stable storage. Exported for the backup tree's copy, which needs the
|
||||
// same durability as the writer below.
|
||||
export const fsyncPath = async (path: string): Promise<void> => {
|
||||
const handle = await open(path, "r");
|
||||
try {
|
||||
await handle.sync();
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
};
|
||||
|
||||
// Stage a write to `destination` atomically and durably, then rename it into
|
||||
// place. `fill` writes the contents into the open temp file handle — either the
|
||||
// whole buffer at once (`writeAtomic`) or chunk by chunk as they decrypt
|
||||
@@ -193,16 +205,15 @@ const stageAtomic = async (
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
// `rename` replaces the destination's directory entry rather than
|
||||
// writing through it: an existing symlink at `destination` is
|
||||
// replaced, not followed, and the new file has the temp file's
|
||||
// permissions, not those of the file it replaced.
|
||||
await rename(tmpPath, destination);
|
||||
// Fsync the directory so the rename itself survives a crash: renaming
|
||||
// over a synced temp file still leaves the new directory entry in the
|
||||
// page cache until the directory is synced.
|
||||
const dirHandle = await open(dir, "r");
|
||||
try {
|
||||
await dirHandle.sync();
|
||||
} finally {
|
||||
await dirHandle.close();
|
||||
}
|
||||
await fsyncPath(dir);
|
||||
} catch (err) {
|
||||
// Best-effort cleanup. A failure to remove the temporary file must
|
||||
// never replace the error that actually explains what went wrong.
|
||||
|
||||
+50
-27
@@ -15,18 +15,35 @@ export interface MetadataBackupOptions {
|
||||
onProgress?: ProgressCallback;
|
||||
}
|
||||
|
||||
// Extract the raw EXIF APP1 segment from JPEG bytes. Returns the EXIF
|
||||
// data buffer (starting after the APP1 length field, at the "Exif\0\0"
|
||||
// header) or undefined if no APP1 marker is found.
|
||||
const extractExifFromJpeg = (buf: Uint8Array): Buffer | undefined => {
|
||||
if (buf[0] !== 0xff || buf[1] !== 0xd8) return undefined;
|
||||
// Find the raw EXIF APP1 segment in JPEG bytes. Returns `exif` (the segment
|
||||
// data, starting at the "Exif\0\0" header) when there is one, nothing when the
|
||||
// bytes are not a JPEG or carry no EXIF, and `error` when the segment layout is
|
||||
// malformed. Each segment length is checked against the bytes that remain and
|
||||
// each step moves forward by at least 4 bytes, so the scan ends on any input.
|
||||
export const extractExifFromJpeg = (
|
||||
buf: Uint8Array,
|
||||
): { exif?: Buffer; error?: string } => {
|
||||
if (buf[0] !== 0xff || buf[1] !== 0xd8) return {};
|
||||
let offset = 2;
|
||||
while (offset < buf.length - 1) {
|
||||
if (buf[offset] !== 0xff) return undefined;
|
||||
while (offset < buf.length) {
|
||||
if (offset + 2 > buf.length)
|
||||
return { error: `truncated segment marker at byte ${offset}` };
|
||||
if (buf[offset] !== 0xff)
|
||||
return { error: `no segment marker at byte ${offset}` };
|
||||
const marker = buf[offset + 1]!;
|
||||
if (marker === 0xda) break; // start of scan, no more markers
|
||||
if (offset + 3 >= buf.length) break;
|
||||
if (marker === 0xda) return {}; // start of scan, no more markers
|
||||
if (offset + 4 > buf.length)
|
||||
return { error: `truncated segment length at byte ${offset}` };
|
||||
const len = (buf[offset + 2]! << 8) | buf[offset + 3]!;
|
||||
// The length counts its own two bytes, so anything under 2 is invalid.
|
||||
if (len < 2)
|
||||
return {
|
||||
error: `segment length ${len} at byte ${offset} is too small`,
|
||||
};
|
||||
if (offset + 2 + len > buf.length)
|
||||
return {
|
||||
error: `segment length ${len} at byte ${offset} runs past the end of the file`,
|
||||
};
|
||||
if (marker === 0xe1) {
|
||||
// APP1 — check for "Exif\0\0" header
|
||||
if (
|
||||
@@ -35,22 +52,26 @@ const extractExifFromJpeg = (buf: Uint8Array): Buffer | undefined => {
|
||||
buf[offset + 6] === 0x69 &&
|
||||
buf[offset + 7] === 0x66
|
||||
) {
|
||||
return Buffer.from(
|
||||
return {
|
||||
exif: Buffer.from(
|
||||
buf.buffer,
|
||||
buf.byteOffset + offset + 4,
|
||||
len - 2,
|
||||
);
|
||||
),
|
||||
};
|
||||
}
|
||||
}
|
||||
offset += 2 + len;
|
||||
}
|
||||
return undefined;
|
||||
return { error: "file ends before the image data" };
|
||||
};
|
||||
|
||||
const extractImageMetadata = (
|
||||
// Extract dimensions, EXIF and XMP from a file's bytes. When the EXIF segment
|
||||
// is malformed or cannot be parsed, the record carries the reason in
|
||||
// `exifError`.
|
||||
export const extractImageMetadata = (
|
||||
fileBytes: Uint8Array,
|
||||
): Record<string, unknown> | undefined => {
|
||||
try {
|
||||
const result: Record<string, unknown> = {};
|
||||
|
||||
// Try to get dimensions from JPEG decode
|
||||
@@ -63,15 +84,19 @@ const extractImageMetadata = (
|
||||
result.width = decoded.width;
|
||||
result.height = decoded.height;
|
||||
} catch {
|
||||
// Not a JPEG or corrupt; still try EXIF extraction
|
||||
// Not every original is a JPEG (PNG, HEIC, video), so a failed decode
|
||||
// is expected and only means no dimensions; a malformed JPEG is still
|
||||
// reported below through `exifError`.
|
||||
}
|
||||
|
||||
const exifBuf = extractExifFromJpeg(fileBytes);
|
||||
if (exifBuf) {
|
||||
const { exif, error } = extractExifFromJpeg(fileBytes);
|
||||
if (error) result.exifError = error;
|
||||
if (exif) {
|
||||
try {
|
||||
result.exif = exifReader(exifBuf);
|
||||
} catch {
|
||||
result.exifRaw = exifBuf.toString("base64");
|
||||
result.exif = exifReader(exif);
|
||||
} catch (err) {
|
||||
result.exifRaw = exif.toString("base64");
|
||||
result.exifError = err instanceof Error ? err.message : String(err);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -91,9 +116,6 @@ const extractImageMetadata = (
|
||||
}
|
||||
|
||||
return Object.keys(result).length > 0 ? result : undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
|
||||
// Read a file's original bytes through the library's content cache and extract
|
||||
@@ -103,13 +125,9 @@ const extractImageMetadata = (
|
||||
const extractExif = async (
|
||||
photo: Photo,
|
||||
): Promise<Record<string, unknown> | undefined> => {
|
||||
try {
|
||||
const { path } = await photo.original();
|
||||
const fileBytes = new Uint8Array(readFileSync(path));
|
||||
return extractImageMetadata(fileBytes);
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
|
||||
// Dump every decrypted metadata layer the account holds into a directory tree
|
||||
@@ -215,8 +233,13 @@ export const runMetadataBackup = async (
|
||||
|
||||
if (wantExif && !writtenFileIDs.has(file.id)) {
|
||||
log(`[${file.metadata.title}] Extracting EXIF...`);
|
||||
try {
|
||||
const exifData = await extractExif(photo);
|
||||
if (exifData) fileMeta.imageMetadata = exifData;
|
||||
} catch (err) {
|
||||
fileMeta.imageMetadataError =
|
||||
err instanceof Error ? err.message : String(err);
|
||||
}
|
||||
}
|
||||
writtenFileIDs.add(file.id);
|
||||
|
||||
|
||||
+25
-12
@@ -88,18 +88,21 @@ const MAX_CAUSE_DEPTH = 8;
|
||||
// errno on the error it throws — it hangs the underlying socket error off
|
||||
// `cause`, sometimes more than one level down — so a classifier that only read
|
||||
// the top-level error would see a bare `Error` and call every dropped
|
||||
// connection permanent.
|
||||
const causeCodes = (err: unknown): string[] => {
|
||||
// connection permanent. `complete` is false when the walk stopped at the
|
||||
// depth limit with more of the chain still below it.
|
||||
const causeCodes = (err: unknown): { codes: string[]; complete: boolean } => {
|
||||
const codes: string[] = [];
|
||||
let current: unknown = err;
|
||||
for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
|
||||
if (current === null || typeof current !== "object") break;
|
||||
if (current === null || typeof current !== "object") {
|
||||
return { codes, complete: true };
|
||||
}
|
||||
const { code, cause } = current as { code?: unknown; cause?: unknown };
|
||||
if (typeof code === "string") codes.push(code);
|
||||
if (cause === current) break;
|
||||
if (cause === current) return { codes, complete: true };
|
||||
current = cause;
|
||||
}
|
||||
return codes;
|
||||
return { codes, complete: current === null || typeof current !== "object" };
|
||||
};
|
||||
|
||||
const isAbort = (err: unknown): boolean => {
|
||||
@@ -145,15 +148,15 @@ export const isRetryable = (err: unknown): boolean => {
|
||||
// have succeeded; the cost of the imprecision is bounded by the attempt
|
||||
// count.
|
||||
if (err instanceof TypeError) return true;
|
||||
return causeCodes(err).some((code) => TRANSPORT_CODES.has(code));
|
||||
return causeCodes(err).codes.some((code) => TRANSPORT_CODES.has(code));
|
||||
};
|
||||
|
||||
// Could the first attempt already have taken effect on the server?
|
||||
//
|
||||
// `isRetryable` is the wrong question for a request that changes state.
|
||||
// quak's non-idempotent calls are `/users/srp/create-session`,
|
||||
// `/users/two-factor/verify` — which consumes one of a small number of 2FA
|
||||
// attempts — and `/files/thumbnail`. They are replayed only on the failures in
|
||||
// `postJSON` and `putJSON` use this for every `POST` and `PUT` listed in the
|
||||
// README under "Endpoints used"; verifying a second factor, for one, consumes
|
||||
// one of a small number of attempts. They are replayed only on the failures in
|
||||
// `CONNECT_CODES`, which establish that no TCP connection to the server ever
|
||||
// existed: there was no address to connect to, or the peer refused the
|
||||
// connection outright. A request byte cannot have been transmitted, so the
|
||||
@@ -162,9 +165,19 @@ export const isRetryable = (err: unknown): boolean => {
|
||||
// Everything else is ambiguous. A 5xx proves the server did process the
|
||||
// request. A reset or a broken pipe can arrive after it was fully sent and
|
||||
// acted on. A routing errno can be delivered on an established socket. A
|
||||
// deadline says nothing at all about the server's state.
|
||||
export const isSafeToReplay = (err: unknown): boolean =>
|
||||
isRetryable(err) && causeCodes(err).some((code) => CONNECT_CODES.has(code));
|
||||
// deadline says nothing at all about the server's state. So every errno in the
|
||||
// cause chain must be a connect errno: one other errno anywhere in the chain
|
||||
// is doubt, and doubt is not replayed. A chain longer than the walk is doubt
|
||||
// too: the links below the limit were never read.
|
||||
export const isSafeToReplay = (err: unknown): boolean => {
|
||||
const { codes, complete } = causeCodes(err);
|
||||
return (
|
||||
isRetryable(err) &&
|
||||
complete &&
|
||||
codes.length > 0 &&
|
||||
codes.every((code) => CONNECT_CODES.has(code))
|
||||
);
|
||||
};
|
||||
|
||||
export interface WithRetryOptions extends RetryOptions {
|
||||
isRetryable?: (err: unknown) => boolean;
|
||||
|
||||
+91
-3
@@ -639,6 +639,24 @@ describe("ApiClient retries", () => {
|
||||
expect(policy.baseDelayMs).toBe(7);
|
||||
expect(policy.maxDelayMs).toBe(11);
|
||||
});
|
||||
|
||||
it("does not let a caller change its settings through that policy", async () => {
|
||||
const { fetch, calls } = scriptedFetch(
|
||||
textResponse("boom", 500),
|
||||
textResponse("boom", 500),
|
||||
textResponse("boom", 500),
|
||||
);
|
||||
const client = new ApiClient({
|
||||
fetch,
|
||||
retry: { ...noWait, attempts: 2 },
|
||||
});
|
||||
|
||||
client.getRetryOptions().attempts = 3;
|
||||
|
||||
expect(client.getRetryOptions().attempts).toBe(2);
|
||||
await expect(client.getJSON("/x")).rejects.toBeInstanceOf(ApiError);
|
||||
expect(calls).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe("ApiClient timeouts", () => {
|
||||
@@ -693,6 +711,51 @@ describe("ApiClient timeouts", () => {
|
||||
expect(new Set(signals).size).toBe(3);
|
||||
}, 5000);
|
||||
|
||||
it("gives every retrying entry point a fresh deadline per attempt", async () => {
|
||||
// A refused connection is retried by every entry point, the
|
||||
// non-idempotent ones included. If the deadline were created once,
|
||||
// outside the retry, both attempts would carry the same signal.
|
||||
const entryPoints: [
|
||||
string,
|
||||
() => Response,
|
||||
(c: ApiClient) => unknown,
|
||||
][] = [
|
||||
["getJSON", () => jsonResponse({}), (c) => c.getJSON("/a")],
|
||||
["postJSON", () => jsonResponse({}), (c) => c.postJSON("/b", {})],
|
||||
["putJSON", () => jsonResponse({}), (c) => c.putJSON("/c", {})],
|
||||
[
|
||||
"putFile",
|
||||
() => new Response(null, { status: 200 }),
|
||||
(c) => c.putFile("https://s3.example/x", new Uint8Array([1])),
|
||||
],
|
||||
[
|
||||
"getFileStream",
|
||||
() => streamResponse(new Uint8Array([1])),
|
||||
(c) => c.getFileStream(1),
|
||||
],
|
||||
[
|
||||
"getThumbnailStream",
|
||||
() => streamResponse(new Uint8Array([1])),
|
||||
(c) => c.getThumbnailStream(1),
|
||||
],
|
||||
];
|
||||
for (const [name, success, call] of entryPoints) {
|
||||
const { fetch, calls } = scriptedFetch(
|
||||
errnoError("ECONNREFUSED", "connect ECONNREFUSED"),
|
||||
success(),
|
||||
);
|
||||
const client = new ApiClient({ fetch, retry: noWait });
|
||||
|
||||
await call(client);
|
||||
|
||||
expect(calls, name).toHaveLength(2);
|
||||
const [first, second] = calls.map((c) => c.init?.signal);
|
||||
expect(first, name).toBeInstanceOf(AbortSignal);
|
||||
expect(second, name).toBeInstanceOf(AbortSignal);
|
||||
expect(second, name).not.toBe(first);
|
||||
}
|
||||
});
|
||||
|
||||
it("recovers when a later attempt answers in time", async () => {
|
||||
const { fetch, calls } = scriptedFetch(HANG, jsonResponse({ ok: 1 }));
|
||||
const client = new ApiClient({
|
||||
@@ -820,9 +883,8 @@ describe("ApiClient error typing", () => {
|
||||
|
||||
describe("ApiClient non-idempotent requests", () => {
|
||||
/**
|
||||
* `postJSON` and `putJSON` carry quak's only requests that change server
|
||||
* state: `/users/srp/create-session`, `/users/two-factor/verify` — which
|
||||
* consumes one of a small number of 2FA attempts — and `/files/thumbnail`.
|
||||
* `postJSON` and `putJSON` carry quak's requests that can change server
|
||||
* state; the README lists them under "Endpoints used".
|
||||
*
|
||||
* They are retried only on a failure that establishes no TCP connection to
|
||||
* the server ever existed — DNS produced no address, or the peer refused
|
||||
@@ -922,4 +984,30 @@ describe("ApiClient non-idempotent requests", () => {
|
||||
await refusedClient.updateThumbnail(1, "key", "header");
|
||||
expect(refused.calls).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("does not follow or replay a redirect on POST or PUT", async () => {
|
||||
// The origin has already received a request it answers with a
|
||||
// redirect, so following it would let a refused connection to the
|
||||
// redirect target pass for a request that never went out.
|
||||
for (const send of [
|
||||
(c: ApiClient) => c.postJSON("/users/ott", {}),
|
||||
(c: ApiClient) => c.putJSON("/files/thumbnail", {}),
|
||||
]) {
|
||||
const { fetch, calls } = scriptedFetch(
|
||||
new Response(null, {
|
||||
status: 307,
|
||||
headers: { location: "https://elsewhere.example/" },
|
||||
}),
|
||||
jsonResponse({}),
|
||||
);
|
||||
const client = new ApiClient({ fetch, retry: noWait });
|
||||
|
||||
const err: unknown = await send(client).catch((e: unknown) => e);
|
||||
|
||||
expect(calls[0]?.init?.redirect).toBe("manual");
|
||||
expect(err).toBeInstanceOf(ApiError);
|
||||
expect((err as ApiError).status).toBe(307);
|
||||
expect(calls).toHaveLength(1);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
+92
-1
@@ -42,15 +42,44 @@ import {
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import type { ContentSource } from "../../src/library/content.js";
|
||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
|
||||
// `open` and `rename` are wrapped to record, in order, every fsync and rename,
|
||||
// so a test can pin the sequence "fsync the temp file, rename, fsync the
|
||||
// directory" that makes a copied original survive a power cut. `vi.hoisted`
|
||||
// because `vi.mock` factories run before module-level constants exist.
|
||||
const fsEvents = vi.hoisted(() => [] as string[]);
|
||||
|
||||
vi.mock("node:fs/promises", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("node:fs/promises")>();
|
||||
return {
|
||||
...actual,
|
||||
open: async (
|
||||
...args: Parameters<typeof actual.open>
|
||||
): Promise<Awaited<ReturnType<typeof actual.open>>> => {
|
||||
const handle = await actual.open(...args);
|
||||
const realSync = handle.sync.bind(handle);
|
||||
handle.sync = async (): Promise<void> => {
|
||||
fsEvents.push(`sync:${String(args[0])}`);
|
||||
await realSync();
|
||||
};
|
||||
return handle;
|
||||
},
|
||||
rename: async (from: string, to: string): Promise<void> => {
|
||||
fsEvents.push(`rename:${to}`);
|
||||
await actual.rename(from, to);
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
const USER_ID = 42;
|
||||
|
||||
// Decrypted-byte length each stub original writes, keyed by fileID.
|
||||
@@ -530,4 +559,66 @@ describe("lib.backup", () => {
|
||||
expect(readLedger(outDir).files["101"]!.attempts).toBe(1);
|
||||
lib.close();
|
||||
});
|
||||
|
||||
it("fsyncs a copied original before the rename and its directory after", async () => {
|
||||
const lib = await openLibrary(stubSource());
|
||||
const outDir = join(root, "backup");
|
||||
const originals = join(outDir, "originals");
|
||||
const dest = join(originals, "100.jpg");
|
||||
fsEvents.length = 0;
|
||||
|
||||
await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
const at = fsEvents.indexOf(`rename:${dest}`);
|
||||
expect(at).toBeGreaterThan(0);
|
||||
expect(fsEvents[at - 1]).toMatch(
|
||||
/^sync:.*\/\.quak-backup-100\.jpg-\d+-[0-9a-z]*\.tmp$/,
|
||||
);
|
||||
expect(fsEvents[at + 1]).toBe(`sync:${originals}`);
|
||||
lib.close();
|
||||
});
|
||||
|
||||
it("removes temp files left by a killed backup but not those of one still running", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const originals = join(outDir, "originals");
|
||||
mkdirSync(originals, { recursive: true });
|
||||
// A child that has already exited: its process ID is not running.
|
||||
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
|
||||
const leftover = `.quak-backup-100.jpg-${exitedPID}-abc123.tmp`;
|
||||
// This test's own process stands in for a backup running at the same
|
||||
// time.
|
||||
const inProgress = `.quak-backup-101.jpg-${process.pid}-def456.tmp`;
|
||||
writeFileSync(join(originals, leftover), "partial");
|
||||
writeFileSync(join(originals, inProgress), "partial");
|
||||
const lib = await openLibrary(stubSource());
|
||||
|
||||
await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
const names = readdirSync(originals);
|
||||
expect(names).not.toContain(leftover);
|
||||
expect(names).toContain(inProgress);
|
||||
lib.close();
|
||||
});
|
||||
|
||||
it("removes leftover temp files in thumbnails/ but not those of a backup still running", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const thumbnails = join(outDir, "thumbnails");
|
||||
mkdirSync(thumbnails, { recursive: true });
|
||||
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
|
||||
const leftover = `.quak-backup-100.jpg-${exitedPID}-abc123.tmp`;
|
||||
const inProgress = `.quak-backup-101.jpg-${process.pid}-def456.tmp`;
|
||||
writeFileSync(join(thumbnails, leftover), "partial");
|
||||
writeFileSync(join(thumbnails, inProgress), "partial");
|
||||
const lib = await openLibrary(stubSource());
|
||||
|
||||
await lib.backup({
|
||||
downloadDirectory: outDir,
|
||||
includeThumbnails: true,
|
||||
});
|
||||
|
||||
const names = readdirSync(thumbnails);
|
||||
expect(names).not.toContain(leftover);
|
||||
expect(names).toContain(inProgress);
|
||||
lib.close();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,397 @@
|
||||
/**
|
||||
* Tests for the CLI commands (`src/cli-commands.ts`, issue #12).
|
||||
*
|
||||
* Each command is called directly with a context whose output streams collect
|
||||
* text, whose session directory is a fresh temp directory, and whose session
|
||||
* loader hands back a fake client. The fake serves two albums and three files
|
||||
* from memory, writes stand-in bytes for originals and thumbnails, and makes no
|
||||
* network calls. The helpers the commands call (`cli-read`, `cli-output`,
|
||||
* backup, thumbnails) have their own tests; these check what each command
|
||||
* prints and the exit code it returns.
|
||||
*/
|
||||
|
||||
import {
|
||||
existsSync,
|
||||
mkdtempSync,
|
||||
readFileSync,
|
||||
rmSync,
|
||||
statSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { describe, it, expect, beforeAll, beforeEach, afterEach } from "vitest";
|
||||
|
||||
import {
|
||||
type CliContext,
|
||||
saveSession,
|
||||
whoamiCommand,
|
||||
logoutCommand,
|
||||
collectionsCommand,
|
||||
filesCommand,
|
||||
getCommand,
|
||||
getThumbCommand,
|
||||
backupCommand,
|
||||
listMissingThumbnailsCommand,
|
||||
} from "../../src/cli-commands.js";
|
||||
import { loadSession } from "../../src/cli-session.js";
|
||||
import type { Client, ClientSnapshot } from "../../src/client.js";
|
||||
import type { ContentSource } from "../../src/library/content.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
import { init } from "../../src/crypto/index.js";
|
||||
|
||||
const USER_ID = 42;
|
||||
|
||||
const collection = (
|
||||
id: number,
|
||||
name: string,
|
||||
isShared = false,
|
||||
): Collection => ({
|
||||
id,
|
||||
ownerID: USER_ID,
|
||||
key: new Uint8Array([id]),
|
||||
name,
|
||||
type: "album",
|
||||
updationTime: 1,
|
||||
isShared,
|
||||
});
|
||||
|
||||
const file = (id: number, collectionID: number, title: string): EnteFile => ({
|
||||
id,
|
||||
collectionID,
|
||||
ownerID: USER_ID,
|
||||
key: new Uint8Array([id & 0xff]),
|
||||
metadata: {
|
||||
title,
|
||||
fileType: "image",
|
||||
creationTime: 1000,
|
||||
modificationTime: 1000,
|
||||
},
|
||||
file: { decryptionHeader: "aGVhZGVy" },
|
||||
thumbnail: { decryptionHeader: "dGh1bWI=" },
|
||||
updationTime: 1,
|
||||
});
|
||||
|
||||
const COLLECTIONS = [collection(1, "Vacation"), collection(2, "Work", true)];
|
||||
|
||||
const FILES: Record<number, EnteFile[]> = {
|
||||
1: [file(100, 1, "beach.jpg"), file(101, 1, "sunset.jpg")],
|
||||
2: [file(200, 2, "diagram.png")],
|
||||
};
|
||||
|
||||
// An original is 7 bytes and a thumbnail 3. `failID` makes that file's
|
||||
// original fail; `emptyThumbID` makes the server report that file's
|
||||
// thumbnail as empty.
|
||||
const fakeClient = (opts: { failID?: number; emptyThumbID?: number } = {}) => {
|
||||
const source: ContentSource = {
|
||||
original: async ({ file: f, destination }) => {
|
||||
if (f.id === opts.failID) throw new Error("HTTP 500 from server");
|
||||
writeFileSync(destination, Buffer.alloc(7, f.id & 0xff));
|
||||
return { bytesWritten: 7 };
|
||||
},
|
||||
thumbnail: async ({ file: f, destination }) => {
|
||||
writeFileSync(destination, Buffer.alloc(3, f.id & 0xff));
|
||||
return { bytesWritten: 3 };
|
||||
},
|
||||
};
|
||||
const fake = {
|
||||
whoami: () => ({ email: "cli@example.com", userID: USER_ID }),
|
||||
collectionsSince: async () => ({
|
||||
collections: COLLECTIONS,
|
||||
deleted: [],
|
||||
cursor: 1,
|
||||
}),
|
||||
filesSince: async (args: { collectionID: number }) => ({
|
||||
files: FILES[args.collectionID] ?? [],
|
||||
deleted: [],
|
||||
cursor: 1,
|
||||
}),
|
||||
contentSource: () => source,
|
||||
getApiClient: () => ({
|
||||
getThumbnailStream: async (fileID: number) =>
|
||||
new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
if (fileID !== opts.emptyThumbID) {
|
||||
controller.enqueue(new Uint8Array(3));
|
||||
}
|
||||
controller.close();
|
||||
},
|
||||
}),
|
||||
}),
|
||||
};
|
||||
// The commands only call the methods above.
|
||||
return fake as unknown as Client;
|
||||
};
|
||||
|
||||
// Collects everything written to it.
|
||||
class Output {
|
||||
text = "";
|
||||
write(text: string): void {
|
||||
this.text += text;
|
||||
}
|
||||
}
|
||||
|
||||
let root: string;
|
||||
let stdout: Output;
|
||||
let stderr: Output;
|
||||
|
||||
const context = (client: Client | null = fakeClient()): CliContext => ({
|
||||
stdout,
|
||||
stderr,
|
||||
sessionDir: join(root, "session"),
|
||||
cacheDir: join(root, "cache"),
|
||||
loadSession: () => client,
|
||||
});
|
||||
|
||||
beforeAll(async () => {
|
||||
await init();
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
root = mkdtempSync(join(tmpdir(), "quak-cli-test-"));
|
||||
stdout = new Output();
|
||||
stderr = new Output();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe("session file", () => {
|
||||
const snapshot: ClientSnapshot = {
|
||||
email: "cli@example.com",
|
||||
userID: USER_ID,
|
||||
token: "token",
|
||||
masterKey: "a",
|
||||
secretKey: "b",
|
||||
publicKey: "c",
|
||||
};
|
||||
|
||||
it("is written with mode 0600 in a directory with mode 0700", () => {
|
||||
const dir = join(root, "new", "session");
|
||||
saveSession(dir, snapshot);
|
||||
expect(statSync(dir).mode & 0o777).toBe(0o700);
|
||||
const path = join(dir, "session.json");
|
||||
expect(statSync(path).mode & 0o777).toBe(0o600);
|
||||
expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot);
|
||||
});
|
||||
|
||||
it("is removed by logout", async () => {
|
||||
const ctx = context();
|
||||
saveSession(ctx.sessionDir, snapshot);
|
||||
expect(await logoutCommand(ctx)).toBe(0);
|
||||
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
|
||||
expect(stderr.text).toBe("Session deleted.\n");
|
||||
});
|
||||
|
||||
it("logout without a session says so and exits 0", async () => {
|
||||
expect(await logoutCommand(context())).toBe(0);
|
||||
expect(stderr.text).toBe("No session found.\n");
|
||||
});
|
||||
|
||||
it("a missing session exits 1 with 'Not logged in'", async () => {
|
||||
const ctx = { ...context(), loadSession };
|
||||
expect(await whoamiCommand(ctx)).toBe(1);
|
||||
expect(stderr.text).toBe(
|
||||
`Not logged in. Run "quak login" first.\n` +
|
||||
`Session file: ${join(ctx.sessionDir, "session.json")}\n`,
|
||||
);
|
||||
expect(stdout.text).toBe("");
|
||||
});
|
||||
|
||||
it("a corrupt session exits 1 and says it is corrupt", async () => {
|
||||
const ctx = { ...context(), loadSession };
|
||||
saveSession(ctx.sessionDir, snapshot);
|
||||
expect(await collectionsCommand(ctx, {})).toBe(1);
|
||||
expect(stderr.text).toContain("is corrupt");
|
||||
expect(stderr.text).toContain(
|
||||
`Run "quak logout" and then "quak login" to replace it.\n`,
|
||||
);
|
||||
expect(stdout.text).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
describe("whoami", () => {
|
||||
it("prints the account as one line of JSON", async () => {
|
||||
expect(await whoamiCommand(context())).toBe(0);
|
||||
expect(stdout.text).toBe(
|
||||
`{"email":"cli@example.com","userID":${USER_ID}}\n`,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("collections", () => {
|
||||
it("prints one tab-separated line per album", async () => {
|
||||
expect(await collectionsCommand(context(), {})).toBe(0);
|
||||
expect(stdout.text).toBe(
|
||||
"1\talbum\tVacation\n" + "2\talbum\tWork (shared)\n",
|
||||
);
|
||||
});
|
||||
|
||||
it("prints a JSON array with --json", async () => {
|
||||
expect(await collectionsCommand(context(), { json: true })).toBe(0);
|
||||
expect(JSON.parse(stdout.text)).toEqual([
|
||||
{
|
||||
id: 1,
|
||||
name: "Vacation",
|
||||
type: "album",
|
||||
ownerID: USER_ID,
|
||||
isShared: false,
|
||||
updationTime: 1,
|
||||
},
|
||||
{
|
||||
id: 2,
|
||||
name: "Work",
|
||||
type: "album",
|
||||
ownerID: USER_ID,
|
||||
isShared: true,
|
||||
updationTime: 1,
|
||||
},
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("files", () => {
|
||||
it("prints one tab-separated line per file", async () => {
|
||||
expect(await filesCommand(context(), { collection: "1" })).toBe(0);
|
||||
expect(stdout.text).toBe(
|
||||
"100\timage\tbeach.jpg\n" + "101\timage\tsunset.jpg\n",
|
||||
);
|
||||
});
|
||||
|
||||
it("prints a JSON array with --json", async () => {
|
||||
const code = await filesCommand(context(), {
|
||||
collection: "2",
|
||||
json: true,
|
||||
});
|
||||
expect(code).toBe(0);
|
||||
expect(JSON.parse(stdout.text)).toEqual([
|
||||
{
|
||||
id: 200,
|
||||
title: "diagram.png",
|
||||
fileType: "image",
|
||||
creationTime: 1000,
|
||||
collectionID: 2,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("exits 1 for an unknown collection", async () => {
|
||||
expect(await filesCommand(context(), { collection: "9" })).toBe(1);
|
||||
expect(stderr.text).toBe("Collection 9 not found\n");
|
||||
});
|
||||
|
||||
it("exits 1 for a collection ID that is not a number", async () => {
|
||||
expect(await filesCommand(context(), { collection: "abc" })).toBe(1);
|
||||
expect(stderr.text).toBe("Invalid collection ID\n");
|
||||
});
|
||||
});
|
||||
|
||||
describe("get and get-thumb", () => {
|
||||
it("get finds a file in any album without --collection", async () => {
|
||||
const out = join(root, "diagram.png");
|
||||
expect(await getCommand(context(), "200", { out })).toBe(0);
|
||||
expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
|
||||
expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
|
||||
});
|
||||
|
||||
it("get-thumb finds a file in any album without --collection", async () => {
|
||||
const out = join(root, "thumb.jpg");
|
||||
expect(await getThumbCommand(context(), "200", { out })).toBe(0);
|
||||
expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
|
||||
expect(stderr.text).toBe(`3 bytes -> ${out}\n`);
|
||||
});
|
||||
|
||||
it("get exits 1 when no album has the file", async () => {
|
||||
const out = join(root, "x");
|
||||
expect(await getCommand(context(), "999", { out })).toBe(1);
|
||||
expect(stderr.text).toBe("File 999 not found\n");
|
||||
expect(existsSync(out)).toBe(false);
|
||||
});
|
||||
|
||||
it("get-thumb exits 1 when no album has the file", async () => {
|
||||
const out = join(root, "x");
|
||||
expect(await getThumbCommand(context(), "999", { out })).toBe(1);
|
||||
expect(stderr.text).toBe("File 999 not found\n");
|
||||
expect(existsSync(out)).toBe(false);
|
||||
});
|
||||
|
||||
it("both exit 1 for a file ID that is not a number", async () => {
|
||||
expect(await getCommand(context(), "abc", {})).toBe(1);
|
||||
expect(await getThumbCommand(context(), "abc", {})).toBe(1);
|
||||
expect(stderr.text).toBe("Invalid file ID\nInvalid file ID\n");
|
||||
});
|
||||
});
|
||||
|
||||
describe("backup", () => {
|
||||
it("exits 0 and prints a summary when every file is saved", async () => {
|
||||
const dir = join(root, "backup");
|
||||
expect(await backupCommand(context(), dir, {})).toBe(0);
|
||||
expect(stderr.text).toContain(
|
||||
"\n--- Backup complete ---\n" +
|
||||
" Total files: 3\n" +
|
||||
" Downloaded: 3\n" +
|
||||
" Skipped: 0\n" +
|
||||
" Failed: 0\n",
|
||||
);
|
||||
expect(stdout.text).toBe("");
|
||||
});
|
||||
|
||||
it("exits 1 and lists the file when one download fails", async () => {
|
||||
const ctx = context(fakeClient({ failID: 101 }));
|
||||
expect(await backupCommand(ctx, join(root, "backup"), {})).toBe(1);
|
||||
expect(stderr.text).toContain(" Failed: 1\n");
|
||||
expect(stderr.text).toContain(
|
||||
"\nFailed files:\n" +
|
||||
" [Vacation] sunset.jpg (id 101): HTTP 500 from server\n",
|
||||
);
|
||||
});
|
||||
|
||||
it("prints the result as JSON with --json, still exiting 1 on a failure", async () => {
|
||||
const ctx = context(fakeClient({ failID: 101 }));
|
||||
const code = await backupCommand(ctx, join(root, "backup"), {
|
||||
json: true,
|
||||
});
|
||||
expect(code).toBe(1);
|
||||
const result = JSON.parse(stdout.text);
|
||||
expect(result).toMatchObject({
|
||||
totalFiles: 3,
|
||||
downloaded: 2,
|
||||
skipped: 0,
|
||||
failed: 1,
|
||||
});
|
||||
expect(result.errors[0].fileID).toBe(101);
|
||||
expect(stderr.text).toBe("Starting backup...\n");
|
||||
});
|
||||
});
|
||||
|
||||
describe("helper list-missing-thumbnails", () => {
|
||||
it("prints one line per file with an empty thumbnail", async () => {
|
||||
const ctx = context(fakeClient({ emptyThumbID: 200 }));
|
||||
expect(await listMissingThumbnailsCommand(ctx, {})).toBe(0);
|
||||
expect(stdout.text).toBe(
|
||||
"200\tdiagram.png\tWork\tempty thumbnail (0 bytes)\n",
|
||||
);
|
||||
expect(stderr.text).toContain("\n1 file(s) with missing thumbnails:\n");
|
||||
});
|
||||
|
||||
it("says so when nothing is missing", async () => {
|
||||
expect(await listMissingThumbnailsCommand(context(), {})).toBe(0);
|
||||
expect(stdout.text).toBe("");
|
||||
expect(stderr.text).toContain("No missing thumbnails found.\n");
|
||||
});
|
||||
|
||||
it("prints a JSON array with --json and no progress", async () => {
|
||||
const ctx = context(fakeClient({ emptyThumbID: 200 }));
|
||||
expect(await listMissingThumbnailsCommand(ctx, { json: true })).toBe(0);
|
||||
expect(JSON.parse(stdout.text)).toEqual([
|
||||
{
|
||||
fileID: 200,
|
||||
title: "diagram.png",
|
||||
collection: "Work",
|
||||
reason: "empty thumbnail (0 bytes)",
|
||||
},
|
||||
]);
|
||||
expect(stderr.text).toBe("");
|
||||
});
|
||||
});
|
||||
@@ -621,5 +621,18 @@ describe("quak backup-metadata", () => {
|
||||
expect(fileMeta.imageMetadata.format).toBe("jpeg");
|
||||
expect(fileMeta.imageMetadata.width).toBe(100);
|
||||
expect(fileMeta.imageMetadata.height).toBe(80);
|
||||
expect(fileMeta.imageMetadataError).toBeUndefined();
|
||||
|
||||
// File 200 has no original on the mock server, so extraction fails
|
||||
// and the reason is recorded instead of the field being left out.
|
||||
const workDir = collDirs.find((d) => d.includes("Work"))!;
|
||||
const failedMeta = JSON.parse(
|
||||
readFileSync(
|
||||
join(outDir, "collections", workDir, "200.json"),
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
expect(failedMeta.imageMetadata).toBeUndefined();
|
||||
expect(failedMeta.imageMetadataError).toEqual(expect.any(String));
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,122 @@
|
||||
/**
|
||||
* Tests for the JPEG EXIF scan behind `quak backup-metadata --exif`.
|
||||
*
|
||||
* The originals come from users' libraries, so a truncated or corrupt JPEG
|
||||
* must neither hang the scan nor throw out of it, and a malformed file must be
|
||||
* told apart from one that simply has no EXIF: the record carries the reason in
|
||||
* `exifError`. Each input below is a short hand-built byte array.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
extractExifFromJpeg,
|
||||
extractImageMetadata,
|
||||
} from "../../src/metadata-backup.js";
|
||||
|
||||
const SOI = [0xff, 0xd8]; // start of image
|
||||
const SOS = [0xff, 0xda, 0x00, 0x02]; // start of scan, where the scan stops
|
||||
const EXIF_HEADER = [0x45, 0x78, 0x69, 0x66, 0x00, 0x00]; // "Exif\0\0"
|
||||
|
||||
// A big-endian TIFF block with one IFD entry: Orientation (0x0112), SHORT, 6.
|
||||
const TIFF_ORIENTATION_6 = [
|
||||
0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08, 0x00, 0x01, 0x01, 0x12,
|
||||
0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00,
|
||||
0x00, 0x00,
|
||||
];
|
||||
|
||||
// An APP1 segment whose length field matches its data.
|
||||
const app1 = (data: number[]): number[] => {
|
||||
const len = data.length + 2;
|
||||
return [0xff, 0xe1, len >> 8, len & 0xff, ...data];
|
||||
};
|
||||
|
||||
const bytes = (...parts: number[][]): Uint8Array =>
|
||||
new Uint8Array(parts.flat());
|
||||
|
||||
describe("extractExifFromJpeg", () => {
|
||||
it("returns the EXIF segment of a valid JPEG", () => {
|
||||
const data = [...EXIF_HEADER, ...TIFF_ORIENTATION_6];
|
||||
const scan = extractExifFromJpeg(bytes(SOI, app1(data), SOS));
|
||||
expect(scan.error).toBeUndefined();
|
||||
expect([...scan.exif!]).toEqual(data);
|
||||
});
|
||||
|
||||
it("returns nothing for a file that is not a JPEG", () => {
|
||||
const png = bytes([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
|
||||
expect(extractExifFromJpeg(png)).toEqual({});
|
||||
});
|
||||
|
||||
it("returns nothing for a JPEG without EXIF", () => {
|
||||
const app0 = [0xff, 0xe0, 0x00, 0x04, 0x00, 0x00];
|
||||
expect(extractExifFromJpeg(bytes(SOI, app0, SOS))).toEqual({});
|
||||
});
|
||||
|
||||
it("reports a JPEG truncated inside a segment header", () => {
|
||||
const scan = extractExifFromJpeg(bytes(SOI, [0xff, 0xe1, 0x00]));
|
||||
expect(scan.exif).toBeUndefined();
|
||||
expect(scan.error).toMatch(/truncated segment length/);
|
||||
});
|
||||
|
||||
it("reports a JPEG that ends before the image data", () => {
|
||||
const app0 = [0xff, 0xe0, 0x00, 0x04, 0x00, 0x00];
|
||||
const scan = extractExifFromJpeg(bytes(SOI, app0));
|
||||
expect(scan.error).toMatch(/ends before the image data/);
|
||||
});
|
||||
|
||||
it("stops on a zero-length segment instead of looping", () => {
|
||||
// A length of 0 would otherwise step the scan by 2 bytes at a time
|
||||
// through the rest of the file, reading garbage as markers.
|
||||
const zero = [0xff, 0xe0, 0x00, 0x00];
|
||||
const scan = extractExifFromJpeg(
|
||||
bytes(SOI, zero, zero, zero, zero, SOS),
|
||||
);
|
||||
expect(scan.error).toMatch(/segment length 0 at byte 2 is too small/);
|
||||
});
|
||||
|
||||
it("stops on a segment length of 1", () => {
|
||||
const scan = extractExifFromJpeg(
|
||||
bytes(SOI, [0xff, 0xe0, 0x00, 0x01], SOS),
|
||||
);
|
||||
expect(scan.error).toMatch(/segment length 1 at byte 2 is too small/);
|
||||
});
|
||||
|
||||
it("reports a segment length that runs past the end of the file", () => {
|
||||
// APP1 claims 0x4000 bytes but only the "Exif\0\0" header follows.
|
||||
const scan = extractExifFromJpeg(
|
||||
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
|
||||
);
|
||||
expect(scan.exif).toBeUndefined();
|
||||
expect(scan.error).toMatch(/runs past the end of the file/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("extractImageMetadata", () => {
|
||||
it("parses EXIF from a valid JPEG", () => {
|
||||
const meta = extractImageMetadata(
|
||||
bytes(SOI, app1([...EXIF_HEADER, ...TIFF_ORIENTATION_6]), SOS),
|
||||
);
|
||||
expect(meta?.exifError).toBeUndefined();
|
||||
expect(meta?.exif).toMatchObject({ Image: { Orientation: 6 } });
|
||||
});
|
||||
|
||||
it("returns nothing for a file that is not a JPEG", () => {
|
||||
const text = new TextEncoder().encode("just some text, not an image");
|
||||
expect(extractImageMetadata(text)).toBeUndefined();
|
||||
});
|
||||
|
||||
it("records the reason when the JPEG is malformed", () => {
|
||||
const meta = extractImageMetadata(
|
||||
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
|
||||
);
|
||||
expect(meta?.exif).toBeUndefined();
|
||||
expect(meta?.exifError).toMatch(/runs past the end of the file/);
|
||||
});
|
||||
|
||||
it("keeps the raw bytes and the reason when EXIF cannot be parsed", () => {
|
||||
const data = [...EXIF_HEADER, 0x58, 0x58];
|
||||
const meta = extractImageMetadata(bytes(SOI, app1(data), SOS));
|
||||
expect(meta?.exif).toBeUndefined();
|
||||
expect(meta?.exifRaw).toBe(Buffer.from(data).toString("base64"));
|
||||
expect(meta?.exifError).toEqual(expect.any(String));
|
||||
});
|
||||
});
|
||||
@@ -48,6 +48,7 @@
|
||||
*/
|
||||
|
||||
import {
|
||||
chmodSync,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
readdirSync,
|
||||
@@ -989,6 +990,51 @@ describe.each(entryPoints)(
|
||||
expect(readFileSync(outPath)).toEqual(Buffer.from(existing));
|
||||
expect(readdirSync(dir)).toEqual(["rename-fails.bin"]);
|
||||
});
|
||||
|
||||
it("fails without creating anything when the destination directory does not exist", async () => {
|
||||
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const { header, ciphertext } = encryptFileBody(
|
||||
patternBytes(64, 33),
|
||||
key,
|
||||
);
|
||||
const { api, file } = fixtureFor(key, header, ciphertext);
|
||||
const dir = freshDir();
|
||||
const outPath = join(dir, "missing", "never.bin");
|
||||
|
||||
await expect(download(api, file, outPath)).rejects.toMatchObject({
|
||||
code: "ENOENT",
|
||||
});
|
||||
|
||||
// The missing directory is not created on the caller's behalf.
|
||||
expect(readdirSync(dir)).toEqual([]);
|
||||
});
|
||||
|
||||
// Root ignores directory permissions, so this cannot fail as root
|
||||
// (the Docker test image runs as root).
|
||||
it.skipIf(process.getuid?.() === 0)(
|
||||
"fails without creating anything when the destination directory is not writable",
|
||||
async () => {
|
||||
const key =
|
||||
sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const { header, ciphertext } = encryptFileBody(
|
||||
patternBytes(64, 34),
|
||||
key,
|
||||
);
|
||||
const { api, file } = fixtureFor(key, header, ciphertext);
|
||||
const dir = freshDir();
|
||||
const outPath = join(dir, "never.bin");
|
||||
chmodSync(dir, 0o500);
|
||||
try {
|
||||
await expect(
|
||||
download(api, file, outPath),
|
||||
).rejects.toMatchObject({ code: "EACCES" });
|
||||
} finally {
|
||||
chmodSync(dir, 0o700);
|
||||
}
|
||||
|
||||
expect(readdirSync(dir)).toEqual([]);
|
||||
},
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
|
||||
@@ -149,8 +149,11 @@ describe("isRetryable: transport failures", () => {
|
||||
});
|
||||
|
||||
it("retries an errno carried on the error itself", () => {
|
||||
// Every errno the classifier names, so none can be reclassified
|
||||
// unnoticed.
|
||||
for (const code of [
|
||||
"ECONNRESET",
|
||||
"ECONNABORTED",
|
||||
"ETIMEDOUT",
|
||||
"EPIPE",
|
||||
"ENOTFOUND",
|
||||
@@ -158,6 +161,8 @@ describe("isRetryable: transport failures", () => {
|
||||
"ECONNREFUSED",
|
||||
"EHOSTUNREACH",
|
||||
"ENETUNREACH",
|
||||
"ENETRESET",
|
||||
"ENETDOWN",
|
||||
]) {
|
||||
expect(isRetryable(errnoError(code))).toBe(true);
|
||||
}
|
||||
@@ -215,6 +220,27 @@ describe("isRetryable: transport failures", () => {
|
||||
looped.cause = looped;
|
||||
expect(isRetryable(looped)).toBe(false);
|
||||
});
|
||||
|
||||
it("terminates on a cause chain that loops through two errors", () => {
|
||||
const first: Error & { cause?: unknown } = new Error("first");
|
||||
const second = new Error("second", { cause: first });
|
||||
first.cause = second;
|
||||
expect(isRetryable(first)).toBe(false);
|
||||
});
|
||||
|
||||
it("reads the error and at most seven causes below it", () => {
|
||||
// The walk is bounded at eight links. An errno at the eighth link is
|
||||
// found; one at the ninth is not.
|
||||
const buried = (causes: number): Error => {
|
||||
let err = errnoError("ECONNRESET");
|
||||
for (let i = 0; i < causes; i++) {
|
||||
err = new Error(`wrapper ${i}`, { cause: err });
|
||||
}
|
||||
return err;
|
||||
};
|
||||
expect(isRetryable(buried(7))).toBe(true);
|
||||
expect(isRetryable(buried(8))).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("isRetryable: stream truncation versus corruption", () => {
|
||||
@@ -279,10 +305,9 @@ describe("isSafeToReplay", () => {
|
||||
* that is not the whole question: the other half is "could the first
|
||||
* attempt already have taken effect on the server?".
|
||||
*
|
||||
* quak's non-idempotent calls are `/users/srp/create-session`,
|
||||
* `/users/two-factor/verify` (which consumes one of a limited number of
|
||||
* 2FA attempts) and `/files/thumbnail`. A blind replay of any of them can
|
||||
* do real damage, so they retry only on the failures that establish no TCP
|
||||
* The calls this guards are the `POST` and `PUT` requests listed in the
|
||||
* README under "Endpoints used". A blind replay of some of them can do
|
||||
* real damage, so they retry only on the failures that establish no TCP
|
||||
* connection to the server ever existed — DNS produced no address, or the
|
||||
* peer refused the connection — and therefore that no request byte can
|
||||
* have been transmitted.
|
||||
@@ -333,6 +358,52 @@ describe("isSafeToReplay", () => {
|
||||
).toBe(false);
|
||||
expect(isSafeToReplay(new TypeError("fetch failed"))).toBe(false);
|
||||
});
|
||||
|
||||
it("does not replay any other errno the classifier names", () => {
|
||||
for (const code of [
|
||||
"ECONNRESET",
|
||||
"ECONNABORTED",
|
||||
"ETIMEDOUT",
|
||||
"EPIPE",
|
||||
"EHOSTUNREACH",
|
||||
"ENETUNREACH",
|
||||
"ENETRESET",
|
||||
"ENETDOWN",
|
||||
]) {
|
||||
expect(isSafeToReplay(errnoError(code))).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("does not replay a chain that also shows the request may have gone out", () => {
|
||||
// A connect errno somewhere in the chain is not enough: any other
|
||||
// errno beside it is doubt, and doubt is not replayed.
|
||||
const reset = Object.assign(
|
||||
new Error("read ECONNRESET", { cause: errnoError("ECONNREFUSED") }),
|
||||
{ code: "ECONNRESET" },
|
||||
);
|
||||
const mixed = new TypeError("fetch failed", { cause: reset });
|
||||
expect(isRetryable(mixed)).toBe(true);
|
||||
expect(isSafeToReplay(mixed)).toBe(false);
|
||||
});
|
||||
|
||||
it("does not replay a chain longer than the walk reads", () => {
|
||||
// Eight connect errnos, then a reset at the ninth link, below the
|
||||
// limit. The walk never sees the reset, so it cannot rule it out.
|
||||
const refusedChain = (below: Error | undefined): Error => {
|
||||
let err = below;
|
||||
for (let i = 0; i < 8; i++) {
|
||||
err = Object.assign(new Error(`refused ${i}`, { cause: err }), {
|
||||
code: "ECONNREFUSED",
|
||||
});
|
||||
}
|
||||
return err as Error;
|
||||
};
|
||||
expect(isSafeToReplay(refusedChain(errnoError("ECONNRESET")))).toBe(
|
||||
false,
|
||||
);
|
||||
// The same eight links with nothing below them are replayable.
|
||||
expect(isSafeToReplay(refusedChain(undefined))).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user