8 Commits
Author SHA1 Message Date
sneak 99a536286d Move CLI commands into testable functions and test them (closes #12)
check / check (push) Successful in 43s
The command bodies in bin/quak.ts become functions in
src/cli-commands.ts that take their options and a context (output
streams, session directory, cache directory, session loader) and return
an exit code. bin/quak.ts wires them to commander and exits with that
code once stdout and stderr have drained; nothing below it calls
process.exit. test/cli/commands.test.ts drives the commands with a fake
client and temp directories. Output is unchanged.

Model: opus-5-5
2026-09-23 00:45:06 +00:00
clawbot ed535be1da Harden the backup tree's atomic copy (closes #22)
check / check (push) Successful in 27s
The backup copy now fsyncs its temp file before the rename and the
directory after it, using the download writer's new fsyncPath helper.
Each backup run deletes .quak-backup-*.tmp files whose process is no
longer running, leaving those of a concurrent backup alone. The rename
sites and the README backup layout state that a symlink at the
destination is replaced and the new file takes the temp file's
permissions, and the README names the temp files. Adds tests for a
missing and an unwritable destination directory for downloadFile and
downloadThumbnail.

Model: opus-5-5
2026-09-23 02:44:46 +02:00
clawbot d545dcd8b1 Harden the retry classifier and pin per-attempt deadlines (closes #80)
check / check (push) Successful in 42s
A POST or PUT is replayed only when every errno in the cause chain is a
connect errno and the walk reached the end of the chain, and
postJSON/putJSON no longer follow redirects, so a redirect is an
ApiError that is not retried. getRetryOptions() returns a copy. New
tests pin every errno the classifier names, the cause-chain depth
limit, a chain deeper than the limit, a two-error cycle, and a fresh
deadline per attempt for every retrying entry point. The README
endpoint list is now the one place naming the requests the replay rule
covers; code comments point to it.

Model: opus-5-5
2026-09-23 02:38:01 +02:00
clawbot 52f58f5d2b Harden the JPEG EXIF scan against malformed input (closes #11)
check / check (push) Successful in 29s
The segment scan behind `backup-metadata --exif` now checks every
segment length against the bytes that remain and stops on lengths
under 2, so a truncated or corrupt original can neither throw nor loop.
A malformed or unparseable EXIF segment is recorded as
`imageMetadata.exifError`, and a failure to read the original as
`imageMetadataError` in the per-file JSON, instead of the field being
silently left out. Tests use short hand-built byte arrays.

Model: opus-5-5
2026-09-23 02:28:15 +02:00
clawbot b44c4ba6d7 Keep make test from collecting tests in nested checkouts (closes #25)
check / check (push) Successful in 33s
vitest does not read .gitignore when finding tests, so a checkout nested
under .claude/ had its whole test/ tree run as part of this suite.
vitest.config.ts adds .claude/** to vitest's default excludes. The new
packaging test plants a nested checkout in a temp directory and fails if
vitest, run with this config, would collect it.

Model: opus-5-5
2026-09-23 02:18:31 +02:00
clawbot d50b296d3a Drop the deprecated @types/libsodium-wrappers-sumo stub (closes #27)
check / check (push) Successful in 46s
The package is an empty stub with no declarations; libsodium-wrappers-sumo
ships its own types. Removed with yarn remove, which regenerated yarn.lock.

Model: opus-5-5
2026-09-23 02:15:08 +02:00
clawbot b7d6ab99f4 Validate session snapshots and wipe keys on logout (closes #10)
check / check (push) Successful in 41s
Client.fromJSON checks every snapshot field and each key's decoded length
and throws an error naming the bad field. toJSON reads the token through a
new ApiClient.getAuthToken and throws when there is none. logout zeroes the
key buffers in place; collectionsSince re-checks for logout after its
request so it never decrypts with zeroed keys. The CLI now reports a
corrupt session file separately from a missing one.

Model: opus-5-5
2026-09-23 02:08:02 +02:00
clawbot 3871d6228e Sanitize file names taken from server metadata (closes #9)
check / check (push) Successful in 37s
A file title or album name decrypted from server data could name a path
outside the chosen directory (`../../.ssh/authorized_keys`). One module,
src/filename.ts, now makes such names safe for `quak get`/`get-thumb`
without `--out`, downloadFile/downloadThumbnail without outPath, and the
backup and metadata backup trees. Originals-cache extensions are limited to
letters and digits. A user-supplied path is still used as is. decryptFile
reads a missing or non-string title as "" and rejects metadata that is not
a JSON object.

Model: opus-5-5
2026-09-23 02:04:31 +02:00
31 changed files with 2384 additions and 579 deletions
+35 -16
View File
@@ -323,6 +323,7 @@ Endpoints used:
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a - `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a
collection; paginate while `hasMore` is true. collection; paginate while `hasMore` is true.
- `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes. - `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes.
- `POST /files/data/fetch`: fetch encrypted ML data for a batch of files.
- `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair). - `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair).
- `PUT /files/thumbnail`: register an uploaded thumbnail's object key. - `PUT /files/thumbnail`: register an uploaded thumbnail's object key.
@@ -377,20 +378,23 @@ headers — `getFileStream` returns as soon as headers arrive, so a deadline tha
only guarded the initial request would leave the same hang one layer down. only guarded the initial request would leave the same hang one layer down.
**Non-idempotent requests are not blindly replayed.** `postJSON` and `putJSON` **Non-idempotent requests are not blindly replayed.** `postJSON` and `putJSON`
reach `/users/srp/create-session`, `/users/two-factor/verify` — which consumes send every `POST` and `PUT` in the endpoint list above; some of them change
one of a small number of second-factor attempts — and `/files/thumbnail`. They server state, and `/users/two-factor/verify` consumes one of a small number of
are retried only on the three failures that establish no TCP connection to the second-factor attempts. They are retried only when every errno in the error's
server ever existed, so no request byte can have been transmitted: `ENOTFOUND` `cause` chain is one of the three that establish no TCP connection to the server
and `EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the ever existed, so no request byte can have been transmitted: `ENOTFOUND` and
peer refused the connection). A 5xx, a mid-flight reset and a deadline are all `EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer
left to the caller, because each of them can happen after the server has already refused the connection). A 5xx, a mid-flight reset and a deadline are all left
acted. The routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are to the caller, because each of them can happen after the server has already
excluded for the same reason, despite looking like connect-time failures: on acted. These two do not follow redirects either: a redirect means the server
Linux an ICMP unreachable arriving mid-flight, or a local interface going down already received the request, so it is reported as an error and not retried. The
after the request was written, delivers them on an already-established socket. routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the
They stay retryable for the idempotent calls. `putFile` is exempt: a presigned same reason, despite looking like connect-time failures: on Linux an ICMP
PUT stores one whole object at one key in one request, so replaying it has no unreachable arriving mid-flight, or a local interface going down after the
partial state to damage. request was written, delivers them on an already-established socket. They stay
retryable for the idempotent calls. `putFile` is exempt: a presigned PUT stores
one whole object at one key in one request, so replaying it has no partial state
to damage.
A download is retried as a whole — request, stream consumption, and decryption — A download is retried as a whole — request, stream consumption, and decryption —
because a socket reset after the response headers have arrived surfaces in the because a socket reset after the response headers have arrived surfaces in the
@@ -422,13 +426,18 @@ decides how to persist sessions.
`client.toJSON()` returns a `ClientSnapshot` (a plain serializable object with `client.toJSON()` returns a `ClientSnapshot` (a plain serializable object with
base64-encoded keys) that the consumer can write to disk, a database, or base64-encoded keys) that the consumer can write to disk, a database, or
whatever else fits their use case. `Client.fromJSON(snapshot)` restores a whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
working client from that snapshot without re-authenticating. working client from that snapshot without re-authenticating; it checks every
field and each key's length first, and throws an error naming the bad field.
`client.logout()` clears the token and zeroes the key buffers in place; every
later call on that client throws.
The CLI stores the snapshot at the platform-appropriate data directory via The CLI stores the snapshot at the platform-appropriate data directory via
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS, `env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
`$XDG_DATA_HOME/quak/session.json` on Linux. The file is written with mode `$XDG_DATA_HOME/quak/session.json` on Linux. The file is written with mode
`0600`. The key material is stored in cleartext in the JSON; treat this file as `0600`. The key material is stored in cleartext in the JSON; treat this file as
you would treat the password itself. you would treat the password itself. A missing file is reported as "not logged
in"; a file that exists but is corrupt is reported as such, naming the bad
field. Both exit with status 1.
### CLI surface ### CLI surface
@@ -485,6 +494,16 @@ appears in. On subsequent runs, existing originals are skipped. If a download
fails, the error is logged and the backup continues with the next file. The exit fails, the error is logged and the backup continues with the next file. The exit
code is non-zero if any files failed. code is non-zero if any files failed.
Each original is copied to a temporary file named
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp` in the same directory, synced
to disk, and renamed into place, so an original is either complete or absent,
even after a power cut. A run that is killed can leave one of these temporary
files behind; the next backup deletes those whose process is no longer running.
Downloads and the content cache use the same scheme with `.quak-<random>.tmp`
names. The rename replaces whatever was at the destination rather than writing
through it: a symlink there is replaced, not followed, and the new file has the
temporary file's permissions, not those of the file it replaced.
## TODO ## TODO
- [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network - [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
+54
View File
@@ -18,6 +18,60 @@ Tag v1.0.0.
# Completed Steps # Completed Steps
- 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies
moved from `bin/quak.ts` into `src/cli-commands.ts` as functions that take
their options and a context (output streams, session directory, cache
directory, session loader) and return an exit code; `bin/quak.ts` only wires
them to commander and exits with the code once stdout and stderr have drained,
so nothing below it calls `process.exit`. `test/cli/commands.test.ts` drives
them with a fake client: session file modes, logout, the missing and corrupt
session paths, and the output and exit code of `whoami`, `collections`,
`files`, `get`, `get-thumb`, `backup` and `helper list-missing-thumbnails`.
- 2026-09-23: Hardened the backup tree's atomic copy (issue 22). `copyAtomic`
fsyncs its temp file before the rename and the directory after it, through the
download writer's `fsyncPath`; each backup run deletes `.quak-backup-*.tmp`
files whose process is no longer running. The README backup layout names the
temp files and states that the rename replaces a symlink and takes the temp
file's permissions. Added tests for a missing and an unwritable destination
directory for `downloadFile` and `downloadThumbnail`.
- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue
11). Every segment length is checked against the remaining bytes and lengths
under 2 stop the scan, so a truncated or corrupt original can neither throw
nor loop. A malformed or unparseable EXIF segment is recorded as
`imageMetadata.exifError`, and a failure to read the original as
`imageMetadataError` in the per-file JSON, instead of the field being left
out.
- 2026-09-22: Hardened the retry classifier (issue 80). A `POST` or `PUT` is
replayed only when every errno in the cause chain is a connect errno, and it
no longer follows redirects. `getRetryOptions()` returns a copy. Tests pin
every errno the classifier names, the cause-chain depth limit, cycle
termination, and a fresh deadline per attempt for every retrying entry point.
The README's endpoint list is the one place that names the requests the replay
rule covers.
- 2026-09-22: Stopped `make test` collecting tests from checkouts nested under
`.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a
nested checkout ran the whole suite again; `vitest.config.ts` now adds
`.claude/**` to vitest's default excludes, and
`test/packaging/nested-checkout.test.ts` plants a nested checkout in a temp
directory and fails if vitest would collect it.
- 2026-09-22: Dropped the deprecated `@types/libsodium-wrappers-sumo` stub from
`devDependencies` (issue 27). It shipped no declarations; the types come from
`libsodium-wrappers-sumo` itself. `yarn.lock` regenerated by `yarn remove`.
- 2026-09-22: Hardened the client session lifecycle (issue 10).
`Client.fromJSON` checks every snapshot field and each key's decoded length
and names the bad field; `toJSON` reads the token through
`ApiClient.getAuthToken` and throws when there is none; `logout` zeroes the
key buffers, and `collectionsSince` re-checks for logout after its request so
it never decrypts with zeroed keys. The CLI reports a corrupt session file
separately from a missing one (`src/cli-session.ts`).
- 2026-09-22: Sanitized file names taken from server metadata (issue 9). A new
`src/filename.ts` holds the one sanitizer, used by `quak get`/`get-thumb`
without `--out`, `downloadFile`/`downloadThumbnail` without `outPath`, and the
backup and metadata backup trees; it removes separators, control characters,
leading dots and Windows device names, and falls back to a name built from the
ID for an empty title. Originals-cache extensions are letters and digits only,
else `.bin`. A user-supplied path is used as is. `decryptFile` reads a missing
or non-string title as "" and rejects metadata that is not a JSON object.
- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage - 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
snippets) to match the shipped cache/API library on `next` (issue 53, issue snippets) to match the shipped cache/API library on `next` (issue 53, issue
13). Documented `Library.open` and its options, the default-read vs `fresh()` 13). Documented `Library.open` and its options, the default-read vs `fresh()`
+49 -380
View File
@@ -1,67 +1,26 @@
#!/usr/bin/env node #!/usr/bin/env node
import { input, password as passwordPrompt } from "@inquirer/prompts";
import { stdout, stderr } from "node:process"; import { stdout, stderr } from "node:process";
import {
copyFileSync,
existsSync,
mkdirSync,
readFileSync,
writeFileSync,
} from "node:fs";
import { join } from "node:path";
import { Command } from "commander"; import { Command } from "commander";
import envPaths from "env-paths"; import envPaths from "env-paths";
import { Client, type ClientSnapshot } from "../src/client.js";
import { init } from "../src/crypto/index.js"; import { init } from "../src/crypto/index.js";
import { Library, type LibraryClient } from "../src/library/index.js";
import { import {
fileListRow, type CliContext,
fileListLine, loginCommand,
originalName, whoamiCommand,
thumbnailName, logoutCommand,
} from "../src/cli-output.js"; collectionsCommand,
import { freshCollections, freshFiles, freshFile } from "../src/cli-read.js"; filesCommand,
import { runMetadataBackup } from "../src/metadata-backup.js"; getCommand,
import { getThumbCommand,
listMissingThumbnails, backupMetadataCommand,
fixMissingThumbnails, backupCommand,
} from "../src/thumbnails.js"; listMissingThumbnailsCommand,
fixMissingThumbnailsCommand,
} from "../src/cli-commands.js";
import { loadSession } from "../src/cli-session.js";
const paths = envPaths("quak", { suffix: "" }); const paths = envPaths("quak", { suffix: "" });
const sessionPath = join(paths.data, "session.json");
const loadSession = (): ClientSnapshot | null => {
if (!existsSync(sessionPath)) return null;
try {
return JSON.parse(readFileSync(sessionPath, "utf-8")) as ClientSnapshot;
} catch {
return null;
}
};
const saveSession = (snapshot: ClientSnapshot): void => {
mkdirSync(paths.data, { recursive: true, mode: 0o700 });
writeFileSync(sessionPath, JSON.stringify(snapshot, null, 2), {
mode: 0o600,
});
};
const requireSession = (): Client => {
const snapshot = loadSession();
if (!snapshot) {
stderr.write(
`Not logged in. Run "quak login" first.\nSession file: ${sessionPath}\n`,
);
process.exit(1);
}
return Client.fromJSON(snapshot);
};
const prompt = async (message: string): Promise<string> => input({ message });
const promptSecret = async (message: string): Promise<string> =>
passwordPrompt({ message, mask: true });
const program = new Command(); const program = new Command();
@@ -75,51 +34,28 @@ program
"(default: the per-user cache directory)", "(default: the per-user cache directory)",
); );
// The `--cache-dir` global, or undefined to let the library pick its per-user const context = (): CliContext => ({
// default keyed by the account id. stdout,
const cacheDirOption = (): string | undefined => stderr,
program.opts<{ cacheDir?: string }>().cacheDir; sessionDir: paths.data,
cacheDir: program.opts<{ cacheDir?: string }>().cacheDir,
// A library client that omits `fetchMLData`, so the point commands below do not loadSession,
// kick the library's background ML backfill: they read metadata, or fetch one
// file's content, and exit. `backup` and `backup-metadata` handle ML on their
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
// originals through the on-disk cache.
const readLibraryClient = (client: Client): LibraryClient => ({
whoami: () => client.whoami(),
collectionsSince: (args) => client.collectionsSince(args),
filesSince: (args) => client.filesSince(args),
contentSource: () => client.contentSource(),
}); });
// Open a library for a single point command: the aggressive background precache // Run a command and exit with its code once stdout/stderr have drained.
// (issue #48) is off — a one-shot `collections` or `get` must not start // Exiting before the drain can truncate piped output, and the library can keep
// downloading the whole account — and the refresh interval is long so no second // the event loop alive after a command returns, so a plain return could hang.
// refresh fires mid-command. const run = async (command: Promise<number>): Promise<void> => {
const openReadLibrary = (client: Client): Promise<Library> => process.exitCode = await command;
Library.open({
client: readLibraryClient(client),
cacheDirectory: cacheDirOption(),
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
});
// Close the library and exit once stdout/stderr have drained. `process.exit`
// alone can truncate buffered piped output, and the library keeps the event
// loop alive with a background refresh, so a plain return could hang; this does
// neither.
const finish = (lib: Library | undefined, code: number): void => {
lib?.close();
const pending = [stdout, stderr].filter((s) => s.writableLength > 0); const pending = [stdout, stderr].filter((s) => s.writableLength > 0);
if (pending.length === 0) { if (pending.length === 0) {
process.exit(code); process.exit();
return; return;
} }
let remaining = pending.length; let remaining = pending.length;
for (const s of pending) { for (const s of pending) {
s.once("drain", () => { s.once("drain", () => {
if (--remaining === 0) process.exit(code); if (--remaining === 0) process.exit();
}); });
} }
}; };
@@ -127,92 +63,25 @@ const finish = (lib: Library | undefined, code: number): void => {
program program
.command("login") .command("login")
.description("Log in to an Ente account and save the session") .description("Log in to an Ente account and save the session")
.action(async () => { .action(() => run(loginCommand(context())));
await init();
const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
const password =
process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
stderr.write("Authenticating...\n");
try {
const client = await Client.login({
email,
password,
totp: async () => prompt("TOTP code: "),
emailOTP: async () => prompt("Email verification code: "),
});
saveSession(client.toJSON());
const info = client.whoami();
stderr.write(`Logged in as ${info.email} (user ${info.userID})\n`);
stderr.write(`Session saved to ${sessionPath}\n`);
} catch (err) {
stderr.write(
`Login failed: ${err instanceof Error ? err.message : err}\n`,
);
process.exit(1);
}
});
program program
.command("whoami") .command("whoami")
.description("Print the logged-in account") .description("Print the logged-in account")
.action(() => { .action(() => run(whoamiCommand(context())));
const client = requireSession();
const info = client.whoami();
stdout.write(JSON.stringify(info) + "\n");
});
program program
.command("logout") .command("logout")
.description("Delete the saved session") .description("Delete the saved session")
.action(async () => { .action(() => run(logoutCommand(context())));
if (existsSync(sessionPath)) {
const { unlinkSync } = await import("node:fs");
unlinkSync(sessionPath);
stderr.write("Session deleted.\n");
} else {
stderr.write("No session found.\n");
}
});
program program
.command("collections") .command("collections")
.description("List all collections (albums)") .description("List all collections (albums)")
.option("--json", "Output as JSON array") .option("--json", "Output as JSON array")
.action(async (opts: { json?: boolean }) => { .action((opts: { json?: boolean }) =>
await init(); run(collectionsCommand(context(), opts)),
const client = requireSession();
const lib = await openReadLibrary(client);
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52): the pre-library CLI printed current state in
// this order, not the albums projection's newest-first order.
const collections = await freshCollections(lib);
if (opts.json) {
stdout.write(
JSON.stringify(
collections.map((c) => ({
id: c.id,
name: c.name,
type: c.type,
ownerID: c.ownerID,
isShared: c.isShared,
updationTime: c.updationTime,
})),
null,
2,
) + "\n",
); );
} else {
for (const c of collections) {
stdout.write(
`${c.id}\t${c.type}\t${c.name}${c.isShared ? " (shared)" : ""}\n`,
);
}
}
finish(lib, 0);
});
program program
.command("files") .command("files")
@@ -222,39 +91,9 @@ program
"Collection ID (from `quak collections`)", "Collection ID (from `quak collections`)",
) )
.option("--json", "Output as JSON array") .option("--json", "Output as JSON array")
.action(async (opts: { collection: string; json?: boolean }) => { .action((opts: { collection: string; json?: boolean }) =>
await init(); run(filesCommand(context(), opts)),
const client = requireSession();
const collectionID = Number(opts.collection);
if (!Number.isFinite(collectionID)) {
stderr.write("Invalid collection ID\n");
process.exit(1);
}
const lib = await openReadLibrary(client);
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52). Each file prints from its own decrypted
// metadata (raw title, microsecond creationTime) via cli-output, and in
// the pre-library CLI's enumeration order, not the projection's
// newest-first order.
const files = await freshFiles(lib, collectionID);
if (!files) {
stderr.write(`Collection ${collectionID} not found\n`);
finish(lib, 1);
return;
}
if (opts.json) {
stdout.write(
JSON.stringify(files.map(fileListRow), null, 2) + "\n",
); );
} else {
for (const file of files) {
stdout.write(fileListLine(file) + "\n");
}
}
finish(lib, 0);
});
program program
.command("get") .command("get")
@@ -262,34 +101,9 @@ program
.argument("<fileID>", "File ID (from `quak files`)") .argument("<fileID>", "File ID (from `quak files`)")
.option("--out <path>", "Output file path") .option("--out <path>", "Output file path")
.option("--collection <id>", "Accepted for compatibility; ignored") .option("--collection <id>", "Accepted for compatibility; ignored")
.action(async (fileIDStr: string, opts: { out?: string }) => { .action((fileID: string, opts: { out?: string }) =>
await init(); run(getCommand(context(), fileID, opts)),
const client = requireSession(); );
const fileID = Number(fileIDStr);
if (!Number.isFinite(fileID)) {
stderr.write("Invalid file ID\n");
process.exit(1);
}
const lib = await openReadLibrary(client);
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
stderr.write(`File ${fileID} not found\n`);
finish(lib, 1);
return;
}
const { photo, file } = resolved;
const result = await photo.original();
// Default name is the file's own title, as the pre-library CLI used
// (not the editedName-preferring projection title) (issue #52).
const outPath = opts.out ?? originalName(file);
copyFileSync(result.path, outPath);
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
finish(lib, 0);
});
program program
.command("get-thumb") .command("get-thumb")
@@ -297,34 +111,9 @@ program
.argument("<fileID>", "File ID (from `quak files`)") .argument("<fileID>", "File ID (from `quak files`)")
.option("--out <path>", "Output file path") .option("--out <path>", "Output file path")
.option("--collection <id>", "Accepted for compatibility; ignored") .option("--collection <id>", "Accepted for compatibility; ignored")
.action(async (fileIDStr: string, opts: { out?: string }) => { .action((fileID: string, opts: { out?: string }) =>
await init(); run(getThumbCommand(context(), fileID, opts)),
const client = requireSession(); );
const fileID = Number(fileIDStr);
if (!Number.isFinite(fileID)) {
stderr.write("Invalid file ID\n");
process.exit(1);
}
const lib = await openReadLibrary(client);
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
stderr.write(`File ${fileID} not found\n`);
finish(lib, 1);
return;
}
const { photo, file } = resolved;
const result = await photo.thumbnail();
// Default name is thumb_<file's own title>, as the pre-library CLI
// used (not the projection title) (issue #52).
const outPath = opts.out ?? thumbnailName(file);
copyFileSync(result.path, outPath);
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
finish(lib, 0);
});
program program
.command("backup-metadata") .command("backup-metadata")
@@ -337,16 +126,9 @@ program
"Download each file and extract full EXIF/IPTC/XMP metadata (slow)", "Download each file and extract full EXIF/IPTC/XMP metadata (slow)",
) )
.option("--all", "Alias for --exif") .option("--all", "Alias for --exif")
.action(async (dir: string, opts: { exif?: boolean; all?: boolean }) => { .action((dir: string, opts: { exif?: boolean; all?: boolean }) =>
await init(); run(backupMetadataCommand(context(), dir, opts)),
const client = requireSession(); );
const lib = await openReadLibrary(client);
await runMetadataBackup(lib, client, dir, {
exif: opts.exif || opts.all,
onProgress: (msg) => stderr.write(msg + "\n"),
});
finish(lib, 0);
});
program program
.command("backup") .command("backup")
@@ -355,43 +137,9 @@ program
) )
.argument("<dir>", "Output directory") .argument("<dir>", "Output directory")
.option("--json", "Print result as JSON instead of human-readable summary") .option("--json", "Print result as JSON instead of human-readable summary")
.action(async (dir: string, opts: { json?: boolean }) => { .action((dir: string, opts: { json?: boolean }) =>
await init(); run(backupCommand(context(), dir, opts)),
const client = requireSession();
stderr.write("Starting backup...\n");
const lib = await Library.open({
client,
downloadDirectory: dir,
cacheDirectory: cacheDirOption(),
});
const result = await lib.backup({
downloadDirectory: dir,
onProgress: (msg) => {
if (!opts.json) stderr.write(msg + "\n");
},
});
if (opts.json) {
stdout.write(JSON.stringify(result, null, 2) + "\n");
} else {
stderr.write("\n--- Backup complete ---\n");
stderr.write(` Total files: ${result.totalFiles}\n`);
stderr.write(` Downloaded: ${result.downloaded}\n`);
stderr.write(` Skipped: ${result.skipped}\n`);
stderr.write(` Failed: ${result.failed}\n`);
if (result.errors.length > 0) {
stderr.write("\nFailed files:\n");
for (const e of result.errors) {
stderr.write(
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
); );
}
}
}
finish(lib, result.failed > 0 ? 1 : 0);
});
const helper = program const helper = program
.command("helper") .command("helper")
@@ -401,32 +149,9 @@ helper
.command("list-missing-thumbnails") .command("list-missing-thumbnails")
.description("List files whose thumbnails are missing or empty") .description("List files whose thumbnails are missing or empty")
.option("--json", "Output as JSON array") .option("--json", "Output as JSON array")
.action(async (opts: { json?: boolean }) => { .action((opts: { json?: boolean }) =>
await init(); run(listMissingThumbnailsCommand(context(), opts)),
const client = requireSession();
const lib = await openReadLibrary(client);
const missing = await listMissingThumbnails(lib, client, (msg) => {
if (!opts.json) stderr.write(msg + "\n");
});
if (opts.json) {
stdout.write(JSON.stringify(missing, null, 2) + "\n");
} else {
if (missing.length === 0) {
stderr.write("No missing thumbnails found.\n");
} else {
stderr.write(
`\n${missing.length} file(s) with missing thumbnails:\n`,
); );
for (const m of missing) {
stdout.write(
`${m.fileID}\t${m.title}\t${m.collection}\t${m.reason}\n`,
);
}
}
}
finish(lib, 0);
});
helper helper
.command("fix-missing-thumbnails") .command("fix-missing-thumbnails")
@@ -438,65 +163,9 @@ helper
"Specific file IDs to fix (default: fix all missing)", "Specific file IDs to fix (default: fix all missing)",
) )
.option("--json", "Output as JSON") .option("--json", "Output as JSON")
.action(async (opts: { file?: string[]; json?: boolean }) => { .action((opts: { file?: string[]; json?: boolean }) =>
await init(); run(fixMissingThumbnailsCommand(context(), opts)),
const client = requireSession();
const lib = await openReadLibrary(client);
let fileIDs: number[];
if (opts.file && opts.file.length > 0) {
fileIDs = opts.file.map(Number).filter(Number.isFinite);
} else {
stderr.write("Scanning for missing thumbnails...\n");
const missing = await listMissingThumbnails(lib, client, (msg) => {
if (!opts.json) stderr.write(msg + "\n");
});
fileIDs = missing.map((m) => m.fileID);
if (fileIDs.length === 0) {
stderr.write("No missing thumbnails found.\n");
finish(lib, 0);
return;
}
stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
}
const results = await fixMissingThumbnails(
lib,
client,
fileIDs,
(msg) => {
if (!opts.json) stderr.write(msg + "\n");
},
); );
if (opts.json) {
stdout.write(JSON.stringify(results, null, 2) + "\n");
} else {
const fixed = results.filter((r) => r.status === "fixed").length;
const skipped = results.filter(
(r) => r.status === "skipped",
).length;
const failed = results.filter((r) => r.status === "failed").length;
stderr.write(`\n--- Done ---\n`);
stderr.write(` Fixed: ${fixed}\n`);
stderr.write(` Skipped: ${skipped}\n`);
stderr.write(` Failed: ${failed}\n`);
if (skipped > 0) {
stderr.write("\nSkipped (unsupported format):\n");
for (const r of results.filter((r) => r.status === "skipped")) {
stderr.write(` ${r.fileID}\t${r.title}\t${r.reason}\n`);
}
}
if (failed > 0) {
stderr.write("\nFailed files:\n");
for (const r of results.filter((r) => r.status === "failed")) {
stderr.write(` ${r.fileID}\t${r.title}\t${r.reason}\n`);
}
}
}
finish(lib, results.some((r) => r.status === "failed") ? 1 : 0);
});
await init(); await init();
program.parse(); program.parse();
-1
View File
@@ -29,7 +29,6 @@
}, },
"devDependencies": { "devDependencies": {
"@eslint/js": "9.38.0", "@eslint/js": "9.38.0",
"@types/libsodium-wrappers-sumo": "0.8.2",
"@types/node": "22.18.13", "@types/node": "22.18.13",
"eslint": "9.38.0", "eslint": "9.38.0",
"prettier": "3.8.1", "prettier": "3.8.1",
+18 -13
View File
@@ -139,11 +139,16 @@ export class ApiClient {
this.token = undefined; this.token = undefined;
} }
getAuthToken(): string | undefined {
return this.token;
}
// The policy this client was configured with, so that a caller wrapping a // The policy this client was configured with, so that a caller wrapping a
// whole operation in its own `withRetry` — the download layer — runs under // whole operation in its own `withRetry` — the download layer — runs under
// the same settings rather than under the library defaults. // the same settings rather than under the library defaults.
// A copy, so the caller cannot change this client's settings through it.
getRetryOptions(): ResolvedRetryOptions { getRetryOptions(): ResolvedRetryOptions {
return this.retry; return { ...this.retry };
} }
private headers(extra?: Record<string, string>): Record<string, string> { private headers(extra?: Record<string, string>): Record<string, string> {
@@ -224,15 +229,15 @@ export class ApiClient {
async postJSON<T>(path: string, body: unknown): Promise<T> { async postJSON<T>(path: string, body: unknown): Promise<T> {
const url = `${this.apiOrigin}${path}`; const url = `${this.apiOrigin}${path}`;
// Idempotency: this reaches `/users/srp/create-session`, // Not idempotent: a POST is replayed only when `isSafeToReplay`
// `/users/two-factor/verify` and `/users/ott`, all of which change // says no request byte can have reached the server. The endpoints
// server state — verifying a second factor consumes one of a small // this covers are listed in the README under "Endpoints used".
// number of attempts. So a POST is replayed only on a failure that //
// establishes no TCP connection to the server ever existed: DNS // Redirects are not followed. The origin has already received the
// produced no address, or the peer refused the connection. A 5xx, a // request when it answers with one, so a connection refused by the
// mid-flight reset, a routing errno (which Linux also delivers on an // redirect target would look replay-safe when it is not. The API has
// established socket) and a timeout are all left to the caller, // no legitimate redirect, so one surfaces as an `ApiError` with its
// because each of them can occur after the server has already acted. // 3xx status, which is not retried.
return withRetry( return withRetry(
async () => { async () => {
const resp = await this._fetch(url, { const resp = await this._fetch(url, {
@@ -241,6 +246,7 @@ export class ApiClient {
"Content-Type": "application/json", "Content-Type": "application/json",
}), }),
body: JSON.stringify(body), body: JSON.stringify(body),
redirect: "manual",
signal: AbortSignal.timeout(this.requestTimeoutMs), signal: AbortSignal.timeout(this.requestTimeoutMs),
}); });
await this.throwIfError(resp); await this.throwIfError(resp);
@@ -299,9 +305,7 @@ export class ApiClient {
async putJSON<T>(path: string, body: unknown): Promise<T> { async putJSON<T>(path: string, body: unknown): Promise<T> {
const url = `${this.apiOrigin}${path}`; const url = `${this.apiOrigin}${path}`;
// Same idempotency rule as `postJSON`, for the same reason: this // Same replay and redirect rules as `postJSON`, for the same reasons.
// reaches `/files/thumbnail`, which registers an uploaded thumbnail
// against a file.
return withRetry( return withRetry(
async () => { async () => {
const resp = await this._fetch(url, { const resp = await this._fetch(url, {
@@ -310,6 +314,7 @@ export class ApiClient {
"Content-Type": "application/json", "Content-Type": "application/json",
}), }),
body: JSON.stringify(body), body: JSON.stringify(body),
redirect: "manual",
signal: AbortSignal.timeout(this.requestTimeoutMs), signal: AbortSignal.timeout(this.requestTimeoutMs),
}); });
await this.throwIfError(resp); await this.throwIfError(resp);
+59 -20
View File
@@ -29,19 +29,21 @@
// rather than counted forever, which would poison a scheduled backup's exit code. // rather than counted forever, which would poison a scheduled backup's exit code.
import { import {
copyFileSync,
lstatSync, lstatSync,
mkdirSync, mkdirSync,
readdirSync,
readFileSync, readFileSync,
readlinkSync, readlinkSync,
renameSync,
rmSync, rmSync,
statSync, statSync,
symlinkSync, symlinkSync,
writeFileSync, writeFileSync,
} from "node:fs"; } from "node:fs";
import { basename, dirname, extname, join, relative } from "node:path"; import { copyFile, rename, rm } from "node:fs/promises";
import { basename, dirname, join, relative } from "node:path";
import { fsyncPath } from "./download/index.js";
import { safeExtension, sanitizeFileName } from "./filename.js";
import type { Collection, EnteFile } from "./model/types.js"; import type { Collection, EnteFile } from "./model/types.js";
export type ProgressCallback = (message: string) => void; export type ProgressCallback = (message: string) => void;
@@ -108,16 +110,11 @@ interface FailureEntry {
const LEDGER_VERSION = 1; const LEDGER_VERSION = 1;
const sanitizePath = (name: string): string =>
name.replace(/[/\\:*?"<>|]/g, "_").replace(/^\.+/, "_");
// The originals/ filename for a file: `<id><ext>`, the extension taken from the // The originals/ filename for a file: `<id><ext>`, the extension taken from the
// title (or `.bin`). Matches the content cache's own naming so a present check // title (or `.bin`). Matches the content cache's own naming so a present check
// lines up with what a fetch would write. // lines up with what a fetch would write.
const originalName = (file: EnteFile): string => { const originalName = (file: EnteFile): string =>
const ext = extname(file.metadata.title || "") || ".bin"; `${file.id}${safeExtension(file.metadata.title)}`;
return `${file.id}${ext}`;
};
// A regular file with content is treated as complete. A zero-byte file is not: // A regular file with content is treated as complete. A zero-byte file is not:
// it is the shape an aborted write leaves and must be re-fetched. // it is the shape an aborted write leaves and must be re-fetched.
@@ -158,8 +155,12 @@ const errorMessage = (err: unknown): string =>
err instanceof Error ? err.message : String(err); err instanceof Error ? err.message : String(err);
// Copy bytes into `dest` via a temp file in the same directory plus rename, so // Copy bytes into `dest` via a temp file in the same directory plus rename, so
// `dest` appears only once it is whole ("present means complete"). // `dest` appears only once it is whole ("present means complete"). As in the
const copyAtomic = (src: string, dest: string): void => { // download writer, the temp file is fsynced before the rename and the directory
// after it, so a power cut cannot leave a correctly named but short original.
// The temp name carries this process's ID so a later run can tell a leftover
// from a copy still in progress (see `removeLeftoverTempFiles`).
const copyAtomic = async (src: string, dest: string): Promise<void> => {
if (src === dest) return; if (src === dest) return;
const tmp = join( const tmp = join(
dirname(dest), dirname(dest),
@@ -168,10 +169,45 @@ const copyAtomic = (src: string, dest: string): void => {
.slice(2)}.tmp`, .slice(2)}.tmp`,
); );
try { try {
copyFileSync(src, tmp); await copyFile(src, tmp);
renameSync(tmp, dest); await fsyncPath(tmp);
// `rename` replaces the destination's directory entry: an existing
// symlink at `dest` is replaced, not followed, and the new file has
// the temp file's permissions (copied from `src`).
await rename(tmp, dest);
await fsyncPath(dirname(dest));
} finally { } finally {
rmSync(tmp, { force: true }); await rm(tmp, { force: true });
}
};
// A process-ID check: signal 0 delivers nothing and only reports whether the
// process exists. EPERM means it exists but belongs to another user.
const isRunning = (pid: number): boolean => {
try {
process.kill(pid, 0);
return true;
} catch (err) {
return (err as NodeJS.ErrnoException).code === "EPERM";
}
};
// Delete the temp files `copyAtomic` leaves behind when a backup is killed
// before its rename. Only files whose process is no longer running are
// removed, so a backup running at the same time keeps its own. A reused
// process ID can only keep a leftover a while longer, never remove a live one.
const removeLeftoverTempFiles = (dir: string): void => {
let names: string[];
try {
names = readdirSync(dir);
} catch {
return;
}
for (const name of names) {
const match = /^\.quak-backup-.*-(\d+)-[0-9a-z]*\.tmp$/.exec(name);
if (match && !isRunning(Number(match[1]))) {
rmSync(join(dir, name), { force: true });
}
} }
}; };
@@ -258,6 +294,8 @@ export const runBackup = async (
mkdirSync(originalsDir, { recursive: true }); mkdirSync(originalsDir, { recursive: true });
mkdirSync(collectionsDir, { recursive: true }); mkdirSync(collectionsDir, { recursive: true });
if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true }); if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true });
removeLeftoverTempFiles(originalsDir);
removeLeftoverTempFiles(thumbnailsDir);
const ledgerPath = join(downloadDirectory, "failures.json"); const ledgerPath = join(downloadDirectory, "failures.json");
const ledger = loadLedger(ledgerPath); const ledger = loadLedger(ledgerPath);
@@ -325,7 +363,7 @@ export const runBackup = async (
try { try {
log(`Fetching original ${file.metadata.title} (${fileID})...`); log(`Fetching original ${file.metadata.title} (${fileID})...`);
const { path } = await lib.original(fileID); const { path } = await lib.original(fileID);
copyAtomic(path, dest); await copyAtomic(path, dest);
downloaded++; downloaded++;
} catch (err) { } catch (err) {
log( log(
@@ -346,7 +384,7 @@ export const runBackup = async (
if (isPresent(dest)) continue; if (isPresent(dest)) continue;
try { try {
const { path } = await lib.thumbnail(fileID); const { path } = await lib.thumbnail(fileID);
copyAtomic(path, dest); await copyAtomic(path, dest);
} catch (err) { } catch (err) {
recordFailure( recordFailure(
file, file,
@@ -370,7 +408,7 @@ export const runBackup = async (
// Then the per-collection symlink trees and JSON. // Then the per-collection symlink trees and JSON.
for (const c of collections) { for (const c of collections) {
const colDirName = sanitizePath(c.name || `collection-${c.id}`); const colDirName = sanitizeFileName(c.name, `collection-${c.id}`);
const colDir = join(collectionsDir, colDirName); const colDir = join(collectionsDir, colDirName);
mkdirSync(colDir, { recursive: true }); mkdirSync(colDir, { recursive: true });
@@ -381,8 +419,9 @@ export const runBackup = async (
if (!includeOriginals) continue; if (!includeOriginals) continue;
const orig = join(originalsDir, originalName(file)); const orig = join(originalsDir, originalName(file));
if (!isPresent(orig)) continue; if (!isPresent(orig)) continue;
const linkName = sanitizePath( const linkName = sanitizeFileName(
file.metadata.title || `file-${file.id}`, file.metadata.title,
`file-${file.id}`,
); );
const linkPath = join(colDir, linkName); const linkPath = join(colDir, linkName);
try { try {
+486
View File
@@ -0,0 +1,486 @@
// The CLI's commands as plain functions.
//
// Each command takes its options and a `CliContext` and resolves to the exit
// code; a thrown error is left to the caller. Nothing here calls
// `process.exit`: `bin/quak.ts` wires these to the command line and exits with
// the returned code once output has drained. Output must stay byte-identical
// (see `cli-output.ts`).
import { input, password as passwordPrompt } from "@inquirer/prompts";
import {
copyFileSync,
existsSync,
mkdirSync,
unlinkSync,
writeFileSync,
} from "node:fs";
import { join } from "node:path";
import { Client, type ClientSnapshot } from "./client.js";
import { init } from "./crypto/index.js";
import { Library, type LibraryClient } from "./library/index.js";
import {
fileListRow,
fileListLine,
originalName,
thumbnailName,
} from "./cli-output.js";
import { freshCollections, freshFiles, freshFile } from "./cli-read.js";
import { runMetadataBackup } from "./metadata-backup.js";
import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js";
export interface CliContext {
stdout: { write(text: string): unknown };
stderr: { write(text: string): unknown };
// Directory holding `session.json`.
sessionDir: string;
// The `--cache-dir` global, or undefined to let the library pick its
// per-user default keyed by the account id.
cacheDir?: string;
// Reads the session file into a client, or null when there is none. The
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
loadSession: (path: string) => Client | null;
}
const sessionPath = (ctx: CliContext): string =>
join(ctx.sessionDir, "session.json");
// Write the session readable by its owner only, in a directory only its owner
// can enter.
export const saveSession = (
sessionDir: string,
snapshot: ClientSnapshot,
): void => {
mkdirSync(sessionDir, { recursive: true, mode: 0o700 });
writeFileSync(
join(sessionDir, "session.json"),
JSON.stringify(snapshot, null, 2),
{ mode: 0o600 },
);
};
// The saved client, or undefined after telling the user why there is none.
const requireSession = (ctx: CliContext): Client | undefined => {
let client: Client | null;
try {
client = ctx.loadSession(sessionPath(ctx));
} catch (err) {
ctx.stderr.write(
`${err instanceof Error ? err.message : err}\n` +
`Run "quak logout" and then "quak login" to replace it.\n`,
);
return undefined;
}
if (!client) {
ctx.stderr.write(
`Not logged in. Run "quak login" first.\nSession file: ${sessionPath(ctx)}\n`,
);
return undefined;
}
return client;
};
// A library client that omits `fetchMLData`, so the point commands below do not
// kick the library's background ML backfill: they read metadata, or fetch one
// file's content, and exit. `backup` and `backup-metadata` handle ML on their
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
// originals through the on-disk cache.
const readLibraryClient = (client: Client): LibraryClient => ({
whoami: () => client.whoami(),
collectionsSince: (args) => client.collectionsSince(args),
filesSince: (args) => client.filesSince(args),
contentSource: () => client.contentSource(),
});
// Open a library for a single point command: the aggressive background precache
// (issue #48) is off — a one-shot `collections` or `get` must not start
// downloading the whole account — and the refresh interval is long so no second
// refresh fires mid-command.
const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
Library.open({
client: readLibraryClient(client),
cacheDirectory: ctx.cacheDir,
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
});
const prompt = async (message: string): Promise<string> => input({ message });
const promptSecret = async (message: string): Promise<string> =>
passwordPrompt({ message, mask: true });
export const loginCommand = async (ctx: CliContext): Promise<number> => {
await init();
const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
const password =
process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
ctx.stderr.write("Authenticating...\n");
try {
const client = await Client.login({
email,
password,
totp: async () => prompt("TOTP code: "),
emailOTP: async () => prompt("Email verification code: "),
});
saveSession(ctx.sessionDir, client.toJSON());
const info = client.whoami();
ctx.stderr.write(`Logged in as ${info.email} (user ${info.userID})\n`);
ctx.stderr.write(`Session saved to ${sessionPath(ctx)}\n`);
} catch (err) {
ctx.stderr.write(
`Login failed: ${err instanceof Error ? err.message : err}\n`,
);
return 1;
}
return 0;
};
export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const info = client.whoami();
ctx.stdout.write(JSON.stringify(info) + "\n");
return 0;
};
export const logoutCommand = async (ctx: CliContext): Promise<number> => {
if (existsSync(sessionPath(ctx))) {
unlinkSync(sessionPath(ctx));
ctx.stderr.write("Session deleted.\n");
} else {
ctx.stderr.write("No session found.\n");
}
return 0;
};
export const collectionsCommand = async (
ctx: CliContext,
opts: { json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52): the pre-library CLI printed current state in
// this order, not the albums projection's newest-first order.
const collections = await freshCollections(lib);
if (opts.json) {
ctx.stdout.write(
JSON.stringify(
collections.map((c) => ({
id: c.id,
name: c.name,
type: c.type,
ownerID: c.ownerID,
isShared: c.isShared,
updationTime: c.updationTime,
})),
null,
2,
) + "\n",
);
} else {
for (const c of collections) {
ctx.stdout.write(
`${c.id}\t${c.type}\t${c.name}${c.isShared ? " (shared)" : ""}\n`,
);
}
}
return 0;
} finally {
lib.close();
}
};
export const filesCommand = async (
ctx: CliContext,
opts: { collection: string; json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const collectionID = Number(opts.collection);
if (!Number.isFinite(collectionID)) {
ctx.stderr.write("Invalid collection ID\n");
return 1;
}
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52). Each file prints from its own decrypted
// metadata (raw title, microsecond creationTime) via cli-output, and in
// the pre-library CLI's enumeration order, not the projection's
// newest-first order.
const files = await freshFiles(lib, collectionID);
if (!files) {
ctx.stderr.write(`Collection ${collectionID} not found\n`);
return 1;
}
if (opts.json) {
ctx.stdout.write(
JSON.stringify(files.map(fileListRow), null, 2) + "\n",
);
} else {
for (const file of files) {
ctx.stdout.write(fileListLine(file) + "\n");
}
}
return 0;
} finally {
lib.close();
}
};
export const getCommand = async (
ctx: CliContext,
fileIDStr: string,
opts: { out?: string },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const fileID = Number(fileIDStr);
if (!Number.isFinite(fileID)) {
ctx.stderr.write("Invalid file ID\n");
return 1;
}
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
ctx.stderr.write(`File ${fileID} not found\n`);
return 1;
}
const { photo, file } = resolved;
const result = await photo.original();
// Default name is the file's own title, as the pre-library CLI used
// (not the editedName-preferring projection title) (issue #52).
const outPath = opts.out ?? originalName(file);
copyFileSync(result.path, outPath);
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
return 0;
} finally {
lib.close();
}
};
export const getThumbCommand = async (
ctx: CliContext,
fileIDStr: string,
opts: { out?: string },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const fileID = Number(fileIDStr);
if (!Number.isFinite(fileID)) {
ctx.stderr.write("Invalid file ID\n");
return 1;
}
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
ctx.stderr.write(`File ${fileID} not found\n`);
return 1;
}
const { photo, file } = resolved;
const result = await photo.thumbnail();
// Default name is thumb_<file's own title>, as the pre-library CLI
// used (not the projection title) (issue #52).
const outPath = opts.out ?? thumbnailName(file);
copyFileSync(result.path, outPath);
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
return 0;
} finally {
lib.close();
}
};
export const backupMetadataCommand = async (
ctx: CliContext,
dir: string,
opts: { exif?: boolean; all?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
await runMetadataBackup(lib, client, dir, {
exif: opts.exif || opts.all,
onProgress: (msg) => ctx.stderr.write(msg + "\n"),
});
return 0;
} finally {
lib.close();
}
};
export const backupCommand = async (
ctx: CliContext,
dir: string,
opts: { json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
ctx.stderr.write("Starting backup...\n");
const lib = await Library.open({
client,
downloadDirectory: dir,
cacheDirectory: ctx.cacheDir,
});
try {
const result = await lib.backup({
downloadDirectory: dir,
onProgress: (msg) => {
if (!opts.json) ctx.stderr.write(msg + "\n");
},
});
if (opts.json) {
ctx.stdout.write(JSON.stringify(result, null, 2) + "\n");
} else {
ctx.stderr.write("\n--- Backup complete ---\n");
ctx.stderr.write(` Total files: ${result.totalFiles}\n`);
ctx.stderr.write(` Downloaded: ${result.downloaded}\n`);
ctx.stderr.write(` Skipped: ${result.skipped}\n`);
ctx.stderr.write(` Failed: ${result.failed}\n`);
if (result.errors.length > 0) {
ctx.stderr.write("\nFailed files:\n");
for (const e of result.errors) {
ctx.stderr.write(
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
);
}
}
}
return result.failed > 0 ? 1 : 0;
} finally {
lib.close();
}
};
export const listMissingThumbnailsCommand = async (
ctx: CliContext,
opts: { json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
const missing = await listMissingThumbnails(lib, client, (msg) => {
if (!opts.json) ctx.stderr.write(msg + "\n");
});
if (opts.json) {
ctx.stdout.write(JSON.stringify(missing, null, 2) + "\n");
} else {
if (missing.length === 0) {
ctx.stderr.write("No missing thumbnails found.\n");
} else {
ctx.stderr.write(
`\n${missing.length} file(s) with missing thumbnails:\n`,
);
for (const m of missing) {
ctx.stdout.write(
`${m.fileID}\t${m.title}\t${m.collection}\t${m.reason}\n`,
);
}
}
}
return 0;
} finally {
lib.close();
}
};
export const fixMissingThumbnailsCommand = async (
ctx: CliContext,
opts: { file?: string[]; json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
let fileIDs: number[];
if (opts.file && opts.file.length > 0) {
fileIDs = opts.file.map(Number).filter(Number.isFinite);
} else {
ctx.stderr.write("Scanning for missing thumbnails...\n");
const missing = await listMissingThumbnails(lib, client, (msg) => {
if (!opts.json) ctx.stderr.write(msg + "\n");
});
fileIDs = missing.map((m) => m.fileID);
if (fileIDs.length === 0) {
ctx.stderr.write("No missing thumbnails found.\n");
return 0;
}
ctx.stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
}
const results = await fixMissingThumbnails(
lib,
client,
fileIDs,
(msg) => {
if (!opts.json) ctx.stderr.write(msg + "\n");
},
);
if (opts.json) {
ctx.stdout.write(JSON.stringify(results, null, 2) + "\n");
} else {
const fixed = results.filter((r) => r.status === "fixed").length;
const skipped = results.filter(
(r) => r.status === "skipped",
).length;
const failed = results.filter((r) => r.status === "failed").length;
ctx.stderr.write(`\n--- Done ---\n`);
ctx.stderr.write(` Fixed: ${fixed}\n`);
ctx.stderr.write(` Skipped: ${skipped}\n`);
ctx.stderr.write(` Failed: ${failed}\n`);
if (skipped > 0) {
ctx.stderr.write("\nSkipped (unsupported format):\n");
for (const r of results.filter((r) => r.status === "skipped")) {
ctx.stderr.write(
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
);
}
}
if (failed > 0) {
ctx.stderr.write("\nFailed files:\n");
for (const r of results.filter((r) => r.status === "failed")) {
ctx.stderr.write(
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
);
}
}
}
return results.some((r) => r.status === "failed") ? 1 : 0;
} finally {
lib.close();
}
};
+6 -3
View File
@@ -9,6 +9,7 @@
// `metadata.title`, and issue #52 requires that output stay byte-identical, so // `metadata.title`, and issue #52 requires that output stay byte-identical, so
// the commands shape their output from the raw `EnteFile` through here. // the commands shape their output from the raw `EnteFile` through here.
import { sanitizeFileName } from "./filename.js";
import type { EnteFile, FileType, Microseconds } from "./model/types.js"; import type { EnteFile, FileType, Microseconds } from "./model/types.js";
// One row of `quak files --json`. // One row of `quak files --json`.
@@ -32,9 +33,11 @@ export const fileListRow = (file: EnteFile): FileListRow => ({
export const fileListLine = (file: EnteFile): string => export const fileListLine = (file: EnteFile): string =>
`${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`; `${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
// Default output path for `quak get` when `--out` is not given. // Default output path for `quak get` when `--out` is not given. The title comes
export const originalName = (file: EnteFile): string => file.metadata.title; // from the server, so it is sanitized; `--out` is the user's and is used as is.
export const originalName = (file: EnteFile): string =>
sanitizeFileName(file.metadata.title, `file-${file.id}`);
// Default output path for `quak get-thumb` when `--out` is not given. // Default output path for `quak get-thumb` when `--out` is not given.
export const thumbnailName = (file: EnteFile): string => export const thumbnailName = (file: EnteFile): string =>
`thumb_${file.metadata.title}`; `thumb_${originalName(file)}`;
+26
View File
@@ -0,0 +1,26 @@
// How the CLI reads its saved session file back into a `Client`.
//
// A missing file means "not logged in" and returns null. A file that exists but
// cannot be read back into a client (bad JSON, a missing field, a key of the
// wrong length) throws an error saying the session file is corrupt, so the CLI
// can tell the user which of the two it is. Needs `init()` first.
import { existsSync, readFileSync } from "node:fs";
import type { ApiClientOptions } from "./api/client.js";
import { Client } from "./client.js";
export const loadSession = (
path: string,
apiOptions?: ApiClientOptions,
): Client | null => {
if (!existsSync(path)) return null;
try {
return Client.fromJSON(
JSON.parse(readFileSync(path, "utf-8")),
apiOptions,
);
} catch (err) {
const reason = err instanceof Error ? err.message : String(err);
throw new Error(`Session file ${path} is corrupt: ${reason}`);
}
};
+62 -11
View File
@@ -125,18 +125,57 @@ export class Client {
); );
} }
static fromJSON( // Restore a client from a `toJSON()` snapshot. The snapshot usually comes
snapshot: ClientSnapshot, // straight from `JSON.parse` of a file on disk, so every field is checked
apiOptions?: ApiClientOptions, // before use; a bad one throws an error naming it. Needs `init()` first.
): Client { static fromJSON(snapshot: unknown, apiOptions?: ApiClientOptions): Client {
const api = new ApiClient({ ...apiOptions, authToken: snapshot.token }); const invalid = (field: string, problem: string): Error =>
new Error(`Invalid session data: ${field} ${problem}`);
if (typeof snapshot !== "object" || snapshot === null) {
throw new Error("Invalid session data: not a JSON object");
}
const s = snapshot as Record<string, unknown>;
for (const field of ["email", "token"]) {
if (typeof s[field] !== "string" || s[field] === "") {
throw invalid(field, "must be a non-empty string");
}
}
if (!Number.isInteger(s.userID)) {
throw invalid("userID", "must be an integer");
}
const key = (field: string): Uint8Array => {
const value = s[field];
if (typeof value !== "string") {
throw invalid(field, "must be a base64 string");
}
let bytes: Uint8Array;
try {
bytes = fromBase64(value);
} catch {
throw invalid(field, "is not valid base64");
}
// The master key (secretbox) and the key pair (box) are all 32 bytes.
if (bytes.length !== 32) {
throw invalid(
field,
`must decode to 32 bytes, got ${bytes.length}`,
);
}
return bytes;
};
const api = new ApiClient({
...apiOptions,
authToken: s.token as string,
});
return new Client( return new Client(
api, api,
snapshot.email, s.email as string,
snapshot.userID, s.userID as number,
fromBase64(snapshot.masterKey), key("masterKey"),
fromBase64(snapshot.secretKey), key("secretKey"),
fromBase64(snapshot.publicKey), key("publicKey"),
); );
} }
@@ -164,19 +203,29 @@ export class Client {
toJSON(): ClientSnapshot { toJSON(): ClientSnapshot {
this.assertLoggedIn(); this.assertLoggedIn();
const token = this.api.getAuthToken();
if (!token) {
throw new Error("Cannot serialize client: it has no auth token");
}
return { return {
email: this.email, email: this.email,
userID: this.userID, userID: this.userID,
token: this.api["token"]!, token,
masterKey: toBase64(this.masterKey), masterKey: toBase64(this.masterKey),
secretKey: toBase64(this.secretKey), secretKey: toBase64(this.secretKey),
publicKey: toBase64(this.publicKey), publicKey: toBase64(this.publicKey),
}; };
} }
// Zeroes the key buffers in place, so any copy of the reference held
// elsewhere is wiped too. Every method checks `assertLoggedIn` before
// touching the keys, so nothing decrypts with the zeroed keys.
logout(): void { logout(): void {
this.loggedOut = true; this.loggedOut = true;
this.api.clearAuthToken(); this.api.clearAuthToken();
this.masterKey.fill(0);
this.secretKey.fill(0);
this.publicKey.fill(0);
} }
// Enumerate collections changed since `sinceTime`. Live collections are // Enumerate collections changed since `sinceTime`. Live collections are
@@ -192,6 +241,8 @@ export class Client {
const { collections: raws } = await this.api.getJSON<{ const { collections: raws } = await this.api.getJSON<{
collections: RawCollection[]; collections: RawCollection[];
}>("/collections/v2", { sinceTime: args.sinceTime }); }>("/collections/v2", { sinceTime: args.sinceTime });
// logout() may have zeroed the keys while the request was in flight.
this.assertLoggedIn();
const collections: Collection[] = []; const collections: Collection[] = [];
const deleted: number[] = []; const deleted: number[] = [];
+25 -8
View File
@@ -11,6 +11,7 @@ import {
streamTagFinal, streamTagFinal,
} from "../crypto/index.js"; } from "../crypto/index.js";
import { TruncatedStreamError } from "../errors.js"; import { TruncatedStreamError } from "../errors.js";
import { sanitizeFileName } from "../filename.js";
import { withRetry } from "../retry.js"; import { withRetry } from "../retry.js";
import type { ApiClient } from "../api/client.js"; import type { ApiClient } from "../api/client.js";
import type { EnteFile } from "../model/types.js"; import type { EnteFile } from "../model/types.js";
@@ -157,6 +158,18 @@ const streamDecrypt = async (
return totalPlain; return totalPlain;
}; };
// Fsync a file or a directory, so its contents (for a directory, its entries)
// are on stable storage. Exported for the backup tree's copy, which needs the
// same durability as the writer below.
export const fsyncPath = async (path: string): Promise<void> => {
const handle = await open(path, "r");
try {
await handle.sync();
} finally {
await handle.close();
}
};
// Stage a write to `destination` atomically and durably, then rename it into // Stage a write to `destination` atomically and durably, then rename it into
// place. `fill` writes the contents into the open temp file handle — either the // place. `fill` writes the contents into the open temp file handle — either the
// whole buffer at once (`writeAtomic`) or chunk by chunk as they decrypt // whole buffer at once (`writeAtomic`) or chunk by chunk as they decrypt
@@ -192,16 +205,15 @@ const stageAtomic = async (
} finally { } finally {
await handle.close(); await handle.close();
} }
// `rename` replaces the destination's directory entry rather than
// writing through it: an existing symlink at `destination` is
// replaced, not followed, and the new file has the temp file's
// permissions, not those of the file it replaced.
await rename(tmpPath, destination); await rename(tmpPath, destination);
// Fsync the directory so the rename itself survives a crash: renaming // Fsync the directory so the rename itself survives a crash: renaming
// over a synced temp file still leaves the new directory entry in the // over a synced temp file still leaves the new directory entry in the
// page cache until the directory is synced. // page cache until the directory is synced.
const dirHandle = await open(dir, "r"); await fsyncPath(dir);
try {
await dirHandle.sync();
} finally {
await dirHandle.close();
}
} catch (err) { } catch (err) {
// Best-effort cleanup. A failure to remove the temporary file must // Best-effort cleanup. A failure to remove the temporary file must
// never replace the error that actually explains what went wrong. // never replace the error that actually explains what went wrong.
@@ -286,7 +298,10 @@ export const downloadFile = async (
outPath?: string, outPath?: string,
onProgress?: ProgressCallback, onProgress?: ProgressCallback,
): Promise<DownloadResult> => { ): Promise<DownloadResult> => {
const resolvedPath = outPath ?? file.metadata.title; // `outPath` is the caller's and is used as is; the title is the server's
// and is sanitized so it can only name a file in the current directory.
const resolvedPath =
outPath ?? sanitizeFileName(file.metadata.title, `file-${file.id}`);
const header = fromBase64(file.file.decryptionHeader); const header = fromBase64(file.file.decryptionHeader);
const bytesWritten = await fetchAndDecrypt( const bytesWritten = await fetchAndDecrypt(
api, api,
@@ -305,7 +320,9 @@ export const downloadThumbnail = async (
outPath?: string, outPath?: string,
onProgress?: ProgressCallback, onProgress?: ProgressCallback,
): Promise<DownloadResult> => { ): Promise<DownloadResult> => {
const resolvedPath = outPath ?? `thumb_${file.metadata.title}`; const resolvedPath =
outPath ??
`thumb_${sanitizeFileName(file.metadata.title, `file-${file.id}`)}`;
const header = fromBase64(file.thumbnail.decryptionHeader); const header = fromBase64(file.thumbnail.decryptionHeader);
const bytesWritten = await fetchAndDecrypt( const bytesWritten = await fetchAndDecrypt(
api, api,
+37
View File
@@ -0,0 +1,37 @@
// File names built from server-supplied metadata.
//
// A file's title and a collection's name are decrypted from data the server
// hands us, and quak does not trust the server. Any name taken from them and
// used on disk goes through here, so it can only ever name one file inside the
// directory the caller chose: never a path, never `..`, never hidden, never a
// Windows device name.
//
// A path the user typed (`--out`, `outPath`) is not passed through here: the
// caller is trusted, the server is not.
import { extname } from "node:path";
// Path separators, characters Windows forbids in file names, and control
// characters (NUL included).
// eslint-disable-next-line no-control-regex
const UNSAFE_CHARACTERS = /[/\\:*?"<>|\x00-\x1f\x7f]/g;
// Names Windows reserves for devices, with or without an extension.
const RESERVED_DEVICE_NAME = /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\.|$)/i;
// `name` made safe to use as a single file name. Each unsafe character becomes
// `_`, a leading run of dots becomes one `_`, and a device name gets a leading
// `_`. A name with none of these comes back unchanged. An empty name becomes
// `fallback`, which the caller derives from the record's ID.
export const sanitizeFileName = (name: string, fallback: string): string => {
if (name === "") return fallback;
const cleaned = name.replace(UNSAFE_CHARACTERS, "_").replace(/^\.+/, "_");
return RESERVED_DEVICE_NAME.test(cleaned) ? `_${cleaned}` : cleaned;
};
// The extension of `title` (".jpg"), or ".bin" when it has none or it holds
// anything but letters and digits.
export const safeExtension = (title: string): string => {
const ext = extname(title);
return /^\.[A-Za-z0-9]+$/.test(ext) ? ext : ".bin";
};
+3 -4
View File
@@ -40,6 +40,7 @@ import {
downloadThumbnail, downloadThumbnail,
type ProgressCallback, type ProgressCallback,
} from "../download/index.js"; } from "../download/index.js";
import { safeExtension } from "../filename.js";
import type { EnteFile } from "../model/types.js"; import type { EnteFile } from "../model/types.js";
import type { Priority, RequestPools } from "./pools.js"; import type { Priority, RequestPools } from "./pools.js";
@@ -209,10 +210,8 @@ class AbortDrop extends Error {
} }
} }
const originalName = (file: EnteFile): string => { const originalName = (file: EnteFile): string =>
const ext = extname(file.metadata.title || "") || ".bin"; `${file.id}${safeExtension(file.metadata.title)}`;
return `${file.id}${ext}`;
};
// The fileID a cache filename encodes, or undefined when the name is not one // The fileID a cache filename encodes, or undefined when the name is not one
// the cache writes (`<digits><ext>`). // the cache writes (`<digits><ext>`).
+52 -31
View File
@@ -4,6 +4,7 @@ import * as jpeg from "jpeg-js";
import exifReader from "exif-reader"; import exifReader from "exif-reader";
import type { Client } from "./client.js"; import type { Client } from "./client.js";
import type { Library, Photo } from "./library/index.js"; import type { Library, Photo } from "./library/index.js";
import { sanitizeFileName } from "./filename.js";
import { fetchMLData } from "./mldata-fetch.js"; import { fetchMLData } from "./mldata-fetch.js";
import type { EnteFile } from "./model/types.js"; import type { EnteFile } from "./model/types.js";
@@ -14,21 +15,35 @@ export interface MetadataBackupOptions {
onProgress?: ProgressCallback; onProgress?: ProgressCallback;
} }
const sanitizePath = (name: string): string => // Find the raw EXIF APP1 segment in JPEG bytes. Returns `exif` (the segment
name.replace(/[/\\:*?"<>|]/g, "_").replace(/^\.+/, "_"); // data, starting at the "Exif\0\0" header) when there is one, nothing when the
// bytes are not a JPEG or carry no EXIF, and `error` when the segment layout is
// Extract the raw EXIF APP1 segment from JPEG bytes. Returns the EXIF // malformed. Each segment length is checked against the bytes that remain and
// data buffer (starting after the APP1 length field, at the "Exif\0\0" // each step moves forward by at least 4 bytes, so the scan ends on any input.
// header) or undefined if no APP1 marker is found. export const extractExifFromJpeg = (
const extractExifFromJpeg = (buf: Uint8Array): Buffer | undefined => { buf: Uint8Array,
if (buf[0] !== 0xff || buf[1] !== 0xd8) return undefined; ): { exif?: Buffer; error?: string } => {
if (buf[0] !== 0xff || buf[1] !== 0xd8) return {};
let offset = 2; let offset = 2;
while (offset < buf.length - 1) { while (offset < buf.length) {
if (buf[offset] !== 0xff) return undefined; if (offset + 2 > buf.length)
return { error: `truncated segment marker at byte ${offset}` };
if (buf[offset] !== 0xff)
return { error: `no segment marker at byte ${offset}` };
const marker = buf[offset + 1]!; const marker = buf[offset + 1]!;
if (marker === 0xda) break; // start of scan, no more markers if (marker === 0xda) return {}; // start of scan, no more markers
if (offset + 3 >= buf.length) break; if (offset + 4 > buf.length)
return { error: `truncated segment length at byte ${offset}` };
const len = (buf[offset + 2]! << 8) | buf[offset + 3]!; const len = (buf[offset + 2]! << 8) | buf[offset + 3]!;
// The length counts its own two bytes, so anything under 2 is invalid.
if (len < 2)
return {
error: `segment length ${len} at byte ${offset} is too small`,
};
if (offset + 2 + len > buf.length)
return {
error: `segment length ${len} at byte ${offset} runs past the end of the file`,
};
if (marker === 0xe1) { if (marker === 0xe1) {
// APP1 — check for "Exif\0\0" header // APP1 — check for "Exif\0\0" header
if ( if (
@@ -37,22 +52,26 @@ const extractExifFromJpeg = (buf: Uint8Array): Buffer | undefined => {
buf[offset + 6] === 0x69 && buf[offset + 6] === 0x69 &&
buf[offset + 7] === 0x66 buf[offset + 7] === 0x66
) { ) {
return Buffer.from( return {
exif: Buffer.from(
buf.buffer, buf.buffer,
buf.byteOffset + offset + 4, buf.byteOffset + offset + 4,
len - 2, len - 2,
); ),
};
} }
} }
offset += 2 + len; offset += 2 + len;
} }
return undefined; return { error: "file ends before the image data" };
}; };
const extractImageMetadata = ( // Extract dimensions, EXIF and XMP from a file's bytes. When the EXIF segment
// is malformed or cannot be parsed, the record carries the reason in
// `exifError`.
export const extractImageMetadata = (
fileBytes: Uint8Array, fileBytes: Uint8Array,
): Record<string, unknown> | undefined => { ): Record<string, unknown> | undefined => {
try {
const result: Record<string, unknown> = {}; const result: Record<string, unknown> = {};
// Try to get dimensions from JPEG decode // Try to get dimensions from JPEG decode
@@ -65,15 +84,19 @@ const extractImageMetadata = (
result.width = decoded.width; result.width = decoded.width;
result.height = decoded.height; result.height = decoded.height;
} catch { } catch {
// Not a JPEG or corrupt; still try EXIF extraction // Not every original is a JPEG (PNG, HEIC, video), so a failed decode
// is expected and only means no dimensions; a malformed JPEG is still
// reported below through `exifError`.
} }
const exifBuf = extractExifFromJpeg(fileBytes); const { exif, error } = extractExifFromJpeg(fileBytes);
if (exifBuf) { if (error) result.exifError = error;
if (exif) {
try { try {
result.exif = exifReader(exifBuf); result.exif = exifReader(exif);
} catch { } catch (err) {
result.exifRaw = exifBuf.toString("base64"); result.exifRaw = exif.toString("base64");
result.exifError = err instanceof Error ? err.message : String(err);
} }
} }
@@ -93,9 +116,6 @@ const extractImageMetadata = (
} }
return Object.keys(result).length > 0 ? result : undefined; return Object.keys(result).length > 0 ? result : undefined;
} catch {
return undefined;
}
}; };
// Read a file's original bytes through the library's content cache and extract // Read a file's original bytes through the library's content cache and extract
@@ -105,13 +125,9 @@ const extractImageMetadata = (
const extractExif = async ( const extractExif = async (
photo: Photo, photo: Photo,
): Promise<Record<string, unknown> | undefined> => { ): Promise<Record<string, unknown> | undefined> => {
try {
const { path } = await photo.original(); const { path } = await photo.original();
const fileBytes = new Uint8Array(readFileSync(path)); const fileBytes = new Uint8Array(readFileSync(path));
return extractImageMetadata(fileBytes); return extractImageMetadata(fileBytes);
} catch {
return undefined;
}
}; };
// Dump every decrypted metadata layer the account holds into a directory tree // Dump every decrypted metadata layer the account holds into a directory tree
@@ -151,7 +167,7 @@ export const runMetadataBackup = async (
const col = lib.getCollection(album.collectionID); const col = lib.getCollection(album.collectionID);
if (!col) continue; if (!col) continue;
const dirName = `${col.id}-${sanitizePath(col.name || "unnamed")}`; const dirName = `${col.id}-${sanitizeFileName(col.name, "unnamed")}`;
const colDir = join(outDir, "collections", dirName); const colDir = join(outDir, "collections", dirName);
mkdirSync(colDir, { recursive: true }); mkdirSync(colDir, { recursive: true });
@@ -217,8 +233,13 @@ export const runMetadataBackup = async (
if (wantExif && !writtenFileIDs.has(file.id)) { if (wantExif && !writtenFileIDs.has(file.id)) {
log(`[${file.metadata.title}] Extracting EXIF...`); log(`[${file.metadata.title}] Extracting EXIF...`);
try {
const exifData = await extractExif(photo); const exifData = await extractExif(photo);
if (exifData) fileMeta.imageMetadata = exifData; if (exifData) fileMeta.imageMetadata = exifData;
} catch (err) {
fileMeta.imageMetadataError =
err instanceof Error ? err.message : String(err);
}
} }
writtenFileIDs.add(file.id); writtenFileIDs.add(file.id);
+10 -1
View File
@@ -98,9 +98,18 @@ export const decryptFile = (
key, key,
); );
const metadataJSON = JSON.parse(new TextDecoder().decode(metadataBytes)); const metadataJSON = JSON.parse(new TextDecoder().decode(metadataBytes));
if (
typeof metadataJSON !== "object" ||
metadataJSON === null ||
Array.isArray(metadataJSON)
) {
throw new Error(`file ${raw.id}: metadata is not a JSON object`);
}
const metadata: FileMetadata = { const metadata: FileMetadata = {
title: metadataJSON.title ?? "", // The server controls this JSON: a title that is missing or not a
// string becomes "", never an arbitrary value.
title: typeof metadataJSON.title === "string" ? metadataJSON.title : "",
fileType: parseFileType(metadataJSON.fileType ?? -1), fileType: parseFileType(metadataJSON.fileType ?? -1),
creationTime: metadataJSON.creationTime ?? 0, creationTime: metadataJSON.creationTime ?? 0,
modificationTime: metadataJSON.modificationTime ?? 0, modificationTime: metadataJSON.modificationTime ?? 0,
+25 -12
View File
@@ -88,18 +88,21 @@ const MAX_CAUSE_DEPTH = 8;
// errno on the error it throws — it hangs the underlying socket error off // errno on the error it throws — it hangs the underlying socket error off
// `cause`, sometimes more than one level down — so a classifier that only read // `cause`, sometimes more than one level down — so a classifier that only read
// the top-level error would see a bare `Error` and call every dropped // the top-level error would see a bare `Error` and call every dropped
// connection permanent. // connection permanent. `complete` is false when the walk stopped at the
const causeCodes = (err: unknown): string[] => { // depth limit with more of the chain still below it.
const causeCodes = (err: unknown): { codes: string[]; complete: boolean } => {
const codes: string[] = []; const codes: string[] = [];
let current: unknown = err; let current: unknown = err;
for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) { for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
if (current === null || typeof current !== "object") break; if (current === null || typeof current !== "object") {
return { codes, complete: true };
}
const { code, cause } = current as { code?: unknown; cause?: unknown }; const { code, cause } = current as { code?: unknown; cause?: unknown };
if (typeof code === "string") codes.push(code); if (typeof code === "string") codes.push(code);
if (cause === current) break; if (cause === current) return { codes, complete: true };
current = cause; current = cause;
} }
return codes; return { codes, complete: current === null || typeof current !== "object" };
}; };
const isAbort = (err: unknown): boolean => { const isAbort = (err: unknown): boolean => {
@@ -145,15 +148,15 @@ export const isRetryable = (err: unknown): boolean => {
// have succeeded; the cost of the imprecision is bounded by the attempt // have succeeded; the cost of the imprecision is bounded by the attempt
// count. // count.
if (err instanceof TypeError) return true; if (err instanceof TypeError) return true;
return causeCodes(err).some((code) => TRANSPORT_CODES.has(code)); return causeCodes(err).codes.some((code) => TRANSPORT_CODES.has(code));
}; };
// Could the first attempt already have taken effect on the server? // Could the first attempt already have taken effect on the server?
// //
// `isRetryable` is the wrong question for a request that changes state. // `isRetryable` is the wrong question for a request that changes state.
// quak's non-idempotent calls are `/users/srp/create-session`, // `postJSON` and `putJSON` use this for every `POST` and `PUT` listed in the
// `/users/two-factor/verify` — which consumes one of a small number of 2FA // README under "Endpoints used"; verifying a second factor, for one, consumes
// attempts — and `/files/thumbnail`. They are replayed only on the failures in // one of a small number of attempts. They are replayed only on the failures in
// `CONNECT_CODES`, which establish that no TCP connection to the server ever // `CONNECT_CODES`, which establish that no TCP connection to the server ever
// existed: there was no address to connect to, or the peer refused the // existed: there was no address to connect to, or the peer refused the
// connection outright. A request byte cannot have been transmitted, so the // connection outright. A request byte cannot have been transmitted, so the
@@ -162,9 +165,19 @@ export const isRetryable = (err: unknown): boolean => {
// Everything else is ambiguous. A 5xx proves the server did process the // Everything else is ambiguous. A 5xx proves the server did process the
// request. A reset or a broken pipe can arrive after it was fully sent and // request. A reset or a broken pipe can arrive after it was fully sent and
// acted on. A routing errno can be delivered on an established socket. A // acted on. A routing errno can be delivered on an established socket. A
// deadline says nothing at all about the server's state. // deadline says nothing at all about the server's state. So every errno in the
export const isSafeToReplay = (err: unknown): boolean => // cause chain must be a connect errno: one other errno anywhere in the chain
isRetryable(err) && causeCodes(err).some((code) => CONNECT_CODES.has(code)); // is doubt, and doubt is not replayed. A chain longer than the walk is doubt
// too: the links below the limit were never read.
export const isSafeToReplay = (err: unknown): boolean => {
const { codes, complete } = causeCodes(err);
return (
isRetryable(err) &&
complete &&
codes.length > 0 &&
codes.every((code) => CONNECT_CODES.has(code))
);
};
export interface WithRetryOptions extends RetryOptions { export interface WithRetryOptions extends RetryOptions {
isRetryable?: (err: unknown) => boolean; isRetryable?: (err: unknown) => boolean;
+91 -3
View File
@@ -639,6 +639,24 @@ describe("ApiClient retries", () => {
expect(policy.baseDelayMs).toBe(7); expect(policy.baseDelayMs).toBe(7);
expect(policy.maxDelayMs).toBe(11); expect(policy.maxDelayMs).toBe(11);
}); });
it("does not let a caller change its settings through that policy", async () => {
const { fetch, calls } = scriptedFetch(
textResponse("boom", 500),
textResponse("boom", 500),
textResponse("boom", 500),
);
const client = new ApiClient({
fetch,
retry: { ...noWait, attempts: 2 },
});
client.getRetryOptions().attempts = 3;
expect(client.getRetryOptions().attempts).toBe(2);
await expect(client.getJSON("/x")).rejects.toBeInstanceOf(ApiError);
expect(calls).toHaveLength(2);
});
}); });
describe("ApiClient timeouts", () => { describe("ApiClient timeouts", () => {
@@ -693,6 +711,51 @@ describe("ApiClient timeouts", () => {
expect(new Set(signals).size).toBe(3); expect(new Set(signals).size).toBe(3);
}, 5000); }, 5000);
it("gives every retrying entry point a fresh deadline per attempt", async () => {
// A refused connection is retried by every entry point, the
// non-idempotent ones included. If the deadline were created once,
// outside the retry, both attempts would carry the same signal.
const entryPoints: [
string,
() => Response,
(c: ApiClient) => unknown,
][] = [
["getJSON", () => jsonResponse({}), (c) => c.getJSON("/a")],
["postJSON", () => jsonResponse({}), (c) => c.postJSON("/b", {})],
["putJSON", () => jsonResponse({}), (c) => c.putJSON("/c", {})],
[
"putFile",
() => new Response(null, { status: 200 }),
(c) => c.putFile("https://s3.example/x", new Uint8Array([1])),
],
[
"getFileStream",
() => streamResponse(new Uint8Array([1])),
(c) => c.getFileStream(1),
],
[
"getThumbnailStream",
() => streamResponse(new Uint8Array([1])),
(c) => c.getThumbnailStream(1),
],
];
for (const [name, success, call] of entryPoints) {
const { fetch, calls } = scriptedFetch(
errnoError("ECONNREFUSED", "connect ECONNREFUSED"),
success(),
);
const client = new ApiClient({ fetch, retry: noWait });
await call(client);
expect(calls, name).toHaveLength(2);
const [first, second] = calls.map((c) => c.init?.signal);
expect(first, name).toBeInstanceOf(AbortSignal);
expect(second, name).toBeInstanceOf(AbortSignal);
expect(second, name).not.toBe(first);
}
});
it("recovers when a later attempt answers in time", async () => { it("recovers when a later attempt answers in time", async () => {
const { fetch, calls } = scriptedFetch(HANG, jsonResponse({ ok: 1 })); const { fetch, calls } = scriptedFetch(HANG, jsonResponse({ ok: 1 }));
const client = new ApiClient({ const client = new ApiClient({
@@ -820,9 +883,8 @@ describe("ApiClient error typing", () => {
describe("ApiClient non-idempotent requests", () => { describe("ApiClient non-idempotent requests", () => {
/** /**
* `postJSON` and `putJSON` carry quak's only requests that change server * `postJSON` and `putJSON` carry quak's requests that can change server
* state: `/users/srp/create-session`, `/users/two-factor/verify` — which * state; the README lists them under "Endpoints used".
* consumes one of a small number of 2FA attempts — and `/files/thumbnail`.
* *
* They are retried only on a failure that establishes no TCP connection to * They are retried only on a failure that establishes no TCP connection to
* the server ever existed — DNS produced no address, or the peer refused * the server ever existed — DNS produced no address, or the peer refused
@@ -922,4 +984,30 @@ describe("ApiClient non-idempotent requests", () => {
await refusedClient.updateThumbnail(1, "key", "header"); await refusedClient.updateThumbnail(1, "key", "header");
expect(refused.calls).toHaveLength(2); expect(refused.calls).toHaveLength(2);
}); });
it("does not follow or replay a redirect on POST or PUT", async () => {
// The origin has already received a request it answers with a
// redirect, so following it would let a refused connection to the
// redirect target pass for a request that never went out.
for (const send of [
(c: ApiClient) => c.postJSON("/users/ott", {}),
(c: ApiClient) => c.putJSON("/files/thumbnail", {}),
]) {
const { fetch, calls } = scriptedFetch(
new Response(null, {
status: 307,
headers: { location: "https://elsewhere.example/" },
}),
jsonResponse({}),
);
const client = new ApiClient({ fetch, retry: noWait });
const err: unknown = await send(client).catch((e: unknown) => e);
expect(calls[0]?.init?.redirect).toBe("manual");
expect(err).toBeInstanceOf(ApiError);
expect((err as ApiError).status).toBe(307);
expect(calls).toHaveLength(1);
}
});
}); });
+145 -3
View File
@@ -36,20 +36,50 @@ import {
lstatSync, lstatSync,
mkdirSync, mkdirSync,
mkdtempSync, mkdtempSync,
readdirSync,
readFileSync, readFileSync,
readlinkSync, readlinkSync,
rmSync, rmSync,
writeFileSync, writeFileSync,
} from "node:fs"; } from "node:fs";
import { spawnSync } from "node:child_process";
import { join } from "node:path"; import { join } from "node:path";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { describe, it, expect, beforeEach, afterEach } from "vitest"; import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import { Library } from "../../src/library/index.js"; import { Library } from "../../src/library/index.js";
import type { ContentSource } from "../../src/library/content.js"; import type { ContentSource } from "../../src/library/content.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js"; import type { CollectionsPage, FilesPage } from "../../src/client.js";
import type { Collection, EnteFile } from "../../src/model/types.js"; import type { Collection, EnteFile } from "../../src/model/types.js";
// `open` and `rename` are wrapped to record, in order, every fsync and rename,
// so a test can pin the sequence "fsync the temp file, rename, fsync the
// directory" that makes a copied original survive a power cut. `vi.hoisted`
// because `vi.mock` factories run before module-level constants exist.
const fsEvents = vi.hoisted(() => [] as string[]);
vi.mock("node:fs/promises", async (importOriginal) => {
const actual = await importOriginal<typeof import("node:fs/promises")>();
return {
...actual,
open: async (
...args: Parameters<typeof actual.open>
): Promise<Awaited<ReturnType<typeof actual.open>>> => {
const handle = await actual.open(...args);
const realSync = handle.sync.bind(handle);
handle.sync = async (): Promise<void> => {
fsEvents.push(`sync:${String(args[0])}`);
await realSync();
};
return handle;
},
rename: async (from: string, to: string): Promise<void> => {
fsEvents.push(`rename:${to}`);
await actual.rename(from, to);
},
};
});
const USER_ID = 42; const USER_ID = 42;
// Decrypted-byte length each stub original writes, keyed by fileID. // Decrypted-byte length each stub original writes, keyed by fileID.
@@ -107,6 +137,29 @@ class MockClient {
} }
} }
// A server that names an album and a file so as to climb out of the backup
// directory.
class HostileClient extends MockClient {
override async collectionsSince(): Promise<CollectionsPage> {
const page = await super.collectionsSince();
return {
...page,
collections: page.collections.length
? [collection(3, "../escape")]
: [],
};
}
override async filesSince(args: {
collectionID: number;
}): Promise<FilesPage> {
const files =
args.collectionID === 3
? [file(300, 3, "../../.ssh/authorized_keys")]
: [];
return { files, deleted: [], cursor: 1 };
}
}
// A content source that writes byte buffers of the expected length and can be // A content source that writes byte buffers of the expected length and can be
// told to fail one fileID's original, to exercise per-file resilience. // told to fail one fileID's original, to exercise per-file resilience.
interface StubSource extends ContentSource { interface StubSource extends ContentSource {
@@ -136,9 +189,12 @@ const stubSource = (): StubSource => {
let root: string; let root: string;
const openLibrary = (source: ContentSource): Promise<Library> => const openLibrary = (
source: ContentSource,
client: MockClient = new MockClient(),
): Promise<Library> =>
Library.open({ Library.open({
client: new MockClient(), client,
cacheDirectory: join(root, "cache"), cacheDirectory: join(root, "cache"),
contentSource: source, contentSource: source,
refreshIntervalSeconds: 3600, refreshIntervalSeconds: 3600,
@@ -243,6 +299,30 @@ describe("lib.backup", () => {
lib.close(); lib.close();
}); });
it("keeps server-supplied album and file names inside the backup", async () => {
const lib = await openLibrary(stubSource(), new HostileClient());
const outDir = join(root, "backup");
const result = await lib.backup({ downloadDirectory: outDir });
expect(result.failed).toBe(0);
// The title has no usable extension, so the original is `.bin`.
expect(existsSync(join(outDir, "originals", "300.bin"))).toBe(true);
const link = join(
outDir,
"collections",
"__escape",
"__.._.ssh_authorized_keys",
);
expect(lstatSync(link).isSymbolicLink()).toBe(true);
expect(existsSync(join(outDir, "collections", "__escape.json"))).toBe(
true,
);
// Nothing landed beside or above the backup directory.
expect(readdirSync(root).sort()).toEqual(["backup", "cache"]);
lib.close();
});
it("is an idempotent no-op when every original is already present", async () => { it("is an idempotent no-op when every original is already present", async () => {
const source = stubSource(); const source = stubSource();
const lib = await openLibrary(source); const lib = await openLibrary(source);
@@ -479,4 +559,66 @@ describe("lib.backup", () => {
expect(readLedger(outDir).files["101"]!.attempts).toBe(1); expect(readLedger(outDir).files["101"]!.attempts).toBe(1);
lib.close(); lib.close();
}); });
it("fsyncs a copied original before the rename and its directory after", async () => {
const lib = await openLibrary(stubSource());
const outDir = join(root, "backup");
const originals = join(outDir, "originals");
const dest = join(originals, "100.jpg");
fsEvents.length = 0;
await lib.backup({ downloadDirectory: outDir });
const at = fsEvents.indexOf(`rename:${dest}`);
expect(at).toBeGreaterThan(0);
expect(fsEvents[at - 1]).toMatch(
/^sync:.*\/\.quak-backup-100\.jpg-\d+-[0-9a-z]*\.tmp$/,
);
expect(fsEvents[at + 1]).toBe(`sync:${originals}`);
lib.close();
});
it("removes temp files left by a killed backup but not those of one still running", async () => {
const outDir = join(root, "backup");
const originals = join(outDir, "originals");
mkdirSync(originals, { recursive: true });
// A child that has already exited: its process ID is not running.
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
const leftover = `.quak-backup-100.jpg-${exitedPID}-abc123.tmp`;
// This test's own process stands in for a backup running at the same
// time.
const inProgress = `.quak-backup-101.jpg-${process.pid}-def456.tmp`;
writeFileSync(join(originals, leftover), "partial");
writeFileSync(join(originals, inProgress), "partial");
const lib = await openLibrary(stubSource());
await lib.backup({ downloadDirectory: outDir });
const names = readdirSync(originals);
expect(names).not.toContain(leftover);
expect(names).toContain(inProgress);
lib.close();
});
it("removes leftover temp files in thumbnails/ but not those of a backup still running", async () => {
const outDir = join(root, "backup");
const thumbnails = join(outDir, "thumbnails");
mkdirSync(thumbnails, { recursive: true });
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
const leftover = `.quak-backup-100.jpg-${exitedPID}-abc123.tmp`;
const inProgress = `.quak-backup-101.jpg-${process.pid}-def456.tmp`;
writeFileSync(join(thumbnails, leftover), "partial");
writeFileSync(join(thumbnails, inProgress), "partial");
const lib = await openLibrary(stubSource());
await lib.backup({
downloadDirectory: outDir,
includeThumbnails: true,
});
const names = readdirSync(thumbnails);
expect(names).not.toContain(leftover);
expect(names).toContain(inProgress);
lib.close();
});
}); });
+397
View File
@@ -0,0 +1,397 @@
/**
* Tests for the CLI commands (`src/cli-commands.ts`, issue #12).
*
* Each command is called directly with a context whose output streams collect
* text, whose session directory is a fresh temp directory, and whose session
* loader hands back a fake client. The fake serves two albums and three files
* from memory, writes stand-in bytes for originals and thumbnails, and makes no
* network calls. The helpers the commands call (`cli-read`, `cli-output`,
* backup, thumbnails) have their own tests; these check what each command
* prints and the exit code it returns.
*/
import {
existsSync,
mkdtempSync,
readFileSync,
rmSync,
statSync,
writeFileSync,
} from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { describe, it, expect, beforeAll, beforeEach, afterEach } from "vitest";
import {
type CliContext,
saveSession,
whoamiCommand,
logoutCommand,
collectionsCommand,
filesCommand,
getCommand,
getThumbCommand,
backupCommand,
listMissingThumbnailsCommand,
} from "../../src/cli-commands.js";
import { loadSession } from "../../src/cli-session.js";
import type { Client, ClientSnapshot } from "../../src/client.js";
import type { ContentSource } from "../../src/library/content.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
import { init } from "../../src/crypto/index.js";
const USER_ID = 42;
const collection = (
id: number,
name: string,
isShared = false,
): Collection => ({
id,
ownerID: USER_ID,
key: new Uint8Array([id]),
name,
type: "album",
updationTime: 1,
isShared,
});
const file = (id: number, collectionID: number, title: string): EnteFile => ({
id,
collectionID,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
metadata: {
title,
fileType: "image",
creationTime: 1000,
modificationTime: 1000,
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime: 1,
});
const COLLECTIONS = [collection(1, "Vacation"), collection(2, "Work", true)];
const FILES: Record<number, EnteFile[]> = {
1: [file(100, 1, "beach.jpg"), file(101, 1, "sunset.jpg")],
2: [file(200, 2, "diagram.png")],
};
// An original is 7 bytes and a thumbnail 3. `failID` makes that file's
// original fail; `emptyThumbID` makes the server report that file's
// thumbnail as empty.
const fakeClient = (opts: { failID?: number; emptyThumbID?: number } = {}) => {
const source: ContentSource = {
original: async ({ file: f, destination }) => {
if (f.id === opts.failID) throw new Error("HTTP 500 from server");
writeFileSync(destination, Buffer.alloc(7, f.id & 0xff));
return { bytesWritten: 7 };
},
thumbnail: async ({ file: f, destination }) => {
writeFileSync(destination, Buffer.alloc(3, f.id & 0xff));
return { bytesWritten: 3 };
},
};
const fake = {
whoami: () => ({ email: "cli@example.com", userID: USER_ID }),
collectionsSince: async () => ({
collections: COLLECTIONS,
deleted: [],
cursor: 1,
}),
filesSince: async (args: { collectionID: number }) => ({
files: FILES[args.collectionID] ?? [],
deleted: [],
cursor: 1,
}),
contentSource: () => source,
getApiClient: () => ({
getThumbnailStream: async (fileID: number) =>
new ReadableStream<Uint8Array>({
start(controller) {
if (fileID !== opts.emptyThumbID) {
controller.enqueue(new Uint8Array(3));
}
controller.close();
},
}),
}),
};
// The commands only call the methods above.
return fake as unknown as Client;
};
// Collects everything written to it.
class Output {
text = "";
write(text: string): void {
this.text += text;
}
}
let root: string;
let stdout: Output;
let stderr: Output;
const context = (client: Client | null = fakeClient()): CliContext => ({
stdout,
stderr,
sessionDir: join(root, "session"),
cacheDir: join(root, "cache"),
loadSession: () => client,
});
beforeAll(async () => {
await init();
});
beforeEach(() => {
root = mkdtempSync(join(tmpdir(), "quak-cli-test-"));
stdout = new Output();
stderr = new Output();
});
afterEach(() => {
rmSync(root, { recursive: true, force: true });
});
describe("session file", () => {
const snapshot: ClientSnapshot = {
email: "cli@example.com",
userID: USER_ID,
token: "token",
masterKey: "a",
secretKey: "b",
publicKey: "c",
};
it("is written with mode 0600 in a directory with mode 0700", () => {
const dir = join(root, "new", "session");
saveSession(dir, snapshot);
expect(statSync(dir).mode & 0o777).toBe(0o700);
const path = join(dir, "session.json");
expect(statSync(path).mode & 0o777).toBe(0o600);
expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot);
});
it("is removed by logout", async () => {
const ctx = context();
saveSession(ctx.sessionDir, snapshot);
expect(await logoutCommand(ctx)).toBe(0);
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
expect(stderr.text).toBe("Session deleted.\n");
});
it("logout without a session says so and exits 0", async () => {
expect(await logoutCommand(context())).toBe(0);
expect(stderr.text).toBe("No session found.\n");
});
it("a missing session exits 1 with 'Not logged in'", async () => {
const ctx = { ...context(), loadSession };
expect(await whoamiCommand(ctx)).toBe(1);
expect(stderr.text).toBe(
`Not logged in. Run "quak login" first.\n` +
`Session file: ${join(ctx.sessionDir, "session.json")}\n`,
);
expect(stdout.text).toBe("");
});
it("a corrupt session exits 1 and says it is corrupt", async () => {
const ctx = { ...context(), loadSession };
saveSession(ctx.sessionDir, snapshot);
expect(await collectionsCommand(ctx, {})).toBe(1);
expect(stderr.text).toContain("is corrupt");
expect(stderr.text).toContain(
`Run "quak logout" and then "quak login" to replace it.\n`,
);
expect(stdout.text).toBe("");
});
});
describe("whoami", () => {
it("prints the account as one line of JSON", async () => {
expect(await whoamiCommand(context())).toBe(0);
expect(stdout.text).toBe(
`{"email":"cli@example.com","userID":${USER_ID}}\n`,
);
});
});
describe("collections", () => {
it("prints one tab-separated line per album", async () => {
expect(await collectionsCommand(context(), {})).toBe(0);
expect(stdout.text).toBe(
"1\talbum\tVacation\n" + "2\talbum\tWork (shared)\n",
);
});
it("prints a JSON array with --json", async () => {
expect(await collectionsCommand(context(), { json: true })).toBe(0);
expect(JSON.parse(stdout.text)).toEqual([
{
id: 1,
name: "Vacation",
type: "album",
ownerID: USER_ID,
isShared: false,
updationTime: 1,
},
{
id: 2,
name: "Work",
type: "album",
ownerID: USER_ID,
isShared: true,
updationTime: 1,
},
]);
});
});
describe("files", () => {
it("prints one tab-separated line per file", async () => {
expect(await filesCommand(context(), { collection: "1" })).toBe(0);
expect(stdout.text).toBe(
"100\timage\tbeach.jpg\n" + "101\timage\tsunset.jpg\n",
);
});
it("prints a JSON array with --json", async () => {
const code = await filesCommand(context(), {
collection: "2",
json: true,
});
expect(code).toBe(0);
expect(JSON.parse(stdout.text)).toEqual([
{
id: 200,
title: "diagram.png",
fileType: "image",
creationTime: 1000,
collectionID: 2,
},
]);
});
it("exits 1 for an unknown collection", async () => {
expect(await filesCommand(context(), { collection: "9" })).toBe(1);
expect(stderr.text).toBe("Collection 9 not found\n");
});
it("exits 1 for a collection ID that is not a number", async () => {
expect(await filesCommand(context(), { collection: "abc" })).toBe(1);
expect(stderr.text).toBe("Invalid collection ID\n");
});
});
describe("get and get-thumb", () => {
it("get finds a file in any album without --collection", async () => {
const out = join(root, "diagram.png");
expect(await getCommand(context(), "200", { out })).toBe(0);
expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
});
it("get-thumb finds a file in any album without --collection", async () => {
const out = join(root, "thumb.jpg");
expect(await getThumbCommand(context(), "200", { out })).toBe(0);
expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
expect(stderr.text).toBe(`3 bytes -> ${out}\n`);
});
it("get exits 1 when no album has the file", async () => {
const out = join(root, "x");
expect(await getCommand(context(), "999", { out })).toBe(1);
expect(stderr.text).toBe("File 999 not found\n");
expect(existsSync(out)).toBe(false);
});
it("get-thumb exits 1 when no album has the file", async () => {
const out = join(root, "x");
expect(await getThumbCommand(context(), "999", { out })).toBe(1);
expect(stderr.text).toBe("File 999 not found\n");
expect(existsSync(out)).toBe(false);
});
it("both exit 1 for a file ID that is not a number", async () => {
expect(await getCommand(context(), "abc", {})).toBe(1);
expect(await getThumbCommand(context(), "abc", {})).toBe(1);
expect(stderr.text).toBe("Invalid file ID\nInvalid file ID\n");
});
});
describe("backup", () => {
it("exits 0 and prints a summary when every file is saved", async () => {
const dir = join(root, "backup");
expect(await backupCommand(context(), dir, {})).toBe(0);
expect(stderr.text).toContain(
"\n--- Backup complete ---\n" +
" Total files: 3\n" +
" Downloaded: 3\n" +
" Skipped: 0\n" +
" Failed: 0\n",
);
expect(stdout.text).toBe("");
});
it("exits 1 and lists the file when one download fails", async () => {
const ctx = context(fakeClient({ failID: 101 }));
expect(await backupCommand(ctx, join(root, "backup"), {})).toBe(1);
expect(stderr.text).toContain(" Failed: 1\n");
expect(stderr.text).toContain(
"\nFailed files:\n" +
" [Vacation] sunset.jpg (id 101): HTTP 500 from server\n",
);
});
it("prints the result as JSON with --json, still exiting 1 on a failure", async () => {
const ctx = context(fakeClient({ failID: 101 }));
const code = await backupCommand(ctx, join(root, "backup"), {
json: true,
});
expect(code).toBe(1);
const result = JSON.parse(stdout.text);
expect(result).toMatchObject({
totalFiles: 3,
downloaded: 2,
skipped: 0,
failed: 1,
});
expect(result.errors[0].fileID).toBe(101);
expect(stderr.text).toBe("Starting backup...\n");
});
});
describe("helper list-missing-thumbnails", () => {
it("prints one line per file with an empty thumbnail", async () => {
const ctx = context(fakeClient({ emptyThumbID: 200 }));
expect(await listMissingThumbnailsCommand(ctx, {})).toBe(0);
expect(stdout.text).toBe(
"200\tdiagram.png\tWork\tempty thumbnail (0 bytes)\n",
);
expect(stderr.text).toContain("\n1 file(s) with missing thumbnails:\n");
});
it("says so when nothing is missing", async () => {
expect(await listMissingThumbnailsCommand(context(), {})).toBe(0);
expect(stdout.text).toBe("");
expect(stderr.text).toContain("No missing thumbnails found.\n");
});
it("prints a JSON array with --json and no progress", async () => {
const ctx = context(fakeClient({ emptyThumbID: 200 }));
expect(await listMissingThumbnailsCommand(ctx, { json: true })).toBe(0);
expect(JSON.parse(stdout.text)).toEqual([
{
fileID: 200,
title: "diagram.png",
collection: "Work",
reason: "empty thumbnail (0 bytes)",
},
]);
expect(stderr.text).toBe("");
});
});
+18 -3
View File
@@ -165,14 +165,15 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
}, },
}; };
// Collection 2: "Work" with no magic metadata // Collection 2: "../Work" with no magic metadata. The server chose a name
// that tries to climb out of the backup directory.
const ck2 = sodium.crypto_secretbox_keygen(); const ck2 = sodium.crypto_secretbox_keygen();
const { ciphertext: encCK2, nonce: ck2N } = encryptSecretbox( const { ciphertext: encCK2, nonce: ck2N } = encryptSecretbox(
ck2, ck2,
masterKey, masterKey,
); );
const { ciphertext: encCN2, nonce: cn2N } = encryptSecretbox( const { ciphertext: encCN2, nonce: cn2N } = encryptSecretbox(
new TextEncoder().encode("Work"), new TextEncoder().encode("../Work"),
ck2, ck2,
); );
const rawColl2 = { const rawColl2 = {
@@ -496,7 +497,8 @@ describe("quak backup-metadata", () => {
await runBackup(outDir); await runBackup(outDir);
const collDirs = readdirSync(join(outDir, "collections")); const collDirs = readdirSync(join(outDir, "collections"));
expect(collDirs.length).toBe(2); // "../Work" is sanitized into one directory name.
expect(collDirs.sort()).toEqual(["10-Vacation", "20-__Work"]);
// Find the Vacation collection dir (prefixed with ID) // Find the Vacation collection dir (prefixed with ID)
const vacDir = collDirs.find((d) => d.includes("Vacation"))!; const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
@@ -619,5 +621,18 @@ describe("quak backup-metadata", () => {
expect(fileMeta.imageMetadata.format).toBe("jpeg"); expect(fileMeta.imageMetadata.format).toBe("jpeg");
expect(fileMeta.imageMetadata.width).toBe(100); expect(fileMeta.imageMetadata.width).toBe(100);
expect(fileMeta.imageMetadata.height).toBe(80); expect(fileMeta.imageMetadata.height).toBe(80);
expect(fileMeta.imageMetadataError).toBeUndefined();
// File 200 has no original on the mock server, so extraction fails
// and the reason is recorded instead of the field being left out.
const workDir = collDirs.find((d) => d.includes("Work"))!;
const failedMeta = JSON.parse(
readFileSync(
join(outDir, "collections", workDir, "200.json"),
"utf-8",
),
);
expect(failedMeta.imageMetadata).toBeUndefined();
expect(failedMeta.imageMetadataError).toEqual(expect.any(String));
}); });
}); });
+122
View File
@@ -0,0 +1,122 @@
/**
* Tests for the JPEG EXIF scan behind `quak backup-metadata --exif`.
*
* The originals come from users' libraries, so a truncated or corrupt JPEG
* must neither hang the scan nor throw out of it, and a malformed file must be
* told apart from one that simply has no EXIF: the record carries the reason in
* `exifError`. Each input below is a short hand-built byte array.
*/
import { describe, expect, it } from "vitest";
import {
extractExifFromJpeg,
extractImageMetadata,
} from "../../src/metadata-backup.js";
const SOI = [0xff, 0xd8]; // start of image
const SOS = [0xff, 0xda, 0x00, 0x02]; // start of scan, where the scan stops
const EXIF_HEADER = [0x45, 0x78, 0x69, 0x66, 0x00, 0x00]; // "Exif\0\0"
// A big-endian TIFF block with one IFD entry: Orientation (0x0112), SHORT, 6.
const TIFF_ORIENTATION_6 = [
0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08, 0x00, 0x01, 0x01, 0x12,
0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00,
];
// An APP1 segment whose length field matches its data.
const app1 = (data: number[]): number[] => {
const len = data.length + 2;
return [0xff, 0xe1, len >> 8, len & 0xff, ...data];
};
const bytes = (...parts: number[][]): Uint8Array =>
new Uint8Array(parts.flat());
describe("extractExifFromJpeg", () => {
it("returns the EXIF segment of a valid JPEG", () => {
const data = [...EXIF_HEADER, ...TIFF_ORIENTATION_6];
const scan = extractExifFromJpeg(bytes(SOI, app1(data), SOS));
expect(scan.error).toBeUndefined();
expect([...scan.exif!]).toEqual(data);
});
it("returns nothing for a file that is not a JPEG", () => {
const png = bytes([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
expect(extractExifFromJpeg(png)).toEqual({});
});
it("returns nothing for a JPEG without EXIF", () => {
const app0 = [0xff, 0xe0, 0x00, 0x04, 0x00, 0x00];
expect(extractExifFromJpeg(bytes(SOI, app0, SOS))).toEqual({});
});
it("reports a JPEG truncated inside a segment header", () => {
const scan = extractExifFromJpeg(bytes(SOI, [0xff, 0xe1, 0x00]));
expect(scan.exif).toBeUndefined();
expect(scan.error).toMatch(/truncated segment length/);
});
it("reports a JPEG that ends before the image data", () => {
const app0 = [0xff, 0xe0, 0x00, 0x04, 0x00, 0x00];
const scan = extractExifFromJpeg(bytes(SOI, app0));
expect(scan.error).toMatch(/ends before the image data/);
});
it("stops on a zero-length segment instead of looping", () => {
// A length of 0 would otherwise step the scan by 2 bytes at a time
// through the rest of the file, reading garbage as markers.
const zero = [0xff, 0xe0, 0x00, 0x00];
const scan = extractExifFromJpeg(
bytes(SOI, zero, zero, zero, zero, SOS),
);
expect(scan.error).toMatch(/segment length 0 at byte 2 is too small/);
});
it("stops on a segment length of 1", () => {
const scan = extractExifFromJpeg(
bytes(SOI, [0xff, 0xe0, 0x00, 0x01], SOS),
);
expect(scan.error).toMatch(/segment length 1 at byte 2 is too small/);
});
it("reports a segment length that runs past the end of the file", () => {
// APP1 claims 0x4000 bytes but only the "Exif\0\0" header follows.
const scan = extractExifFromJpeg(
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
);
expect(scan.exif).toBeUndefined();
expect(scan.error).toMatch(/runs past the end of the file/);
});
});
describe("extractImageMetadata", () => {
it("parses EXIF from a valid JPEG", () => {
const meta = extractImageMetadata(
bytes(SOI, app1([...EXIF_HEADER, ...TIFF_ORIENTATION_6]), SOS),
);
expect(meta?.exifError).toBeUndefined();
expect(meta?.exif).toMatchObject({ Image: { Orientation: 6 } });
});
it("returns nothing for a file that is not a JPEG", () => {
const text = new TextEncoder().encode("just some text, not an image");
expect(extractImageMetadata(text)).toBeUndefined();
});
it("records the reason when the JPEG is malformed", () => {
const meta = extractImageMetadata(
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
);
expect(meta?.exif).toBeUndefined();
expect(meta?.exifError).toMatch(/runs past the end of the file/);
});
it("keeps the raw bytes and the reason when EXIF cannot be parsed", () => {
const data = [...EXIF_HEADER, 0x58, 0x58];
const meta = extractImageMetadata(bytes(SOI, app1(data), SOS));
expect(meta?.exif).toBeUndefined();
expect(meta?.exifRaw).toBe(Buffer.from(data).toString("base64"));
expect(meta?.exifError).toEqual(expect.any(String));
});
});
+18
View File
@@ -64,6 +64,24 @@ describe("CLI file output (issue #52)", () => {
expect(thumbnailName(renamedFile)).toBe(`thumb_${RAW_TITLE}`); expect(thumbnailName(renamedFile)).toBe(`thumb_${RAW_TITLE}`);
}); });
it("sanitizes the title when naming `quak get` downloads", () => {
// Without `--out`, the server-supplied title names the file, so it must
// not be able to point outside the working directory.
const hostile = {
...renamedFile,
metadata: { ...renamedFile.metadata, title: "../../.bashrc" },
};
expect(originalName(hostile)).toBe("__.._.bashrc");
expect(thumbnailName(hostile)).toBe("thumb___.._.bashrc");
const untitled = {
...renamedFile,
metadata: { ...renamedFile.metadata, title: "" },
};
expect(originalName(untitled)).toBe("file-100");
expect(thumbnailName(untitled)).toBe("thumb_file-100");
});
it("does not use the editedName/editedTime projection", () => { it("does not use the editedName/editedTime projection", () => {
const record = deriveRecords([], [renamedFile]).photos.get(100); const record = deriveRecords([], [renamedFile]).photos.get(100);
// The projection prefers the edits and reports milliseconds; the CLI // The projection prefers the edits and reports milliseconds; the CLI
+197
View File
@@ -0,0 +1,197 @@
/**
* Tests for the client session lifecycle: `toJSON`, `fromJSON`, `logout`, and
* the CLI's `loadSession`, which reads the saved session file back into a
* client.
*/
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import sodium from "libsodium-wrappers-sumo";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { init, toBase64 } from "../../src/crypto/index.js";
import { Client, type ClientSnapshot } from "../../src/client.js";
import { loadSession } from "../../src/cli-session.js";
const validSnapshot = (): ClientSnapshot => {
const kp = sodium.crypto_box_keypair();
return {
email: "user@example.com",
userID: 42,
token: "test-token",
masterKey: toBase64(sodium.crypto_secretbox_keygen()),
secretKey: toBase64(kp.privateKey),
publicKey: toBase64(kp.publicKey),
};
};
// The client's key buffers are private; the tests read them to prove that
// logout wipes them.
const keyBuffers = (client: Client): Uint8Array[] => [
client["masterKey"],
client["secretKey"],
client["publicKey"],
];
beforeAll(async () => {
await init();
});
describe("Client.toJSON", () => {
it("round-trips through fromJSON unchanged", () => {
const snapshot = validSnapshot();
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
});
it("throws instead of emitting a snapshot without a token", () => {
const client = Client.fromJSON(validSnapshot());
client.getApiClient().clearAuthToken();
expect(() => client.toJSON()).toThrow(/no auth token/);
});
});
describe("Client.fromJSON", () => {
const shortKey = toBase64(new Uint8Array(16));
it.each([
["email", undefined],
["email", 7],
["email", ""],
["token", undefined],
["token", null],
["token", ""],
["userID", undefined],
["userID", "42"],
["userID", 4.2],
["masterKey", undefined],
["masterKey", 7],
["masterKey", "not base64!"],
["masterKey", shortKey],
["secretKey", undefined],
["secretKey", "not base64!"],
["secretKey", shortKey],
["publicKey", undefined],
["publicKey", "not base64!"],
["publicKey", shortKey],
])("rejects %s = %j, naming the field", (field, value) => {
const snapshot: Record<string, unknown> = { ...validSnapshot() };
snapshot[field] = value;
expect(() => Client.fromJSON(snapshot)).toThrow(
new RegExp(`^Invalid session data: ${field} `),
);
});
it.each([null, "a string", 42])("rejects a non-object %j", (value) => {
expect(() => Client.fromJSON(value)).toThrow(
/^Invalid session data: not a JSON object/,
);
});
});
describe("Client.logout", () => {
it("zeroes the key buffers and clears the token", () => {
const client = Client.fromJSON(validSnapshot());
const api = client.getApiClient();
const keys = keyBuffers(client);
client.logout();
for (const key of keys) {
expect(key.length).toBe(32);
expect(key.every((b) => b === 0)).toBe(true);
}
expect(api.getAuthToken()).toBeUndefined();
});
it("makes every later operation throw", async () => {
const client = Client.fromJSON(validSnapshot());
client.logout();
expect(() => client.whoami()).toThrow(/logged out/);
expect(() => client.toJSON()).toThrow(/logged out/);
expect(() => client.getApiClient()).toThrow(/logged out/);
expect(() => client.contentSource()).toThrow(/logged out/);
await expect(client.listCollections()).rejects.toThrow(/logged out/);
await expect(client.collectionsSince({ sinceTime: 0 })).rejects.toThrow(
/logged out/,
);
await expect(
client.filesSince({
collectionID: 1,
collectionKey: new Uint8Array(32),
sinceTime: 0,
}),
).rejects.toThrow(/logged out/);
await expect(
client.fetchMLData({ fileIDs: [1], fileKeys: new Map() }),
).rejects.toThrow(/logged out/);
});
it("stops a listing in flight from decrypting with the zeroed keys", async () => {
// The server answers only after the client has logged out. If the
// listing went on to decrypt this row with all-zero keys it would fail
// with a decryption error, not the logged-out one.
const row = {
id: 1,
owner: { id: 42 },
encryptedKey: toBase64(new Uint8Array(48)),
keyDecryptionNonce: toBase64(new Uint8Array(24)),
updationTime: 1,
};
const client: Client = Client.fromJSON(validSnapshot(), {
fetch: async () => {
client.logout();
return new Response(JSON.stringify({ collections: [row] }), {
status: 200,
headers: { "content-type": "application/json" },
});
},
});
await expect(client.listCollections()).rejects.toThrow(/logged out/);
});
});
describe("loadSession", () => {
let dir: string;
beforeAll(() => {
dir = mkdtempSync(join(tmpdir(), "quak-session-test-"));
});
afterAll(() => {
rmSync(dir, { recursive: true, force: true });
});
it("returns null when there is no session file", () => {
expect(loadSession(join(dir, "missing.json"))).toBeNull();
});
it("restores a client from a valid session file", () => {
const path = join(dir, "valid.json");
writeFileSync(path, JSON.stringify(validSnapshot()));
expect(loadSession(path)!.whoami()).toEqual({
email: "user@example.com",
userID: 42,
});
});
it("says the file is corrupt when it is not JSON", () => {
const path = join(dir, "truncated.json");
writeFileSync(path, '{"email": "user@exa');
expect(() => loadSession(path)).toThrow(
`Session file ${path} is corrupt`,
);
});
it("says the file is corrupt and names the bad field", () => {
const path = join(dir, "bad-key.json");
writeFileSync(
path,
JSON.stringify({ ...validSnapshot(), secretKey: "AAAA" }),
);
expect(() => loadSession(path)).toThrow(
new RegExp(`^Session file ${path} is corrupt: .*secretKey`),
);
});
});
+124 -11
View File
@@ -48,7 +48,9 @@
*/ */
import { import {
chmodSync,
existsSync, existsSync,
mkdirSync,
readdirSync, readdirSync,
readFileSync, readFileSync,
rmSync, rmSync,
@@ -501,6 +503,22 @@ const entryPoints = [
{ name: "downloadThumbnail", download: downloadThumbnail }, { name: "downloadThumbnail", download: downloadThumbnail },
]; ];
// With no `outPath`, the destination is named after `metadata.title`, relative
// to the working directory. Such tests run inside a temporary directory:
// `make check` must not create files in the repo root.
const inDirectory = async <T>(
dir: string,
run: () => Promise<T>,
): Promise<T> => {
const previous = process.cwd();
process.chdir(dir);
try {
return await run();
} finally {
process.chdir(previous);
}
};
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Tests // Tests
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -536,26 +554,59 @@ describe("downloadFile", () => {
}); });
it("uses metadata.title as filename when outPath is omitted", async () => { it("uses metadata.title as filename when outPath is omitted", async () => {
// With no `outPath`, the destination is `metadata.title`, used
// verbatim as a path. The title here is therefore given inside the
// test's temporary directory: a bare relative name would resolve
// against the process working directory, i.e. the repo root, and
// `make check` must not create files in the repo — a failure between
// the write and any cleanup would leave one behind.
const plaintext = new Uint8Array([1, 2, 3]); const plaintext = new Uint8Array([1, 2, 3]);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen(); const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(plaintext, key); const { header, ciphertext } = encryptFileBody(plaintext, key);
const thumbPush = const thumbPush =
sodium.crypto_secretstream_xchacha20poly1305_init_push(key); sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const file = buildMockEnteFile(key, header, thumbPush.header); const file = buildMockEnteFile(key, header, thumbPush.header);
const titlePath = join(testDir, "fallback-name.png"); file.metadata.title = "fallback-name.png";
file.metadata.title = titlePath; const dir = mkdtempSync(join(testDir, "title-"));
const api = new ApiClient({ fetch: mockFetchForBody(ciphertext) }); const api = new ApiClient({ fetch: mockFetchForBody(ciphertext) });
const result = await downloadFile(api, file); const result = await inDirectory(dir, () => downloadFile(api, file));
expect(result.path).toBe(titlePath); expect(result.path).toBe("fallback-name.png");
expect(readFileSync(result.path)).toEqual(Buffer.from(plaintext)); expect(readFileSync(join(dir, "fallback-name.png"))).toEqual(
Buffer.from(plaintext),
);
});
it("keeps a hostile title inside the working directory", async () => {
// The server controls the title. `../escaped.png` must not write to
// the parent directory; it becomes one file name in the current one.
const { api, file } = fixtureFor(
multiChunkKey,
multiChunk.header,
multiChunk.body,
);
file.metadata.title = "../escaped.png";
const parent = mkdtempSync(join(testDir, "hostile-"));
const dir = join(parent, "cwd");
mkdirSync(dir);
const result = await inDirectory(dir, () => downloadFile(api, file));
expect(result.path).toBe("__escaped.png");
expect(readdirSync(dir)).toEqual(["__escaped.png"]);
expect(readdirSync(parent)).toEqual(["cwd"]);
});
it("uses an explicit outPath verbatim, even one with ..", async () => {
// The caller is trusted: its path is not sanitized.
const { api, file } = fixtureFor(
multiChunkKey,
multiChunk.header,
multiChunk.body,
);
const dir = mkdtempSync(join(testDir, "explicit-"));
mkdirSync(join(dir, "sub"));
const outPath = join(dir, "sub", "..", "explicit.bin");
const result = await downloadFile(api, file, outPath);
expect(result.path).toBe(outPath);
expect(existsSync(join(dir, "explicit.bin"))).toBe(true);
}); });
it("handles a larger single-chunk file (random binary payload)", async () => { it("handles a larger single-chunk file (random binary payload)", async () => {
@@ -617,6 +668,23 @@ describe("downloadThumbnail", () => {
expect(result).toEqual({ path: outPath, bytesWritten: 4 }); expect(result).toEqual({ path: outPath, bytesWritten: 4 });
expect(readFileSync(outPath)).toEqual(Buffer.from(plaintext)); expect(readFileSync(outPath)).toEqual(Buffer.from(plaintext));
}); });
it("names the thumbnail thumb_ plus the sanitized title", async () => {
const { api, file } = fixtureFor(
multiChunkKey,
multiChunk.header,
multiChunk.body,
);
file.metadata.title = "/etc/passwd";
const dir = mkdtempSync(join(testDir, "thumb-title-"));
const result = await inDirectory(dir, () =>
downloadThumbnail(api, file),
);
expect(result.path).toBe("thumb__etc_passwd");
expect(readdirSync(dir)).toEqual(["thumb__etc_passwd"]);
});
}); });
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
@@ -922,6 +990,51 @@ describe.each(entryPoints)(
expect(readFileSync(outPath)).toEqual(Buffer.from(existing)); expect(readFileSync(outPath)).toEqual(Buffer.from(existing));
expect(readdirSync(dir)).toEqual(["rename-fails.bin"]); expect(readdirSync(dir)).toEqual(["rename-fails.bin"]);
}); });
it("fails without creating anything when the destination directory does not exist", async () => {
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(
patternBytes(64, 33),
key,
);
const { api, file } = fixtureFor(key, header, ciphertext);
const dir = freshDir();
const outPath = join(dir, "missing", "never.bin");
await expect(download(api, file, outPath)).rejects.toMatchObject({
code: "ENOENT",
});
// The missing directory is not created on the caller's behalf.
expect(readdirSync(dir)).toEqual([]);
});
// Root ignores directory permissions, so this cannot fail as root
// (the Docker test image runs as root).
it.skipIf(process.getuid?.() === 0)(
"fails without creating anything when the destination directory is not writable",
async () => {
const key =
sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(
patternBytes(64, 34),
key,
);
const { api, file } = fixtureFor(key, header, ciphertext);
const dir = freshDir();
const outPath = join(dir, "never.bin");
chmodSync(dir, 0o500);
try {
await expect(
download(api, file, outPath),
).rejects.toMatchObject({ code: "EACCES" });
} finally {
chmodSync(dir, 0o700);
}
expect(readdirSync(dir)).toEqual([]);
},
);
}, },
); );
+93
View File
@@ -0,0 +1,93 @@
// File names built from server-supplied metadata.
//
// quak does not trust the server. A file's title and a collection's name are
// decrypted from data the server hands us, and a hostile server (or a
// compromised account) can set them to anything. quak uses them to name files
// on disk: `quak get` without `--out`, `downloadFile` without `outPath`, the
// backup's symlink and collection directories, and the extension of every file
// in the originals cache. Each of those goes through `sanitizeFileName` or
// `safeExtension`, so a title can only ever name one file inside the directory
// the caller chose.
//
// A path the user supplies (`--out`, `outPath`) is never sanitized: the caller
// is trusted, the server is not.
import { describe, expect, it } from "vitest";
import { safeExtension, sanitizeFileName } from "../../src/filename.js";
const FALLBACK = "file-42";
describe("sanitizeFileName", () => {
it("passes a normal title through unchanged", () => {
expect(sanitizeFileName("IMG_0001.HEIC", FALLBACK)).toBe(
"IMG_0001.HEIC",
);
expect(sanitizeFileName("Holiday 2024 (1).jpg", FALLBACK)).toBe(
"Holiday 2024 (1).jpg",
);
expect(sanitizeFileName("café.jpg", FALLBACK)).toBe("café.jpg");
});
it("cannot climb out of the directory with ../", () => {
// Without sanitizing, this would overwrite the user's SSH keys.
expect(sanitizeFileName("../../.ssh/authorized_keys", FALLBACK)).toBe(
"__.._.ssh_authorized_keys",
);
expect(sanitizeFileName("..", FALLBACK)).toBe("_");
expect(sanitizeFileName("..\\..\\x", FALLBACK)).toBe("__.._x");
});
it("cannot name an absolute path", () => {
expect(sanitizeFileName("/etc/passwd", FALLBACK)).toBe("_etc_passwd");
expect(sanitizeFileName("C:\\Windows\\x.dll", FALLBACK)).toBe(
"C__Windows_x.dll",
);
});
it("replaces embedded separators, so the name stays one file", () => {
expect(sanitizeFileName("a/b\\c.jpg", FALLBACK)).toBe("a_b_c.jpg");
});
it("replaces NUL and other control characters", () => {
// A NUL truncates the path in C code and makes Node's fs throw.
expect(sanitizeFileName("evil\0.jpg", FALLBACK)).toBe("evil_.jpg");
expect(sanitizeFileName("line\nbreak\x7f.jpg", FALLBACK)).toBe(
"line_break_.jpg",
);
});
it("does not produce a hidden file", () => {
expect(sanitizeFileName(".bashrc", FALLBACK)).toBe("_bashrc");
});
it("does not produce a Windows device name", () => {
expect(sanitizeFileName("CON", FALLBACK)).toBe("_CON");
expect(sanitizeFileName("nul.txt", FALLBACK)).toBe("_nul.txt");
expect(sanitizeFileName("LPT1", FALLBACK)).toBe("_LPT1");
// Only the exact names are reserved.
expect(sanitizeFileName("console.jpg", FALLBACK)).toBe("console.jpg");
});
it("falls back to the given name for an empty title", () => {
expect(sanitizeFileName("", FALLBACK)).toBe(FALLBACK);
});
});
describe("safeExtension", () => {
it("keeps a normal extension", () => {
expect(safeExtension("IMG_0001.HEIC")).toBe(".HEIC");
expect(safeExtension("clip.mp4")).toBe(".mp4");
});
it("uses .bin when there is no extension", () => {
expect(safeExtension("")).toBe(".bin");
expect(safeExtension("README")).toBe(".bin");
});
it("uses .bin when the extension holds anything but letters and digits", () => {
expect(safeExtension("x.j\\..\\pg")).toBe(".bin");
expect(safeExtension("x.jp g")).toBe(".bin");
expect(safeExtension("x.jpg\0")).toBe(".bin");
});
});
+19
View File
@@ -226,6 +226,25 @@ describe("ContentCache.original / thumbnail", () => {
expect(skips).toEqual(["skipped"]); expect(skips).toEqual(["skipped"]);
}); });
it("takes only a letters-and-digits extension from the title", async () => {
// The title comes from the server; an extension such as `.\..\x`
// must not reach the cache file name, so it becomes `.bin`.
const { cache } = buildCache({
files: [file(1, "a.jpg"), file(2, "b.\\..\\x"), file(3, "")],
});
await cache.open();
expect((await cache.original(1)).path).toBe(
join(cacheDir, "originals", "1.jpg"),
);
expect((await cache.original(2)).path).toBe(
join(cacheDir, "originals", "2.bin"),
);
expect((await cache.original(3)).path).toBe(
join(cacheDir, "originals", "3.bin"),
);
});
it("serves a file already present in the download directory without fetching", async () => { it("serves a file already present in the download directory without fetching", async () => {
const downloadDirectory = join(root, "backup"); const downloadDirectory = join(root, "backup");
mkdirSync(join(downloadDirectory, "originals"), { recursive: true }); mkdirSync(join(downloadDirectory, "originals"), { recursive: true });
+33 -3
View File
@@ -146,10 +146,13 @@ const buildSharedRawCollection = (
const buildRawFile = ( const buildRawFile = (
collectionKey: Uint8Array, collectionKey: Uint8Array,
opts?: { opts?: {
title?: string; // Any JSON value; `undefined` leaves the title out of the metadata.
title?: unknown;
fileType?: number; fileType?: number;
creationTime?: number; creationTime?: number;
info?: { fileSize?: number; thumbSize?: number }; info?: { fileSize?: number; thumbSize?: number };
// Replaces the whole metadata JSON value.
metadata?: unknown;
}, },
): RawEnteFile => { ): RawEnteFile => {
const fileKey = sodium.crypto_secretbox_keygen(); const fileKey = sodium.crypto_secretbox_keygen();
@@ -158,8 +161,8 @@ const buildRawFile = (
collectionKey, collectionKey,
); );
const metadata = { const defaultMetadata = {
title: opts?.title ?? "IMG_0001.jpg", title: opts && "title" in opts ? opts.title : "IMG_0001.jpg",
fileType: opts?.fileType ?? 0, fileType: opts?.fileType ?? 0,
creationTime: opts?.creationTime ?? 1700000000000000, creationTime: opts?.creationTime ?? 1700000000000000,
modificationTime: 1700000000000000, modificationTime: 1700000000000000,
@@ -167,6 +170,8 @@ const buildRawFile = (
longitude: 2.3522, longitude: 2.3522,
hash: "abcdef1234567890", hash: "abcdef1234567890",
}; };
const metadata =
opts && "metadata" in opts ? opts.metadata : defaultMetadata;
// File metadata is encrypted as a single-chunk secretstream blob // File metadata is encrypted as a single-chunk secretstream blob
// (not secretbox). The decryptionHeader is the secretstream init header. // (not secretbox). The decryptionHeader is the secretstream init header.
const metadataBytes = new TextEncoder().encode(JSON.stringify(metadata)); const metadataBytes = new TextEncoder().encode(JSON.stringify(metadata));
@@ -321,6 +326,31 @@ describe("model.decryptFile", () => {
expect(file.metadata.longitude).toBeCloseTo(2.3522); expect(file.metadata.longitude).toBeCloseTo(2.3522);
}); });
it("reads a missing or non-string title as an empty string", () => {
// The server controls the metadata JSON. A title that is not a
// string must not reach code that builds file names from it.
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
for (const title of [undefined, null, 42, ["a"], { x: "../y" }]) {
const file = decryptFile(
buildRawFile(collectionKey, { title }),
collectionKey,
);
expect(file.metadata.title).toBe("");
}
});
it("rejects metadata that is not a JSON object", () => {
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
for (const metadata of [null, "IMG_0001.jpg", 7, []]) {
const raw = buildRawFile(collectionKey, { metadata });
expect(() => decryptFile(raw, collectionKey)).toThrow(
"file 200: metadata is not a JSON object",
);
}
});
it("maps fileType numbers to FileType strings", () => { it("maps fileType numbers to FileType strings", () => {
// Ente uses: 0=image, 1=video, 2=livePhoto // Ente uses: 0=image, 1=video, 2=livePhoto
const masterKey = sodium.crypto_secretbox_keygen(); const masterKey = sodium.crypto_secretbox_keygen();
+50
View File
@@ -0,0 +1,50 @@
// A checkout nested under `.claude/` must not add its tests to this suite.
// The test plants one in a temporary directory next to a real test file and
// asks vitest, with this repo's config, which test files it would run.
import { afterEach, describe, expect, it } from "vitest";
import { execFileSync } from "node:child_process";
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
let root = "";
afterEach(() => {
rmSync(root, { recursive: true, force: true });
});
const writeTest = (path: string): void => {
mkdirSync(join(root, path, ".."), { recursive: true });
writeFileSync(
join(root, path),
'import { it } from "vitest";\nit("runs", () => {});\n',
);
};
describe("vitest.config.ts", () => {
it("does not collect tests from a checkout nested under .claude/", () => {
root = mkdtempSync(join(tmpdir(), "quak-nested-checkout-"));
writeTest("test/real.test.ts");
writeTest(".claude/worktrees/other/test/real.test.ts");
const output = execFileSync(
process.execPath,
[
join(repoRoot, "node_modules/vitest/vitest.mjs"),
"list",
"--filesOnly",
"--config",
join(repoRoot, "vitest.config.ts"),
"--root",
root,
],
{ cwd: root, encoding: "utf-8" },
);
const files = output.split("\n").filter((line) => line !== "");
expect(files).toEqual(["test/real.test.ts"]);
});
});
+75 -4
View File
@@ -149,8 +149,11 @@ describe("isRetryable: transport failures", () => {
}); });
it("retries an errno carried on the error itself", () => { it("retries an errno carried on the error itself", () => {
// Every errno the classifier names, so none can be reclassified
// unnoticed.
for (const code of [ for (const code of [
"ECONNRESET", "ECONNRESET",
"ECONNABORTED",
"ETIMEDOUT", "ETIMEDOUT",
"EPIPE", "EPIPE",
"ENOTFOUND", "ENOTFOUND",
@@ -158,6 +161,8 @@ describe("isRetryable: transport failures", () => {
"ECONNREFUSED", "ECONNREFUSED",
"EHOSTUNREACH", "EHOSTUNREACH",
"ENETUNREACH", "ENETUNREACH",
"ENETRESET",
"ENETDOWN",
]) { ]) {
expect(isRetryable(errnoError(code))).toBe(true); expect(isRetryable(errnoError(code))).toBe(true);
} }
@@ -215,6 +220,27 @@ describe("isRetryable: transport failures", () => {
looped.cause = looped; looped.cause = looped;
expect(isRetryable(looped)).toBe(false); expect(isRetryable(looped)).toBe(false);
}); });
it("terminates on a cause chain that loops through two errors", () => {
const first: Error & { cause?: unknown } = new Error("first");
const second = new Error("second", { cause: first });
first.cause = second;
expect(isRetryable(first)).toBe(false);
});
it("reads the error and at most seven causes below it", () => {
// The walk is bounded at eight links. An errno at the eighth link is
// found; one at the ninth is not.
const buried = (causes: number): Error => {
let err = errnoError("ECONNRESET");
for (let i = 0; i < causes; i++) {
err = new Error(`wrapper ${i}`, { cause: err });
}
return err;
};
expect(isRetryable(buried(7))).toBe(true);
expect(isRetryable(buried(8))).toBe(false);
});
}); });
describe("isRetryable: stream truncation versus corruption", () => { describe("isRetryable: stream truncation versus corruption", () => {
@@ -279,10 +305,9 @@ describe("isSafeToReplay", () => {
* that is not the whole question: the other half is "could the first * that is not the whole question: the other half is "could the first
* attempt already have taken effect on the server?". * attempt already have taken effect on the server?".
* *
* quak's non-idempotent calls are `/users/srp/create-session`, * The calls this guards are the `POST` and `PUT` requests listed in the
* `/users/two-factor/verify` (which consumes one of a limited number of * README under "Endpoints used". A blind replay of some of them can do
* 2FA attempts) and `/files/thumbnail`. A blind replay of any of them can * real damage, so they retry only on the failures that establish no TCP
* do real damage, so they retry only on the failures that establish no TCP
* connection to the server ever existed — DNS produced no address, or the * connection to the server ever existed — DNS produced no address, or the
* peer refused the connection — and therefore that no request byte can * peer refused the connection — and therefore that no request byte can
* have been transmitted. * have been transmitted.
@@ -333,6 +358,52 @@ describe("isSafeToReplay", () => {
).toBe(false); ).toBe(false);
expect(isSafeToReplay(new TypeError("fetch failed"))).toBe(false); expect(isSafeToReplay(new TypeError("fetch failed"))).toBe(false);
}); });
it("does not replay any other errno the classifier names", () => {
for (const code of [
"ECONNRESET",
"ECONNABORTED",
"ETIMEDOUT",
"EPIPE",
"EHOSTUNREACH",
"ENETUNREACH",
"ENETRESET",
"ENETDOWN",
]) {
expect(isSafeToReplay(errnoError(code))).toBe(false);
}
});
it("does not replay a chain that also shows the request may have gone out", () => {
// A connect errno somewhere in the chain is not enough: any other
// errno beside it is doubt, and doubt is not replayed.
const reset = Object.assign(
new Error("read ECONNRESET", { cause: errnoError("ECONNREFUSED") }),
{ code: "ECONNRESET" },
);
const mixed = new TypeError("fetch failed", { cause: reset });
expect(isRetryable(mixed)).toBe(true);
expect(isSafeToReplay(mixed)).toBe(false);
});
it("does not replay a chain longer than the walk reads", () => {
// Eight connect errnos, then a reset at the ninth link, below the
// limit. The walk never sees the reset, so it cannot rule it out.
const refusedChain = (below: Error | undefined): Error => {
let err = below;
for (let i = 0; i < 8; i++) {
err = Object.assign(new Error(`refused ${i}`, { cause: err }), {
code: "ECONNREFUSED",
});
}
return err as Error;
};
expect(isSafeToReplay(refusedChain(errnoError("ECONNRESET")))).toBe(
false,
);
// The same eight links with nothing below them are replayable.
expect(isSafeToReplay(refusedChain(undefined))).toBe(true);
});
}); });
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
+10
View File
@@ -0,0 +1,10 @@
import { configDefaults, defineConfig } from "vitest/config";
// vitest does not read .gitignore when looking for tests. A checkout nested
// under .claude/ has its own test/ tree, and without this exclude the suite
// runs once per nested checkout and still reports success.
export default defineConfig({
test: {
exclude: [...configDefaults.exclude, ".claude/**"],
},
});
+1 -8
View File
@@ -528,13 +528,6 @@
resolved "https://registry.yarnpkg.com/@types/json-schema/-/json-schema-7.0.15.tgz#596a1747233694d50f6ad8a7869fcb6f56cf5841" resolved "https://registry.yarnpkg.com/@types/json-schema/-/json-schema-7.0.15.tgz#596a1747233694d50f6ad8a7869fcb6f56cf5841"
integrity sha512-5+fP8P8MFNC+AyZCDxrB2pkZFPGzqQWUzpSeuuVLvm8VMcorNYavBqoFcxK8bQz4Qsbn4oUEEem4wDLfcysGHA== integrity sha512-5+fP8P8MFNC+AyZCDxrB2pkZFPGzqQWUzpSeuuVLvm8VMcorNYavBqoFcxK8bQz4Qsbn4oUEEem4wDLfcysGHA==
"@types/libsodium-wrappers-sumo@0.8.2":
version "0.8.2"
resolved "https://registry.yarnpkg.com/@types/libsodium-wrappers-sumo/-/libsodium-wrappers-sumo-0.8.2.tgz#488e8747fbb982fe901020b5afeaddfa63da6830"
integrity sha512-uFOBpg/r21hExVlh2ty8YpDfSR+Yy3Jn8XS4+SSjitbhTxdYq+pBz/49XRxyUFe8SzqujHf/Wu0/O4d+FUtNfQ==
dependencies:
libsodium-wrappers-sumo "*"
"@types/node@22.18.13": "@types/node@22.18.13":
version "22.18.13" version "22.18.13"
resolved "https://registry.yarnpkg.com/@types/node/-/node-22.18.13.tgz#a037c4f474b860be660e05dbe92a9ef945472e28" resolved "https://registry.yarnpkg.com/@types/node/-/node-22.18.13.tgz#a037c4f474b860be660e05dbe92a9ef945472e28"
@@ -1249,7 +1242,7 @@ libsodium-sumo@^0.8.0:
resolved "https://registry.yarnpkg.com/libsodium-sumo/-/libsodium-sumo-0.8.4.tgz#6d4687781fa0ad398af14a7df872d5c27cf8cd31" resolved "https://registry.yarnpkg.com/libsodium-sumo/-/libsodium-sumo-0.8.4.tgz#6d4687781fa0ad398af14a7df872d5c27cf8cd31"
integrity sha512-TMtHShQfVVsaxDygyapvUC3o7YsPgXa/hRWeIgzyFz6w5k/1hirGptCxp1U7XwW3rCskaTTYKgV10v86UiGgNw== integrity sha512-TMtHShQfVVsaxDygyapvUC3o7YsPgXa/hRWeIgzyFz6w5k/1hirGptCxp1U7XwW3rCskaTTYKgV10v86UiGgNw==
libsodium-wrappers-sumo@*, libsodium-wrappers-sumo@0.8.4: libsodium-wrappers-sumo@0.8.4:
version "0.8.4" version "0.8.4"
resolved "https://registry.yarnpkg.com/libsodium-wrappers-sumo/-/libsodium-wrappers-sumo-0.8.4.tgz#6656a3e7e0551ecce08ddee4bfb501a092eac6fa" resolved "https://registry.yarnpkg.com/libsodium-wrappers-sumo/-/libsodium-wrappers-sumo-0.8.4.tgz#6656a3e7e0551ecce08ddee4bfb501a092eac6fa"
integrity sha512-ql7hcgulKZ3ekfa2DGAogcCKsWU0diA/0nArz1CFzh93WQdb46/Kj18ka/Hifq6uA3Ush34Pc6vU/6HXeRwUkg== integrity sha512-ql7hcgulKZ3ekfa2DGAogcCKsWU0diA/0nArz1CFzh93WQdb46/Kj18ka/Hifq6uA3Ush34Pc6vU/6HXeRwUkg==