Compare commits
15
Commits
next
..
59e65e2a41
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
59e65e2a41 | ||
|
|
d05b53d560 | ||
|
|
0ca8887f52 | ||
|
|
f1836ced57 | ||
|
|
c75c4f987c | ||
|
|
28a2beeab8 | ||
|
|
2b410c3ed6 | ||
|
|
d07692897b | ||
|
|
ed535be1da | ||
|
|
d545dcd8b1 | ||
|
|
52f58f5d2b | ||
|
|
b44c4ba6d7 | ||
|
|
d50b296d3a | ||
|
|
b7d6ab99f4 | ||
|
|
3871d6228e |
@@ -73,7 +73,7 @@ if (photo) {
|
||||
console.log(`original at ${path}`);
|
||||
}
|
||||
|
||||
lib.close();
|
||||
await lib.close();
|
||||
```
|
||||
|
||||
The lower-level `Client` (login, session serialization, and the raw
|
||||
@@ -323,6 +323,7 @@ Endpoints used:
|
||||
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a
|
||||
collection; paginate while `hasMore` is true.
|
||||
- `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes.
|
||||
- `POST /files/data/fetch`: fetch encrypted ML data for a batch of files.
|
||||
- `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair).
|
||||
- `PUT /files/thumbnail`: register an uploaded thumbnail's object key.
|
||||
|
||||
@@ -362,35 +363,41 @@ and a half seconds of waiting. `sleep` and `random` are injectable through the
|
||||
same option, which is how the test suite exercises the whole policy without
|
||||
waiting.
|
||||
|
||||
Two deadlines, applied with `AbortSignal.timeout()` and renewed for each
|
||||
attempt:
|
||||
Two deadlines, renewed for each attempt:
|
||||
|
||||
| Option | Default | Applies to |
|
||||
| ------------------- | -------- | ------------------------------------------- |
|
||||
| `requestTimeoutMs` | `30000` | `getJSON`, `postJSON`, `putJSON`, `putFile` |
|
||||
| `downloadTimeoutMs` | `600000` | file and thumbnail body transfers |
|
||||
| Option | Default | Applies to | Kind |
|
||||
| ------------------- | ------- | ------------------------------------------- | ------------------------------------- |
|
||||
| `requestTimeoutMs` | `30000` | `getJSON`, `postJSON`, `putJSON`, `putFile` | the whole request |
|
||||
| `downloadTimeoutMs` | `60000` | file and thumbnail downloads | idle: no bytes received for this long |
|
||||
|
||||
They are separate because one number cannot serve both: a value short enough to
|
||||
keep a hung API call from stalling a backup would cancel a legitimate
|
||||
multi-gigabyte download. The download deadline covers the body, not just the
|
||||
headers — `getFileStream` returns as soon as headers arrive, so a deadline that
|
||||
only guarded the initial request would leave the same hang one layer down.
|
||||
They are different kinds because a download's length depends on the file and the
|
||||
link: a whole-transfer deadline short enough to catch a hung connection would
|
||||
cancel a large video on a slow link that is still making progress. The download
|
||||
deadline restarts every time bytes arrive, so a slow download runs as long as it
|
||||
keeps moving, and one that stalls is aborted after 60 seconds of silence. It
|
||||
covers the wait for the headers and the body — `getFileStream` returns as soon
|
||||
as headers arrive, so a deadline that only guarded the initial request would
|
||||
leave the same hang one layer down. There is no limit on the total length of a
|
||||
download.
|
||||
|
||||
**Non-idempotent requests are not blindly replayed.** `postJSON` and `putJSON`
|
||||
reach `/users/srp/create-session`, `/users/two-factor/verify` — which consumes
|
||||
one of a small number of second-factor attempts — and `/files/thumbnail`. They
|
||||
are retried only on the three failures that establish no TCP connection to the
|
||||
server ever existed, so no request byte can have been transmitted: `ENOTFOUND`
|
||||
and `EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the
|
||||
peer refused the connection). A 5xx, a mid-flight reset and a deadline are all
|
||||
left to the caller, because each of them can happen after the server has already
|
||||
acted. The routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are
|
||||
excluded for the same reason, despite looking like connect-time failures: on
|
||||
Linux an ICMP unreachable arriving mid-flight, or a local interface going down
|
||||
after the request was written, delivers them on an already-established socket.
|
||||
They stay retryable for the idempotent calls. `putFile` is exempt: a presigned
|
||||
PUT stores one whole object at one key in one request, so replaying it has no
|
||||
partial state to damage.
|
||||
send every `POST` and `PUT` in the endpoint list above; some of them change
|
||||
server state, and `/users/two-factor/verify` consumes one of a small number of
|
||||
second-factor attempts. They are retried only when every errno in the error's
|
||||
`cause` chain is one of the three that establish no TCP connection to the server
|
||||
ever existed, so no request byte can have been transmitted: `ENOTFOUND` and
|
||||
`EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer
|
||||
refused the connection). A 5xx, a mid-flight reset and a deadline are all left
|
||||
to the caller, because each of them can happen after the server has already
|
||||
acted. These two do not follow redirects either: a redirect means the server
|
||||
already received the request, so it is reported as an error and not retried. The
|
||||
routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the
|
||||
same reason, despite looking like connect-time failures: on Linux an ICMP
|
||||
unreachable arriving mid-flight, or a local interface going down after the
|
||||
request was written, delivers them on an already-established socket. They stay
|
||||
retryable for the idempotent calls. `putFile` is exempt: a presigned PUT stores
|
||||
one whole object at one key in one request, so replaying it has no partial state
|
||||
to damage.
|
||||
|
||||
A download is retried as a whole — request, stream consumption, and decryption —
|
||||
because a socket reset after the response headers have arrived surfaces in the
|
||||
@@ -422,13 +429,18 @@ decides how to persist sessions.
|
||||
`client.toJSON()` returns a `ClientSnapshot` (a plain serializable object with
|
||||
base64-encoded keys) that the consumer can write to disk, a database, or
|
||||
whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
|
||||
working client from that snapshot without re-authenticating.
|
||||
working client from that snapshot without re-authenticating; it checks every
|
||||
field and each key's length first, and throws an error naming the bad field.
|
||||
`client.logout()` clears the token and zeroes the key buffers in place; every
|
||||
later call on that client throws.
|
||||
|
||||
The CLI stores the snapshot at the platform-appropriate data directory via
|
||||
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
|
||||
`$XDG_DATA_HOME/quak/session.json` on Linux. The file is written with mode
|
||||
`0600`. The key material is stored in cleartext in the JSON; treat this file as
|
||||
you would treat the password itself.
|
||||
you would treat the password itself. A missing file is reported as "not logged
|
||||
in"; a file that exists but is corrupt is reported as such, naming the bad
|
||||
field. Both exit with status 1.
|
||||
|
||||
### CLI surface
|
||||
|
||||
@@ -485,6 +497,16 @@ appears in. On subsequent runs, existing originals are skipped. If a download
|
||||
fails, the error is logged and the backup continues with the next file. The exit
|
||||
code is non-zero if any files failed.
|
||||
|
||||
Each original is copied to a temporary file named
|
||||
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp` in the same directory, synced
|
||||
to disk, and renamed into place, so an original is either complete or absent,
|
||||
even after a power cut. A run that is killed can leave one of these temporary
|
||||
files behind; the next backup deletes those whose process is no longer running.
|
||||
Downloads and the content cache use the same scheme with `.quak-<random>.tmp`
|
||||
names. The rename replaces whatever was at the destination rather than writing
|
||||
through it: a symlink there is replaced, not followed, and the new file has the
|
||||
temporary file's permissions, not those of the file it replaced.
|
||||
|
||||
## TODO
|
||||
|
||||
- [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
|
||||
@@ -541,7 +563,10 @@ and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).
|
||||
`lib.status()` returns a `LibraryStatus` (collection/file counts, last
|
||||
refresh/ML times and errors, originals usage and effective limit, precache
|
||||
progress, and `closed`). `lib.close()` stops the background timer; it is
|
||||
idempotent, and an in-flight refresh is left to finish.
|
||||
idempotent, and an in-flight refresh is left to finish. The promise it returns
|
||||
resolves once that refresh (including its cache write), the ML data fetch and
|
||||
the precache fetches already running have all finished, so the cache directory
|
||||
can then be removed.
|
||||
|
||||
### Default reads vs. fresh reads
|
||||
|
||||
@@ -652,10 +677,13 @@ Under `cacheDirectory`:
|
||||
```
|
||||
|
||||
A stored file appears only via an atomic temp-then-rename, so its presence means
|
||||
it is complete. The design also calls for a content-hash comparison against
|
||||
`FileMetadata.hash` on each fetched original; that check is deferred (issue
|
||||
https://git.eeqj.de/sneak/quak/issues/68) because the exact hash construction
|
||||
cannot yet be confirmed against the repo's fixtures.
|
||||
it is complete. Every downloaded original (by `quak get`, the cache, or
|
||||
`backup`) whose metadata records a content hash (`FileMetadata.hash`) is hashed
|
||||
as it is written: unkeyed BLAKE2b with a 64-byte output, standard base64. For a
|
||||
live photo, which is stored as a ZIP, the image and the video are hashed
|
||||
separately and joined as `<imageHash>:<videoHash>`. A mismatch stores nothing
|
||||
and fails the download with an error naming the file ID. An original with no
|
||||
recorded hash, from a very old client, is stored unchecked.
|
||||
|
||||
### Key types by source file
|
||||
|
||||
|
||||
@@ -18,6 +18,102 @@ Tag v1.0.0.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-09-23: Checked downloaded originals against their recorded content hash
|
||||
(issue 68). `downloadFile`, which `quak get`, the content cache and backup all
|
||||
use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and
|
||||
stores nothing on a mismatch, failing with an error naming the file ID. A live
|
||||
photo ZIP is unpacked as it streams with `fflate` and its image and video
|
||||
hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older
|
||||
clients' `imageHash` and `videoHash` fields for live photos. A file with no
|
||||
recorded hash is stored unchecked.
|
||||
- 2026-09-23: Single-sourced the version string (issue 5). `package.json` is the
|
||||
only place it is written: `src/index.ts` imports it for `VERSION` and
|
||||
`bin/quak.ts` passes `VERSION` to commander. tsc copies `package.json` to
|
||||
`dist/package.json`, so the import resolves from the built output too, and
|
||||
`script/build` runs the built CLI with `--version` to prove it. A test checks
|
||||
that `VERSION` and `quak --version` both equal the `package.json` version.
|
||||
- 2026-09-23: Tested that `Library.close()` waits for the originals precache
|
||||
(issue 93). The test that holds a precache fetch open while `close()` runs now
|
||||
runs once with only the thumbnail fill and once with only the originals fill,
|
||||
so dropping either wait from `Precache.close()` fails a test.
|
||||
- 2026-09-23: Fixed two intermittently failing library tests (issue 90).
|
||||
`Library.close()` now returns a promise that resolves once an in-flight
|
||||
refresh (including its cache write), the ML data fetch and running precache
|
||||
sweeps have finished; the library tests await it, so `afterEach` no longer
|
||||
removes the cache directory while something is still writing into it. The
|
||||
precache test waits for both fills to report "done" instead of for its stub
|
||||
source to be called, which happened before the cache recorded the file.
|
||||
- 2026-09-23: Pinned three guards reviewers found untested (issue 89). The
|
||||
download idle deadline's timer is unref'd, so it can never keep the process
|
||||
alive, and a test checks no timer is left after a download completes or fails.
|
||||
A test covers the rejection of `#` in a request path. The EXIF scan compares
|
||||
the `Exif` header only in an APP1 segment of length 8 or more, so it never
|
||||
reads the next segment's bytes, with a test for a short one.
|
||||
- 2026-09-23: Made the download deadline an idle deadline (issue 24).
|
||||
`downloadTimeoutMs` now aborts a file or thumbnail download only after no
|
||||
bytes have arrived for that long, default 60 seconds, instead of bounding the
|
||||
whole transfer at 10 minutes, so a slow download that keeps making progress
|
||||
completes. A download that fails before reading the whole body cancels it, so
|
||||
a failed file no longer holds its connection.
|
||||
- 2026-09-23: Every `ApiClient` request URL is now built by one function next to
|
||||
the class (issue 18), so a self-hosted `apiOrigin` with a base path keeps it
|
||||
on every request, a path works with or without a leading slash, and query
|
||||
parameters are percent-encoded. A path containing `?` or `#` is rejected with
|
||||
an error instead of being silently cut.
|
||||
- 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies
|
||||
moved from `bin/quak.ts` into `src/cli-commands.ts` as functions that take
|
||||
their options and a context (output streams, session directory, cache
|
||||
directory, session loader) and return an exit code; `bin/quak.ts` only wires
|
||||
them to commander and exits with the code once stdout and stderr have drained,
|
||||
so nothing below it calls `process.exit`. `test/cli/commands.test.ts` drives
|
||||
them with a fake client: session file modes, logout, the missing and corrupt
|
||||
session paths, and the output and exit code of `whoami`, `collections`,
|
||||
`files`, `get`, `get-thumb`, `backup` and `helper list-missing-thumbnails`.
|
||||
- 2026-09-23: Hardened the backup tree's atomic copy (issue 22). `copyAtomic`
|
||||
fsyncs its temp file before the rename and the directory after it, through the
|
||||
download writer's `fsyncPath`; each backup run deletes `.quak-backup-*.tmp`
|
||||
files whose process is no longer running. The README backup layout names the
|
||||
temp files and states that the rename replaces a symlink and takes the temp
|
||||
file's permissions. Added tests for a missing and an unwritable destination
|
||||
directory for `downloadFile` and `downloadThumbnail`.
|
||||
- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue
|
||||
11). Every segment length is checked against the remaining bytes and lengths
|
||||
under 2 stop the scan, so a truncated or corrupt original can neither throw
|
||||
nor loop. A malformed or unparseable EXIF segment is recorded as
|
||||
`imageMetadata.exifError`, and a failure to read the original as
|
||||
`imageMetadataError` in the per-file JSON, instead of the field being left
|
||||
out.
|
||||
- 2026-09-22: Hardened the retry classifier (issue 80). A `POST` or `PUT` is
|
||||
replayed only when every errno in the cause chain is a connect errno, and it
|
||||
no longer follows redirects. `getRetryOptions()` returns a copy. Tests pin
|
||||
every errno the classifier names, the cause-chain depth limit, cycle
|
||||
termination, and a fresh deadline per attempt for every retrying entry point.
|
||||
The README's endpoint list is the one place that names the requests the replay
|
||||
rule covers.
|
||||
- 2026-09-22: Stopped `make test` collecting tests from checkouts nested under
|
||||
`.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a
|
||||
nested checkout ran the whole suite again; `vitest.config.ts` now adds
|
||||
`.claude/**` to vitest's default excludes, and
|
||||
`test/packaging/nested-checkout.test.ts` plants a nested checkout in a temp
|
||||
directory and fails if vitest would collect it.
|
||||
- 2026-09-22: Dropped the deprecated `@types/libsodium-wrappers-sumo` stub from
|
||||
`devDependencies` (issue 27). It shipped no declarations; the types come from
|
||||
`libsodium-wrappers-sumo` itself. `yarn.lock` regenerated by `yarn remove`.
|
||||
- 2026-09-22: Hardened the client session lifecycle (issue 10).
|
||||
`Client.fromJSON` checks every snapshot field and each key's decoded length
|
||||
and names the bad field; `toJSON` reads the token through
|
||||
`ApiClient.getAuthToken` and throws when there is none; `logout` zeroes the
|
||||
key buffers, and `collectionsSince` re-checks for logout after its request so
|
||||
it never decrypts with zeroed keys. The CLI reports a corrupt session file
|
||||
separately from a missing one (`src/cli-session.ts`).
|
||||
- 2026-09-22: Sanitized file names taken from server metadata (issue 9). A new
|
||||
`src/filename.ts` holds the one sanitizer, used by `quak get`/`get-thumb`
|
||||
without `--out`, `downloadFile`/`downloadThumbnail` without `outPath`, and the
|
||||
backup and metadata backup trees; it removes separators, control characters,
|
||||
leading dots and Windows device names, and falls back to a name built from the
|
||||
ID for an empty title. Originals-cache extensions are letters and digits only,
|
||||
else `.bin`. A user-supplied path is used as is. `decryptFile` reads a missing
|
||||
or non-string title as "" and rejects metadata that is not a JSON object.
|
||||
- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
|
||||
snippets) to match the shipped cache/API library on `next` (issue 53, issue
|
||||
13). Documented `Library.open` and its options, the default-read vs `fresh()`
|
||||
|
||||
+51
-381
@@ -1,125 +1,62 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { input, password as passwordPrompt } from "@inquirer/prompts";
|
||||
import { stdout, stderr } from "node:process";
|
||||
import {
|
||||
copyFileSync,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
readFileSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { Command } from "commander";
|
||||
import envPaths from "env-paths";
|
||||
import { Client, type ClientSnapshot } from "../src/client.js";
|
||||
import { init } from "../src/crypto/index.js";
|
||||
import { Library, type LibraryClient } from "../src/library/index.js";
|
||||
import {
|
||||
fileListRow,
|
||||
fileListLine,
|
||||
originalName,
|
||||
thumbnailName,
|
||||
} from "../src/cli-output.js";
|
||||
import { freshCollections, freshFiles, freshFile } from "../src/cli-read.js";
|
||||
import { runMetadataBackup } from "../src/metadata-backup.js";
|
||||
import {
|
||||
listMissingThumbnails,
|
||||
fixMissingThumbnails,
|
||||
} from "../src/thumbnails.js";
|
||||
type CliContext,
|
||||
loginCommand,
|
||||
whoamiCommand,
|
||||
logoutCommand,
|
||||
collectionsCommand,
|
||||
filesCommand,
|
||||
getCommand,
|
||||
getThumbCommand,
|
||||
backupMetadataCommand,
|
||||
backupCommand,
|
||||
listMissingThumbnailsCommand,
|
||||
fixMissingThumbnailsCommand,
|
||||
} from "../src/cli-commands.js";
|
||||
import { loadSession } from "../src/cli-session.js";
|
||||
import { VERSION } from "../src/index.js";
|
||||
|
||||
const paths = envPaths("quak", { suffix: "" });
|
||||
const sessionPath = join(paths.data, "session.json");
|
||||
|
||||
const loadSession = (): ClientSnapshot | null => {
|
||||
if (!existsSync(sessionPath)) return null;
|
||||
try {
|
||||
return JSON.parse(readFileSync(sessionPath, "utf-8")) as ClientSnapshot;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
};
|
||||
|
||||
const saveSession = (snapshot: ClientSnapshot): void => {
|
||||
mkdirSync(paths.data, { recursive: true, mode: 0o700 });
|
||||
writeFileSync(sessionPath, JSON.stringify(snapshot, null, 2), {
|
||||
mode: 0o600,
|
||||
});
|
||||
};
|
||||
|
||||
const requireSession = (): Client => {
|
||||
const snapshot = loadSession();
|
||||
if (!snapshot) {
|
||||
stderr.write(
|
||||
`Not logged in. Run "quak login" first.\nSession file: ${sessionPath}\n`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
return Client.fromJSON(snapshot);
|
||||
};
|
||||
|
||||
const prompt = async (message: string): Promise<string> => input({ message });
|
||||
|
||||
const promptSecret = async (message: string): Promise<string> =>
|
||||
passwordPrompt({ message, mask: true });
|
||||
|
||||
const program = new Command();
|
||||
|
||||
program
|
||||
.name("quak")
|
||||
.description("CLI for the Ente end-to-end encrypted photo service")
|
||||
.version("0.0.0")
|
||||
.version(VERSION)
|
||||
.option(
|
||||
"--cache-dir <path>",
|
||||
"Directory for the local metadata/content cache " +
|
||||
"(default: the per-user cache directory)",
|
||||
);
|
||||
|
||||
// The `--cache-dir` global, or undefined to let the library pick its per-user
|
||||
// default keyed by the account id.
|
||||
const cacheDirOption = (): string | undefined =>
|
||||
program.opts<{ cacheDir?: string }>().cacheDir;
|
||||
|
||||
// A library client that omits `fetchMLData`, so the point commands below do not
|
||||
// kick the library's background ML backfill: they read metadata, or fetch one
|
||||
// file's content, and exit. `backup` and `backup-metadata` handle ML on their
|
||||
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
|
||||
// originals through the on-disk cache.
|
||||
const readLibraryClient = (client: Client): LibraryClient => ({
|
||||
whoami: () => client.whoami(),
|
||||
collectionsSince: (args) => client.collectionsSince(args),
|
||||
filesSince: (args) => client.filesSince(args),
|
||||
contentSource: () => client.contentSource(),
|
||||
const context = (): CliContext => ({
|
||||
stdout,
|
||||
stderr,
|
||||
sessionDir: paths.data,
|
||||
cacheDir: program.opts<{ cacheDir?: string }>().cacheDir,
|
||||
loadSession,
|
||||
});
|
||||
|
||||
// Open a library for a single point command: the aggressive background precache
|
||||
// (issue #48) is off — a one-shot `collections` or `get` must not start
|
||||
// downloading the whole account — and the refresh interval is long so no second
|
||||
// refresh fires mid-command.
|
||||
const openReadLibrary = (client: Client): Promise<Library> =>
|
||||
Library.open({
|
||||
client: readLibraryClient(client),
|
||||
cacheDirectory: cacheDirOption(),
|
||||
refreshIntervalSeconds: 3600,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
|
||||
// Close the library and exit once stdout/stderr have drained. `process.exit`
|
||||
// alone can truncate buffered piped output, and the library keeps the event
|
||||
// loop alive with a background refresh, so a plain return could hang; this does
|
||||
// neither.
|
||||
const finish = (lib: Library | undefined, code: number): void => {
|
||||
lib?.close();
|
||||
// Run a command and exit with its code once stdout/stderr have drained.
|
||||
// Exiting before the drain can truncate piped output, and the library can keep
|
||||
// the event loop alive after a command returns, so a plain return could hang.
|
||||
const run = async (command: Promise<number>): Promise<void> => {
|
||||
process.exitCode = await command;
|
||||
const pending = [stdout, stderr].filter((s) => s.writableLength > 0);
|
||||
if (pending.length === 0) {
|
||||
process.exit(code);
|
||||
process.exit();
|
||||
return;
|
||||
}
|
||||
let remaining = pending.length;
|
||||
for (const s of pending) {
|
||||
s.once("drain", () => {
|
||||
if (--remaining === 0) process.exit(code);
|
||||
if (--remaining === 0) process.exit();
|
||||
});
|
||||
}
|
||||
};
|
||||
@@ -127,92 +64,25 @@ const finish = (lib: Library | undefined, code: number): void => {
|
||||
program
|
||||
.command("login")
|
||||
.description("Log in to an Ente account and save the session")
|
||||
.action(async () => {
|
||||
await init();
|
||||
const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
|
||||
const password =
|
||||
process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
|
||||
|
||||
stderr.write("Authenticating...\n");
|
||||
try {
|
||||
const client = await Client.login({
|
||||
email,
|
||||
password,
|
||||
totp: async () => prompt("TOTP code: "),
|
||||
emailOTP: async () => prompt("Email verification code: "),
|
||||
});
|
||||
|
||||
saveSession(client.toJSON());
|
||||
const info = client.whoami();
|
||||
stderr.write(`Logged in as ${info.email} (user ${info.userID})\n`);
|
||||
stderr.write(`Session saved to ${sessionPath}\n`);
|
||||
} catch (err) {
|
||||
stderr.write(
|
||||
`Login failed: ${err instanceof Error ? err.message : err}\n`,
|
||||
);
|
||||
process.exit(1);
|
||||
}
|
||||
});
|
||||
.action(() => run(loginCommand(context())));
|
||||
|
||||
program
|
||||
.command("whoami")
|
||||
.description("Print the logged-in account")
|
||||
.action(() => {
|
||||
const client = requireSession();
|
||||
const info = client.whoami();
|
||||
stdout.write(JSON.stringify(info) + "\n");
|
||||
});
|
||||
.action(() => run(whoamiCommand(context())));
|
||||
|
||||
program
|
||||
.command("logout")
|
||||
.description("Delete the saved session")
|
||||
.action(async () => {
|
||||
if (existsSync(sessionPath)) {
|
||||
const { unlinkSync } = await import("node:fs");
|
||||
unlinkSync(sessionPath);
|
||||
stderr.write("Session deleted.\n");
|
||||
} else {
|
||||
stderr.write("No session found.\n");
|
||||
}
|
||||
});
|
||||
.action(() => run(logoutCommand(context())));
|
||||
|
||||
program
|
||||
.command("collections")
|
||||
.description("List all collections (albums)")
|
||||
.option("--json", "Output as JSON array")
|
||||
.action(async (opts: { json?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const lib = await openReadLibrary(client);
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52): the pre-library CLI printed current state in
|
||||
// this order, not the albums projection's newest-first order.
|
||||
const collections = await freshCollections(lib);
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(
|
||||
JSON.stringify(
|
||||
collections.map((c) => ({
|
||||
id: c.id,
|
||||
name: c.name,
|
||||
type: c.type,
|
||||
ownerID: c.ownerID,
|
||||
isShared: c.isShared,
|
||||
updationTime: c.updationTime,
|
||||
})),
|
||||
null,
|
||||
2,
|
||||
) + "\n",
|
||||
.action((opts: { json?: boolean }) =>
|
||||
run(collectionsCommand(context(), opts)),
|
||||
);
|
||||
} else {
|
||||
for (const c of collections) {
|
||||
stdout.write(
|
||||
`${c.id}\t${c.type}\t${c.name}${c.isShared ? " (shared)" : ""}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
finish(lib, 0);
|
||||
});
|
||||
|
||||
program
|
||||
.command("files")
|
||||
@@ -222,39 +92,9 @@ program
|
||||
"Collection ID (from `quak collections`)",
|
||||
)
|
||||
.option("--json", "Output as JSON array")
|
||||
.action(async (opts: { collection: string; json?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const collectionID = Number(opts.collection);
|
||||
if (!Number.isFinite(collectionID)) {
|
||||
stderr.write("Invalid collection ID\n");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(client);
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52). Each file prints from its own decrypted
|
||||
// metadata (raw title, microsecond creationTime) via cli-output, and in
|
||||
// the pre-library CLI's enumeration order, not the projection's
|
||||
// newest-first order.
|
||||
const files = await freshFiles(lib, collectionID);
|
||||
if (!files) {
|
||||
stderr.write(`Collection ${collectionID} not found\n`);
|
||||
finish(lib, 1);
|
||||
return;
|
||||
}
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(
|
||||
JSON.stringify(files.map(fileListRow), null, 2) + "\n",
|
||||
.action((opts: { collection: string; json?: boolean }) =>
|
||||
run(filesCommand(context(), opts)),
|
||||
);
|
||||
} else {
|
||||
for (const file of files) {
|
||||
stdout.write(fileListLine(file) + "\n");
|
||||
}
|
||||
}
|
||||
finish(lib, 0);
|
||||
});
|
||||
|
||||
program
|
||||
.command("get")
|
||||
@@ -262,34 +102,9 @@ program
|
||||
.argument("<fileID>", "File ID (from `quak files`)")
|
||||
.option("--out <path>", "Output file path")
|
||||
.option("--collection <id>", "Accepted for compatibility; ignored")
|
||||
.action(async (fileIDStr: string, opts: { out?: string }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
stderr.write("Invalid file ID\n");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(client);
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
stderr.write(`File ${fileID} not found\n`);
|
||||
finish(lib, 1);
|
||||
return;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await photo.original();
|
||||
// Default name is the file's own title, as the pre-library CLI used
|
||||
// (not the editedName-preferring projection title) (issue #52).
|
||||
const outPath = opts.out ?? originalName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
finish(lib, 0);
|
||||
});
|
||||
.action((fileID: string, opts: { out?: string }) =>
|
||||
run(getCommand(context(), fileID, opts)),
|
||||
);
|
||||
|
||||
program
|
||||
.command("get-thumb")
|
||||
@@ -297,34 +112,9 @@ program
|
||||
.argument("<fileID>", "File ID (from `quak files`)")
|
||||
.option("--out <path>", "Output file path")
|
||||
.option("--collection <id>", "Accepted for compatibility; ignored")
|
||||
.action(async (fileIDStr: string, opts: { out?: string }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
stderr.write("Invalid file ID\n");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(client);
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
stderr.write(`File ${fileID} not found\n`);
|
||||
finish(lib, 1);
|
||||
return;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await photo.thumbnail();
|
||||
// Default name is thumb_<file's own title>, as the pre-library CLI
|
||||
// used (not the projection title) (issue #52).
|
||||
const outPath = opts.out ?? thumbnailName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
finish(lib, 0);
|
||||
});
|
||||
.action((fileID: string, opts: { out?: string }) =>
|
||||
run(getThumbCommand(context(), fileID, opts)),
|
||||
);
|
||||
|
||||
program
|
||||
.command("backup-metadata")
|
||||
@@ -337,16 +127,9 @@ program
|
||||
"Download each file and extract full EXIF/IPTC/XMP metadata (slow)",
|
||||
)
|
||||
.option("--all", "Alias for --exif")
|
||||
.action(async (dir: string, opts: { exif?: boolean; all?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const lib = await openReadLibrary(client);
|
||||
await runMetadataBackup(lib, client, dir, {
|
||||
exif: opts.exif || opts.all,
|
||||
onProgress: (msg) => stderr.write(msg + "\n"),
|
||||
});
|
||||
finish(lib, 0);
|
||||
});
|
||||
.action((dir: string, opts: { exif?: boolean; all?: boolean }) =>
|
||||
run(backupMetadataCommand(context(), dir, opts)),
|
||||
);
|
||||
|
||||
program
|
||||
.command("backup")
|
||||
@@ -355,43 +138,9 @@ program
|
||||
)
|
||||
.argument("<dir>", "Output directory")
|
||||
.option("--json", "Print result as JSON instead of human-readable summary")
|
||||
.action(async (dir: string, opts: { json?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
|
||||
stderr.write("Starting backup...\n");
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
downloadDirectory: dir,
|
||||
cacheDirectory: cacheDirOption(),
|
||||
});
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: dir,
|
||||
onProgress: (msg) => {
|
||||
if (!opts.json) stderr.write(msg + "\n");
|
||||
},
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(JSON.stringify(result, null, 2) + "\n");
|
||||
} else {
|
||||
stderr.write("\n--- Backup complete ---\n");
|
||||
stderr.write(` Total files: ${result.totalFiles}\n`);
|
||||
stderr.write(` Downloaded: ${result.downloaded}\n`);
|
||||
stderr.write(` Skipped: ${result.skipped}\n`);
|
||||
stderr.write(` Failed: ${result.failed}\n`);
|
||||
if (result.errors.length > 0) {
|
||||
stderr.write("\nFailed files:\n");
|
||||
for (const e of result.errors) {
|
||||
stderr.write(
|
||||
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
|
||||
.action((dir: string, opts: { json?: boolean }) =>
|
||||
run(backupCommand(context(), dir, opts)),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
finish(lib, result.failed > 0 ? 1 : 0);
|
||||
});
|
||||
|
||||
const helper = program
|
||||
.command("helper")
|
||||
@@ -401,32 +150,9 @@ helper
|
||||
.command("list-missing-thumbnails")
|
||||
.description("List files whose thumbnails are missing or empty")
|
||||
.option("--json", "Output as JSON array")
|
||||
.action(async (opts: { json?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const lib = await openReadLibrary(client);
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
if (!opts.json) stderr.write(msg + "\n");
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(JSON.stringify(missing, null, 2) + "\n");
|
||||
} else {
|
||||
if (missing.length === 0) {
|
||||
stderr.write("No missing thumbnails found.\n");
|
||||
} else {
|
||||
stderr.write(
|
||||
`\n${missing.length} file(s) with missing thumbnails:\n`,
|
||||
.action((opts: { json?: boolean }) =>
|
||||
run(listMissingThumbnailsCommand(context(), opts)),
|
||||
);
|
||||
for (const m of missing) {
|
||||
stdout.write(
|
||||
`${m.fileID}\t${m.title}\t${m.collection}\t${m.reason}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
finish(lib, 0);
|
||||
});
|
||||
|
||||
helper
|
||||
.command("fix-missing-thumbnails")
|
||||
@@ -438,65 +164,9 @@ helper
|
||||
"Specific file IDs to fix (default: fix all missing)",
|
||||
)
|
||||
.option("--json", "Output as JSON")
|
||||
.action(async (opts: { file?: string[]; json?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const lib = await openReadLibrary(client);
|
||||
|
||||
let fileIDs: number[];
|
||||
if (opts.file && opts.file.length > 0) {
|
||||
fileIDs = opts.file.map(Number).filter(Number.isFinite);
|
||||
} else {
|
||||
stderr.write("Scanning for missing thumbnails...\n");
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
if (!opts.json) stderr.write(msg + "\n");
|
||||
});
|
||||
fileIDs = missing.map((m) => m.fileID);
|
||||
if (fileIDs.length === 0) {
|
||||
stderr.write("No missing thumbnails found.\n");
|
||||
finish(lib, 0);
|
||||
return;
|
||||
}
|
||||
stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
|
||||
}
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
fileIDs,
|
||||
(msg) => {
|
||||
if (!opts.json) stderr.write(msg + "\n");
|
||||
},
|
||||
.action((opts: { file?: string[]; json?: boolean }) =>
|
||||
run(fixMissingThumbnailsCommand(context(), opts)),
|
||||
);
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(JSON.stringify(results, null, 2) + "\n");
|
||||
} else {
|
||||
const fixed = results.filter((r) => r.status === "fixed").length;
|
||||
const skipped = results.filter(
|
||||
(r) => r.status === "skipped",
|
||||
).length;
|
||||
const failed = results.filter((r) => r.status === "failed").length;
|
||||
stderr.write(`\n--- Done ---\n`);
|
||||
stderr.write(` Fixed: ${fixed}\n`);
|
||||
stderr.write(` Skipped: ${skipped}\n`);
|
||||
stderr.write(` Failed: ${failed}\n`);
|
||||
if (skipped > 0) {
|
||||
stderr.write("\nSkipped (unsupported format):\n");
|
||||
for (const r of results.filter((r) => r.status === "skipped")) {
|
||||
stderr.write(` ${r.fileID}\t${r.title}\t${r.reason}\n`);
|
||||
}
|
||||
}
|
||||
if (failed > 0) {
|
||||
stderr.write("\nFailed files:\n");
|
||||
for (const r of results.filter((r) => r.status === "failed")) {
|
||||
stderr.write(` ${r.fileID}\t${r.title}\t${r.reason}\n`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
finish(lib, results.some((r) => r.status === "failed") ? 1 : 0);
|
||||
});
|
||||
|
||||
await init();
|
||||
program.parse();
|
||||
|
||||
+1
-1
@@ -29,7 +29,6 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@eslint/js": "9.38.0",
|
||||
"@types/libsodium-wrappers-sumo": "0.8.2",
|
||||
"@types/node": "22.18.13",
|
||||
"eslint": "9.38.0",
|
||||
"prettier": "3.8.1",
|
||||
@@ -43,6 +42,7 @@
|
||||
"env-paths": "4.0.0",
|
||||
"exif-reader": "2.0.3",
|
||||
"fast-srp-hap": "2.0.4",
|
||||
"fflate": "0.8.3",
|
||||
"jpeg-js": "0.4.4",
|
||||
"libsodium-wrappers-sumo": "0.8.4"
|
||||
}
|
||||
|
||||
@@ -45,10 +45,24 @@ for (const bin of bins) {
|
||||
'
|
||||
}
|
||||
|
||||
# src/index.ts imports ../package.json for the version, which tsc copies to
|
||||
# dist/package.json. Running the built CLI proves that import resolves from
|
||||
# dist/ and reports the version package.json declares.
|
||||
verify_version() {
|
||||
built="$(node dist/bin/quak.js --version)"
|
||||
declared="$(node -p 'require("./package.json").version')"
|
||||
if [ "$built" != "$declared" ]; then
|
||||
echo "build: dist/bin/quak.js reports $built, package.json declares $declared" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "build: dist/bin/quak.js reports version $built"
|
||||
}
|
||||
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
yarn run tsc
|
||||
verify_entrypoints
|
||||
verify_version
|
||||
}
|
||||
|
||||
main "$@"
|
||||
|
||||
+103
-53
@@ -19,14 +19,12 @@ const DEFAULT_FILES_ORIGIN = "https://files.ente.io";
|
||||
const DEFAULT_THUMBS_ORIGIN = "https://thumbnails.ente.io";
|
||||
const CLIENT_PACKAGE = "berlin.sneak.quak";
|
||||
|
||||
// Two deadlines rather than one, because a single number cannot serve both
|
||||
// jobs. Thirty seconds is generous for a JSON call and short enough that a
|
||||
// hung API connection cannot stall a backup for long. A file body is a
|
||||
// different shape of problem: the deadline has to cover the whole transfer,
|
||||
// which for a large video on a slow link is minutes, so a value sane for JSON
|
||||
// would cancel legitimate downloads.
|
||||
// Two deadlines of different kinds. `requestTimeoutMs` bounds a whole JSON
|
||||
// call. `downloadTimeoutMs` is an idle deadline: a file or thumbnail download
|
||||
// is aborted only when no bytes have arrived for that long, so a large video on
|
||||
// a slow link that keeps making progress is never cut off.
|
||||
export const DEFAULT_REQUEST_TIMEOUT_MS = 30_000;
|
||||
export const DEFAULT_DOWNLOAD_TIMEOUT_MS = 600_000;
|
||||
export const DEFAULT_DOWNLOAD_TIMEOUT_MS = 60_000;
|
||||
|
||||
export interface ApiClientOptions {
|
||||
apiOrigin?: string;
|
||||
@@ -48,18 +46,44 @@ export interface StreamOptions {
|
||||
retry?: boolean;
|
||||
}
|
||||
|
||||
// Enforce a deadline over a response body, not merely over its headers.
|
||||
// An abort signal that fires once `ms` pass without a call to `restart`. It
|
||||
// aborts with a `TimeoutError`, the same reason `AbortSignal.timeout()` gives,
|
||||
// so the retry classifier treats an idle download exactly as it treats any
|
||||
// other deadline. `stop` must be called when the download ends. The timer is
|
||||
// unref'd, so even one left running never keeps the process alive.
|
||||
const idleDeadline = (ms: number) => {
|
||||
const controller = new AbortController();
|
||||
let timer: ReturnType<typeof setTimeout> | undefined;
|
||||
const stop = (): void => clearTimeout(timer);
|
||||
const restart = (): void => {
|
||||
stop();
|
||||
timer = setTimeout(() => {
|
||||
controller.abort(
|
||||
new DOMException(
|
||||
`download stalled: no bytes received for ${ms} ms`,
|
||||
"TimeoutError",
|
||||
),
|
||||
);
|
||||
}, ms);
|
||||
timer.unref();
|
||||
};
|
||||
restart();
|
||||
return { signal: controller.signal, restart, stop };
|
||||
};
|
||||
|
||||
// Enforce the idle deadline over a response body, not merely over its headers.
|
||||
//
|
||||
// `getFileStream` returns as soon as headers arrive; the bytes are pulled
|
||||
// later, in the download layer. Whether the signal passed to `fetch` also
|
||||
// tears down the body afterwards is up to the fetch implementation, so this
|
||||
// wrapper makes it a property of quak instead: every read races the signal,
|
||||
// and an abort errors the stream with the abort reason — which the retry
|
||||
// classifier recognises.
|
||||
// each chunk that arrives restarts the deadline, and an abort errors the
|
||||
// stream with the abort reason — which the retry classifier recognises.
|
||||
const deadlineStream = (
|
||||
body: ReadableStream<Uint8Array>,
|
||||
signal: AbortSignal,
|
||||
deadline: ReturnType<typeof idleDeadline>,
|
||||
): ReadableStream<Uint8Array> => {
|
||||
const { signal } = deadline;
|
||||
const reader = body.getReader();
|
||||
let rejectOnAbort: (reason: unknown) => void = () => undefined;
|
||||
const aborted = new Promise<never>((_resolve, reject) => {
|
||||
@@ -73,7 +97,10 @@ const deadlineStream = (
|
||||
const onAbort = (): void => rejectOnAbort(signal.reason);
|
||||
if (signal.aborted) onAbort();
|
||||
else signal.addEventListener("abort", onAbort, { once: true });
|
||||
const release = (): void => signal.removeEventListener("abort", onAbort);
|
||||
const release = (): void => {
|
||||
deadline.stop();
|
||||
signal.removeEventListener("abort", onAbort);
|
||||
};
|
||||
|
||||
return new ReadableStream<Uint8Array>({
|
||||
async pull(controller) {
|
||||
@@ -84,6 +111,7 @@ const deadlineStream = (
|
||||
controller.close();
|
||||
return;
|
||||
}
|
||||
deadline.restart();
|
||||
controller.enqueue(next.value);
|
||||
} catch (err) {
|
||||
release();
|
||||
@@ -98,6 +126,30 @@ const deadlineStream = (
|
||||
});
|
||||
};
|
||||
|
||||
// The one place a request URL is built. `origin` may carry a base path (a
|
||||
// self-hosted server behind a prefix) and may end in a slash; `path` may or
|
||||
// may not start with one. Query parameters go only through `query`, which
|
||||
// percent-encodes them: a `?` or `#` in `path` is an error, because
|
||||
// `new URL` would otherwise quietly treat what follows as something else.
|
||||
const buildURL = (
|
||||
origin: string,
|
||||
path: string,
|
||||
query?: Record<string, string | number | undefined>,
|
||||
): string => {
|
||||
if (path.includes("?") || path.includes("#")) {
|
||||
throw new Error(
|
||||
`request path must not contain "?" or "#"; pass query parameters separately: ${path}`,
|
||||
);
|
||||
}
|
||||
const url = new URL(
|
||||
`${origin.replace(/\/+$/, "")}/${path.replace(/^\/+/, "")}`,
|
||||
);
|
||||
for (const [k, v] of Object.entries(query ?? {})) {
|
||||
if (v !== undefined) url.searchParams.set(k, String(v));
|
||||
}
|
||||
return url.href;
|
||||
};
|
||||
|
||||
export class ApiClient {
|
||||
private readonly apiOrigin: string;
|
||||
private readonly isCustomOrigin: boolean;
|
||||
@@ -139,11 +191,16 @@ export class ApiClient {
|
||||
this.token = undefined;
|
||||
}
|
||||
|
||||
getAuthToken(): string | undefined {
|
||||
return this.token;
|
||||
}
|
||||
|
||||
// The policy this client was configured with, so that a caller wrapping a
|
||||
// whole operation in its own `withRetry` — the download layer — runs under
|
||||
// the same settings rather than under the library defaults.
|
||||
// A copy, so the caller cannot change this client's settings through it.
|
||||
getRetryOptions(): ResolvedRetryOptions {
|
||||
return this.retry;
|
||||
return { ...this.retry };
|
||||
}
|
||||
|
||||
private headers(extra?: Record<string, string>): Record<string, string> {
|
||||
@@ -197,22 +254,10 @@ export class ApiClient {
|
||||
path: string,
|
||||
query?: Record<string, string | number | undefined>,
|
||||
): Promise<T> {
|
||||
const url = new URL(path, this.apiOrigin + "/");
|
||||
// new URL with a base resolves relative paths; ensure we keep the
|
||||
// origin from apiOrigin even when path starts with /
|
||||
url.protocol = new URL(this.apiOrigin).protocol;
|
||||
url.host = new URL(this.apiOrigin).host;
|
||||
url.pathname = path;
|
||||
if (query) {
|
||||
for (const [k, v] of Object.entries(query)) {
|
||||
if (v !== undefined) {
|
||||
url.searchParams.set(k, String(v));
|
||||
}
|
||||
}
|
||||
}
|
||||
const url = buildURL(this.apiOrigin, path, query);
|
||||
// A GET changes nothing, so it is retried under the full policy.
|
||||
return withRetry(async () => {
|
||||
const resp = await this._fetch(url.href, {
|
||||
const resp = await this._fetch(url, {
|
||||
method: "GET",
|
||||
headers: this.headers(),
|
||||
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
||||
@@ -223,16 +268,16 @@ export class ApiClient {
|
||||
}
|
||||
|
||||
async postJSON<T>(path: string, body: unknown): Promise<T> {
|
||||
const url = `${this.apiOrigin}${path}`;
|
||||
// Idempotency: this reaches `/users/srp/create-session`,
|
||||
// `/users/two-factor/verify` and `/users/ott`, all of which change
|
||||
// server state — verifying a second factor consumes one of a small
|
||||
// number of attempts. So a POST is replayed only on a failure that
|
||||
// establishes no TCP connection to the server ever existed: DNS
|
||||
// produced no address, or the peer refused the connection. A 5xx, a
|
||||
// mid-flight reset, a routing errno (which Linux also delivers on an
|
||||
// established socket) and a timeout are all left to the caller,
|
||||
// because each of them can occur after the server has already acted.
|
||||
const url = buildURL(this.apiOrigin, path);
|
||||
// Not idempotent: a POST is replayed only when `isSafeToReplay`
|
||||
// says no request byte can have reached the server. The endpoints
|
||||
// this covers are listed in the README under "Endpoints used".
|
||||
//
|
||||
// Redirects are not followed. The origin has already received the
|
||||
// request when it answers with one, so a connection refused by the
|
||||
// redirect target would look replay-safe when it is not. The API has
|
||||
// no legitimate redirect, so one surfaces as an `ApiError` with its
|
||||
// 3xx status, which is not retried.
|
||||
return withRetry(
|
||||
async () => {
|
||||
const resp = await this._fetch(url, {
|
||||
@@ -241,6 +286,7 @@ export class ApiClient {
|
||||
"Content-Type": "application/json",
|
||||
}),
|
||||
body: JSON.stringify(body),
|
||||
redirect: "manual",
|
||||
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
||||
});
|
||||
await this.throwIfError(resp);
|
||||
@@ -255,8 +301,8 @@ export class ApiClient {
|
||||
opts?: StreamOptions,
|
||||
): Promise<ReadableStream<Uint8Array>> {
|
||||
const url = this.isCustomOrigin
|
||||
? `${this.apiOrigin}/files/download/${fileID}`
|
||||
: `${this.filesOrigin}/?fileID=${fileID}`;
|
||||
? buildURL(this.apiOrigin, `/files/download/${fileID}`)
|
||||
: buildURL(this.filesOrigin, "/", { fileID });
|
||||
return this.streamRequest(url, opts);
|
||||
}
|
||||
|
||||
@@ -298,10 +344,8 @@ export class ApiClient {
|
||||
}
|
||||
|
||||
async putJSON<T>(path: string, body: unknown): Promise<T> {
|
||||
const url = `${this.apiOrigin}${path}`;
|
||||
// Same idempotency rule as `postJSON`, for the same reason: this
|
||||
// reaches `/files/thumbnail`, which registers an uploaded thumbnail
|
||||
// against a file.
|
||||
const url = buildURL(this.apiOrigin, path);
|
||||
// Same replay and redirect rules as `postJSON`, for the same reasons.
|
||||
return withRetry(
|
||||
async () => {
|
||||
const resp = await this._fetch(url, {
|
||||
@@ -310,6 +354,7 @@ export class ApiClient {
|
||||
"Content-Type": "application/json",
|
||||
}),
|
||||
body: JSON.stringify(body),
|
||||
redirect: "manual",
|
||||
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
||||
});
|
||||
await this.throwIfError(resp);
|
||||
@@ -335,8 +380,8 @@ export class ApiClient {
|
||||
opts?: StreamOptions,
|
||||
): Promise<ReadableStream<Uint8Array>> {
|
||||
const url = this.isCustomOrigin
|
||||
? `${this.apiOrigin}/files/preview/${fileID}`
|
||||
: `${this.thumbsOrigin}/?fileID=${fileID}`;
|
||||
? buildURL(this.apiOrigin, `/files/preview/${fileID}`)
|
||||
: buildURL(this.thumbsOrigin, "/", { fileID });
|
||||
return this.streamRequest(url, opts);
|
||||
}
|
||||
|
||||
@@ -345,22 +390,27 @@ export class ApiClient {
|
||||
opts?: StreamOptions,
|
||||
): Promise<ReadableStream<Uint8Array>> {
|
||||
const once = async (): Promise<ReadableStream<Uint8Array>> => {
|
||||
// A fresh deadline per attempt, so a retry gets the whole budget
|
||||
// rather than the remainder of the one that just expired.
|
||||
const signal = AbortSignal.timeout(this.downloadTimeoutMs);
|
||||
// A fresh deadline per attempt. It also covers the wait for the
|
||||
// headers, when no bytes have arrived either.
|
||||
const deadline = idleDeadline(this.downloadTimeoutMs);
|
||||
try {
|
||||
const resp = await this._fetch(url, {
|
||||
method: "GET",
|
||||
headers: this.headers(),
|
||||
signal,
|
||||
signal: deadline.signal,
|
||||
});
|
||||
await this.throwIfError(resp);
|
||||
if (!resp.body) {
|
||||
// Carries the status, and is not retryable: a response that
|
||||
// arrived without a body is malformed, and asking again
|
||||
// produces the same malformed response.
|
||||
// Carries the status, and is not retryable: a response
|
||||
// that arrived without a body is malformed, and asking
|
||||
// again produces the same malformed response.
|
||||
throw new ApiError("response body is null", resp.status);
|
||||
}
|
||||
return deadlineStream(resp.body, signal);
|
||||
return deadlineStream(resp.body, deadline);
|
||||
} catch (err) {
|
||||
deadline.stop();
|
||||
throw err;
|
||||
}
|
||||
};
|
||||
return opts?.retry === false ? once() : withRetry(once, this.retry);
|
||||
}
|
||||
|
||||
+59
-20
@@ -29,19 +29,21 @@
|
||||
// rather than counted forever, which would poison a scheduled backup's exit code.
|
||||
|
||||
import {
|
||||
copyFileSync,
|
||||
lstatSync,
|
||||
mkdirSync,
|
||||
readdirSync,
|
||||
readFileSync,
|
||||
readlinkSync,
|
||||
renameSync,
|
||||
rmSync,
|
||||
statSync,
|
||||
symlinkSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { basename, dirname, extname, join, relative } from "node:path";
|
||||
import { copyFile, rename, rm } from "node:fs/promises";
|
||||
import { basename, dirname, join, relative } from "node:path";
|
||||
|
||||
import { fsyncPath } from "./download/index.js";
|
||||
import { safeExtension, sanitizeFileName } from "./filename.js";
|
||||
import type { Collection, EnteFile } from "./model/types.js";
|
||||
|
||||
export type ProgressCallback = (message: string) => void;
|
||||
@@ -108,16 +110,11 @@ interface FailureEntry {
|
||||
|
||||
const LEDGER_VERSION = 1;
|
||||
|
||||
const sanitizePath = (name: string): string =>
|
||||
name.replace(/[/\\:*?"<>|]/g, "_").replace(/^\.+/, "_");
|
||||
|
||||
// The originals/ filename for a file: `<id><ext>`, the extension taken from the
|
||||
// title (or `.bin`). Matches the content cache's own naming so a present check
|
||||
// lines up with what a fetch would write.
|
||||
const originalName = (file: EnteFile): string => {
|
||||
const ext = extname(file.metadata.title || "") || ".bin";
|
||||
return `${file.id}${ext}`;
|
||||
};
|
||||
const originalName = (file: EnteFile): string =>
|
||||
`${file.id}${safeExtension(file.metadata.title)}`;
|
||||
|
||||
// A regular file with content is treated as complete. A zero-byte file is not:
|
||||
// it is the shape an aborted write leaves and must be re-fetched.
|
||||
@@ -158,8 +155,12 @@ const errorMessage = (err: unknown): string =>
|
||||
err instanceof Error ? err.message : String(err);
|
||||
|
||||
// Copy bytes into `dest` via a temp file in the same directory plus rename, so
|
||||
// `dest` appears only once it is whole ("present means complete").
|
||||
const copyAtomic = (src: string, dest: string): void => {
|
||||
// `dest` appears only once it is whole ("present means complete"). As in the
|
||||
// download writer, the temp file is fsynced before the rename and the directory
|
||||
// after it, so a power cut cannot leave a correctly named but short original.
|
||||
// The temp name carries this process's ID so a later run can tell a leftover
|
||||
// from a copy still in progress (see `removeLeftoverTempFiles`).
|
||||
const copyAtomic = async (src: string, dest: string): Promise<void> => {
|
||||
if (src === dest) return;
|
||||
const tmp = join(
|
||||
dirname(dest),
|
||||
@@ -168,10 +169,45 @@ const copyAtomic = (src: string, dest: string): void => {
|
||||
.slice(2)}.tmp`,
|
||||
);
|
||||
try {
|
||||
copyFileSync(src, tmp);
|
||||
renameSync(tmp, dest);
|
||||
await copyFile(src, tmp);
|
||||
await fsyncPath(tmp);
|
||||
// `rename` replaces the destination's directory entry: an existing
|
||||
// symlink at `dest` is replaced, not followed, and the new file has
|
||||
// the temp file's permissions (copied from `src`).
|
||||
await rename(tmp, dest);
|
||||
await fsyncPath(dirname(dest));
|
||||
} finally {
|
||||
rmSync(tmp, { force: true });
|
||||
await rm(tmp, { force: true });
|
||||
}
|
||||
};
|
||||
|
||||
// A process-ID check: signal 0 delivers nothing and only reports whether the
|
||||
// process exists. EPERM means it exists but belongs to another user.
|
||||
const isRunning = (pid: number): boolean => {
|
||||
try {
|
||||
process.kill(pid, 0);
|
||||
return true;
|
||||
} catch (err) {
|
||||
return (err as NodeJS.ErrnoException).code === "EPERM";
|
||||
}
|
||||
};
|
||||
|
||||
// Delete the temp files `copyAtomic` leaves behind when a backup is killed
|
||||
// before its rename. Only files whose process is no longer running are
|
||||
// removed, so a backup running at the same time keeps its own. A reused
|
||||
// process ID can only keep a leftover a while longer, never remove a live one.
|
||||
const removeLeftoverTempFiles = (dir: string): void => {
|
||||
let names: string[];
|
||||
try {
|
||||
names = readdirSync(dir);
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
for (const name of names) {
|
||||
const match = /^\.quak-backup-.*-(\d+)-[0-9a-z]*\.tmp$/.exec(name);
|
||||
if (match && !isRunning(Number(match[1]))) {
|
||||
rmSync(join(dir, name), { force: true });
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
@@ -258,6 +294,8 @@ export const runBackup = async (
|
||||
mkdirSync(originalsDir, { recursive: true });
|
||||
mkdirSync(collectionsDir, { recursive: true });
|
||||
if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true });
|
||||
removeLeftoverTempFiles(originalsDir);
|
||||
removeLeftoverTempFiles(thumbnailsDir);
|
||||
|
||||
const ledgerPath = join(downloadDirectory, "failures.json");
|
||||
const ledger = loadLedger(ledgerPath);
|
||||
@@ -325,7 +363,7 @@ export const runBackup = async (
|
||||
try {
|
||||
log(`Fetching original ${file.metadata.title} (${fileID})...`);
|
||||
const { path } = await lib.original(fileID);
|
||||
copyAtomic(path, dest);
|
||||
await copyAtomic(path, dest);
|
||||
downloaded++;
|
||||
} catch (err) {
|
||||
log(
|
||||
@@ -346,7 +384,7 @@ export const runBackup = async (
|
||||
if (isPresent(dest)) continue;
|
||||
try {
|
||||
const { path } = await lib.thumbnail(fileID);
|
||||
copyAtomic(path, dest);
|
||||
await copyAtomic(path, dest);
|
||||
} catch (err) {
|
||||
recordFailure(
|
||||
file,
|
||||
@@ -370,7 +408,7 @@ export const runBackup = async (
|
||||
|
||||
// Then the per-collection symlink trees and JSON.
|
||||
for (const c of collections) {
|
||||
const colDirName = sanitizePath(c.name || `collection-${c.id}`);
|
||||
const colDirName = sanitizeFileName(c.name, `collection-${c.id}`);
|
||||
const colDir = join(collectionsDir, colDirName);
|
||||
mkdirSync(colDir, { recursive: true });
|
||||
|
||||
@@ -381,8 +419,9 @@ export const runBackup = async (
|
||||
if (!includeOriginals) continue;
|
||||
const orig = join(originalsDir, originalName(file));
|
||||
if (!isPresent(orig)) continue;
|
||||
const linkName = sanitizePath(
|
||||
file.metadata.title || `file-${file.id}`,
|
||||
const linkName = sanitizeFileName(
|
||||
file.metadata.title,
|
||||
`file-${file.id}`,
|
||||
);
|
||||
const linkPath = join(colDir, linkName);
|
||||
try {
|
||||
|
||||
@@ -0,0 +1,486 @@
|
||||
// The CLI's commands as plain functions.
|
||||
//
|
||||
// Each command takes its options and a `CliContext` and resolves to the exit
|
||||
// code; a thrown error is left to the caller. Nothing here calls
|
||||
// `process.exit`: `bin/quak.ts` wires these to the command line and exits with
|
||||
// the returned code once output has drained. Output must stay byte-identical
|
||||
// (see `cli-output.ts`).
|
||||
|
||||
import { input, password as passwordPrompt } from "@inquirer/prompts";
|
||||
import {
|
||||
copyFileSync,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
unlinkSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { Client, type ClientSnapshot } from "./client.js";
|
||||
import { init } from "./crypto/index.js";
|
||||
import { Library, type LibraryClient } from "./library/index.js";
|
||||
import {
|
||||
fileListRow,
|
||||
fileListLine,
|
||||
originalName,
|
||||
thumbnailName,
|
||||
} from "./cli-output.js";
|
||||
import { freshCollections, freshFiles, freshFile } from "./cli-read.js";
|
||||
import { runMetadataBackup } from "./metadata-backup.js";
|
||||
import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js";
|
||||
|
||||
export interface CliContext {
|
||||
stdout: { write(text: string): unknown };
|
||||
stderr: { write(text: string): unknown };
|
||||
// Directory holding `session.json`.
|
||||
sessionDir: string;
|
||||
// The `--cache-dir` global, or undefined to let the library pick its
|
||||
// per-user default keyed by the account id.
|
||||
cacheDir?: string;
|
||||
// Reads the session file into a client, or null when there is none. The
|
||||
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
|
||||
loadSession: (path: string) => Client | null;
|
||||
}
|
||||
|
||||
const sessionPath = (ctx: CliContext): string =>
|
||||
join(ctx.sessionDir, "session.json");
|
||||
|
||||
// Write the session readable by its owner only, in a directory only its owner
|
||||
// can enter.
|
||||
export const saveSession = (
|
||||
sessionDir: string,
|
||||
snapshot: ClientSnapshot,
|
||||
): void => {
|
||||
mkdirSync(sessionDir, { recursive: true, mode: 0o700 });
|
||||
writeFileSync(
|
||||
join(sessionDir, "session.json"),
|
||||
JSON.stringify(snapshot, null, 2),
|
||||
{ mode: 0o600 },
|
||||
);
|
||||
};
|
||||
|
||||
// The saved client, or undefined after telling the user why there is none.
|
||||
const requireSession = (ctx: CliContext): Client | undefined => {
|
||||
let client: Client | null;
|
||||
try {
|
||||
client = ctx.loadSession(sessionPath(ctx));
|
||||
} catch (err) {
|
||||
ctx.stderr.write(
|
||||
`${err instanceof Error ? err.message : err}\n` +
|
||||
`Run "quak logout" and then "quak login" to replace it.\n`,
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
if (!client) {
|
||||
ctx.stderr.write(
|
||||
`Not logged in. Run "quak login" first.\nSession file: ${sessionPath(ctx)}\n`,
|
||||
);
|
||||
return undefined;
|
||||
}
|
||||
return client;
|
||||
};
|
||||
|
||||
// A library client that omits `fetchMLData`, so the point commands below do not
|
||||
// kick the library's background ML backfill: they read metadata, or fetch one
|
||||
// file's content, and exit. `backup` and `backup-metadata` handle ML on their
|
||||
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
|
||||
// originals through the on-disk cache.
|
||||
const readLibraryClient = (client: Client): LibraryClient => ({
|
||||
whoami: () => client.whoami(),
|
||||
collectionsSince: (args) => client.collectionsSince(args),
|
||||
filesSince: (args) => client.filesSince(args),
|
||||
contentSource: () => client.contentSource(),
|
||||
});
|
||||
|
||||
// Open a library for a single point command: the aggressive background precache
|
||||
// (issue #48) is off — a one-shot `collections` or `get` must not start
|
||||
// downloading the whole account — and the refresh interval is long so no second
|
||||
// refresh fires mid-command.
|
||||
const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
|
||||
Library.open({
|
||||
client: readLibraryClient(client),
|
||||
cacheDirectory: ctx.cacheDir,
|
||||
refreshIntervalSeconds: 3600,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
|
||||
const prompt = async (message: string): Promise<string> => input({ message });
|
||||
|
||||
const promptSecret = async (message: string): Promise<string> =>
|
||||
passwordPrompt({ message, mask: true });
|
||||
|
||||
export const loginCommand = async (ctx: CliContext): Promise<number> => {
|
||||
await init();
|
||||
const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
|
||||
const password =
|
||||
process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
|
||||
|
||||
ctx.stderr.write("Authenticating...\n");
|
||||
try {
|
||||
const client = await Client.login({
|
||||
email,
|
||||
password,
|
||||
totp: async () => prompt("TOTP code: "),
|
||||
emailOTP: async () => prompt("Email verification code: "),
|
||||
});
|
||||
|
||||
saveSession(ctx.sessionDir, client.toJSON());
|
||||
const info = client.whoami();
|
||||
ctx.stderr.write(`Logged in as ${info.email} (user ${info.userID})\n`);
|
||||
ctx.stderr.write(`Session saved to ${sessionPath(ctx)}\n`);
|
||||
} catch (err) {
|
||||
ctx.stderr.write(
|
||||
`Login failed: ${err instanceof Error ? err.message : err}\n`,
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
return 0;
|
||||
};
|
||||
|
||||
export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const info = client.whoami();
|
||||
ctx.stdout.write(JSON.stringify(info) + "\n");
|
||||
return 0;
|
||||
};
|
||||
|
||||
export const logoutCommand = async (ctx: CliContext): Promise<number> => {
|
||||
if (existsSync(sessionPath(ctx))) {
|
||||
unlinkSync(sessionPath(ctx));
|
||||
ctx.stderr.write("Session deleted.\n");
|
||||
} else {
|
||||
ctx.stderr.write("No session found.\n");
|
||||
}
|
||||
return 0;
|
||||
};
|
||||
|
||||
export const collectionsCommand = async (
|
||||
ctx: CliContext,
|
||||
opts: { json?: boolean },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52): the pre-library CLI printed current state in
|
||||
// this order, not the albums projection's newest-first order.
|
||||
const collections = await freshCollections(lib);
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(
|
||||
JSON.stringify(
|
||||
collections.map((c) => ({
|
||||
id: c.id,
|
||||
name: c.name,
|
||||
type: c.type,
|
||||
ownerID: c.ownerID,
|
||||
isShared: c.isShared,
|
||||
updationTime: c.updationTime,
|
||||
})),
|
||||
null,
|
||||
2,
|
||||
) + "\n",
|
||||
);
|
||||
} else {
|
||||
for (const c of collections) {
|
||||
ctx.stdout.write(
|
||||
`${c.id}\t${c.type}\t${c.name}${c.isShared ? " (shared)" : ""}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
export const filesCommand = async (
|
||||
ctx: CliContext,
|
||||
opts: { collection: string; json?: boolean },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const collectionID = Number(opts.collection);
|
||||
if (!Number.isFinite(collectionID)) {
|
||||
ctx.stderr.write("Invalid collection ID\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52). Each file prints from its own decrypted
|
||||
// metadata (raw title, microsecond creationTime) via cli-output, and in
|
||||
// the pre-library CLI's enumeration order, not the projection's
|
||||
// newest-first order.
|
||||
const files = await freshFiles(lib, collectionID);
|
||||
if (!files) {
|
||||
ctx.stderr.write(`Collection ${collectionID} not found\n`);
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(
|
||||
JSON.stringify(files.map(fileListRow), null, 2) + "\n",
|
||||
);
|
||||
} else {
|
||||
for (const file of files) {
|
||||
ctx.stdout.write(fileListLine(file) + "\n");
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
export const getCommand = async (
|
||||
ctx: CliContext,
|
||||
fileIDStr: string,
|
||||
opts: { out?: string },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
ctx.stderr.write("Invalid file ID\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
ctx.stderr.write(`File ${fileID} not found\n`);
|
||||
return 1;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await photo.original();
|
||||
// Default name is the file's own title, as the pre-library CLI used
|
||||
// (not the editedName-preferring projection title) (issue #52).
|
||||
const outPath = opts.out ?? originalName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
return 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
export const getThumbCommand = async (
|
||||
ctx: CliContext,
|
||||
fileIDStr: string,
|
||||
opts: { out?: string },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
ctx.stderr.write("Invalid file ID\n");
|
||||
return 1;
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
ctx.stderr.write(`File ${fileID} not found\n`);
|
||||
return 1;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await photo.thumbnail();
|
||||
// Default name is thumb_<file's own title>, as the pre-library CLI
|
||||
// used (not the projection title) (issue #52).
|
||||
const outPath = opts.out ?? thumbnailName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
return 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
export const backupMetadataCommand = async (
|
||||
ctx: CliContext,
|
||||
dir: string,
|
||||
opts: { exif?: boolean; all?: boolean },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
await runMetadataBackup(lib, client, dir, {
|
||||
exif: opts.exif || opts.all,
|
||||
onProgress: (msg) => ctx.stderr.write(msg + "\n"),
|
||||
});
|
||||
return 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
export const backupCommand = async (
|
||||
ctx: CliContext,
|
||||
dir: string,
|
||||
opts: { json?: boolean },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
|
||||
ctx.stderr.write("Starting backup...\n");
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
downloadDirectory: dir,
|
||||
cacheDirectory: ctx.cacheDir,
|
||||
});
|
||||
try {
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: dir,
|
||||
onProgress: (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
},
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(JSON.stringify(result, null, 2) + "\n");
|
||||
} else {
|
||||
ctx.stderr.write("\n--- Backup complete ---\n");
|
||||
ctx.stderr.write(` Total files: ${result.totalFiles}\n`);
|
||||
ctx.stderr.write(` Downloaded: ${result.downloaded}\n`);
|
||||
ctx.stderr.write(` Skipped: ${result.skipped}\n`);
|
||||
ctx.stderr.write(` Failed: ${result.failed}\n`);
|
||||
if (result.errors.length > 0) {
|
||||
ctx.stderr.write("\nFailed files:\n");
|
||||
for (const e of result.errors) {
|
||||
ctx.stderr.write(
|
||||
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return result.failed > 0 ? 1 : 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
export const listMissingThumbnailsCommand = async (
|
||||
ctx: CliContext,
|
||||
opts: { json?: boolean },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(JSON.stringify(missing, null, 2) + "\n");
|
||||
} else {
|
||||
if (missing.length === 0) {
|
||||
ctx.stderr.write("No missing thumbnails found.\n");
|
||||
} else {
|
||||
ctx.stderr.write(
|
||||
`\n${missing.length} file(s) with missing thumbnails:\n`,
|
||||
);
|
||||
for (const m of missing) {
|
||||
ctx.stdout.write(
|
||||
`${m.fileID}\t${m.title}\t${m.collection}\t${m.reason}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
return 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
export const fixMissingThumbnailsCommand = async (
|
||||
ctx: CliContext,
|
||||
opts: { file?: string[]; json?: boolean },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 1;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
let fileIDs: number[];
|
||||
if (opts.file && opts.file.length > 0) {
|
||||
fileIDs = opts.file.map(Number).filter(Number.isFinite);
|
||||
} else {
|
||||
ctx.stderr.write("Scanning for missing thumbnails...\n");
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
});
|
||||
fileIDs = missing.map((m) => m.fileID);
|
||||
if (fileIDs.length === 0) {
|
||||
ctx.stderr.write("No missing thumbnails found.\n");
|
||||
return 0;
|
||||
}
|
||||
ctx.stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
|
||||
}
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
fileIDs,
|
||||
(msg) => {
|
||||
if (!opts.json) ctx.stderr.write(msg + "\n");
|
||||
},
|
||||
);
|
||||
|
||||
if (opts.json) {
|
||||
ctx.stdout.write(JSON.stringify(results, null, 2) + "\n");
|
||||
} else {
|
||||
const fixed = results.filter((r) => r.status === "fixed").length;
|
||||
const skipped = results.filter(
|
||||
(r) => r.status === "skipped",
|
||||
).length;
|
||||
const failed = results.filter((r) => r.status === "failed").length;
|
||||
ctx.stderr.write(`\n--- Done ---\n`);
|
||||
ctx.stderr.write(` Fixed: ${fixed}\n`);
|
||||
ctx.stderr.write(` Skipped: ${skipped}\n`);
|
||||
ctx.stderr.write(` Failed: ${failed}\n`);
|
||||
if (skipped > 0) {
|
||||
ctx.stderr.write("\nSkipped (unsupported format):\n");
|
||||
for (const r of results.filter((r) => r.status === "skipped")) {
|
||||
ctx.stderr.write(
|
||||
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
if (failed > 0) {
|
||||
ctx.stderr.write("\nFailed files:\n");
|
||||
for (const r of results.filter((r) => r.status === "failed")) {
|
||||
ctx.stderr.write(
|
||||
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return results.some((r) => r.status === "failed") ? 1 : 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
};
|
||||
+6
-3
@@ -9,6 +9,7 @@
|
||||
// `metadata.title`, and issue #52 requires that output stay byte-identical, so
|
||||
// the commands shape their output from the raw `EnteFile` through here.
|
||||
|
||||
import { sanitizeFileName } from "./filename.js";
|
||||
import type { EnteFile, FileType, Microseconds } from "./model/types.js";
|
||||
|
||||
// One row of `quak files --json`.
|
||||
@@ -32,9 +33,11 @@ export const fileListRow = (file: EnteFile): FileListRow => ({
|
||||
export const fileListLine = (file: EnteFile): string =>
|
||||
`${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
|
||||
|
||||
// Default output path for `quak get` when `--out` is not given.
|
||||
export const originalName = (file: EnteFile): string => file.metadata.title;
|
||||
// Default output path for `quak get` when `--out` is not given. The title comes
|
||||
// from the server, so it is sanitized; `--out` is the user's and is used as is.
|
||||
export const originalName = (file: EnteFile): string =>
|
||||
sanitizeFileName(file.metadata.title, `file-${file.id}`);
|
||||
|
||||
// Default output path for `quak get-thumb` when `--out` is not given.
|
||||
export const thumbnailName = (file: EnteFile): string =>
|
||||
`thumb_${file.metadata.title}`;
|
||||
`thumb_${originalName(file)}`;
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
// How the CLI reads its saved session file back into a `Client`.
|
||||
//
|
||||
// A missing file means "not logged in" and returns null. A file that exists but
|
||||
// cannot be read back into a client (bad JSON, a missing field, a key of the
|
||||
// wrong length) throws an error saying the session file is corrupt, so the CLI
|
||||
// can tell the user which of the two it is. Needs `init()` first.
|
||||
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import type { ApiClientOptions } from "./api/client.js";
|
||||
import { Client } from "./client.js";
|
||||
|
||||
export const loadSession = (
|
||||
path: string,
|
||||
apiOptions?: ApiClientOptions,
|
||||
): Client | null => {
|
||||
if (!existsSync(path)) return null;
|
||||
try {
|
||||
return Client.fromJSON(
|
||||
JSON.parse(readFileSync(path, "utf-8")),
|
||||
apiOptions,
|
||||
);
|
||||
} catch (err) {
|
||||
const reason = err instanceof Error ? err.message : String(err);
|
||||
throw new Error(`Session file ${path} is corrupt: ${reason}`);
|
||||
}
|
||||
};
|
||||
+62
-11
@@ -125,18 +125,57 @@ export class Client {
|
||||
);
|
||||
}
|
||||
|
||||
static fromJSON(
|
||||
snapshot: ClientSnapshot,
|
||||
apiOptions?: ApiClientOptions,
|
||||
): Client {
|
||||
const api = new ApiClient({ ...apiOptions, authToken: snapshot.token });
|
||||
// Restore a client from a `toJSON()` snapshot. The snapshot usually comes
|
||||
// straight from `JSON.parse` of a file on disk, so every field is checked
|
||||
// before use; a bad one throws an error naming it. Needs `init()` first.
|
||||
static fromJSON(snapshot: unknown, apiOptions?: ApiClientOptions): Client {
|
||||
const invalid = (field: string, problem: string): Error =>
|
||||
new Error(`Invalid session data: ${field} ${problem}`);
|
||||
|
||||
if (typeof snapshot !== "object" || snapshot === null) {
|
||||
throw new Error("Invalid session data: not a JSON object");
|
||||
}
|
||||
const s = snapshot as Record<string, unknown>;
|
||||
for (const field of ["email", "token"]) {
|
||||
if (typeof s[field] !== "string" || s[field] === "") {
|
||||
throw invalid(field, "must be a non-empty string");
|
||||
}
|
||||
}
|
||||
if (!Number.isInteger(s.userID)) {
|
||||
throw invalid("userID", "must be an integer");
|
||||
}
|
||||
const key = (field: string): Uint8Array => {
|
||||
const value = s[field];
|
||||
if (typeof value !== "string") {
|
||||
throw invalid(field, "must be a base64 string");
|
||||
}
|
||||
let bytes: Uint8Array;
|
||||
try {
|
||||
bytes = fromBase64(value);
|
||||
} catch {
|
||||
throw invalid(field, "is not valid base64");
|
||||
}
|
||||
// The master key (secretbox) and the key pair (box) are all 32 bytes.
|
||||
if (bytes.length !== 32) {
|
||||
throw invalid(
|
||||
field,
|
||||
`must decode to 32 bytes, got ${bytes.length}`,
|
||||
);
|
||||
}
|
||||
return bytes;
|
||||
};
|
||||
|
||||
const api = new ApiClient({
|
||||
...apiOptions,
|
||||
authToken: s.token as string,
|
||||
});
|
||||
return new Client(
|
||||
api,
|
||||
snapshot.email,
|
||||
snapshot.userID,
|
||||
fromBase64(snapshot.masterKey),
|
||||
fromBase64(snapshot.secretKey),
|
||||
fromBase64(snapshot.publicKey),
|
||||
s.email as string,
|
||||
s.userID as number,
|
||||
key("masterKey"),
|
||||
key("secretKey"),
|
||||
key("publicKey"),
|
||||
);
|
||||
}
|
||||
|
||||
@@ -164,19 +203,29 @@ export class Client {
|
||||
|
||||
toJSON(): ClientSnapshot {
|
||||
this.assertLoggedIn();
|
||||
const token = this.api.getAuthToken();
|
||||
if (!token) {
|
||||
throw new Error("Cannot serialize client: it has no auth token");
|
||||
}
|
||||
return {
|
||||
email: this.email,
|
||||
userID: this.userID,
|
||||
token: this.api["token"]!,
|
||||
token,
|
||||
masterKey: toBase64(this.masterKey),
|
||||
secretKey: toBase64(this.secretKey),
|
||||
publicKey: toBase64(this.publicKey),
|
||||
};
|
||||
}
|
||||
|
||||
// Zeroes the key buffers in place, so any copy of the reference held
|
||||
// elsewhere is wiped too. Every method checks `assertLoggedIn` before
|
||||
// touching the keys, so nothing decrypts with the zeroed keys.
|
||||
logout(): void {
|
||||
this.loggedOut = true;
|
||||
this.api.clearAuthToken();
|
||||
this.masterKey.fill(0);
|
||||
this.secretKey.fill(0);
|
||||
this.publicKey.fill(0);
|
||||
}
|
||||
|
||||
// Enumerate collections changed since `sinceTime`. Live collections are
|
||||
@@ -192,6 +241,8 @@ export class Client {
|
||||
const { collections: raws } = await this.api.getJSON<{
|
||||
collections: RawCollection[];
|
||||
}>("/collections/v2", { sinceTime: args.sinceTime });
|
||||
// logout() may have zeroed the keys while the request was in flight.
|
||||
this.assertLoggedIn();
|
||||
|
||||
const collections: Collection[] = [];
|
||||
const deleted: number[] = [];
|
||||
|
||||
@@ -0,0 +1,22 @@
|
||||
import sodium, { type StateAddress } from "libsodium-wrappers-sumo";
|
||||
import { toBase64 } from "./encoding.js";
|
||||
|
||||
// The content hash an uploading client records in a file's metadata: unkeyed
|
||||
// BLAKE2b with a 64-byte output over the original's bytes, fed in chunks, as
|
||||
// standard base64 with padding. Named after the upstream client's functions.
|
||||
// The output length is read at call time for the same reason as
|
||||
// `streamTagFinal` in stream.ts: libsodium sets its constants only once ready.
|
||||
|
||||
export const chunkHashInit = (): StateAddress =>
|
||||
sodium.crypto_generichash_init(null, sodium.crypto_generichash_BYTES_MAX);
|
||||
|
||||
export const chunkHashUpdate = (state: StateAddress, chunk: Uint8Array): void =>
|
||||
sodium.crypto_generichash_update(state, chunk);
|
||||
|
||||
export const chunkHashFinal = (state: StateAddress): string =>
|
||||
toBase64(
|
||||
sodium.crypto_generichash_final(
|
||||
state,
|
||||
sodium.crypto_generichash_BYTES_MAX,
|
||||
),
|
||||
);
|
||||
@@ -7,6 +7,7 @@ export {
|
||||
} from "./encoding.js";
|
||||
export { deriveKEK, deriveLoginSubkey } from "./kdf.js";
|
||||
export { decryptBox, decryptSealed } from "./box.js";
|
||||
export { chunkHashFinal, chunkHashInit, chunkHashUpdate } from "./hash.js";
|
||||
export {
|
||||
decryptBlob,
|
||||
encryptBlob,
|
||||
|
||||
+163
-19
@@ -2,7 +2,11 @@ import { randomUUID } from "node:crypto";
|
||||
import { open, rename, rm } from "node:fs/promises";
|
||||
import type { FileHandle } from "node:fs/promises";
|
||||
import { dirname, join } from "node:path";
|
||||
import { Unzip, UnzipInflate } from "fflate";
|
||||
import {
|
||||
chunkHashFinal,
|
||||
chunkHashInit,
|
||||
chunkHashUpdate,
|
||||
fromBase64,
|
||||
initStreamPull,
|
||||
pullStreamChunk,
|
||||
@@ -11,6 +15,7 @@ import {
|
||||
streamTagFinal,
|
||||
} from "../crypto/index.js";
|
||||
import { TruncatedStreamError } from "../errors.js";
|
||||
import { sanitizeFileName } from "../filename.js";
|
||||
import { withRetry } from "../retry.js";
|
||||
import type { ApiClient } from "../api/client.js";
|
||||
import type { EnteFile } from "../model/types.js";
|
||||
@@ -97,6 +102,7 @@ const streamDecrypt = async (
|
||||
onProgress?.(totalPlain);
|
||||
};
|
||||
|
||||
try {
|
||||
for (;;) {
|
||||
const { done, value } = await reader.read();
|
||||
if (value && value.length > 0) {
|
||||
@@ -106,8 +112,9 @@ const streamDecrypt = async (
|
||||
|
||||
while (pendingBytes >= ENC_CHUNK_SIZE) {
|
||||
const encChunk = takeContiguous(ENC_CHUNK_SIZE);
|
||||
// A whole chunk that fails to authenticate while the stream carries
|
||||
// on is corruption, not truncation; that error propagates unchanged.
|
||||
// A whole chunk that fails to authenticate while the stream
|
||||
// carries on is corruption, not truncation; that error
|
||||
// propagates unchanged.
|
||||
const { plaintext, tag } = pullStreamChunk(state, encChunk);
|
||||
await consume(plaintext, tag);
|
||||
}
|
||||
@@ -117,14 +124,15 @@ const streamDecrypt = async (
|
||||
const buffer = takeContiguous(pendingBytes);
|
||||
// Whatever is left over once every whole chunk has been
|
||||
// consumed must be the stream's final chunk, and a final
|
||||
// chunk that actually arrived in full authenticates. If it
|
||||
// does not, the body stopped part-way through a chunk — the
|
||||
// ordinary shape of a dropped connection. Poly1305 cannot
|
||||
// tell a partial chunk from a corrupt one, so this is
|
||||
// reported as the truncation it almost always is, with the
|
||||
// authentication failure kept as the error's cause. Only the
|
||||
// pull is guarded: a sink failure on a chunk that did
|
||||
// authenticate is a disk error, not a truncation.
|
||||
// chunk that actually arrived in full authenticates. If
|
||||
// it does not, the body stopped part-way through a chunk
|
||||
// — the ordinary shape of a dropped connection. Poly1305
|
||||
// cannot tell a partial chunk from a corrupt one, so this
|
||||
// is reported as the truncation it almost always is, with
|
||||
// the authentication failure kept as the error's cause.
|
||||
// Only the pull is guarded: a sink failure on a chunk
|
||||
// that did authenticate is a disk error, not a
|
||||
// truncation.
|
||||
let pulled;
|
||||
try {
|
||||
pulled = pullStreamChunk(state, buffer);
|
||||
@@ -139,6 +147,9 @@ const streamDecrypt = async (
|
||||
break;
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
reader.releaseLock();
|
||||
}
|
||||
|
||||
// Only the last chunk of a secretstream carries TAG_FINAL. Everything a
|
||||
// dropped connection did deliver still decrypts and authenticates, so the
|
||||
@@ -157,6 +168,18 @@ const streamDecrypt = async (
|
||||
return totalPlain;
|
||||
};
|
||||
|
||||
// Fsync a file or a directory, so its contents (for a directory, its entries)
|
||||
// are on stable storage. Exported for the backup tree's copy, which needs the
|
||||
// same durability as the writer below.
|
||||
export const fsyncPath = async (path: string): Promise<void> => {
|
||||
const handle = await open(path, "r");
|
||||
try {
|
||||
await handle.sync();
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
};
|
||||
|
||||
// Stage a write to `destination` atomically and durably, then rename it into
|
||||
// place. `fill` writes the contents into the open temp file handle — either the
|
||||
// whole buffer at once (`writeAtomic`) or chunk by chunk as they decrypt
|
||||
@@ -192,16 +215,15 @@ const stageAtomic = async (
|
||||
} finally {
|
||||
await handle.close();
|
||||
}
|
||||
// `rename` replaces the destination's directory entry rather than
|
||||
// writing through it: an existing symlink at `destination` is
|
||||
// replaced, not followed, and the new file has the temp file's
|
||||
// permissions, not those of the file it replaced.
|
||||
await rename(tmpPath, destination);
|
||||
// Fsync the directory so the rename itself survives a crash: renaming
|
||||
// over a synced temp file still leaves the new directory entry in the
|
||||
// page cache until the directory is synced.
|
||||
const dirHandle = await open(dir, "r");
|
||||
try {
|
||||
await dirHandle.sync();
|
||||
} finally {
|
||||
await dirHandle.close();
|
||||
}
|
||||
await fsyncPath(dir);
|
||||
} catch (err) {
|
||||
// Best-effort cleanup. A failure to remove the temporary file must
|
||||
// never replace the error that actually explains what went wrong.
|
||||
@@ -219,31 +241,139 @@ export const writeAtomic = async (
|
||||
): Promise<void> =>
|
||||
stageAtomic(destination, (handle) => handle.writeFile(plaintext));
|
||||
|
||||
// Hashes an original's bytes as they are decrypted, for comparison with the
|
||||
// hash its uploader recorded.
|
||||
interface ContentHasher {
|
||||
update: (plaintext: Uint8Array) => void;
|
||||
digest: () => string;
|
||||
}
|
||||
|
||||
const fileHasher = (): ContentHasher => {
|
||||
const state = chunkHashInit();
|
||||
return {
|
||||
update: (plaintext) => chunkHashUpdate(state, plaintext),
|
||||
digest: () => chunkHashFinal(state),
|
||||
};
|
||||
};
|
||||
|
||||
// A live photo is stored as a ZIP of its image and its video, and its recorded
|
||||
// hash is `<imageHash>:<videoHash>`, each over that part's own bytes. Like the
|
||||
// upstream client's decoder, this takes the first entries whose names start
|
||||
// with `image` and `video`.
|
||||
//
|
||||
// The ZIP is chosen by its uploader and may expand enormously, so entries are
|
||||
// hashed as they decompress and never held. fflate's `Unzip` inflates each
|
||||
// push in one piece, and deflate expands at most about 1000-fold, so the ZIP
|
||||
// is pushed in 4 KiB slices to keep each decompressed piece near 4 MiB, one
|
||||
// plaintext chunk. Every entry is started, even one that is not hashed,
|
||||
// because fflate keeps an unstarted entry's data in memory.
|
||||
const livePhotoHasher = (fileID: number): ContentHasher => {
|
||||
const sliceSize = 4096;
|
||||
const fail = (message: string, cause?: unknown): Error =>
|
||||
new Error(`download: file ${fileID}: ${message}`, { cause });
|
||||
const claimed = new Set<string>();
|
||||
const hashes = new Map<string, string>();
|
||||
const unzip = new Unzip((entry) => {
|
||||
const part = ["image", "video"].find((p) => entry.name.startsWith(p));
|
||||
const target =
|
||||
part === undefined || claimed.has(part)
|
||||
? undefined
|
||||
: { part, state: chunkHashInit() };
|
||||
if (target !== undefined) claimed.add(target.part);
|
||||
entry.ondata = (err, data, final) => {
|
||||
if (err) throw err;
|
||||
if (target === undefined) return;
|
||||
chunkHashUpdate(target.state, data);
|
||||
if (final) hashes.set(target.part, chunkHashFinal(target.state));
|
||||
};
|
||||
entry.start();
|
||||
});
|
||||
unzip.register(UnzipInflate);
|
||||
// fflate reports a bad ZIP by throwing, sometimes a TypeError, which the
|
||||
// retry would take for a network failure; a bad ZIP is never retried.
|
||||
const push = (data: Uint8Array, final: boolean): void => {
|
||||
try {
|
||||
unzip.push(data, final);
|
||||
} catch (err) {
|
||||
throw fail("live photo is not a readable ZIP", err);
|
||||
}
|
||||
};
|
||||
return {
|
||||
update: (plaintext) => {
|
||||
for (let i = 0; i < plaintext.length; i += sliceSize) {
|
||||
push(plaintext.subarray(i, i + sliceSize), false);
|
||||
}
|
||||
},
|
||||
digest: () => {
|
||||
push(new Uint8Array(0), true);
|
||||
const image = hashes.get("image");
|
||||
const video = hashes.get("video");
|
||||
if (image === undefined || video === undefined) {
|
||||
throw fail(
|
||||
"live photo ZIP does not hold both an image and a video",
|
||||
);
|
||||
}
|
||||
return `${image}:${video}`;
|
||||
},
|
||||
};
|
||||
};
|
||||
|
||||
// Decrypt `stream` straight to `destination`, one plaintext chunk at a time,
|
||||
// under the atomic writer's temp-then-rename discipline. Memory stays bounded
|
||||
// by the chunk size: each decrypted chunk is written to the temp file and
|
||||
// dropped. The rename happens only after the stream authenticates as terminated
|
||||
// on TAG_FINAL; a truncated stream throws and leaves the destination untouched.
|
||||
// Returns the plaintext length written.
|
||||
//
|
||||
// `original` is the file whose original this is (none for a thumbnail, which
|
||||
// has no recorded hash). When its metadata has a hash, the decrypted bytes
|
||||
// must match it or nothing is stored. Both a plain file and a live photo's
|
||||
// parts are hashed as they stream. The mismatch error is not retried.
|
||||
const decryptToTemp = async (
|
||||
destination: string,
|
||||
stream: ReadableStream<Uint8Array>,
|
||||
header: Uint8Array,
|
||||
key: Uint8Array,
|
||||
onProgress?: ProgressCallback,
|
||||
original?: EnteFile,
|
||||
): Promise<number> => {
|
||||
const expected = original?.metadata.hash;
|
||||
const hasher =
|
||||
original === undefined || expected === undefined
|
||||
? undefined
|
||||
: original.metadata.fileType === "livePhoto"
|
||||
? livePhotoHasher(original.id)
|
||||
: fileHasher();
|
||||
let bytesWritten = 0;
|
||||
try {
|
||||
await stageAtomic(destination, async (handle) => {
|
||||
bytesWritten = await streamDecrypt(
|
||||
stream,
|
||||
header,
|
||||
key,
|
||||
async (plaintext) => {
|
||||
hasher?.update(plaintext);
|
||||
await handle.write(plaintext);
|
||||
},
|
||||
onProgress,
|
||||
);
|
||||
if (original === undefined || hasher === undefined) return;
|
||||
const actual = hasher.digest();
|
||||
if (actual !== expected) {
|
||||
throw new Error(
|
||||
`download: file ${original.id}: content hash ${actual} does not match the hash its uploader recorded, ${expected}`,
|
||||
);
|
||||
}
|
||||
});
|
||||
} catch (err) {
|
||||
// Cancel the body so its connection is closed now rather than held
|
||||
// until the stream is garbage collected. A backup run carries on past
|
||||
// a failed file, so without this every failure would hold a socket.
|
||||
// This covers every failure, including a temp file that cannot be
|
||||
// opened and a header that is rejected before the body is read.
|
||||
await stream.cancel(err).catch(() => undefined);
|
||||
throw err;
|
||||
}
|
||||
return bytesWritten;
|
||||
};
|
||||
|
||||
@@ -274,10 +404,18 @@ const fetchAndDecrypt = async (
|
||||
key: Uint8Array,
|
||||
destination: string,
|
||||
onProgress?: ProgressCallback,
|
||||
original?: EnteFile,
|
||||
): Promise<number> =>
|
||||
withRetry(async () => {
|
||||
const stream = await openStream();
|
||||
return decryptToTemp(destination, stream, header, key, onProgress);
|
||||
return decryptToTemp(
|
||||
destination,
|
||||
stream,
|
||||
header,
|
||||
key,
|
||||
onProgress,
|
||||
original,
|
||||
);
|
||||
}, api.getRetryOptions());
|
||||
|
||||
export const downloadFile = async (
|
||||
@@ -286,7 +424,10 @@ export const downloadFile = async (
|
||||
outPath?: string,
|
||||
onProgress?: ProgressCallback,
|
||||
): Promise<DownloadResult> => {
|
||||
const resolvedPath = outPath ?? file.metadata.title;
|
||||
// `outPath` is the caller's and is used as is; the title is the server's
|
||||
// and is sanitized so it can only name a file in the current directory.
|
||||
const resolvedPath =
|
||||
outPath ?? sanitizeFileName(file.metadata.title, `file-${file.id}`);
|
||||
const header = fromBase64(file.file.decryptionHeader);
|
||||
const bytesWritten = await fetchAndDecrypt(
|
||||
api,
|
||||
@@ -295,6 +436,7 @@ export const downloadFile = async (
|
||||
file.key,
|
||||
resolvedPath,
|
||||
onProgress,
|
||||
file,
|
||||
);
|
||||
return { path: resolvedPath, bytesWritten };
|
||||
};
|
||||
@@ -305,7 +447,9 @@ export const downloadThumbnail = async (
|
||||
outPath?: string,
|
||||
onProgress?: ProgressCallback,
|
||||
): Promise<DownloadResult> => {
|
||||
const resolvedPath = outPath ?? `thumb_${file.metadata.title}`;
|
||||
const resolvedPath =
|
||||
outPath ??
|
||||
`thumb_${sanitizeFileName(file.metadata.title, `file-${file.id}`)}`;
|
||||
const header = fromBase64(file.thumbnail.decryptionHeader);
|
||||
const bytesWritten = await fetchAndDecrypt(
|
||||
api,
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
// File names built from server-supplied metadata.
|
||||
//
|
||||
// A file's title and a collection's name are decrypted from data the server
|
||||
// hands us, and quak does not trust the server. Any name taken from them and
|
||||
// used on disk goes through here, so it can only ever name one file inside the
|
||||
// directory the caller chose: never a path, never `..`, never hidden, never a
|
||||
// Windows device name.
|
||||
//
|
||||
// A path the user typed (`--out`, `outPath`) is not passed through here: the
|
||||
// caller is trusted, the server is not.
|
||||
|
||||
import { extname } from "node:path";
|
||||
|
||||
// Path separators, characters Windows forbids in file names, and control
|
||||
// characters (NUL included).
|
||||
// eslint-disable-next-line no-control-regex
|
||||
const UNSAFE_CHARACTERS = /[/\\:*?"<>|\x00-\x1f\x7f]/g;
|
||||
|
||||
// Names Windows reserves for devices, with or without an extension.
|
||||
const RESERVED_DEVICE_NAME = /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\.|$)/i;
|
||||
|
||||
// `name` made safe to use as a single file name. Each unsafe character becomes
|
||||
// `_`, a leading run of dots becomes one `_`, and a device name gets a leading
|
||||
// `_`. A name with none of these comes back unchanged. An empty name becomes
|
||||
// `fallback`, which the caller derives from the record's ID.
|
||||
export const sanitizeFileName = (name: string, fallback: string): string => {
|
||||
if (name === "") return fallback;
|
||||
const cleaned = name.replace(UNSAFE_CHARACTERS, "_").replace(/^\.+/, "_");
|
||||
return RESERVED_DEVICE_NAME.test(cleaned) ? `_${cleaned}` : cleaned;
|
||||
};
|
||||
|
||||
// The extension of `title` (".jpg"), or ".bin" when it has none or it holds
|
||||
// anything but letters and digits.
|
||||
export const safeExtension = (title: string): string => {
|
||||
const ext = extname(title);
|
||||
return /^\.[A-Za-z0-9]+$/.test(ext) ? ext : ".bin";
|
||||
};
|
||||
+5
-1
@@ -1,4 +1,8 @@
|
||||
export const VERSION = "0.0.0";
|
||||
// package.json is the one place the version is written. tsc copies it to
|
||||
// dist/package.json, so this path resolves from source and from dist/src/.
|
||||
import pkg from "../package.json" with { type: "json" };
|
||||
|
||||
export const VERSION: string = pkg.version;
|
||||
|
||||
export {
|
||||
Client,
|
||||
|
||||
+9
-10
@@ -15,12 +15,12 @@
|
||||
// Integrity. The reused streaming decrypt is the enforced guarantee: every
|
||||
// chunk is authenticated and the writer renames the file into place only once
|
||||
// the stream ends on TAG_FINAL, so a truncated or corrupt fetch throws and
|
||||
// nothing is stored. On top of that this module refuses to record a stored file
|
||||
// that came out empty. The design also asks for a content-hash comparison
|
||||
// against `FileMetadata.hash` (with a `fileSize` fallback); that is deferred —
|
||||
// see the PR — because the exact hash construction cannot be confirmed against
|
||||
// the repo's fixtures and `FileBlob.size` is the encrypted object size, not the
|
||||
// decrypted length this layer has.
|
||||
// nothing is stored. For an original whose metadata records a content hash
|
||||
// (`FileMetadata.hash`), the writer also hashes the decrypted bytes and stores
|
||||
// nothing if they differ, failing the fetch with an error naming the file. An
|
||||
// original with no recorded hash is stored unchecked, as the upstream client
|
||||
// does; thumbnails have none. On top of that this module refuses to record a
|
||||
// stored file that came out empty.
|
||||
|
||||
import { existsSync, statSync } from "node:fs";
|
||||
import {
|
||||
@@ -40,6 +40,7 @@ import {
|
||||
downloadThumbnail,
|
||||
type ProgressCallback,
|
||||
} from "../download/index.js";
|
||||
import { safeExtension } from "../filename.js";
|
||||
import type { EnteFile } from "../model/types.js";
|
||||
import type { Priority, RequestPools } from "./pools.js";
|
||||
|
||||
@@ -209,10 +210,8 @@ class AbortDrop extends Error {
|
||||
}
|
||||
}
|
||||
|
||||
const originalName = (file: EnteFile): string => {
|
||||
const ext = extname(file.metadata.title || "") || ".bin";
|
||||
return `${file.id}${ext}`;
|
||||
};
|
||||
const originalName = (file: EnteFile): string =>
|
||||
`${file.id}${safeExtension(file.metadata.title)}`;
|
||||
|
||||
// The fileID a cache filename encodes, or undefined when the name is not one
|
||||
// the cache writes (`<digits><ext>`).
|
||||
|
||||
+17
-10
@@ -266,8 +266,9 @@ export class Library {
|
||||
// that, a fresh read propagates it.
|
||||
private cycle?: Promise<void>;
|
||||
// Guards the ML fetch pass so a slow backfill never runs twice at once; a
|
||||
// refresh whose pass is still running kicks nothing new.
|
||||
private mlFetching = false;
|
||||
// refresh whose pass is still running kicks nothing new. Holds the running
|
||||
// pass, so `close()` can wait for it.
|
||||
private mlFetch?: Promise<void>;
|
||||
private closed = false;
|
||||
private lastRefreshAt?: number;
|
||||
private lastError?: string;
|
||||
@@ -560,14 +561,21 @@ export class Library {
|
||||
}
|
||||
|
||||
// Stop the background timer. Idempotent. An in-flight refresh is left to
|
||||
// finish; it will not schedule another cycle once closed.
|
||||
close(): void {
|
||||
// finish; it will not schedule another cycle once closed. The returned
|
||||
// promise resolves once that refresh (including its cache write), the ML
|
||||
// fetch pass and the precache fetches already running have all finished,
|
||||
// so a caller can then remove the cache directory. A refresh failure is
|
||||
// reported through `status()`, not thrown here.
|
||||
async close(): Promise<void> {
|
||||
this.closed = true;
|
||||
this.precache?.close();
|
||||
const precacheClosed = this.precache?.close();
|
||||
if (this.timer !== undefined) {
|
||||
clearTimeout(this.timer);
|
||||
this.timer = undefined;
|
||||
}
|
||||
await this.cycle?.catch(() => {});
|
||||
await this.mlFetch;
|
||||
await precacheClosed;
|
||||
}
|
||||
|
||||
private scheduleNext(): void {
|
||||
@@ -631,7 +639,9 @@ export class Library {
|
||||
// outside the refresh's success/failure so a fetch or disk problem
|
||||
// there never marks the metadata refresh failed, and it is not
|
||||
// awaited so it never stalls the refresh interval.
|
||||
void this.runMLFetch();
|
||||
this.mlFetch ??= this.runMLFetch().finally(() => {
|
||||
this.mlFetch = undefined;
|
||||
});
|
||||
} catch (err) {
|
||||
const error = err instanceof Error ? err.message : String(err);
|
||||
this.lastError = error;
|
||||
@@ -745,13 +755,12 @@ export class Library {
|
||||
// Bind so the call keeps the client as its receiver when invoked
|
||||
// through the pool below.
|
||||
const fetchMLData = this.client.fetchMLData?.bind(this.client);
|
||||
if (!mldata || !fetchMLData || this.closed || this.mlFetching) return;
|
||||
if (!mldata || !fetchMLData || this.closed) return;
|
||||
|
||||
const files = this.uniqueFiles();
|
||||
const needed = mldata.neededFor(files);
|
||||
if (needed.length === 0) return;
|
||||
|
||||
this.mlFetching = true;
|
||||
this.emit({ operation: "fetchMLData", status: "started" });
|
||||
try {
|
||||
const fileKeys = new Map<number, Uint8Array>();
|
||||
@@ -784,8 +793,6 @@ export class Library {
|
||||
const error = err instanceof Error ? err.message : String(err);
|
||||
this.lastMLError = error;
|
||||
this.emit({ operation: "fetchMLData", status: "failed", error });
|
||||
} finally {
|
||||
this.mlFetching = false;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+16
-12
@@ -92,9 +92,10 @@ export class Precache {
|
||||
private pinned = new Set<number>();
|
||||
|
||||
// A sweep runs at most once per fill at a time; a re-kick while one runs is
|
||||
// a no-op, and the next refresh re-kicks after it finishes.
|
||||
private thumbRunning = false;
|
||||
private originalsRunning = false;
|
||||
// a no-op, and the next refresh re-kicks after it finishes. Each holds the
|
||||
// running sweep, so `close()` can wait for it.
|
||||
private thumbSweep?: Promise<void>;
|
||||
private originalsSweep?: Promise<void>;
|
||||
private readonly aborter = new AbortController();
|
||||
private closed = false;
|
||||
|
||||
@@ -191,15 +192,19 @@ export class Precache {
|
||||
}
|
||||
|
||||
// Stop the fills. In-flight fetches are left to settle; queued ones drop.
|
||||
close(): void {
|
||||
// Resolves once both sweeps have finished, so nothing is still writing.
|
||||
async close(): Promise<void> {
|
||||
this.closed = true;
|
||||
this.aborter.abort();
|
||||
await Promise.all([
|
||||
this.thumbSweep?.catch(() => {}),
|
||||
this.originalsSweep?.catch(() => {}),
|
||||
]);
|
||||
}
|
||||
|
||||
private kickThumbnails(): void {
|
||||
if (this.thumbRunning) return;
|
||||
this.thumbRunning = true;
|
||||
void this.sweep(
|
||||
if (this.thumbSweep) return;
|
||||
this.thumbSweep = this.sweep(
|
||||
"precacheThumbnails",
|
||||
() => this.thumbOrder,
|
||||
(id) => this.cache!.pathsFor(id).thumbnailPath !== undefined,
|
||||
@@ -211,14 +216,13 @@ export class Precache {
|
||||
signal: this.aborter.signal,
|
||||
}),
|
||||
).finally(() => {
|
||||
this.thumbRunning = false;
|
||||
this.thumbSweep = undefined;
|
||||
});
|
||||
}
|
||||
|
||||
private kickOriginals(): void {
|
||||
if (this.originalsRunning) return;
|
||||
this.originalsRunning = true;
|
||||
void this.sweep(
|
||||
if (this.originalsSweep) return;
|
||||
this.originalsSweep = this.sweep(
|
||||
"precacheOriginals",
|
||||
() => this.originalsOrder,
|
||||
(id) => this.cache!.pathsFor(id).originalPath !== undefined,
|
||||
@@ -229,7 +233,7 @@ export class Precache {
|
||||
signal: this.aborter.signal,
|
||||
}),
|
||||
).finally(() => {
|
||||
this.originalsRunning = false;
|
||||
this.originalsSweep = undefined;
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
+56
-32
@@ -4,6 +4,7 @@ import * as jpeg from "jpeg-js";
|
||||
import exifReader from "exif-reader";
|
||||
import type { Client } from "./client.js";
|
||||
import type { Library, Photo } from "./library/index.js";
|
||||
import { sanitizeFileName } from "./filename.js";
|
||||
import { fetchMLData } from "./mldata-fetch.js";
|
||||
import type { EnteFile } from "./model/types.js";
|
||||
|
||||
@@ -14,45 +15,66 @@ export interface MetadataBackupOptions {
|
||||
onProgress?: ProgressCallback;
|
||||
}
|
||||
|
||||
const sanitizePath = (name: string): string =>
|
||||
name.replace(/[/\\:*?"<>|]/g, "_").replace(/^\.+/, "_");
|
||||
|
||||
// Extract the raw EXIF APP1 segment from JPEG bytes. Returns the EXIF
|
||||
// data buffer (starting after the APP1 length field, at the "Exif\0\0"
|
||||
// header) or undefined if no APP1 marker is found.
|
||||
const extractExifFromJpeg = (buf: Uint8Array): Buffer | undefined => {
|
||||
if (buf[0] !== 0xff || buf[1] !== 0xd8) return undefined;
|
||||
// Find the raw EXIF APP1 segment in JPEG bytes. Returns `exif` (the segment
|
||||
// data, starting at the "Exif\0\0" header) when there is one, nothing when the
|
||||
// bytes are not a JPEG or carry no EXIF, and `error` when the segment layout is
|
||||
// malformed. Each segment length is checked against the bytes that remain and
|
||||
// each step moves forward by at least 4 bytes, so the scan ends on any input.
|
||||
export const extractExifFromJpeg = (
|
||||
buf: Uint8Array,
|
||||
): { exif?: Buffer; error?: string } => {
|
||||
if (buf[0] !== 0xff || buf[1] !== 0xd8) return {};
|
||||
let offset = 2;
|
||||
while (offset < buf.length - 1) {
|
||||
if (buf[offset] !== 0xff) return undefined;
|
||||
while (offset < buf.length) {
|
||||
if (offset + 2 > buf.length)
|
||||
return { error: `truncated segment marker at byte ${offset}` };
|
||||
if (buf[offset] !== 0xff)
|
||||
return { error: `no segment marker at byte ${offset}` };
|
||||
const marker = buf[offset + 1]!;
|
||||
if (marker === 0xda) break; // start of scan, no more markers
|
||||
if (offset + 3 >= buf.length) break;
|
||||
if (marker === 0xda) return {}; // start of scan, no more markers
|
||||
if (offset + 4 > buf.length)
|
||||
return { error: `truncated segment length at byte ${offset}` };
|
||||
const len = (buf[offset + 2]! << 8) | buf[offset + 3]!;
|
||||
// The length counts its own two bytes, so anything under 2 is invalid.
|
||||
if (len < 2)
|
||||
return {
|
||||
error: `segment length ${len} at byte ${offset} is too small`,
|
||||
};
|
||||
if (offset + 2 + len > buf.length)
|
||||
return {
|
||||
error: `segment length ${len} at byte ${offset} runs past the end of the file`,
|
||||
};
|
||||
if (marker === 0xe1) {
|
||||
// APP1 — check for "Exif\0\0" header
|
||||
// APP1 — check for "Exif\0\0" header. A length under 8 cannot hold
|
||||
// the six-byte header, so the segment is not EXIF; below 6 the
|
||||
// bytes compared would also lie past the segment.
|
||||
if (
|
||||
len >= 8 &&
|
||||
buf[offset + 4] === 0x45 &&
|
||||
buf[offset + 5] === 0x78 &&
|
||||
buf[offset + 6] === 0x69 &&
|
||||
buf[offset + 7] === 0x66
|
||||
) {
|
||||
return Buffer.from(
|
||||
return {
|
||||
exif: Buffer.from(
|
||||
buf.buffer,
|
||||
buf.byteOffset + offset + 4,
|
||||
len - 2,
|
||||
);
|
||||
),
|
||||
};
|
||||
}
|
||||
}
|
||||
offset += 2 + len;
|
||||
}
|
||||
return undefined;
|
||||
return { error: "file ends before the image data" };
|
||||
};
|
||||
|
||||
const extractImageMetadata = (
|
||||
// Extract dimensions, EXIF and XMP from a file's bytes. When the EXIF segment
|
||||
// is malformed or cannot be parsed, the record carries the reason in
|
||||
// `exifError`.
|
||||
export const extractImageMetadata = (
|
||||
fileBytes: Uint8Array,
|
||||
): Record<string, unknown> | undefined => {
|
||||
try {
|
||||
const result: Record<string, unknown> = {};
|
||||
|
||||
// Try to get dimensions from JPEG decode
|
||||
@@ -65,15 +87,19 @@ const extractImageMetadata = (
|
||||
result.width = decoded.width;
|
||||
result.height = decoded.height;
|
||||
} catch {
|
||||
// Not a JPEG or corrupt; still try EXIF extraction
|
||||
// Not every original is a JPEG (PNG, HEIC, video), so a failed decode
|
||||
// is expected and only means no dimensions; a malformed JPEG is still
|
||||
// reported below through `exifError`.
|
||||
}
|
||||
|
||||
const exifBuf = extractExifFromJpeg(fileBytes);
|
||||
if (exifBuf) {
|
||||
const { exif, error } = extractExifFromJpeg(fileBytes);
|
||||
if (error) result.exifError = error;
|
||||
if (exif) {
|
||||
try {
|
||||
result.exif = exifReader(exifBuf);
|
||||
} catch {
|
||||
result.exifRaw = exifBuf.toString("base64");
|
||||
result.exif = exifReader(exif);
|
||||
} catch (err) {
|
||||
result.exifRaw = exif.toString("base64");
|
||||
result.exifError = err instanceof Error ? err.message : String(err);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -93,9 +119,6 @@ const extractImageMetadata = (
|
||||
}
|
||||
|
||||
return Object.keys(result).length > 0 ? result : undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
|
||||
// Read a file's original bytes through the library's content cache and extract
|
||||
@@ -105,13 +128,9 @@ const extractImageMetadata = (
|
||||
const extractExif = async (
|
||||
photo: Photo,
|
||||
): Promise<Record<string, unknown> | undefined> => {
|
||||
try {
|
||||
const { path } = await photo.original();
|
||||
const fileBytes = new Uint8Array(readFileSync(path));
|
||||
return extractImageMetadata(fileBytes);
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
};
|
||||
|
||||
// Dump every decrypted metadata layer the account holds into a directory tree
|
||||
@@ -151,7 +170,7 @@ export const runMetadataBackup = async (
|
||||
const col = lib.getCollection(album.collectionID);
|
||||
if (!col) continue;
|
||||
|
||||
const dirName = `${col.id}-${sanitizePath(col.name || "unnamed")}`;
|
||||
const dirName = `${col.id}-${sanitizeFileName(col.name, "unnamed")}`;
|
||||
const colDir = join(outDir, "collections", dirName);
|
||||
mkdirSync(colDir, { recursive: true });
|
||||
|
||||
@@ -217,8 +236,13 @@ export const runMetadataBackup = async (
|
||||
|
||||
if (wantExif && !writtenFileIDs.has(file.id)) {
|
||||
log(`[${file.metadata.title}] Extracting EXIF...`);
|
||||
try {
|
||||
const exifData = await extractExif(photo);
|
||||
if (exifData) fileMeta.imageMetadata = exifData;
|
||||
} catch (err) {
|
||||
fileMeta.imageMetadataError =
|
||||
err instanceof Error ? err.message : String(err);
|
||||
}
|
||||
}
|
||||
writtenFileIDs.add(file.id);
|
||||
|
||||
|
||||
+33
-2
@@ -34,6 +34,28 @@ const FILE_TYPE_MAP: Record<number, FileType> = {
|
||||
|
||||
const parseFileType = (n: number): FileType => FILE_TYPE_MAP[n] ?? "unknown";
|
||||
|
||||
// The hash the uploading client recorded for the original's bytes, read the
|
||||
// way the upstream client's `metadataHash` reads it: `hash` if present,
|
||||
// otherwise, for a live photo from an older client that wrote the two parts
|
||||
// separately, `<imageHash>:<videoHash>`. A field that is not a non-empty
|
||||
// string counts as absent, and a file with no hash at all is normal.
|
||||
const expectedHash = (json: Record<string, unknown>): string | undefined => {
|
||||
const text = (v: unknown): string | undefined =>
|
||||
typeof v === "string" && v !== "" ? v : undefined;
|
||||
const hash = text(json.hash);
|
||||
if (hash !== undefined) return hash;
|
||||
const imageHash = text(json.imageHash);
|
||||
const videoHash = text(json.videoHash);
|
||||
if (
|
||||
json.fileType === 2 &&
|
||||
imageHash !== undefined &&
|
||||
videoHash !== undefined
|
||||
) {
|
||||
return `${imageHash}:${videoHash}`;
|
||||
}
|
||||
return undefined;
|
||||
};
|
||||
|
||||
export const decryptCollection = (
|
||||
raw: RawCollection,
|
||||
keys: KeyMaterial,
|
||||
@@ -98,15 +120,24 @@ export const decryptFile = (
|
||||
key,
|
||||
);
|
||||
const metadataJSON = JSON.parse(new TextDecoder().decode(metadataBytes));
|
||||
if (
|
||||
typeof metadataJSON !== "object" ||
|
||||
metadataJSON === null ||
|
||||
Array.isArray(metadataJSON)
|
||||
) {
|
||||
throw new Error(`file ${raw.id}: metadata is not a JSON object`);
|
||||
}
|
||||
|
||||
const metadata: FileMetadata = {
|
||||
title: metadataJSON.title ?? "",
|
||||
// The server controls this JSON: a title that is missing or not a
|
||||
// string becomes "", never an arbitrary value.
|
||||
title: typeof metadataJSON.title === "string" ? metadataJSON.title : "",
|
||||
fileType: parseFileType(metadataJSON.fileType ?? -1),
|
||||
creationTime: metadataJSON.creationTime ?? 0,
|
||||
modificationTime: metadataJSON.modificationTime ?? 0,
|
||||
latitude: metadataJSON.latitude,
|
||||
longitude: metadataJSON.longitude,
|
||||
hash: metadataJSON.hash,
|
||||
hash: expectedHash(metadataJSON),
|
||||
};
|
||||
|
||||
const magicMetadata = decryptMagicMetadata(raw.magicMetadata, key);
|
||||
|
||||
@@ -29,6 +29,9 @@ export interface FileMetadata {
|
||||
modificationTime: Microseconds;
|
||||
latitude?: number;
|
||||
longitude?: number;
|
||||
// The content hash the uploader recorded (see `expectedHash` in
|
||||
// decrypt.ts); `downloadFile` refuses an original that does not match it.
|
||||
// Absent for files from very old clients.
|
||||
hash?: string;
|
||||
}
|
||||
|
||||
|
||||
+25
-12
@@ -88,18 +88,21 @@ const MAX_CAUSE_DEPTH = 8;
|
||||
// errno on the error it throws — it hangs the underlying socket error off
|
||||
// `cause`, sometimes more than one level down — so a classifier that only read
|
||||
// the top-level error would see a bare `Error` and call every dropped
|
||||
// connection permanent.
|
||||
const causeCodes = (err: unknown): string[] => {
|
||||
// connection permanent. `complete` is false when the walk stopped at the
|
||||
// depth limit with more of the chain still below it.
|
||||
const causeCodes = (err: unknown): { codes: string[]; complete: boolean } => {
|
||||
const codes: string[] = [];
|
||||
let current: unknown = err;
|
||||
for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
|
||||
if (current === null || typeof current !== "object") break;
|
||||
if (current === null || typeof current !== "object") {
|
||||
return { codes, complete: true };
|
||||
}
|
||||
const { code, cause } = current as { code?: unknown; cause?: unknown };
|
||||
if (typeof code === "string") codes.push(code);
|
||||
if (cause === current) break;
|
||||
if (cause === current) return { codes, complete: true };
|
||||
current = cause;
|
||||
}
|
||||
return codes;
|
||||
return { codes, complete: current === null || typeof current !== "object" };
|
||||
};
|
||||
|
||||
const isAbort = (err: unknown): boolean => {
|
||||
@@ -145,15 +148,15 @@ export const isRetryable = (err: unknown): boolean => {
|
||||
// have succeeded; the cost of the imprecision is bounded by the attempt
|
||||
// count.
|
||||
if (err instanceof TypeError) return true;
|
||||
return causeCodes(err).some((code) => TRANSPORT_CODES.has(code));
|
||||
return causeCodes(err).codes.some((code) => TRANSPORT_CODES.has(code));
|
||||
};
|
||||
|
||||
// Could the first attempt already have taken effect on the server?
|
||||
//
|
||||
// `isRetryable` is the wrong question for a request that changes state.
|
||||
// quak's non-idempotent calls are `/users/srp/create-session`,
|
||||
// `/users/two-factor/verify` — which consumes one of a small number of 2FA
|
||||
// attempts — and `/files/thumbnail`. They are replayed only on the failures in
|
||||
// `postJSON` and `putJSON` use this for every `POST` and `PUT` listed in the
|
||||
// README under "Endpoints used"; verifying a second factor, for one, consumes
|
||||
// one of a small number of attempts. They are replayed only on the failures in
|
||||
// `CONNECT_CODES`, which establish that no TCP connection to the server ever
|
||||
// existed: there was no address to connect to, or the peer refused the
|
||||
// connection outright. A request byte cannot have been transmitted, so the
|
||||
@@ -162,9 +165,19 @@ export const isRetryable = (err: unknown): boolean => {
|
||||
// Everything else is ambiguous. A 5xx proves the server did process the
|
||||
// request. A reset or a broken pipe can arrive after it was fully sent and
|
||||
// acted on. A routing errno can be delivered on an established socket. A
|
||||
// deadline says nothing at all about the server's state.
|
||||
export const isSafeToReplay = (err: unknown): boolean =>
|
||||
isRetryable(err) && causeCodes(err).some((code) => CONNECT_CODES.has(code));
|
||||
// deadline says nothing at all about the server's state. So every errno in the
|
||||
// cause chain must be a connect errno: one other errno anywhere in the chain
|
||||
// is doubt, and doubt is not replayed. A chain longer than the walk is doubt
|
||||
// too: the links below the limit were never read.
|
||||
export const isSafeToReplay = (err: unknown): boolean => {
|
||||
const { codes, complete } = causeCodes(err);
|
||||
return (
|
||||
isRetryable(err) &&
|
||||
complete &&
|
||||
codes.length > 0 &&
|
||||
codes.every((code) => CONNECT_CODES.has(code))
|
||||
);
|
||||
};
|
||||
|
||||
export interface WithRetryOptions extends RetryOptions {
|
||||
isRetryable?: (err: unknown) => boolean;
|
||||
|
||||
+313
-14
@@ -38,14 +38,18 @@
|
||||
* the network. The fake records every call for assertion.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { describe, expect, it, vi } from "vitest";
|
||||
import {
|
||||
ApiClient,
|
||||
ApiError,
|
||||
DEFAULT_DOWNLOAD_TIMEOUT_MS,
|
||||
DEFAULT_REQUEST_TIMEOUT_MS,
|
||||
} from "../../src/api/client.js";
|
||||
import type { RetryOptions } from "../../src/retry.js";
|
||||
import {
|
||||
isRetryable,
|
||||
isSafeToReplay,
|
||||
type RetryOptions,
|
||||
} from "../../src/retry.js";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Test helpers
|
||||
@@ -385,6 +389,103 @@ describe("ApiClient custom origins", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("ApiClient request URLs", () => {
|
||||
it("accepts a path with or without a leading slash", async () => {
|
||||
const { fetch, calls } = recordingFetch(
|
||||
jsonResponse({}),
|
||||
jsonResponse({}),
|
||||
jsonResponse({}),
|
||||
);
|
||||
const client = new ApiClient({ fetch });
|
||||
await client.getJSON("health");
|
||||
await client.postJSON("users/ott", {});
|
||||
await client.putJSON("/files/thumbnail", {});
|
||||
|
||||
expect(calls.map((c) => c.url)).toEqual([
|
||||
"https://api.ente.io/health",
|
||||
"https://api.ente.io/users/ott",
|
||||
"https://api.ente.io/files/thumbnail",
|
||||
]);
|
||||
});
|
||||
|
||||
it("accepts an apiOrigin with a trailing slash", async () => {
|
||||
const { fetch, calls } = recordingFetch(jsonResponse({}));
|
||||
const client = new ApiClient({
|
||||
fetch,
|
||||
apiOrigin: "https://my-ente.example.com/",
|
||||
});
|
||||
await client.getJSON("/health");
|
||||
|
||||
expect(calls[0]!.url).toBe("https://my-ente.example.com/health");
|
||||
});
|
||||
|
||||
it("keeps a base path in a self-hosted apiOrigin for every request", async () => {
|
||||
const body = new Uint8Array([1]);
|
||||
const { fetch, calls } = recordingFetch(
|
||||
jsonResponse({}),
|
||||
jsonResponse({}),
|
||||
jsonResponse({}),
|
||||
streamResponse(body),
|
||||
streamResponse(body),
|
||||
);
|
||||
const client = new ApiClient({
|
||||
fetch,
|
||||
apiOrigin: "https://example.com/ente/",
|
||||
});
|
||||
await client.getJSON("/collections/v2", { sinceTime: 0 });
|
||||
await client.postJSON("/users/ott", {});
|
||||
await client.putJSON("/files/thumbnail", {});
|
||||
await client.getFileStream(99);
|
||||
await client.getThumbnailStream(77);
|
||||
|
||||
expect(calls.map((c) => c.url)).toEqual([
|
||||
"https://example.com/ente/collections/v2?sinceTime=0",
|
||||
"https://example.com/ente/users/ott",
|
||||
"https://example.com/ente/files/thumbnail",
|
||||
"https://example.com/ente/files/download/99",
|
||||
"https://example.com/ente/files/preview/77",
|
||||
]);
|
||||
});
|
||||
|
||||
it("percent-encodes query parameters and skips undefined ones", async () => {
|
||||
const { fetch, calls } = recordingFetch(jsonResponse({}));
|
||||
const client = new ApiClient({ fetch });
|
||||
await client.getJSON("/search", {
|
||||
q: "a&b=c/d é",
|
||||
limit: 5,
|
||||
cursor: undefined,
|
||||
});
|
||||
|
||||
const url = new URL(calls[0]!.url);
|
||||
expect(url.pathname).toBe("/search");
|
||||
expect(url.search).toBe("?q=a%26b%3Dc%2Fd+%C3%A9&limit=5");
|
||||
expect(url.searchParams.get("q")).toBe("a&b=c/d é");
|
||||
});
|
||||
|
||||
it("rejects a path that carries its own query string", async () => {
|
||||
const { fetch, calls } = recordingFetch();
|
||||
const client = new ApiClient({ fetch });
|
||||
|
||||
await expect(client.getJSON("/diff?sinceTime=0")).rejects.toThrow(
|
||||
/must not contain "\?"/,
|
||||
);
|
||||
await expect(client.postJSON("/users/ott?x=1", {})).rejects.toThrow(
|
||||
/must not contain "\?"/,
|
||||
);
|
||||
expect(calls).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("rejects a path that carries a fragment", async () => {
|
||||
const { fetch, calls } = recordingFetch();
|
||||
const client = new ApiClient({ fetch });
|
||||
|
||||
await expect(client.getJSON("/diff#top")).rejects.toThrow(
|
||||
/must not contain "\?" or "#"/,
|
||||
);
|
||||
expect(calls).toHaveLength(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("ApiError", () => {
|
||||
it("throws ApiError on 4xx with status, code, requestID", async () => {
|
||||
const { fetch } = recordingFetch(
|
||||
@@ -639,17 +740,33 @@ describe("ApiClient retries", () => {
|
||||
expect(policy.baseDelayMs).toBe(7);
|
||||
expect(policy.maxDelayMs).toBe(11);
|
||||
});
|
||||
|
||||
it("does not let a caller change its settings through that policy", async () => {
|
||||
const { fetch, calls } = scriptedFetch(
|
||||
textResponse("boom", 500),
|
||||
textResponse("boom", 500),
|
||||
textResponse("boom", 500),
|
||||
);
|
||||
const client = new ApiClient({
|
||||
fetch,
|
||||
retry: { ...noWait, attempts: 2 },
|
||||
});
|
||||
|
||||
client.getRetryOptions().attempts = 3;
|
||||
|
||||
expect(client.getRetryOptions().attempts).toBe(2);
|
||||
await expect(client.getJSON("/x")).rejects.toBeInstanceOf(ApiError);
|
||||
expect(calls).toHaveLength(2);
|
||||
});
|
||||
});
|
||||
|
||||
describe("ApiClient timeouts", () => {
|
||||
it("ships bounded default deadlines", () => {
|
||||
// Asserted here so the README and the code cannot drift. Two numbers
|
||||
// rather than one, because a deadline that is sane for a JSON call is
|
||||
// nowhere near enough for a multi-gigabyte body, and a deadline long
|
||||
// enough for that body would let a hung API call stall a backup for
|
||||
// ten minutes.
|
||||
// Asserted here so the README and the code cannot drift. The request
|
||||
// deadline bounds a whole JSON call; the download deadline is an idle
|
||||
// one, measured from the last byte that arrived.
|
||||
expect(DEFAULT_REQUEST_TIMEOUT_MS).toBe(30_000);
|
||||
expect(DEFAULT_DOWNLOAD_TIMEOUT_MS).toBe(600_000);
|
||||
expect(DEFAULT_DOWNLOAD_TIMEOUT_MS).toBe(60_000);
|
||||
});
|
||||
|
||||
it("attaches an abort signal to every request", async () => {
|
||||
@@ -693,6 +810,51 @@ describe("ApiClient timeouts", () => {
|
||||
expect(new Set(signals).size).toBe(3);
|
||||
}, 5000);
|
||||
|
||||
it("gives every retrying entry point a fresh deadline per attempt", async () => {
|
||||
// A refused connection is retried by every entry point, the
|
||||
// non-idempotent ones included. If the deadline were created once,
|
||||
// outside the retry, both attempts would carry the same signal.
|
||||
const entryPoints: [
|
||||
string,
|
||||
() => Response,
|
||||
(c: ApiClient) => unknown,
|
||||
][] = [
|
||||
["getJSON", () => jsonResponse({}), (c) => c.getJSON("/a")],
|
||||
["postJSON", () => jsonResponse({}), (c) => c.postJSON("/b", {})],
|
||||
["putJSON", () => jsonResponse({}), (c) => c.putJSON("/c", {})],
|
||||
[
|
||||
"putFile",
|
||||
() => new Response(null, { status: 200 }),
|
||||
(c) => c.putFile("https://s3.example/x", new Uint8Array([1])),
|
||||
],
|
||||
[
|
||||
"getFileStream",
|
||||
() => streamResponse(new Uint8Array([1])),
|
||||
(c) => c.getFileStream(1),
|
||||
],
|
||||
[
|
||||
"getThumbnailStream",
|
||||
() => streamResponse(new Uint8Array([1])),
|
||||
(c) => c.getThumbnailStream(1),
|
||||
],
|
||||
];
|
||||
for (const [name, success, call] of entryPoints) {
|
||||
const { fetch, calls } = scriptedFetch(
|
||||
errnoError("ECONNREFUSED", "connect ECONNREFUSED"),
|
||||
success(),
|
||||
);
|
||||
const client = new ApiClient({ fetch, retry: noWait });
|
||||
|
||||
await call(client);
|
||||
|
||||
expect(calls, name).toHaveLength(2);
|
||||
const [first, second] = calls.map((c) => c.init?.signal);
|
||||
expect(first, name).toBeInstanceOf(AbortSignal);
|
||||
expect(second, name).toBeInstanceOf(AbortSignal);
|
||||
expect(second, name).not.toBe(first);
|
||||
}
|
||||
});
|
||||
|
||||
it("recovers when a later attempt answers in time", async () => {
|
||||
const { fetch, calls } = scriptedFetch(HANG, jsonResponse({ ok: 1 }));
|
||||
const client = new ApiClient({
|
||||
@@ -718,6 +880,11 @@ describe("ApiClient timeouts", () => {
|
||||
// that never produces a chunk and never observes the signal, so the
|
||||
// only thing that can unblock the read is quak's own enforcement of
|
||||
// the deadline over the stream it hands out.
|
||||
//
|
||||
// The clock is faked, so the test runs under the real default
|
||||
// deadline and waits for nothing.
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
const stalling = new Response(
|
||||
new ReadableStream<Uint8Array>({
|
||||
pull: () => new Promise<void>(() => {}),
|
||||
@@ -727,16 +894,77 @@ describe("ApiClient timeouts", () => {
|
||||
const { fetch } = scriptedFetch(stalling);
|
||||
const client = new ApiClient({
|
||||
fetch,
|
||||
downloadTimeoutMs: 20,
|
||||
retry: { ...noWait, attempts: 1 },
|
||||
});
|
||||
|
||||
const stream = await client.getFileStream(42);
|
||||
const err: unknown = await readAll(stream).catch((e: unknown) => e);
|
||||
let settled = false;
|
||||
const result = readAll(stream).then(
|
||||
(n) => n,
|
||||
(e: unknown) => e,
|
||||
);
|
||||
void result.finally(() => {
|
||||
settled = true;
|
||||
});
|
||||
|
||||
await vi.advanceTimersByTimeAsync(DEFAULT_DOWNLOAD_TIMEOUT_MS - 1);
|
||||
expect(settled).toBe(false);
|
||||
await vi.advanceTimersByTimeAsync(1);
|
||||
|
||||
const err = await result;
|
||||
expect(err).toBeInstanceOf(Error);
|
||||
expect((err as Error).name).toBe("TimeoutError");
|
||||
}, 5000);
|
||||
// Classified as every deadline is: retried by the idempotent
|
||||
// downloads, never replayed for a POST or PUT.
|
||||
expect(isRetryable(err)).toBe(true);
|
||||
expect(isSafeToReplay(err)).toBe(false);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it("does not abort a slow body that keeps making progress", async () => {
|
||||
// A deadline over the whole transfer would cut off a large video on a
|
||||
// slow link however steadily it was arriving. The deadline restarts
|
||||
// with every chunk, so a body that sends one byte every 600 ms for
|
||||
// well over the 1000 ms deadline completes.
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
let sent = 0;
|
||||
const trickling = new Response(
|
||||
new ReadableStream<Uint8Array>({
|
||||
async pull(controller) {
|
||||
await new Promise((resolve) =>
|
||||
setTimeout(resolve, 600),
|
||||
);
|
||||
if (sent === 10) {
|
||||
controller.close();
|
||||
return;
|
||||
}
|
||||
controller.enqueue(new Uint8Array([sent++]));
|
||||
},
|
||||
}),
|
||||
{ status: 200 },
|
||||
);
|
||||
const { fetch } = scriptedFetch(trickling);
|
||||
const client = new ApiClient({
|
||||
fetch,
|
||||
downloadTimeoutMs: 1000,
|
||||
retry: { ...noWait, attempts: 1 },
|
||||
});
|
||||
|
||||
const stream = await client.getFileStream(42);
|
||||
const result = readAll(stream).then(
|
||||
(n) => n,
|
||||
(e: unknown) => e,
|
||||
);
|
||||
await vi.advanceTimersByTimeAsync(11 * 600);
|
||||
|
||||
expect(await result).toBe(10);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it("lets a body that arrives in time through untouched", async () => {
|
||||
// The counterpart to the previous test: enforcing the deadline over
|
||||
@@ -762,6 +990,52 @@ describe("ApiClient timeouts", () => {
|
||||
}
|
||||
expect(joined).toEqual(payload);
|
||||
});
|
||||
|
||||
it("leaves no timer pending after a download completes or fails", async () => {
|
||||
vi.useFakeTimers();
|
||||
try {
|
||||
const { fetch } = scriptedFetch(
|
||||
streamResponse(new Uint8Array([1, 2, 3])),
|
||||
textResponse("gone", 404),
|
||||
);
|
||||
const client = new ApiClient({
|
||||
fetch,
|
||||
retry: { ...noWait, attempts: 1 },
|
||||
});
|
||||
|
||||
expect(await readAll(await client.getFileStream(1))).toBe(3);
|
||||
expect(vi.getTimerCount()).toBe(0);
|
||||
|
||||
await expect(client.getFileStream(2)).rejects.toBeInstanceOf(
|
||||
ApiError,
|
||||
);
|
||||
expect(vi.getTimerCount()).toBe(0);
|
||||
} finally {
|
||||
vi.useRealTimers();
|
||||
}
|
||||
});
|
||||
|
||||
it("never lets the download timer keep the process alive", async () => {
|
||||
const spy = vi.spyOn(globalThis, "setTimeout");
|
||||
try {
|
||||
const { fetch } = scriptedFetch(
|
||||
streamResponse(new Uint8Array([1])),
|
||||
);
|
||||
const client = new ApiClient({
|
||||
fetch,
|
||||
downloadTimeoutMs: 12_345,
|
||||
retry: noWait,
|
||||
});
|
||||
|
||||
const stream = await client.getFileStream(1);
|
||||
const i = spy.mock.calls.findIndex((call) => call[1] === 12_345);
|
||||
const timer = spy.mock.results[i]!.value as NodeJS.Timeout;
|
||||
expect(timer.hasRef()).toBe(false);
|
||||
await stream.cancel();
|
||||
} finally {
|
||||
spy.mockRestore();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("ApiClient error typing", () => {
|
||||
@@ -820,9 +1094,8 @@ describe("ApiClient error typing", () => {
|
||||
|
||||
describe("ApiClient non-idempotent requests", () => {
|
||||
/**
|
||||
* `postJSON` and `putJSON` carry quak's only requests that change server
|
||||
* state: `/users/srp/create-session`, `/users/two-factor/verify` — which
|
||||
* consumes one of a small number of 2FA attempts — and `/files/thumbnail`.
|
||||
* `postJSON` and `putJSON` carry quak's requests that can change server
|
||||
* state; the README lists them under "Endpoints used".
|
||||
*
|
||||
* They are retried only on a failure that establishes no TCP connection to
|
||||
* the server ever existed — DNS produced no address, or the peer refused
|
||||
@@ -922,4 +1195,30 @@ describe("ApiClient non-idempotent requests", () => {
|
||||
await refusedClient.updateThumbnail(1, "key", "header");
|
||||
expect(refused.calls).toHaveLength(2);
|
||||
});
|
||||
|
||||
it("does not follow or replay a redirect on POST or PUT", async () => {
|
||||
// The origin has already received a request it answers with a
|
||||
// redirect, so following it would let a refused connection to the
|
||||
// redirect target pass for a request that never went out.
|
||||
for (const send of [
|
||||
(c: ApiClient) => c.postJSON("/users/ott", {}),
|
||||
(c: ApiClient) => c.putJSON("/files/thumbnail", {}),
|
||||
]) {
|
||||
const { fetch, calls } = scriptedFetch(
|
||||
new Response(null, {
|
||||
status: 307,
|
||||
headers: { location: "https://elsewhere.example/" },
|
||||
}),
|
||||
jsonResponse({}),
|
||||
);
|
||||
const client = new ApiClient({ fetch, retry: noWait });
|
||||
|
||||
const err: unknown = await send(client).catch((e: unknown) => e);
|
||||
|
||||
expect(calls[0]?.init?.redirect).toBe("manual");
|
||||
expect(err).toBeInstanceOf(ApiError);
|
||||
expect((err as ApiError).status).toBe(307);
|
||||
expect(calls).toHaveLength(1);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
+145
-3
@@ -36,20 +36,50 @@ import {
|
||||
lstatSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
readdirSync,
|
||||
readFileSync,
|
||||
readlinkSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import type { ContentSource } from "../../src/library/content.js";
|
||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
|
||||
// `open` and `rename` are wrapped to record, in order, every fsync and rename,
|
||||
// so a test can pin the sequence "fsync the temp file, rename, fsync the
|
||||
// directory" that makes a copied original survive a power cut. `vi.hoisted`
|
||||
// because `vi.mock` factories run before module-level constants exist.
|
||||
const fsEvents = vi.hoisted(() => [] as string[]);
|
||||
|
||||
vi.mock("node:fs/promises", async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import("node:fs/promises")>();
|
||||
return {
|
||||
...actual,
|
||||
open: async (
|
||||
...args: Parameters<typeof actual.open>
|
||||
): Promise<Awaited<ReturnType<typeof actual.open>>> => {
|
||||
const handle = await actual.open(...args);
|
||||
const realSync = handle.sync.bind(handle);
|
||||
handle.sync = async (): Promise<void> => {
|
||||
fsEvents.push(`sync:${String(args[0])}`);
|
||||
await realSync();
|
||||
};
|
||||
return handle;
|
||||
},
|
||||
rename: async (from: string, to: string): Promise<void> => {
|
||||
fsEvents.push(`rename:${to}`);
|
||||
await actual.rename(from, to);
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
const USER_ID = 42;
|
||||
|
||||
// Decrypted-byte length each stub original writes, keyed by fileID.
|
||||
@@ -107,6 +137,29 @@ class MockClient {
|
||||
}
|
||||
}
|
||||
|
||||
// A server that names an album and a file so as to climb out of the backup
|
||||
// directory.
|
||||
class HostileClient extends MockClient {
|
||||
override async collectionsSince(): Promise<CollectionsPage> {
|
||||
const page = await super.collectionsSince();
|
||||
return {
|
||||
...page,
|
||||
collections: page.collections.length
|
||||
? [collection(3, "../escape")]
|
||||
: [],
|
||||
};
|
||||
}
|
||||
override async filesSince(args: {
|
||||
collectionID: number;
|
||||
}): Promise<FilesPage> {
|
||||
const files =
|
||||
args.collectionID === 3
|
||||
? [file(300, 3, "../../.ssh/authorized_keys")]
|
||||
: [];
|
||||
return { files, deleted: [], cursor: 1 };
|
||||
}
|
||||
}
|
||||
|
||||
// A content source that writes byte buffers of the expected length and can be
|
||||
// told to fail one fileID's original, to exercise per-file resilience.
|
||||
interface StubSource extends ContentSource {
|
||||
@@ -136,9 +189,12 @@ const stubSource = (): StubSource => {
|
||||
|
||||
let root: string;
|
||||
|
||||
const openLibrary = (source: ContentSource): Promise<Library> =>
|
||||
const openLibrary = (
|
||||
source: ContentSource,
|
||||
client: MockClient = new MockClient(),
|
||||
): Promise<Library> =>
|
||||
Library.open({
|
||||
client: new MockClient(),
|
||||
client,
|
||||
cacheDirectory: join(root, "cache"),
|
||||
contentSource: source,
|
||||
refreshIntervalSeconds: 3600,
|
||||
@@ -243,6 +299,30 @@ describe("lib.backup", () => {
|
||||
lib.close();
|
||||
});
|
||||
|
||||
it("keeps server-supplied album and file names inside the backup", async () => {
|
||||
const lib = await openLibrary(stubSource(), new HostileClient());
|
||||
const outDir = join(root, "backup");
|
||||
|
||||
const result = await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
// The title has no usable extension, so the original is `.bin`.
|
||||
expect(existsSync(join(outDir, "originals", "300.bin"))).toBe(true);
|
||||
const link = join(
|
||||
outDir,
|
||||
"collections",
|
||||
"__escape",
|
||||
"__.._.ssh_authorized_keys",
|
||||
);
|
||||
expect(lstatSync(link).isSymbolicLink()).toBe(true);
|
||||
expect(existsSync(join(outDir, "collections", "__escape.json"))).toBe(
|
||||
true,
|
||||
);
|
||||
// Nothing landed beside or above the backup directory.
|
||||
expect(readdirSync(root).sort()).toEqual(["backup", "cache"]);
|
||||
lib.close();
|
||||
});
|
||||
|
||||
it("is an idempotent no-op when every original is already present", async () => {
|
||||
const source = stubSource();
|
||||
const lib = await openLibrary(source);
|
||||
@@ -479,4 +559,66 @@ describe("lib.backup", () => {
|
||||
expect(readLedger(outDir).files["101"]!.attempts).toBe(1);
|
||||
lib.close();
|
||||
});
|
||||
|
||||
it("fsyncs a copied original before the rename and its directory after", async () => {
|
||||
const lib = await openLibrary(stubSource());
|
||||
const outDir = join(root, "backup");
|
||||
const originals = join(outDir, "originals");
|
||||
const dest = join(originals, "100.jpg");
|
||||
fsEvents.length = 0;
|
||||
|
||||
await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
const at = fsEvents.indexOf(`rename:${dest}`);
|
||||
expect(at).toBeGreaterThan(0);
|
||||
expect(fsEvents[at - 1]).toMatch(
|
||||
/^sync:.*\/\.quak-backup-100\.jpg-\d+-[0-9a-z]*\.tmp$/,
|
||||
);
|
||||
expect(fsEvents[at + 1]).toBe(`sync:${originals}`);
|
||||
lib.close();
|
||||
});
|
||||
|
||||
it("removes temp files left by a killed backup but not those of one still running", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const originals = join(outDir, "originals");
|
||||
mkdirSync(originals, { recursive: true });
|
||||
// A child that has already exited: its process ID is not running.
|
||||
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
|
||||
const leftover = `.quak-backup-100.jpg-${exitedPID}-abc123.tmp`;
|
||||
// This test's own process stands in for a backup running at the same
|
||||
// time.
|
||||
const inProgress = `.quak-backup-101.jpg-${process.pid}-def456.tmp`;
|
||||
writeFileSync(join(originals, leftover), "partial");
|
||||
writeFileSync(join(originals, inProgress), "partial");
|
||||
const lib = await openLibrary(stubSource());
|
||||
|
||||
await lib.backup({ downloadDirectory: outDir });
|
||||
|
||||
const names = readdirSync(originals);
|
||||
expect(names).not.toContain(leftover);
|
||||
expect(names).toContain(inProgress);
|
||||
lib.close();
|
||||
});
|
||||
|
||||
it("removes leftover temp files in thumbnails/ but not those of a backup still running", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const thumbnails = join(outDir, "thumbnails");
|
||||
mkdirSync(thumbnails, { recursive: true });
|
||||
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
|
||||
const leftover = `.quak-backup-100.jpg-${exitedPID}-abc123.tmp`;
|
||||
const inProgress = `.quak-backup-101.jpg-${process.pid}-def456.tmp`;
|
||||
writeFileSync(join(thumbnails, leftover), "partial");
|
||||
writeFileSync(join(thumbnails, inProgress), "partial");
|
||||
const lib = await openLibrary(stubSource());
|
||||
|
||||
await lib.backup({
|
||||
downloadDirectory: outDir,
|
||||
includeThumbnails: true,
|
||||
});
|
||||
|
||||
const names = readdirSync(thumbnails);
|
||||
expect(names).not.toContain(leftover);
|
||||
expect(names).toContain(inProgress);
|
||||
lib.close();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,397 @@
|
||||
/**
|
||||
* Tests for the CLI commands (`src/cli-commands.ts`, issue #12).
|
||||
*
|
||||
* Each command is called directly with a context whose output streams collect
|
||||
* text, whose session directory is a fresh temp directory, and whose session
|
||||
* loader hands back a fake client. The fake serves two albums and three files
|
||||
* from memory, writes stand-in bytes for originals and thumbnails, and makes no
|
||||
* network calls. The helpers the commands call (`cli-read`, `cli-output`,
|
||||
* backup, thumbnails) have their own tests; these check what each command
|
||||
* prints and the exit code it returns.
|
||||
*/
|
||||
|
||||
import {
|
||||
existsSync,
|
||||
mkdtempSync,
|
||||
readFileSync,
|
||||
rmSync,
|
||||
statSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { describe, it, expect, beforeAll, beforeEach, afterEach } from "vitest";
|
||||
|
||||
import {
|
||||
type CliContext,
|
||||
saveSession,
|
||||
whoamiCommand,
|
||||
logoutCommand,
|
||||
collectionsCommand,
|
||||
filesCommand,
|
||||
getCommand,
|
||||
getThumbCommand,
|
||||
backupCommand,
|
||||
listMissingThumbnailsCommand,
|
||||
} from "../../src/cli-commands.js";
|
||||
import { loadSession } from "../../src/cli-session.js";
|
||||
import type { Client, ClientSnapshot } from "../../src/client.js";
|
||||
import type { ContentSource } from "../../src/library/content.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
import { init } from "../../src/crypto/index.js";
|
||||
|
||||
const USER_ID = 42;
|
||||
|
||||
const collection = (
|
||||
id: number,
|
||||
name: string,
|
||||
isShared = false,
|
||||
): Collection => ({
|
||||
id,
|
||||
ownerID: USER_ID,
|
||||
key: new Uint8Array([id]),
|
||||
name,
|
||||
type: "album",
|
||||
updationTime: 1,
|
||||
isShared,
|
||||
});
|
||||
|
||||
const file = (id: number, collectionID: number, title: string): EnteFile => ({
|
||||
id,
|
||||
collectionID,
|
||||
ownerID: USER_ID,
|
||||
key: new Uint8Array([id & 0xff]),
|
||||
metadata: {
|
||||
title,
|
||||
fileType: "image",
|
||||
creationTime: 1000,
|
||||
modificationTime: 1000,
|
||||
},
|
||||
file: { decryptionHeader: "aGVhZGVy" },
|
||||
thumbnail: { decryptionHeader: "dGh1bWI=" },
|
||||
updationTime: 1,
|
||||
});
|
||||
|
||||
const COLLECTIONS = [collection(1, "Vacation"), collection(2, "Work", true)];
|
||||
|
||||
const FILES: Record<number, EnteFile[]> = {
|
||||
1: [file(100, 1, "beach.jpg"), file(101, 1, "sunset.jpg")],
|
||||
2: [file(200, 2, "diagram.png")],
|
||||
};
|
||||
|
||||
// An original is 7 bytes and a thumbnail 3. `failID` makes that file's
|
||||
// original fail; `emptyThumbID` makes the server report that file's
|
||||
// thumbnail as empty.
|
||||
const fakeClient = (opts: { failID?: number; emptyThumbID?: number } = {}) => {
|
||||
const source: ContentSource = {
|
||||
original: async ({ file: f, destination }) => {
|
||||
if (f.id === opts.failID) throw new Error("HTTP 500 from server");
|
||||
writeFileSync(destination, Buffer.alloc(7, f.id & 0xff));
|
||||
return { bytesWritten: 7 };
|
||||
},
|
||||
thumbnail: async ({ file: f, destination }) => {
|
||||
writeFileSync(destination, Buffer.alloc(3, f.id & 0xff));
|
||||
return { bytesWritten: 3 };
|
||||
},
|
||||
};
|
||||
const fake = {
|
||||
whoami: () => ({ email: "cli@example.com", userID: USER_ID }),
|
||||
collectionsSince: async () => ({
|
||||
collections: COLLECTIONS,
|
||||
deleted: [],
|
||||
cursor: 1,
|
||||
}),
|
||||
filesSince: async (args: { collectionID: number }) => ({
|
||||
files: FILES[args.collectionID] ?? [],
|
||||
deleted: [],
|
||||
cursor: 1,
|
||||
}),
|
||||
contentSource: () => source,
|
||||
getApiClient: () => ({
|
||||
getThumbnailStream: async (fileID: number) =>
|
||||
new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
if (fileID !== opts.emptyThumbID) {
|
||||
controller.enqueue(new Uint8Array(3));
|
||||
}
|
||||
controller.close();
|
||||
},
|
||||
}),
|
||||
}),
|
||||
};
|
||||
// The commands only call the methods above.
|
||||
return fake as unknown as Client;
|
||||
};
|
||||
|
||||
// Collects everything written to it.
|
||||
class Output {
|
||||
text = "";
|
||||
write(text: string): void {
|
||||
this.text += text;
|
||||
}
|
||||
}
|
||||
|
||||
let root: string;
|
||||
let stdout: Output;
|
||||
let stderr: Output;
|
||||
|
||||
const context = (client: Client | null = fakeClient()): CliContext => ({
|
||||
stdout,
|
||||
stderr,
|
||||
sessionDir: join(root, "session"),
|
||||
cacheDir: join(root, "cache"),
|
||||
loadSession: () => client,
|
||||
});
|
||||
|
||||
beforeAll(async () => {
|
||||
await init();
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
root = mkdtempSync(join(tmpdir(), "quak-cli-test-"));
|
||||
stdout = new Output();
|
||||
stderr = new Output();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe("session file", () => {
|
||||
const snapshot: ClientSnapshot = {
|
||||
email: "cli@example.com",
|
||||
userID: USER_ID,
|
||||
token: "token",
|
||||
masterKey: "a",
|
||||
secretKey: "b",
|
||||
publicKey: "c",
|
||||
};
|
||||
|
||||
it("is written with mode 0600 in a directory with mode 0700", () => {
|
||||
const dir = join(root, "new", "session");
|
||||
saveSession(dir, snapshot);
|
||||
expect(statSync(dir).mode & 0o777).toBe(0o700);
|
||||
const path = join(dir, "session.json");
|
||||
expect(statSync(path).mode & 0o777).toBe(0o600);
|
||||
expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot);
|
||||
});
|
||||
|
||||
it("is removed by logout", async () => {
|
||||
const ctx = context();
|
||||
saveSession(ctx.sessionDir, snapshot);
|
||||
expect(await logoutCommand(ctx)).toBe(0);
|
||||
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
|
||||
expect(stderr.text).toBe("Session deleted.\n");
|
||||
});
|
||||
|
||||
it("logout without a session says so and exits 0", async () => {
|
||||
expect(await logoutCommand(context())).toBe(0);
|
||||
expect(stderr.text).toBe("No session found.\n");
|
||||
});
|
||||
|
||||
it("a missing session exits 1 with 'Not logged in'", async () => {
|
||||
const ctx = { ...context(), loadSession };
|
||||
expect(await whoamiCommand(ctx)).toBe(1);
|
||||
expect(stderr.text).toBe(
|
||||
`Not logged in. Run "quak login" first.\n` +
|
||||
`Session file: ${join(ctx.sessionDir, "session.json")}\n`,
|
||||
);
|
||||
expect(stdout.text).toBe("");
|
||||
});
|
||||
|
||||
it("a corrupt session exits 1 and says it is corrupt", async () => {
|
||||
const ctx = { ...context(), loadSession };
|
||||
saveSession(ctx.sessionDir, snapshot);
|
||||
expect(await collectionsCommand(ctx, {})).toBe(1);
|
||||
expect(stderr.text).toContain("is corrupt");
|
||||
expect(stderr.text).toContain(
|
||||
`Run "quak logout" and then "quak login" to replace it.\n`,
|
||||
);
|
||||
expect(stdout.text).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
describe("whoami", () => {
|
||||
it("prints the account as one line of JSON", async () => {
|
||||
expect(await whoamiCommand(context())).toBe(0);
|
||||
expect(stdout.text).toBe(
|
||||
`{"email":"cli@example.com","userID":${USER_ID}}\n`,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("collections", () => {
|
||||
it("prints one tab-separated line per album", async () => {
|
||||
expect(await collectionsCommand(context(), {})).toBe(0);
|
||||
expect(stdout.text).toBe(
|
||||
"1\talbum\tVacation\n" + "2\talbum\tWork (shared)\n",
|
||||
);
|
||||
});
|
||||
|
||||
it("prints a JSON array with --json", async () => {
|
||||
expect(await collectionsCommand(context(), { json: true })).toBe(0);
|
||||
expect(JSON.parse(stdout.text)).toEqual([
|
||||
{
|
||||
id: 1,
|
||||
name: "Vacation",
|
||||
type: "album",
|
||||
ownerID: USER_ID,
|
||||
isShared: false,
|
||||
updationTime: 1,
|
||||
},
|
||||
{
|
||||
id: 2,
|
||||
name: "Work",
|
||||
type: "album",
|
||||
ownerID: USER_ID,
|
||||
isShared: true,
|
||||
updationTime: 1,
|
||||
},
|
||||
]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("files", () => {
|
||||
it("prints one tab-separated line per file", async () => {
|
||||
expect(await filesCommand(context(), { collection: "1" })).toBe(0);
|
||||
expect(stdout.text).toBe(
|
||||
"100\timage\tbeach.jpg\n" + "101\timage\tsunset.jpg\n",
|
||||
);
|
||||
});
|
||||
|
||||
it("prints a JSON array with --json", async () => {
|
||||
const code = await filesCommand(context(), {
|
||||
collection: "2",
|
||||
json: true,
|
||||
});
|
||||
expect(code).toBe(0);
|
||||
expect(JSON.parse(stdout.text)).toEqual([
|
||||
{
|
||||
id: 200,
|
||||
title: "diagram.png",
|
||||
fileType: "image",
|
||||
creationTime: 1000,
|
||||
collectionID: 2,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("exits 1 for an unknown collection", async () => {
|
||||
expect(await filesCommand(context(), { collection: "9" })).toBe(1);
|
||||
expect(stderr.text).toBe("Collection 9 not found\n");
|
||||
});
|
||||
|
||||
it("exits 1 for a collection ID that is not a number", async () => {
|
||||
expect(await filesCommand(context(), { collection: "abc" })).toBe(1);
|
||||
expect(stderr.text).toBe("Invalid collection ID\n");
|
||||
});
|
||||
});
|
||||
|
||||
describe("get and get-thumb", () => {
|
||||
it("get finds a file in any album without --collection", async () => {
|
||||
const out = join(root, "diagram.png");
|
||||
expect(await getCommand(context(), "200", { out })).toBe(0);
|
||||
expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
|
||||
expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
|
||||
});
|
||||
|
||||
it("get-thumb finds a file in any album without --collection", async () => {
|
||||
const out = join(root, "thumb.jpg");
|
||||
expect(await getThumbCommand(context(), "200", { out })).toBe(0);
|
||||
expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
|
||||
expect(stderr.text).toBe(`3 bytes -> ${out}\n`);
|
||||
});
|
||||
|
||||
it("get exits 1 when no album has the file", async () => {
|
||||
const out = join(root, "x");
|
||||
expect(await getCommand(context(), "999", { out })).toBe(1);
|
||||
expect(stderr.text).toBe("File 999 not found\n");
|
||||
expect(existsSync(out)).toBe(false);
|
||||
});
|
||||
|
||||
it("get-thumb exits 1 when no album has the file", async () => {
|
||||
const out = join(root, "x");
|
||||
expect(await getThumbCommand(context(), "999", { out })).toBe(1);
|
||||
expect(stderr.text).toBe("File 999 not found\n");
|
||||
expect(existsSync(out)).toBe(false);
|
||||
});
|
||||
|
||||
it("both exit 1 for a file ID that is not a number", async () => {
|
||||
expect(await getCommand(context(), "abc", {})).toBe(1);
|
||||
expect(await getThumbCommand(context(), "abc", {})).toBe(1);
|
||||
expect(stderr.text).toBe("Invalid file ID\nInvalid file ID\n");
|
||||
});
|
||||
});
|
||||
|
||||
describe("backup", () => {
|
||||
it("exits 0 and prints a summary when every file is saved", async () => {
|
||||
const dir = join(root, "backup");
|
||||
expect(await backupCommand(context(), dir, {})).toBe(0);
|
||||
expect(stderr.text).toContain(
|
||||
"\n--- Backup complete ---\n" +
|
||||
" Total files: 3\n" +
|
||||
" Downloaded: 3\n" +
|
||||
" Skipped: 0\n" +
|
||||
" Failed: 0\n",
|
||||
);
|
||||
expect(stdout.text).toBe("");
|
||||
});
|
||||
|
||||
it("exits 1 and lists the file when one download fails", async () => {
|
||||
const ctx = context(fakeClient({ failID: 101 }));
|
||||
expect(await backupCommand(ctx, join(root, "backup"), {})).toBe(1);
|
||||
expect(stderr.text).toContain(" Failed: 1\n");
|
||||
expect(stderr.text).toContain(
|
||||
"\nFailed files:\n" +
|
||||
" [Vacation] sunset.jpg (id 101): HTTP 500 from server\n",
|
||||
);
|
||||
});
|
||||
|
||||
it("prints the result as JSON with --json, still exiting 1 on a failure", async () => {
|
||||
const ctx = context(fakeClient({ failID: 101 }));
|
||||
const code = await backupCommand(ctx, join(root, "backup"), {
|
||||
json: true,
|
||||
});
|
||||
expect(code).toBe(1);
|
||||
const result = JSON.parse(stdout.text);
|
||||
expect(result).toMatchObject({
|
||||
totalFiles: 3,
|
||||
downloaded: 2,
|
||||
skipped: 0,
|
||||
failed: 1,
|
||||
});
|
||||
expect(result.errors[0].fileID).toBe(101);
|
||||
expect(stderr.text).toBe("Starting backup...\n");
|
||||
});
|
||||
});
|
||||
|
||||
describe("helper list-missing-thumbnails", () => {
|
||||
it("prints one line per file with an empty thumbnail", async () => {
|
||||
const ctx = context(fakeClient({ emptyThumbID: 200 }));
|
||||
expect(await listMissingThumbnailsCommand(ctx, {})).toBe(0);
|
||||
expect(stdout.text).toBe(
|
||||
"200\tdiagram.png\tWork\tempty thumbnail (0 bytes)\n",
|
||||
);
|
||||
expect(stderr.text).toContain("\n1 file(s) with missing thumbnails:\n");
|
||||
});
|
||||
|
||||
it("says so when nothing is missing", async () => {
|
||||
expect(await listMissingThumbnailsCommand(context(), {})).toBe(0);
|
||||
expect(stdout.text).toBe("");
|
||||
expect(stderr.text).toContain("No missing thumbnails found.\n");
|
||||
});
|
||||
|
||||
it("prints a JSON array with --json and no progress", async () => {
|
||||
const ctx = context(fakeClient({ emptyThumbID: 200 }));
|
||||
expect(await listMissingThumbnailsCommand(ctx, { json: true })).toBe(0);
|
||||
expect(JSON.parse(stdout.text)).toEqual([
|
||||
{
|
||||
fileID: 200,
|
||||
title: "diagram.png",
|
||||
collection: "Work",
|
||||
reason: "empty thumbnail (0 bytes)",
|
||||
},
|
||||
]);
|
||||
expect(stderr.text).toBe("");
|
||||
});
|
||||
});
|
||||
@@ -165,14 +165,15 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
|
||||
},
|
||||
};
|
||||
|
||||
// Collection 2: "Work" with no magic metadata
|
||||
// Collection 2: "../Work" with no magic metadata. The server chose a name
|
||||
// that tries to climb out of the backup directory.
|
||||
const ck2 = sodium.crypto_secretbox_keygen();
|
||||
const { ciphertext: encCK2, nonce: ck2N } = encryptSecretbox(
|
||||
ck2,
|
||||
masterKey,
|
||||
);
|
||||
const { ciphertext: encCN2, nonce: cn2N } = encryptSecretbox(
|
||||
new TextEncoder().encode("Work"),
|
||||
new TextEncoder().encode("../Work"),
|
||||
ck2,
|
||||
);
|
||||
const rawColl2 = {
|
||||
@@ -496,7 +497,8 @@ describe("quak backup-metadata", () => {
|
||||
await runBackup(outDir);
|
||||
|
||||
const collDirs = readdirSync(join(outDir, "collections"));
|
||||
expect(collDirs.length).toBe(2);
|
||||
// "../Work" is sanitized into one directory name.
|
||||
expect(collDirs.sort()).toEqual(["10-Vacation", "20-__Work"]);
|
||||
|
||||
// Find the Vacation collection dir (prefixed with ID)
|
||||
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
|
||||
@@ -619,5 +621,18 @@ describe("quak backup-metadata", () => {
|
||||
expect(fileMeta.imageMetadata.format).toBe("jpeg");
|
||||
expect(fileMeta.imageMetadata.width).toBe(100);
|
||||
expect(fileMeta.imageMetadata.height).toBe(80);
|
||||
expect(fileMeta.imageMetadataError).toBeUndefined();
|
||||
|
||||
// File 200 has no original on the mock server, so extraction fails
|
||||
// and the reason is recorded instead of the field being left out.
|
||||
const workDir = collDirs.find((d) => d.includes("Work"))!;
|
||||
const failedMeta = JSON.parse(
|
||||
readFileSync(
|
||||
join(outDir, "collections", workDir, "200.json"),
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
expect(failedMeta.imageMetadata).toBeUndefined();
|
||||
expect(failedMeta.imageMetadataError).toEqual(expect.any(String));
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
/**
|
||||
* Tests for the JPEG EXIF scan behind `quak backup-metadata --exif`.
|
||||
*
|
||||
* The originals come from users' libraries, so a truncated or corrupt JPEG
|
||||
* must neither hang the scan nor throw out of it, and a malformed file must be
|
||||
* told apart from one that simply has no EXIF: the record carries the reason in
|
||||
* `exifError`. Each input below is a short hand-built byte array.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
extractExifFromJpeg,
|
||||
extractImageMetadata,
|
||||
} from "../../src/metadata-backup.js";
|
||||
|
||||
const SOI = [0xff, 0xd8]; // start of image
|
||||
const SOS = [0xff, 0xda, 0x00, 0x02]; // start of scan, where the scan stops
|
||||
const EXIF_HEADER = [0x45, 0x78, 0x69, 0x66, 0x00, 0x00]; // "Exif\0\0"
|
||||
|
||||
// A big-endian TIFF block with one IFD entry: Orientation (0x0112), SHORT, 6.
|
||||
const TIFF_ORIENTATION_6 = [
|
||||
0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08, 0x00, 0x01, 0x01, 0x12,
|
||||
0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00,
|
||||
0x00, 0x00,
|
||||
];
|
||||
|
||||
// An APP1 segment whose length field matches its data.
|
||||
const app1 = (data: number[]): number[] => {
|
||||
const len = data.length + 2;
|
||||
return [0xff, 0xe1, len >> 8, len & 0xff, ...data];
|
||||
};
|
||||
|
||||
const bytes = (...parts: number[][]): Uint8Array =>
|
||||
new Uint8Array(parts.flat());
|
||||
|
||||
describe("extractExifFromJpeg", () => {
|
||||
it("returns the EXIF segment of a valid JPEG", () => {
|
||||
const data = [...EXIF_HEADER, ...TIFF_ORIENTATION_6];
|
||||
const scan = extractExifFromJpeg(bytes(SOI, app1(data), SOS));
|
||||
expect(scan.error).toBeUndefined();
|
||||
expect([...scan.exif!]).toEqual(data);
|
||||
});
|
||||
|
||||
it("returns nothing for a file that is not a JPEG", () => {
|
||||
const png = bytes([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
|
||||
expect(extractExifFromJpeg(png)).toEqual({});
|
||||
});
|
||||
|
||||
it("returns nothing for a JPEG without EXIF", () => {
|
||||
const app0 = [0xff, 0xe0, 0x00, 0x04, 0x00, 0x00];
|
||||
expect(extractExifFromJpeg(bytes(SOI, app0, SOS))).toEqual({});
|
||||
});
|
||||
|
||||
it("ignores an APP1 segment too short to hold the Exif header", () => {
|
||||
// A length under 8 cannot hold the six-byte "Exif\0\0" header, so the
|
||||
// segment is not EXIF. This one has length 7 and holds only "Exif\0",
|
||||
// which the old code, lacking the length check, returned as EXIF.
|
||||
const short = app1(EXIF_HEADER.slice(0, 5));
|
||||
expect(extractExifFromJpeg(bytes(SOI, short, SOS))).toEqual({});
|
||||
});
|
||||
|
||||
it("accepts an APP1 segment of length 8 holding just the Exif header", () => {
|
||||
const scan = extractExifFromJpeg(bytes(SOI, app1(EXIF_HEADER), SOS));
|
||||
expect(scan.error).toBeUndefined();
|
||||
expect([...scan.exif!]).toEqual(EXIF_HEADER);
|
||||
});
|
||||
|
||||
it("reports a JPEG truncated inside a segment header", () => {
|
||||
const scan = extractExifFromJpeg(bytes(SOI, [0xff, 0xe1, 0x00]));
|
||||
expect(scan.exif).toBeUndefined();
|
||||
expect(scan.error).toMatch(/truncated segment length/);
|
||||
});
|
||||
|
||||
it("reports a JPEG that ends before the image data", () => {
|
||||
const app0 = [0xff, 0xe0, 0x00, 0x04, 0x00, 0x00];
|
||||
const scan = extractExifFromJpeg(bytes(SOI, app0));
|
||||
expect(scan.error).toMatch(/ends before the image data/);
|
||||
});
|
||||
|
||||
it("stops on a zero-length segment instead of looping", () => {
|
||||
// A length of 0 would otherwise step the scan by 2 bytes at a time
|
||||
// through the rest of the file, reading garbage as markers.
|
||||
const zero = [0xff, 0xe0, 0x00, 0x00];
|
||||
const scan = extractExifFromJpeg(
|
||||
bytes(SOI, zero, zero, zero, zero, SOS),
|
||||
);
|
||||
expect(scan.error).toMatch(/segment length 0 at byte 2 is too small/);
|
||||
});
|
||||
|
||||
it("stops on a segment length of 1", () => {
|
||||
const scan = extractExifFromJpeg(
|
||||
bytes(SOI, [0xff, 0xe0, 0x00, 0x01], SOS),
|
||||
);
|
||||
expect(scan.error).toMatch(/segment length 1 at byte 2 is too small/);
|
||||
});
|
||||
|
||||
it("reports a segment length that runs past the end of the file", () => {
|
||||
// APP1 claims 0x4000 bytes but only the "Exif\0\0" header follows.
|
||||
const scan = extractExifFromJpeg(
|
||||
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
|
||||
);
|
||||
expect(scan.exif).toBeUndefined();
|
||||
expect(scan.error).toMatch(/runs past the end of the file/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("extractImageMetadata", () => {
|
||||
it("parses EXIF from a valid JPEG", () => {
|
||||
const meta = extractImageMetadata(
|
||||
bytes(SOI, app1([...EXIF_HEADER, ...TIFF_ORIENTATION_6]), SOS),
|
||||
);
|
||||
expect(meta?.exifError).toBeUndefined();
|
||||
expect(meta?.exif).toMatchObject({ Image: { Orientation: 6 } });
|
||||
});
|
||||
|
||||
it("returns nothing for a file that is not a JPEG", () => {
|
||||
const text = new TextEncoder().encode("just some text, not an image");
|
||||
expect(extractImageMetadata(text)).toBeUndefined();
|
||||
});
|
||||
|
||||
it("records the reason when the JPEG is malformed", () => {
|
||||
const meta = extractImageMetadata(
|
||||
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
|
||||
);
|
||||
expect(meta?.exif).toBeUndefined();
|
||||
expect(meta?.exifError).toMatch(/runs past the end of the file/);
|
||||
});
|
||||
|
||||
it("keeps the raw bytes and the reason when EXIF cannot be parsed", () => {
|
||||
const data = [...EXIF_HEADER, 0x58, 0x58];
|
||||
const meta = extractImageMetadata(bytes(SOI, app1(data), SOS));
|
||||
expect(meta?.exif).toBeUndefined();
|
||||
expect(meta?.exifRaw).toBe(Buffer.from(data).toString("base64"));
|
||||
expect(meta?.exifError).toEqual(expect.any(String));
|
||||
});
|
||||
});
|
||||
@@ -64,6 +64,24 @@ describe("CLI file output (issue #52)", () => {
|
||||
expect(thumbnailName(renamedFile)).toBe(`thumb_${RAW_TITLE}`);
|
||||
});
|
||||
|
||||
it("sanitizes the title when naming `quak get` downloads", () => {
|
||||
// Without `--out`, the server-supplied title names the file, so it must
|
||||
// not be able to point outside the working directory.
|
||||
const hostile = {
|
||||
...renamedFile,
|
||||
metadata: { ...renamedFile.metadata, title: "../../.bashrc" },
|
||||
};
|
||||
expect(originalName(hostile)).toBe("__.._.bashrc");
|
||||
expect(thumbnailName(hostile)).toBe("thumb___.._.bashrc");
|
||||
|
||||
const untitled = {
|
||||
...renamedFile,
|
||||
metadata: { ...renamedFile.metadata, title: "" },
|
||||
};
|
||||
expect(originalName(untitled)).toBe("file-100");
|
||||
expect(thumbnailName(untitled)).toBe("thumb_file-100");
|
||||
});
|
||||
|
||||
it("does not use the editedName/editedTime projection", () => {
|
||||
const record = deriveRecords([], [renamedFile]).photos.get(100);
|
||||
// The projection prefers the edits and reports milliseconds; the CLI
|
||||
|
||||
@@ -0,0 +1,197 @@
|
||||
/**
|
||||
* Tests for the client session lifecycle: `toJSON`, `fromJSON`, `logout`, and
|
||||
* the CLI's `loadSession`, which reads the saved session file back into a
|
||||
* client.
|
||||
*/
|
||||
|
||||
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import sodium from "libsodium-wrappers-sumo";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { init, toBase64 } from "../../src/crypto/index.js";
|
||||
import { Client, type ClientSnapshot } from "../../src/client.js";
|
||||
import { loadSession } from "../../src/cli-session.js";
|
||||
|
||||
const validSnapshot = (): ClientSnapshot => {
|
||||
const kp = sodium.crypto_box_keypair();
|
||||
return {
|
||||
email: "user@example.com",
|
||||
userID: 42,
|
||||
token: "test-token",
|
||||
masterKey: toBase64(sodium.crypto_secretbox_keygen()),
|
||||
secretKey: toBase64(kp.privateKey),
|
||||
publicKey: toBase64(kp.publicKey),
|
||||
};
|
||||
};
|
||||
|
||||
// The client's key buffers are private; the tests read them to prove that
|
||||
// logout wipes them.
|
||||
const keyBuffers = (client: Client): Uint8Array[] => [
|
||||
client["masterKey"],
|
||||
client["secretKey"],
|
||||
client["publicKey"],
|
||||
];
|
||||
|
||||
beforeAll(async () => {
|
||||
await init();
|
||||
});
|
||||
|
||||
describe("Client.toJSON", () => {
|
||||
it("round-trips through fromJSON unchanged", () => {
|
||||
const snapshot = validSnapshot();
|
||||
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
|
||||
});
|
||||
|
||||
it("throws instead of emitting a snapshot without a token", () => {
|
||||
const client = Client.fromJSON(validSnapshot());
|
||||
client.getApiClient().clearAuthToken();
|
||||
expect(() => client.toJSON()).toThrow(/no auth token/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Client.fromJSON", () => {
|
||||
const shortKey = toBase64(new Uint8Array(16));
|
||||
|
||||
it.each([
|
||||
["email", undefined],
|
||||
["email", 7],
|
||||
["email", ""],
|
||||
["token", undefined],
|
||||
["token", null],
|
||||
["token", ""],
|
||||
["userID", undefined],
|
||||
["userID", "42"],
|
||||
["userID", 4.2],
|
||||
["masterKey", undefined],
|
||||
["masterKey", 7],
|
||||
["masterKey", "not base64!"],
|
||||
["masterKey", shortKey],
|
||||
["secretKey", undefined],
|
||||
["secretKey", "not base64!"],
|
||||
["secretKey", shortKey],
|
||||
["publicKey", undefined],
|
||||
["publicKey", "not base64!"],
|
||||
["publicKey", shortKey],
|
||||
])("rejects %s = %j, naming the field", (field, value) => {
|
||||
const snapshot: Record<string, unknown> = { ...validSnapshot() };
|
||||
snapshot[field] = value;
|
||||
expect(() => Client.fromJSON(snapshot)).toThrow(
|
||||
new RegExp(`^Invalid session data: ${field} `),
|
||||
);
|
||||
});
|
||||
|
||||
it.each([null, "a string", 42])("rejects a non-object %j", (value) => {
|
||||
expect(() => Client.fromJSON(value)).toThrow(
|
||||
/^Invalid session data: not a JSON object/,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Client.logout", () => {
|
||||
it("zeroes the key buffers and clears the token", () => {
|
||||
const client = Client.fromJSON(validSnapshot());
|
||||
const api = client.getApiClient();
|
||||
const keys = keyBuffers(client);
|
||||
|
||||
client.logout();
|
||||
|
||||
for (const key of keys) {
|
||||
expect(key.length).toBe(32);
|
||||
expect(key.every((b) => b === 0)).toBe(true);
|
||||
}
|
||||
expect(api.getAuthToken()).toBeUndefined();
|
||||
});
|
||||
|
||||
it("makes every later operation throw", async () => {
|
||||
const client = Client.fromJSON(validSnapshot());
|
||||
client.logout();
|
||||
|
||||
expect(() => client.whoami()).toThrow(/logged out/);
|
||||
expect(() => client.toJSON()).toThrow(/logged out/);
|
||||
expect(() => client.getApiClient()).toThrow(/logged out/);
|
||||
expect(() => client.contentSource()).toThrow(/logged out/);
|
||||
await expect(client.listCollections()).rejects.toThrow(/logged out/);
|
||||
await expect(client.collectionsSince({ sinceTime: 0 })).rejects.toThrow(
|
||||
/logged out/,
|
||||
);
|
||||
await expect(
|
||||
client.filesSince({
|
||||
collectionID: 1,
|
||||
collectionKey: new Uint8Array(32),
|
||||
sinceTime: 0,
|
||||
}),
|
||||
).rejects.toThrow(/logged out/);
|
||||
await expect(
|
||||
client.fetchMLData({ fileIDs: [1], fileKeys: new Map() }),
|
||||
).rejects.toThrow(/logged out/);
|
||||
});
|
||||
|
||||
it("stops a listing in flight from decrypting with the zeroed keys", async () => {
|
||||
// The server answers only after the client has logged out. If the
|
||||
// listing went on to decrypt this row with all-zero keys it would fail
|
||||
// with a decryption error, not the logged-out one.
|
||||
const row = {
|
||||
id: 1,
|
||||
owner: { id: 42 },
|
||||
encryptedKey: toBase64(new Uint8Array(48)),
|
||||
keyDecryptionNonce: toBase64(new Uint8Array(24)),
|
||||
updationTime: 1,
|
||||
};
|
||||
const client: Client = Client.fromJSON(validSnapshot(), {
|
||||
fetch: async () => {
|
||||
client.logout();
|
||||
return new Response(JSON.stringify({ collections: [row] }), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
},
|
||||
});
|
||||
|
||||
await expect(client.listCollections()).rejects.toThrow(/logged out/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("loadSession", () => {
|
||||
let dir: string;
|
||||
|
||||
beforeAll(() => {
|
||||
dir = mkdtempSync(join(tmpdir(), "quak-session-test-"));
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("returns null when there is no session file", () => {
|
||||
expect(loadSession(join(dir, "missing.json"))).toBeNull();
|
||||
});
|
||||
|
||||
it("restores a client from a valid session file", () => {
|
||||
const path = join(dir, "valid.json");
|
||||
writeFileSync(path, JSON.stringify(validSnapshot()));
|
||||
expect(loadSession(path)!.whoami()).toEqual({
|
||||
email: "user@example.com",
|
||||
userID: 42,
|
||||
});
|
||||
});
|
||||
|
||||
it("says the file is corrupt when it is not JSON", () => {
|
||||
const path = join(dir, "truncated.json");
|
||||
writeFileSync(path, '{"email": "user@exa');
|
||||
expect(() => loadSession(path)).toThrow(
|
||||
`Session file ${path} is corrupt`,
|
||||
);
|
||||
});
|
||||
|
||||
it("says the file is corrupt and names the bad field", () => {
|
||||
const path = join(dir, "bad-key.json");
|
||||
writeFileSync(
|
||||
path,
|
||||
JSON.stringify({ ...validSnapshot(), secretKey: "AAAA" }),
|
||||
);
|
||||
expect(() => loadSession(path)).toThrow(
|
||||
new RegExp(`^Session file ${path} is corrupt: .*secretKey`),
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,33 @@
|
||||
import { beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
chunkHashFinal,
|
||||
chunkHashInit,
|
||||
chunkHashUpdate,
|
||||
init,
|
||||
} from "../../src/crypto/index.js";
|
||||
|
||||
beforeAll(async () => {
|
||||
await init();
|
||||
});
|
||||
|
||||
describe("content hash", () => {
|
||||
// RFC 7693 Appendix A: BLAKE2b-512 of "abc".
|
||||
const abc = Buffer.from(
|
||||
"ba80a53f981c4d0d6a2797b69f12f6e94c212f14685ac4b74b12bb6fdbffa2d1" +
|
||||
"7d87c5392aab792dc252d5de4533cc9518d38aa8dbf1925ab92386edd4009923",
|
||||
"hex",
|
||||
).toString("base64");
|
||||
|
||||
it("is unkeyed BLAKE2b-512 in standard base64", () => {
|
||||
const state = chunkHashInit();
|
||||
chunkHashUpdate(state, new TextEncoder().encode("abc"));
|
||||
expect(chunkHashFinal(state)).toBe(abc);
|
||||
});
|
||||
|
||||
it("gives the same hash when the input arrives in chunks", () => {
|
||||
const state = chunkHashInit();
|
||||
chunkHashUpdate(state, new TextEncoder().encode("a"));
|
||||
chunkHashUpdate(state, new TextEncoder().encode("bc"));
|
||||
expect(chunkHashFinal(state)).toBe(abc);
|
||||
});
|
||||
});
|
||||
+355
-11
@@ -48,7 +48,9 @@
|
||||
*/
|
||||
|
||||
import {
|
||||
chmodSync,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
readdirSync,
|
||||
readFileSync,
|
||||
rmSync,
|
||||
@@ -59,6 +61,7 @@ import { dirname, join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { createHash } from "node:crypto";
|
||||
import sodium from "libsodium-wrappers-sumo";
|
||||
import { zipSync } from "fflate";
|
||||
import {
|
||||
beforeAll,
|
||||
beforeEach,
|
||||
@@ -186,7 +189,31 @@ vi.mock("node:fs/promises", async (importOriginal) => {
|
||||
};
|
||||
});
|
||||
|
||||
/**
|
||||
* `chunkHashUpdate` is wrapped to record the length of every piece hashed, so
|
||||
* a test can show that a live photo entry reaches the hash in pieces far
|
||||
* smaller than the entry, rather than decompressed whole first.
|
||||
*/
|
||||
const hashHook = vi.hoisted(() => ({
|
||||
lengths: [] as number[],
|
||||
}));
|
||||
|
||||
vi.mock("../../src/crypto/index.js", async (importOriginal) => {
|
||||
const actual =
|
||||
await importOriginal<typeof import("../../src/crypto/index.js")>();
|
||||
return {
|
||||
...actual,
|
||||
chunkHashUpdate: (
|
||||
...args: Parameters<typeof actual.chunkHashUpdate>
|
||||
): void => {
|
||||
hashHook.lengths.push(args[1].length);
|
||||
actual.chunkHashUpdate(...args);
|
||||
},
|
||||
};
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
hashHook.lengths.length = 0;
|
||||
renameHook.calls.length = 0;
|
||||
renameHook.failWith = null;
|
||||
durabilityHook.events.length = 0;
|
||||
@@ -501,6 +528,22 @@ const entryPoints = [
|
||||
{ name: "downloadThumbnail", download: downloadThumbnail },
|
||||
];
|
||||
|
||||
// With no `outPath`, the destination is named after `metadata.title`, relative
|
||||
// to the working directory. Such tests run inside a temporary directory:
|
||||
// `make check` must not create files in the repo root.
|
||||
const inDirectory = async <T>(
|
||||
dir: string,
|
||||
run: () => Promise<T>,
|
||||
): Promise<T> => {
|
||||
const previous = process.cwd();
|
||||
process.chdir(dir);
|
||||
try {
|
||||
return await run();
|
||||
} finally {
|
||||
process.chdir(previous);
|
||||
}
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Tests
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -536,26 +579,59 @@ describe("downloadFile", () => {
|
||||
});
|
||||
|
||||
it("uses metadata.title as filename when outPath is omitted", async () => {
|
||||
// With no `outPath`, the destination is `metadata.title`, used
|
||||
// verbatim as a path. The title here is therefore given inside the
|
||||
// test's temporary directory: a bare relative name would resolve
|
||||
// against the process working directory, i.e. the repo root, and
|
||||
// `make check` must not create files in the repo — a failure between
|
||||
// the write and any cleanup would leave one behind.
|
||||
const plaintext = new Uint8Array([1, 2, 3]);
|
||||
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const { header, ciphertext } = encryptFileBody(plaintext, key);
|
||||
const thumbPush =
|
||||
sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
|
||||
const file = buildMockEnteFile(key, header, thumbPush.header);
|
||||
const titlePath = join(testDir, "fallback-name.png");
|
||||
file.metadata.title = titlePath;
|
||||
file.metadata.title = "fallback-name.png";
|
||||
const dir = mkdtempSync(join(testDir, "title-"));
|
||||
|
||||
const api = new ApiClient({ fetch: mockFetchForBody(ciphertext) });
|
||||
const result = await downloadFile(api, file);
|
||||
const result = await inDirectory(dir, () => downloadFile(api, file));
|
||||
|
||||
expect(result.path).toBe(titlePath);
|
||||
expect(readFileSync(result.path)).toEqual(Buffer.from(plaintext));
|
||||
expect(result.path).toBe("fallback-name.png");
|
||||
expect(readFileSync(join(dir, "fallback-name.png"))).toEqual(
|
||||
Buffer.from(plaintext),
|
||||
);
|
||||
});
|
||||
|
||||
it("keeps a hostile title inside the working directory", async () => {
|
||||
// The server controls the title. `../escaped.png` must not write to
|
||||
// the parent directory; it becomes one file name in the current one.
|
||||
const { api, file } = fixtureFor(
|
||||
multiChunkKey,
|
||||
multiChunk.header,
|
||||
multiChunk.body,
|
||||
);
|
||||
file.metadata.title = "../escaped.png";
|
||||
const parent = mkdtempSync(join(testDir, "hostile-"));
|
||||
const dir = join(parent, "cwd");
|
||||
mkdirSync(dir);
|
||||
|
||||
const result = await inDirectory(dir, () => downloadFile(api, file));
|
||||
|
||||
expect(result.path).toBe("__escaped.png");
|
||||
expect(readdirSync(dir)).toEqual(["__escaped.png"]);
|
||||
expect(readdirSync(parent)).toEqual(["cwd"]);
|
||||
});
|
||||
|
||||
it("uses an explicit outPath verbatim, even one with ..", async () => {
|
||||
// The caller is trusted: its path is not sanitized.
|
||||
const { api, file } = fixtureFor(
|
||||
multiChunkKey,
|
||||
multiChunk.header,
|
||||
multiChunk.body,
|
||||
);
|
||||
const dir = mkdtempSync(join(testDir, "explicit-"));
|
||||
mkdirSync(join(dir, "sub"));
|
||||
const outPath = join(dir, "sub", "..", "explicit.bin");
|
||||
|
||||
const result = await downloadFile(api, file, outPath);
|
||||
|
||||
expect(result.path).toBe(outPath);
|
||||
expect(existsSync(join(dir, "explicit.bin"))).toBe(true);
|
||||
});
|
||||
|
||||
it("handles a larger single-chunk file (random binary payload)", async () => {
|
||||
@@ -617,6 +693,23 @@ describe("downloadThumbnail", () => {
|
||||
expect(result).toEqual({ path: outPath, bytesWritten: 4 });
|
||||
expect(readFileSync(outPath)).toEqual(Buffer.from(plaintext));
|
||||
});
|
||||
|
||||
it("names the thumbnail thumb_ plus the sanitized title", async () => {
|
||||
const { api, file } = fixtureFor(
|
||||
multiChunkKey,
|
||||
multiChunk.header,
|
||||
multiChunk.body,
|
||||
);
|
||||
file.metadata.title = "/etc/passwd";
|
||||
const dir = mkdtempSync(join(testDir, "thumb-title-"));
|
||||
|
||||
const result = await inDirectory(dir, () =>
|
||||
downloadThumbnail(api, file),
|
||||
);
|
||||
|
||||
expect(result.path).toBe("thumb__etc_passwd");
|
||||
expect(readdirSync(dir)).toEqual(["thumb__etc_passwd"]);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -922,6 +1015,51 @@ describe.each(entryPoints)(
|
||||
expect(readFileSync(outPath)).toEqual(Buffer.from(existing));
|
||||
expect(readdirSync(dir)).toEqual(["rename-fails.bin"]);
|
||||
});
|
||||
|
||||
it("fails without creating anything when the destination directory does not exist", async () => {
|
||||
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const { header, ciphertext } = encryptFileBody(
|
||||
patternBytes(64, 33),
|
||||
key,
|
||||
);
|
||||
const { api, file } = fixtureFor(key, header, ciphertext);
|
||||
const dir = freshDir();
|
||||
const outPath = join(dir, "missing", "never.bin");
|
||||
|
||||
await expect(download(api, file, outPath)).rejects.toMatchObject({
|
||||
code: "ENOENT",
|
||||
});
|
||||
|
||||
// The missing directory is not created on the caller's behalf.
|
||||
expect(readdirSync(dir)).toEqual([]);
|
||||
});
|
||||
|
||||
// Root ignores directory permissions, so this cannot fail as root
|
||||
// (the Docker test image runs as root).
|
||||
it.skipIf(process.getuid?.() === 0)(
|
||||
"fails without creating anything when the destination directory is not writable",
|
||||
async () => {
|
||||
const key =
|
||||
sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const { header, ciphertext } = encryptFileBody(
|
||||
patternBytes(64, 34),
|
||||
key,
|
||||
);
|
||||
const { api, file } = fixtureFor(key, header, ciphertext);
|
||||
const dir = freshDir();
|
||||
const outPath = join(dir, "never.bin");
|
||||
chmodSync(dir, 0o500);
|
||||
try {
|
||||
await expect(
|
||||
download(api, file, outPath),
|
||||
).rejects.toMatchObject({ code: "EACCES" });
|
||||
} finally {
|
||||
chmodSync(dir, 0o700);
|
||||
}
|
||||
|
||||
expect(readdirSync(dir)).toEqual([]);
|
||||
},
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
@@ -1224,6 +1362,102 @@ describe("download retries: corruption is not retried", () => {
|
||||
|
||||
expect(requests()).toBe(1);
|
||||
});
|
||||
|
||||
it("cancels the response body when decryption fails", async () => {
|
||||
// A backup run carries on past a failed file, so a body left open on
|
||||
// failure would hold its connection until garbage collection, once
|
||||
// per failed file. This body delivers a corrupt chunk and then stays
|
||||
// open, so only a cancel from the downloader can close it.
|
||||
const corrupted = Uint8Array.from(multiChunk.body);
|
||||
corrupted[10] ^= 0xff;
|
||||
let cancelled = false;
|
||||
const fetch = (async () =>
|
||||
new Response(
|
||||
new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
controller.enqueue(corrupted);
|
||||
},
|
||||
cancel() {
|
||||
cancelled = true;
|
||||
},
|
||||
}),
|
||||
{ status: 200 },
|
||||
)) as typeof globalThis.fetch;
|
||||
const api = new ApiClient({ fetch, retry: { ...noWait, attempts: 1 } });
|
||||
const file = buildMockEnteFile(
|
||||
multiChunkKey,
|
||||
multiChunk.header,
|
||||
multiChunk.header,
|
||||
);
|
||||
const outPath = join(mkdtempSync(join(testDir, "cancel-")), "c.bin");
|
||||
|
||||
await expect(downloadFile(api, file, outPath)).rejects.toThrow(
|
||||
/authentication failed/i,
|
||||
);
|
||||
|
||||
expect(cancelled).toBe(true);
|
||||
});
|
||||
|
||||
it("cancels the response body when the temp file cannot be opened", async () => {
|
||||
// The download fails before a byte of the body is read, so the body
|
||||
// is still open and only a cancel from the downloader can close it.
|
||||
let cancelled = false;
|
||||
const fetch = (async () =>
|
||||
new Response(
|
||||
new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
controller.enqueue(multiChunk.body);
|
||||
},
|
||||
cancel() {
|
||||
cancelled = true;
|
||||
},
|
||||
}),
|
||||
{ status: 200 },
|
||||
)) as typeof globalThis.fetch;
|
||||
const api = new ApiClient({ fetch, retry: { ...noWait, attempts: 1 } });
|
||||
const file = buildMockEnteFile(
|
||||
multiChunkKey,
|
||||
multiChunk.header,
|
||||
multiChunk.header,
|
||||
);
|
||||
const outPath = join(
|
||||
mkdtempSync(join(testDir, "cancel-")),
|
||||
"missing",
|
||||
"c.bin",
|
||||
);
|
||||
|
||||
await expect(downloadFile(api, file, outPath)).rejects.toMatchObject({
|
||||
code: "ENOENT",
|
||||
});
|
||||
|
||||
expect(cancelled).toBe(true);
|
||||
});
|
||||
|
||||
it("cancels the response body when the header is malformed", async () => {
|
||||
// The header is rejected before the body is read, so the body is
|
||||
// still open and only a cancel from the downloader can close it.
|
||||
let cancelled = false;
|
||||
const fetch = (async () =>
|
||||
new Response(
|
||||
new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
controller.enqueue(multiChunk.body);
|
||||
},
|
||||
cancel() {
|
||||
cancelled = true;
|
||||
},
|
||||
}),
|
||||
{ status: 200 },
|
||||
)) as typeof globalThis.fetch;
|
||||
const api = new ApiClient({ fetch, retry: { ...noWait, attempts: 1 } });
|
||||
const shortHeader = multiChunk.header.subarray(0, 5);
|
||||
const file = buildMockEnteFile(multiChunkKey, shortHeader, shortHeader);
|
||||
const outPath = join(mkdtempSync(join(testDir, "cancel-")), "c.bin");
|
||||
|
||||
await expect(downloadFile(api, file, outPath)).rejects.toThrow();
|
||||
|
||||
expect(cancelled).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -1404,3 +1638,113 @@ describe.each(entryPoints)("$name progress", ({ name, download }) => {
|
||||
expectSameBytes(readFileSync(outPath), plaintext);
|
||||
});
|
||||
});
|
||||
|
||||
describe("downloadFile content hash", () => {
|
||||
// Node's own BLAKE2b-512 is the reference, so these tests do not depend
|
||||
// on the code under test to compute what they expect.
|
||||
const blake2b = (bytes: Uint8Array): string =>
|
||||
createHash("blake2b512").update(bytes).digest("base64");
|
||||
|
||||
// Serve `plaintext` encrypted as file 999 with the given metadata. Four
|
||||
// responses are scripted so a retried mismatch would show in `requests`.
|
||||
const setup = (plaintext: Uint8Array, metadata: Partial<FileMetadata>) => {
|
||||
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const { header, ciphertext } = encryptFileBody(plaintext, key);
|
||||
const file = buildMockEnteFile(key, header, header);
|
||||
file.metadata = { ...file.metadata, ...metadata };
|
||||
const body = { kind: "body", bytes: ciphertext } as const;
|
||||
const { fetch, requests } = scriptedCdnFetch(body, body, body, body);
|
||||
const api = new ApiClient({ fetch, retry: { ...noWait, attempts: 4 } });
|
||||
const dir = mkdtempSync(join(testDir, "hash-"));
|
||||
const outPath = join(dir, "f.bin");
|
||||
return {
|
||||
run: () => downloadFile(api, file, outPath),
|
||||
dir,
|
||||
outPath,
|
||||
requests,
|
||||
};
|
||||
};
|
||||
|
||||
const livePhotoZip = zipSync({
|
||||
"image.heic": patternBytes(500, 81),
|
||||
"video.mov": patternBytes(900, 82),
|
||||
});
|
||||
const livePhotoHash = `${blake2b(patternBytes(500, 81))}:${blake2b(patternBytes(900, 82))}`;
|
||||
|
||||
it("stores a file whose hash matches", async () => {
|
||||
const plaintext = patternBytes(700, 80);
|
||||
const t = setup(plaintext, { hash: blake2b(plaintext) });
|
||||
|
||||
await t.run();
|
||||
|
||||
expectSameBytes(readFileSync(t.outPath), plaintext);
|
||||
});
|
||||
|
||||
it("rejects a mismatch, stores nothing, names the file and does not retry", async () => {
|
||||
const t = setup(patternBytes(700, 80), {
|
||||
hash: blake2b(patternBytes(700, 79)),
|
||||
});
|
||||
|
||||
await expect(t.run()).rejects.toThrow(
|
||||
/file 999: content hash .* does not match/,
|
||||
);
|
||||
|
||||
expect(readdirSync(t.dir)).toEqual([]);
|
||||
expect(t.requests()).toBe(1);
|
||||
});
|
||||
|
||||
it("stores a file with no recorded hash unchecked", async () => {
|
||||
const plaintext = patternBytes(700, 80);
|
||||
const t = setup(plaintext, { hash: undefined });
|
||||
|
||||
await t.run();
|
||||
|
||||
expectSameBytes(readFileSync(t.outPath), plaintext);
|
||||
});
|
||||
|
||||
it("stores a live photo whose image and video hashes match", async () => {
|
||||
const t = setup(livePhotoZip, {
|
||||
fileType: "livePhoto",
|
||||
hash: livePhotoHash,
|
||||
});
|
||||
|
||||
await t.run();
|
||||
|
||||
expectSameBytes(readFileSync(t.outPath), livePhotoZip);
|
||||
});
|
||||
|
||||
it("hashes a large live photo entry as it decompresses, never whole", async () => {
|
||||
// 64 MiB of zeros deflates to a few kilobytes, the shape of a ZIP
|
||||
// that would exhaust memory if expanded whole.
|
||||
const image = new Uint8Array(64 * 1024 * 1024);
|
||||
const video = patternBytes(900, 83);
|
||||
const zip = zipSync({ "image.heic": image, "video.mov": video });
|
||||
const t = setup(zip, {
|
||||
fileType: "livePhoto",
|
||||
hash: `${blake2b(image)}:${blake2b(video)}`,
|
||||
});
|
||||
|
||||
await t.run();
|
||||
|
||||
expectSameBytes(readFileSync(t.outPath), zip);
|
||||
const hashed = hashHook.lengths.reduce((a, b) => a + b, 0);
|
||||
expect(hashed).toBe(image.length + video.length);
|
||||
expect(Math.max(...hashHook.lengths)).toBeLessThanOrEqual(
|
||||
2 * STREAM_CHUNK_SIZE,
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects a live photo whose hash does not match", async () => {
|
||||
// The whole ZIP's hash is not the recorded one: each part is hashed.
|
||||
const t = setup(livePhotoZip, {
|
||||
fileType: "livePhoto",
|
||||
hash: blake2b(livePhotoZip),
|
||||
});
|
||||
|
||||
await expect(t.run()).rejects.toThrow(
|
||||
/file 999: content hash .* does not match/,
|
||||
);
|
||||
|
||||
expect(readdirSync(t.dir)).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
// File names built from server-supplied metadata.
|
||||
//
|
||||
// quak does not trust the server. A file's title and a collection's name are
|
||||
// decrypted from data the server hands us, and a hostile server (or a
|
||||
// compromised account) can set them to anything. quak uses them to name files
|
||||
// on disk: `quak get` without `--out`, `downloadFile` without `outPath`, the
|
||||
// backup's symlink and collection directories, and the extension of every file
|
||||
// in the originals cache. Each of those goes through `sanitizeFileName` or
|
||||
// `safeExtension`, so a title can only ever name one file inside the directory
|
||||
// the caller chose.
|
||||
//
|
||||
// A path the user supplies (`--out`, `outPath`) is never sanitized: the caller
|
||||
// is trusted, the server is not.
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { safeExtension, sanitizeFileName } from "../../src/filename.js";
|
||||
|
||||
const FALLBACK = "file-42";
|
||||
|
||||
describe("sanitizeFileName", () => {
|
||||
it("passes a normal title through unchanged", () => {
|
||||
expect(sanitizeFileName("IMG_0001.HEIC", FALLBACK)).toBe(
|
||||
"IMG_0001.HEIC",
|
||||
);
|
||||
expect(sanitizeFileName("Holiday 2024 (1).jpg", FALLBACK)).toBe(
|
||||
"Holiday 2024 (1).jpg",
|
||||
);
|
||||
expect(sanitizeFileName("café.jpg", FALLBACK)).toBe("café.jpg");
|
||||
});
|
||||
|
||||
it("cannot climb out of the directory with ../", () => {
|
||||
// Without sanitizing, this would overwrite the user's SSH keys.
|
||||
expect(sanitizeFileName("../../.ssh/authorized_keys", FALLBACK)).toBe(
|
||||
"__.._.ssh_authorized_keys",
|
||||
);
|
||||
expect(sanitizeFileName("..", FALLBACK)).toBe("_");
|
||||
expect(sanitizeFileName("..\\..\\x", FALLBACK)).toBe("__.._x");
|
||||
});
|
||||
|
||||
it("cannot name an absolute path", () => {
|
||||
expect(sanitizeFileName("/etc/passwd", FALLBACK)).toBe("_etc_passwd");
|
||||
expect(sanitizeFileName("C:\\Windows\\x.dll", FALLBACK)).toBe(
|
||||
"C__Windows_x.dll",
|
||||
);
|
||||
});
|
||||
|
||||
it("replaces embedded separators, so the name stays one file", () => {
|
||||
expect(sanitizeFileName("a/b\\c.jpg", FALLBACK)).toBe("a_b_c.jpg");
|
||||
});
|
||||
|
||||
it("replaces NUL and other control characters", () => {
|
||||
// A NUL truncates the path in C code and makes Node's fs throw.
|
||||
expect(sanitizeFileName("evil\0.jpg", FALLBACK)).toBe("evil_.jpg");
|
||||
expect(sanitizeFileName("line\nbreak\x7f.jpg", FALLBACK)).toBe(
|
||||
"line_break_.jpg",
|
||||
);
|
||||
});
|
||||
|
||||
it("does not produce a hidden file", () => {
|
||||
expect(sanitizeFileName(".bashrc", FALLBACK)).toBe("_bashrc");
|
||||
});
|
||||
|
||||
it("does not produce a Windows device name", () => {
|
||||
expect(sanitizeFileName("CON", FALLBACK)).toBe("_CON");
|
||||
expect(sanitizeFileName("nul.txt", FALLBACK)).toBe("_nul.txt");
|
||||
expect(sanitizeFileName("LPT1", FALLBACK)).toBe("_LPT1");
|
||||
// Only the exact names are reserved.
|
||||
expect(sanitizeFileName("console.jpg", FALLBACK)).toBe("console.jpg");
|
||||
});
|
||||
|
||||
it("falls back to the given name for an empty title", () => {
|
||||
expect(sanitizeFileName("", FALLBACK)).toBe(FALLBACK);
|
||||
});
|
||||
});
|
||||
|
||||
describe("safeExtension", () => {
|
||||
it("keeps a normal extension", () => {
|
||||
expect(safeExtension("IMG_0001.HEIC")).toBe(".HEIC");
|
||||
expect(safeExtension("clip.mp4")).toBe(".mp4");
|
||||
});
|
||||
|
||||
it("uses .bin when there is no extension", () => {
|
||||
expect(safeExtension("")).toBe(".bin");
|
||||
expect(safeExtension("README")).toBe(".bin");
|
||||
});
|
||||
|
||||
it("uses .bin when the extension holds anything but letters and digits", () => {
|
||||
expect(safeExtension("x.j\\..\\pg")).toBe(".bin");
|
||||
expect(safeExtension("x.jp g")).toBe(".bin");
|
||||
expect(safeExtension("x.jpg\0")).toBe(".bin");
|
||||
});
|
||||
});
|
||||
@@ -116,7 +116,7 @@ describe("Library content wiring", () => {
|
||||
expect(lib.photos.byID({ fileID: 1 })!.record().thumbnailPath).toBe(
|
||||
result.path,
|
||||
);
|
||||
lib.close();
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it("drives thumbnails.ensure through the cache", async () => {
|
||||
@@ -139,7 +139,7 @@ describe("Library content wiring", () => {
|
||||
expect(results).toEqual([
|
||||
{ fileID: 1, path: join(root, "cache", "thumbnails", "1.jpg") },
|
||||
]);
|
||||
lib.close();
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it("throws from content methods when opened without a content source", async () => {
|
||||
@@ -155,6 +155,6 @@ describe("Library content wiring", () => {
|
||||
await expect(
|
||||
lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
|
||||
).rejects.toThrow(/content cache/i);
|
||||
lib.close();
|
||||
await lib.close();
|
||||
});
|
||||
});
|
||||
|
||||
@@ -226,6 +226,25 @@ describe("ContentCache.original / thumbnail", () => {
|
||||
expect(skips).toEqual(["skipped"]);
|
||||
});
|
||||
|
||||
it("takes only a letters-and-digits extension from the title", async () => {
|
||||
// The title comes from the server; an extension such as `.\..\x`
|
||||
// must not reach the cache file name, so it becomes `.bin`.
|
||||
const { cache } = buildCache({
|
||||
files: [file(1, "a.jpg"), file(2, "b.\\..\\x"), file(3, "")],
|
||||
});
|
||||
await cache.open();
|
||||
|
||||
expect((await cache.original(1)).path).toBe(
|
||||
join(cacheDir, "originals", "1.jpg"),
|
||||
);
|
||||
expect((await cache.original(2)).path).toBe(
|
||||
join(cacheDir, "originals", "2.bin"),
|
||||
);
|
||||
expect((await cache.original(3)).path).toBe(
|
||||
join(cacheDir, "originals", "3.bin"),
|
||||
);
|
||||
});
|
||||
|
||||
it("serves a file already present in the download directory without fetching", async () => {
|
||||
const downloadDirectory = join(root, "backup");
|
||||
mkdirSync(join(downloadDirectory, "originals"), { recursive: true });
|
||||
|
||||
@@ -179,7 +179,7 @@ describe("Library.fresh", () => {
|
||||
// And the change is now live for the default namespaces too.
|
||||
expect(lib.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -228,7 +228,7 @@ describe("Library.fresh", () => {
|
||||
expect(reads.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
|
||||
}
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -266,7 +266,7 @@ describe("Library.fresh", () => {
|
||||
);
|
||||
expect(lib.status().lastError).toBeUndefined();
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -17,7 +17,8 @@
|
||||
* failure surfaces via `onProgress` ("failed") and `status()`, and a later
|
||||
* success clears the error. `open()` itself resolves even when the first
|
||||
* refresh fails (offline start from cache).
|
||||
* 5. `close()` stops the timer and is idempotent.
|
||||
* 5. `close()` stops the timer and is idempotent, and its promise resolves
|
||||
* only once an in-flight refresh has written the cache file.
|
||||
* 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id.
|
||||
* 7. `open()` branches on the cache: an empty cache awaits the first refresh
|
||||
* (it has nothing to serve yet); an existing cache serves its copy at once
|
||||
@@ -37,6 +38,11 @@
|
||||
* short interval and `vi.waitFor`: a fake clock cannot settle the real
|
||||
* fsync-and-rename cache write, and empty diffs never write, so the eventual
|
||||
* state is stable to poll for.
|
||||
*
|
||||
* A refresh changes RAM before it writes the cache file, so a polled state can
|
||||
* be visible while that write is still running. Every test therefore awaits
|
||||
* `close()`, which waits for the in-flight refresh, before `afterEach` removes
|
||||
* the directory.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
@@ -193,7 +199,7 @@ describe("Library.open and background refresh", () => {
|
||||
expect(reloaded.getFile(1, 1001)?.id).toBe(1001);
|
||||
expect(reloaded.collectionsSinceTime).toBe(100);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -224,7 +230,7 @@ describe("Library.open and background refresh", () => {
|
||||
expect(client.collectionsSinceTimes.length).toBe(collectionCalls);
|
||||
expect(client.filesCalls.length).toBe(fileCalls);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -271,7 +277,7 @@ describe("Library.open and background refresh", () => {
|
||||
{ timeout: 2000, interval: 5 },
|
||||
);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -303,7 +309,7 @@ describe("Library.open and background refresh", () => {
|
||||
// never re-fetched.
|
||||
expect(client.filesCalls).toEqual([]);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -357,7 +363,7 @@ describe("Library.open and background refresh", () => {
|
||||
{ timeout: 2000, interval: 5 },
|
||||
);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -402,7 +408,7 @@ describe("Library.open and background refresh", () => {
|
||||
await new Promise((r) => setTimeout(r, FAST_INTERVAL * 1000 * 4));
|
||||
expect(saveSpy).toHaveBeenCalledTimes(2);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
saveSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
@@ -462,7 +468,7 @@ describe("Library.open and background refresh", () => {
|
||||
{ timeout: 2000, interval: 5 },
|
||||
);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -488,7 +494,7 @@ describe("Library.open and background refresh", () => {
|
||||
),
|
||||
).toBe(true);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -502,9 +508,14 @@ describe("Library.open and background refresh", () => {
|
||||
seed.putFile(file(1001, 1, 400));
|
||||
await seed.save();
|
||||
|
||||
// The server never answers this run's first refresh.
|
||||
// The server does not answer this run's first refresh until the test
|
||||
// is done with it.
|
||||
let answerFirstFetch: (page: CollectionsPage) => void = () => {};
|
||||
const client = new MockClient();
|
||||
client.collectionsSince = () => new Promise<CollectionsPage>(() => {});
|
||||
client.collectionsSince = () =>
|
||||
new Promise<CollectionsPage>((resolve) => {
|
||||
answerFirstFetch = resolve;
|
||||
});
|
||||
|
||||
// open() must resolve from the cache without blocking on the network,
|
||||
// and reads must serve the seeded copy.
|
||||
@@ -517,7 +528,9 @@ describe("Library.open and background refresh", () => {
|
||||
expect(lib.status().lastRefreshAt).toBeUndefined();
|
||||
expect(lib.status().lastError).toBeUndefined();
|
||||
} finally {
|
||||
lib.close();
|
||||
// close() waits for the outstanding refresh, so let it finish.
|
||||
answerFirstFetch({ collections: [], deleted: [], cursor: 500 });
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -564,7 +577,7 @@ describe("Library.open and background refresh", () => {
|
||||
expect(lib.listFiles(1).map((f) => f.id)).toEqual([1001]);
|
||||
expect(lib.status().lastRefreshAt).toBeGreaterThan(0);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -619,7 +632,7 @@ describe("Library.open and background refresh", () => {
|
||||
);
|
||||
expect(reloaded.getFile(1, 1001)?.id).toBe(1001);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
saveSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
@@ -634,8 +647,8 @@ describe("Library.open and background refresh", () => {
|
||||
});
|
||||
const callsAfterOpen = client.collectionsSinceTimes.length;
|
||||
|
||||
lib.close();
|
||||
lib.close(); // second close must not throw
|
||||
await lib.close();
|
||||
await lib.close(); // second close must not throw
|
||||
expect(lib.status().closed).toBe(true);
|
||||
|
||||
// No further refreshes fire once closed.
|
||||
@@ -643,6 +656,65 @@ describe("Library.open and background refresh", () => {
|
||||
expect(client.collectionsSinceTimes.length).toBe(callsAfterOpen);
|
||||
});
|
||||
|
||||
it("close() resolves only after an in-flight refresh has written the cache", async () => {
|
||||
const path = join(cacheDirectory, "metadata.json");
|
||||
const seed = await MetadataStore.load(path);
|
||||
seed.userID = USER_ID;
|
||||
seed.collectionsSinceTime = 500;
|
||||
seed.putCollection(collection(1, 400));
|
||||
seed.putFile(file(1001, 1, 400));
|
||||
await seed.save();
|
||||
|
||||
const client = new MockClient();
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 600)],
|
||||
deleted: [],
|
||||
cursor: 600,
|
||||
});
|
||||
client.filesFor(1, {
|
||||
files: [file(1002, 1, 600)],
|
||||
deleted: [],
|
||||
cursor: 600,
|
||||
});
|
||||
|
||||
// Hold the refresh's cache write until the test releases it.
|
||||
const realSave = MetadataStore.prototype.save;
|
||||
let releaseSave: () => void = () => {};
|
||||
const saveHeld = new Promise<void>((resolve) => {
|
||||
releaseSave = resolve;
|
||||
});
|
||||
const saveSpy = vi
|
||||
.spyOn(MetadataStore.prototype, "save")
|
||||
.mockImplementation(async function (this: MetadataStore) {
|
||||
await saveHeld;
|
||||
return realSave.call(this);
|
||||
});
|
||||
|
||||
const lib = await Library.open({ client, cacheDirectory });
|
||||
try {
|
||||
await vi.waitFor(() => expect(saveSpy).toHaveBeenCalled(), {
|
||||
timeout: 2000,
|
||||
interval: 5,
|
||||
});
|
||||
|
||||
let closed = false;
|
||||
const closing = lib.close().then(() => {
|
||||
closed = true;
|
||||
});
|
||||
await new Promise((r) => setTimeout(r, 50));
|
||||
expect(closed).toBe(false);
|
||||
|
||||
releaseSave();
|
||||
await closing;
|
||||
const reloaded = await MetadataStore.load(path);
|
||||
expect(reloaded.getFile(1, 1002)?.id).toBe(1002);
|
||||
} finally {
|
||||
releaseSave();
|
||||
await lib.close();
|
||||
saveSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
|
||||
it("defaults cacheDirectory to the env-paths cache dir plus user id", async () => {
|
||||
const xdg = join(dir, "xdg-cache");
|
||||
const prev = process.env.XDG_CACHE_HOME;
|
||||
@@ -659,7 +731,7 @@ describe("Library.open and background refresh", () => {
|
||||
expect(lib.cacheDirectory.startsWith(xdg)).toBe(true);
|
||||
expect(lib.cacheDirectory.endsWith(String(USER_ID))).toBe(true);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
} finally {
|
||||
if (prev === undefined) delete process.env.XDG_CACHE_HOME;
|
||||
|
||||
@@ -374,7 +374,7 @@ describe("Library ML-data fetch on refresh", () => {
|
||||
await new Promise((r) => setTimeout(r, FAST_INTERVAL * 1000 * 5));
|
||||
expect(client.mlFetchCalls.length).toBe(callsAfterFirst);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -443,7 +443,59 @@ describe("Library ML-data fetch on refresh", () => {
|
||||
expect(client.mlFetchCalls.length).toBeGreaterThan(callsBefore);
|
||||
expect(client.mlFetchCalls.flat()).toContain(1001);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("close() resolves only after a running ML data fetch has stored its payloads", async () => {
|
||||
const client = new MLMockClient();
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 100)],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
client.filesFor(1, {
|
||||
files: [file(1001, 1, 90)],
|
||||
deleted: [],
|
||||
cursor: 90,
|
||||
});
|
||||
client.mlByFile.set(1001, payload([0.5, 0.25, 0.75]));
|
||||
|
||||
// Hold the ML data fetch open until the test releases it.
|
||||
let release!: () => void;
|
||||
const held = new Promise<void>((r) => (release = r));
|
||||
let fetchStarted!: () => void;
|
||||
const started = new Promise<void>((r) => (fetchStarted = r));
|
||||
const realFetch = client.fetchMLData.bind(client);
|
||||
client.fetchMLData = async (args) => {
|
||||
fetchStarted();
|
||||
await held;
|
||||
return realFetch(args);
|
||||
};
|
||||
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: 3600,
|
||||
});
|
||||
try {
|
||||
await started;
|
||||
|
||||
let closed = false;
|
||||
const closing = lib.close().then(() => {
|
||||
closed = true;
|
||||
});
|
||||
await new Promise((r) => setTimeout(r, 20));
|
||||
expect(closed).toBe(false);
|
||||
|
||||
release();
|
||||
await closing;
|
||||
expect(
|
||||
existsSync(join(cacheDirectory, "mldata", "1001.json")),
|
||||
).toBe(true);
|
||||
} finally {
|
||||
release();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -402,35 +402,97 @@ describe("Precache through Library.open", () => {
|
||||
}
|
||||
|
||||
it("starts both precaches from open() and reports them in status()", async () => {
|
||||
const thumbFetched = new Set<number>();
|
||||
const origFetched = new Set<number>();
|
||||
const source: ContentSource = {
|
||||
original: async ({ file: f, destination }) => {
|
||||
origFetched.add(f.id);
|
||||
original: async ({ destination }) => {
|
||||
await writeFile(destination, Buffer.alloc(10, 1));
|
||||
return { bytesWritten: 10 };
|
||||
},
|
||||
thumbnail: async ({ file: f, destination }) => {
|
||||
thumbFetched.add(f.id);
|
||||
thumbnail: async ({ destination }) => {
|
||||
await writeFile(destination, Buffer.alloc(10, 1));
|
||||
return { bytesWritten: 10 };
|
||||
},
|
||||
};
|
||||
// Each fill reports "done" once the cache has recorded its files. The
|
||||
// source returning is not enough: the cache records a file only after
|
||||
// it has checked it on disk.
|
||||
const finished = new Set<string>();
|
||||
let bothFinished!: () => void;
|
||||
const precached = new Promise<void>((r) => (bothFinished = r));
|
||||
const lib = await Library.open({
|
||||
client: new MockClient(),
|
||||
cacheDirectory: join(root, "cache"),
|
||||
contentSource: source,
|
||||
refreshIntervalSeconds: 3600,
|
||||
onProgress: (e) => {
|
||||
if (
|
||||
e.status === "done" &&
|
||||
(e.operation === "precacheThumbnails" ||
|
||||
e.operation === "precacheOriginals")
|
||||
) {
|
||||
finished.add(e.operation);
|
||||
if (finished.size === 2) bothFinished();
|
||||
}
|
||||
},
|
||||
});
|
||||
|
||||
// Every file's thumbnail is precached; the favorite (file 3) and the
|
||||
// week's files (1, 2) all have their originals precached.
|
||||
await until(() => thumbFetched.size === 3 && origFetched.size === 3);
|
||||
await precached;
|
||||
const status = lib.status();
|
||||
expect(status.thumbnailsTotal).toBe(3);
|
||||
expect(status.thumbnailsCached).toBe(3);
|
||||
expect(status.originalsPinned).toBe(3);
|
||||
expect(status.originalsCached).toBe(3);
|
||||
lib.close();
|
||||
await lib.close();
|
||||
});
|
||||
|
||||
it.each(["thumbnail", "original"] as const)(
|
||||
"close() resolves only after a running %s precache fetch has written its file",
|
||||
async (kind) => {
|
||||
// Only the fill under test runs, and each of its fetches waits
|
||||
// until the test releases it.
|
||||
let release!: () => void;
|
||||
const held = new Promise<void>((r) => (release = r));
|
||||
let fetchStarted!: (destination: string) => void;
|
||||
const started = new Promise<string>((r) => (fetchStarted = r));
|
||||
const fetch = async ({ destination }: { destination: string }) => {
|
||||
fetchStarted(destination);
|
||||
await held;
|
||||
await writeFile(destination, Buffer.alloc(10, 1));
|
||||
return { bytesWritten: 10 };
|
||||
};
|
||||
const unused = async () => {
|
||||
throw new Error("this fill is turned off");
|
||||
};
|
||||
const source: ContentSource =
|
||||
kind === "thumbnail"
|
||||
? { original: unused, thumbnail: fetch }
|
||||
: { original: fetch, thumbnail: unused };
|
||||
const lib = await Library.open({
|
||||
client: new MockClient(),
|
||||
cacheDirectory: join(root, "cache"),
|
||||
contentSource: source,
|
||||
refreshIntervalSeconds: 3600,
|
||||
precacheThumbnails: kind === "thumbnail",
|
||||
precacheOriginals: kind === "original",
|
||||
});
|
||||
try {
|
||||
const destination = await started;
|
||||
|
||||
let closed = false;
|
||||
const closing = lib.close().then(() => {
|
||||
closed = true;
|
||||
});
|
||||
await new Promise((r) => setTimeout(r, 20));
|
||||
expect(closed).toBe(false);
|
||||
|
||||
release();
|
||||
await closing;
|
||||
expect(existsSync(destination)).toBe(true);
|
||||
} finally {
|
||||
release();
|
||||
await lib.close();
|
||||
}
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
@@ -531,7 +531,7 @@ describe("Library exposes the read surface over its live store", () => {
|
||||
expect(client.collectionsCalls).toBe(collectionsBefore);
|
||||
expect(client.filesCalls).toBe(filesBefore);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -159,7 +159,7 @@ describe("Library.snapshot and Library.subscribe", () => {
|
||||
for (const p of snap.photos) expect("key" in p).toBe(false);
|
||||
for (const a of snap.albums) expect("key" in a).toBe(false);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -219,7 +219,7 @@ describe("Library.snapshot and Library.subscribe", () => {
|
||||
expect(change.refreshedAt).toBeGreaterThan(0);
|
||||
} finally {
|
||||
unsubscribe();
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -251,7 +251,7 @@ describe("Library.snapshot and Library.subscribe", () => {
|
||||
expect(changes).toEqual([]);
|
||||
} finally {
|
||||
unsubscribe();
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
@@ -297,7 +297,7 @@ describe("Library.snapshot and Library.subscribe", () => {
|
||||
);
|
||||
expect(changes).toEqual([]);
|
||||
} finally {
|
||||
lib.close();
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -146,10 +146,13 @@ const buildSharedRawCollection = (
|
||||
const buildRawFile = (
|
||||
collectionKey: Uint8Array,
|
||||
opts?: {
|
||||
title?: string;
|
||||
// Any JSON value; `undefined` leaves the title out of the metadata.
|
||||
title?: unknown;
|
||||
fileType?: number;
|
||||
creationTime?: number;
|
||||
info?: { fileSize?: number; thumbSize?: number };
|
||||
// Replaces the whole metadata JSON value.
|
||||
metadata?: unknown;
|
||||
},
|
||||
): RawEnteFile => {
|
||||
const fileKey = sodium.crypto_secretbox_keygen();
|
||||
@@ -158,8 +161,8 @@ const buildRawFile = (
|
||||
collectionKey,
|
||||
);
|
||||
|
||||
const metadata = {
|
||||
title: opts?.title ?? "IMG_0001.jpg",
|
||||
const defaultMetadata = {
|
||||
title: opts && "title" in opts ? opts.title : "IMG_0001.jpg",
|
||||
fileType: opts?.fileType ?? 0,
|
||||
creationTime: opts?.creationTime ?? 1700000000000000,
|
||||
modificationTime: 1700000000000000,
|
||||
@@ -167,6 +170,8 @@ const buildRawFile = (
|
||||
longitude: 2.3522,
|
||||
hash: "abcdef1234567890",
|
||||
};
|
||||
const metadata =
|
||||
opts && "metadata" in opts ? opts.metadata : defaultMetadata;
|
||||
// File metadata is encrypted as a single-chunk secretstream blob
|
||||
// (not secretbox). The decryptionHeader is the secretstream init header.
|
||||
const metadataBytes = new TextEncoder().encode(JSON.stringify(metadata));
|
||||
@@ -321,6 +326,71 @@ describe("model.decryptFile", () => {
|
||||
expect(file.metadata.longitude).toBeCloseTo(2.3522);
|
||||
});
|
||||
|
||||
it("reads a missing or non-string title as an empty string", () => {
|
||||
// The server controls the metadata JSON. A title that is not a
|
||||
// string must not reach code that builds file names from it.
|
||||
const masterKey = sodium.crypto_secretbox_keygen();
|
||||
const { collectionKey } = buildRawCollection(masterKey);
|
||||
for (const title of [undefined, null, 42, ["a"], { x: "../y" }]) {
|
||||
const file = decryptFile(
|
||||
buildRawFile(collectionKey, { title }),
|
||||
collectionKey,
|
||||
);
|
||||
expect(file.metadata.title).toBe("");
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects metadata that is not a JSON object", () => {
|
||||
const masterKey = sodium.crypto_secretbox_keygen();
|
||||
const { collectionKey } = buildRawCollection(masterKey);
|
||||
for (const metadata of [null, "IMG_0001.jpg", 7, []]) {
|
||||
const raw = buildRawFile(collectionKey, { metadata });
|
||||
expect(() => decryptFile(raw, collectionKey)).toThrow(
|
||||
"file 200: metadata is not a JSON object",
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
it("reads the recorded content hash, joining an older live photo's two parts", () => {
|
||||
const masterKey = sodium.crypto_secretbox_keygen();
|
||||
const { collectionKey } = buildRawCollection(masterKey);
|
||||
const hashOf = (metadata: Record<string, unknown>) =>
|
||||
decryptFile(
|
||||
buildRawFile(collectionKey, {
|
||||
metadata: { title: "x", ...metadata },
|
||||
}),
|
||||
collectionKey,
|
||||
).metadata.hash;
|
||||
|
||||
expect(hashOf({ fileType: 0, hash: "H" })).toBe("H");
|
||||
expect(
|
||||
hashOf({ fileType: 2, hash: "H", imageHash: "I", videoHash: "V" }),
|
||||
).toBe("H");
|
||||
expect(hashOf({ fileType: 2, imageHash: "I", videoHash: "V" })).toBe(
|
||||
"I:V",
|
||||
);
|
||||
expect(hashOf({ fileType: 2, imageHash: "I" })).toBeUndefined();
|
||||
expect(
|
||||
hashOf({ fileType: 0, imageHash: "I", videoHash: "V" }),
|
||||
).toBeUndefined();
|
||||
expect(hashOf({ fileType: 0 })).toBeUndefined();
|
||||
expect(hashOf({ fileType: 0, hash: 42 })).toBeUndefined();
|
||||
expect(
|
||||
hashOf({ fileType: 2, imageHash: "I", videoHash: 7 }),
|
||||
).toBeUndefined();
|
||||
// An empty string counts as absent, not as a hash to match.
|
||||
expect(hashOf({ fileType: 0, hash: "" })).toBeUndefined();
|
||||
expect(
|
||||
hashOf({ fileType: 2, hash: "", imageHash: "I", videoHash: "V" }),
|
||||
).toBe("I:V");
|
||||
expect(
|
||||
hashOf({ fileType: 2, imageHash: "", videoHash: "V" }),
|
||||
).toBeUndefined();
|
||||
expect(
|
||||
hashOf({ fileType: 2, imageHash: "I", videoHash: "" }),
|
||||
).toBeUndefined();
|
||||
});
|
||||
|
||||
it("maps fileType numbers to FileType strings", () => {
|
||||
// Ente uses: 0=image, 1=video, 2=livePhoto
|
||||
const masterKey = sodium.crypto_secretbox_keygen();
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
// A checkout nested under `.claude/` must not add its tests to this suite.
|
||||
// The test plants one in a temporary directory next to a real test file and
|
||||
// asks vitest, with this repo's config, which test files it would run.
|
||||
import { afterEach, describe, expect, it } from "vitest";
|
||||
import { execFileSync } from "node:child_process";
|
||||
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
|
||||
|
||||
let root = "";
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(root, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
const writeTest = (path: string): void => {
|
||||
mkdirSync(join(root, path, ".."), { recursive: true });
|
||||
writeFileSync(
|
||||
join(root, path),
|
||||
'import { it } from "vitest";\nit("runs", () => {});\n',
|
||||
);
|
||||
};
|
||||
|
||||
describe("vitest.config.ts", () => {
|
||||
it("does not collect tests from a checkout nested under .claude/", () => {
|
||||
root = mkdtempSync(join(tmpdir(), "quak-nested-checkout-"));
|
||||
writeTest("test/real.test.ts");
|
||||
writeTest(".claude/worktrees/other/test/real.test.ts");
|
||||
|
||||
const output = execFileSync(
|
||||
process.execPath,
|
||||
[
|
||||
join(repoRoot, "node_modules/vitest/vitest.mjs"),
|
||||
"list",
|
||||
"--filesOnly",
|
||||
"--config",
|
||||
join(repoRoot, "vitest.config.ts"),
|
||||
"--root",
|
||||
root,
|
||||
],
|
||||
{ cwd: root, encoding: "utf-8" },
|
||||
);
|
||||
|
||||
const files = output.split("\n").filter((line) => line !== "");
|
||||
expect(files).toEqual(["test/real.test.ts"]);
|
||||
});
|
||||
});
|
||||
@@ -149,8 +149,11 @@ describe("isRetryable: transport failures", () => {
|
||||
});
|
||||
|
||||
it("retries an errno carried on the error itself", () => {
|
||||
// Every errno the classifier names, so none can be reclassified
|
||||
// unnoticed.
|
||||
for (const code of [
|
||||
"ECONNRESET",
|
||||
"ECONNABORTED",
|
||||
"ETIMEDOUT",
|
||||
"EPIPE",
|
||||
"ENOTFOUND",
|
||||
@@ -158,6 +161,8 @@ describe("isRetryable: transport failures", () => {
|
||||
"ECONNREFUSED",
|
||||
"EHOSTUNREACH",
|
||||
"ENETUNREACH",
|
||||
"ENETRESET",
|
||||
"ENETDOWN",
|
||||
]) {
|
||||
expect(isRetryable(errnoError(code))).toBe(true);
|
||||
}
|
||||
@@ -215,6 +220,27 @@ describe("isRetryable: transport failures", () => {
|
||||
looped.cause = looped;
|
||||
expect(isRetryable(looped)).toBe(false);
|
||||
});
|
||||
|
||||
it("terminates on a cause chain that loops through two errors", () => {
|
||||
const first: Error & { cause?: unknown } = new Error("first");
|
||||
const second = new Error("second", { cause: first });
|
||||
first.cause = second;
|
||||
expect(isRetryable(first)).toBe(false);
|
||||
});
|
||||
|
||||
it("reads the error and at most seven causes below it", () => {
|
||||
// The walk is bounded at eight links. An errno at the eighth link is
|
||||
// found; one at the ninth is not.
|
||||
const buried = (causes: number): Error => {
|
||||
let err = errnoError("ECONNRESET");
|
||||
for (let i = 0; i < causes; i++) {
|
||||
err = new Error(`wrapper ${i}`, { cause: err });
|
||||
}
|
||||
return err;
|
||||
};
|
||||
expect(isRetryable(buried(7))).toBe(true);
|
||||
expect(isRetryable(buried(8))).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("isRetryable: stream truncation versus corruption", () => {
|
||||
@@ -279,10 +305,9 @@ describe("isSafeToReplay", () => {
|
||||
* that is not the whole question: the other half is "could the first
|
||||
* attempt already have taken effect on the server?".
|
||||
*
|
||||
* quak's non-idempotent calls are `/users/srp/create-session`,
|
||||
* `/users/two-factor/verify` (which consumes one of a limited number of
|
||||
* 2FA attempts) and `/files/thumbnail`. A blind replay of any of them can
|
||||
* do real damage, so they retry only on the failures that establish no TCP
|
||||
* The calls this guards are the `POST` and `PUT` requests listed in the
|
||||
* README under "Endpoints used". A blind replay of some of them can do
|
||||
* real damage, so they retry only on the failures that establish no TCP
|
||||
* connection to the server ever existed — DNS produced no address, or the
|
||||
* peer refused the connection — and therefore that no request byte can
|
||||
* have been transmitted.
|
||||
@@ -333,6 +358,52 @@ describe("isSafeToReplay", () => {
|
||||
).toBe(false);
|
||||
expect(isSafeToReplay(new TypeError("fetch failed"))).toBe(false);
|
||||
});
|
||||
|
||||
it("does not replay any other errno the classifier names", () => {
|
||||
for (const code of [
|
||||
"ECONNRESET",
|
||||
"ECONNABORTED",
|
||||
"ETIMEDOUT",
|
||||
"EPIPE",
|
||||
"EHOSTUNREACH",
|
||||
"ENETUNREACH",
|
||||
"ENETRESET",
|
||||
"ENETDOWN",
|
||||
]) {
|
||||
expect(isSafeToReplay(errnoError(code))).toBe(false);
|
||||
}
|
||||
});
|
||||
|
||||
it("does not replay a chain that also shows the request may have gone out", () => {
|
||||
// A connect errno somewhere in the chain is not enough: any other
|
||||
// errno beside it is doubt, and doubt is not replayed.
|
||||
const reset = Object.assign(
|
||||
new Error("read ECONNRESET", { cause: errnoError("ECONNREFUSED") }),
|
||||
{ code: "ECONNRESET" },
|
||||
);
|
||||
const mixed = new TypeError("fetch failed", { cause: reset });
|
||||
expect(isRetryable(mixed)).toBe(true);
|
||||
expect(isSafeToReplay(mixed)).toBe(false);
|
||||
});
|
||||
|
||||
it("does not replay a chain longer than the walk reads", () => {
|
||||
// Eight connect errnos, then a reset at the ninth link, below the
|
||||
// limit. The walk never sees the reset, so it cannot rule it out.
|
||||
const refusedChain = (below: Error | undefined): Error => {
|
||||
let err = below;
|
||||
for (let i = 0; i < 8; i++) {
|
||||
err = Object.assign(new Error(`refused ${i}`, { cause: err }), {
|
||||
code: "ECONNREFUSED",
|
||||
});
|
||||
}
|
||||
return err as Error;
|
||||
};
|
||||
expect(isSafeToReplay(refusedChain(errnoError("ECONNRESET")))).toBe(
|
||||
false,
|
||||
);
|
||||
// The same eight links with nothing below them are replayable.
|
||||
expect(isSafeToReplay(refusedChain(undefined))).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
+39
-5
@@ -1,9 +1,43 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { readFileSync } from "node:fs";
|
||||
import { VERSION } from "../src/index.js";
|
||||
|
||||
describe("quak", () => {
|
||||
it("exports a version string", () => {
|
||||
expect(typeof VERSION).toBe("string");
|
||||
expect(VERSION.length).toBeGreaterThan(0);
|
||||
const packageVersion = (
|
||||
JSON.parse(
|
||||
readFileSync(new URL("../package.json", import.meta.url), "utf-8"),
|
||||
) as { version: string }
|
||||
).version;
|
||||
|
||||
class ExitCalled extends Error {}
|
||||
|
||||
describe("version", () => {
|
||||
const argv = process.argv;
|
||||
|
||||
afterEach(() => {
|
||||
process.argv = argv;
|
||||
vi.restoreAllMocks();
|
||||
});
|
||||
|
||||
it("exports the version from package.json", () => {
|
||||
expect(VERSION).toBe(packageVersion);
|
||||
});
|
||||
|
||||
// Runs bin/quak.ts with --version. commander prints the version and then
|
||||
// calls process.exit, which is stubbed to throw so the test survives.
|
||||
it("reports the version from package.json in quak --version", async () => {
|
||||
const printed: string[] = [];
|
||||
vi.spyOn(process.stdout, "write").mockImplementation((chunk) => {
|
||||
printed.push(String(chunk));
|
||||
return true;
|
||||
});
|
||||
vi.spyOn(process, "exit").mockImplementation(() => {
|
||||
throw new ExitCalled();
|
||||
});
|
||||
process.argv = ["node", "quak", "--version"];
|
||||
|
||||
await expect(import("../bin/quak.js")).rejects.toBeInstanceOf(
|
||||
ExitCalled,
|
||||
);
|
||||
expect(printed.join("").trim()).toBe(packageVersion);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
import { configDefaults, defineConfig } from "vitest/config";
|
||||
|
||||
// vitest does not read .gitignore when looking for tests. A checkout nested
|
||||
// under .claude/ has its own test/ tree, and without this exclude the suite
|
||||
// runs once per nested checkout and still reports success.
|
||||
export default defineConfig({
|
||||
test: {
|
||||
exclude: [...configDefaults.exclude, ".claude/**"],
|
||||
},
|
||||
});
|
||||
@@ -528,13 +528,6 @@
|
||||
resolved "https://registry.yarnpkg.com/@types/json-schema/-/json-schema-7.0.15.tgz#596a1747233694d50f6ad8a7869fcb6f56cf5841"
|
||||
integrity sha512-5+fP8P8MFNC+AyZCDxrB2pkZFPGzqQWUzpSeuuVLvm8VMcorNYavBqoFcxK8bQz4Qsbn4oUEEem4wDLfcysGHA==
|
||||
|
||||
"@types/libsodium-wrappers-sumo@0.8.2":
|
||||
version "0.8.2"
|
||||
resolved "https://registry.yarnpkg.com/@types/libsodium-wrappers-sumo/-/libsodium-wrappers-sumo-0.8.2.tgz#488e8747fbb982fe901020b5afeaddfa63da6830"
|
||||
integrity sha512-uFOBpg/r21hExVlh2ty8YpDfSR+Yy3Jn8XS4+SSjitbhTxdYq+pBz/49XRxyUFe8SzqujHf/Wu0/O4d+FUtNfQ==
|
||||
dependencies:
|
||||
libsodium-wrappers-sumo "*"
|
||||
|
||||
"@types/node@22.18.13":
|
||||
version "22.18.13"
|
||||
resolved "https://registry.yarnpkg.com/@types/node/-/node-22.18.13.tgz#a037c4f474b860be660e05dbe92a9ef945472e28"
|
||||
@@ -1076,6 +1069,11 @@ fastq@^1.6.0:
|
||||
dependencies:
|
||||
reusify "^1.0.4"
|
||||
|
||||
fflate@0.8.3:
|
||||
version "0.8.3"
|
||||
resolved "https://registry.yarnpkg.com/fflate/-/fflate-0.8.3.tgz#bc27d8eb30343d4d512abb03480202ce65d825fc"
|
||||
integrity sha512-tbZNuJrLwGUp3zshBtdy4W+ORxZuIh8a5ilyIEQDC5rY1f3U20JMry0Ll3WBzU58EZKsEuJFXhb5gwv8CsPvgA==
|
||||
|
||||
file-entry-cache@^8.0.0:
|
||||
version "8.0.0"
|
||||
resolved "https://registry.yarnpkg.com/file-entry-cache/-/file-entry-cache-8.0.0.tgz#7787bddcf1131bffb92636c69457bbc0edd6d81f"
|
||||
@@ -1249,7 +1247,7 @@ libsodium-sumo@^0.8.0:
|
||||
resolved "https://registry.yarnpkg.com/libsodium-sumo/-/libsodium-sumo-0.8.4.tgz#6d4687781fa0ad398af14a7df872d5c27cf8cd31"
|
||||
integrity sha512-TMtHShQfVVsaxDygyapvUC3o7YsPgXa/hRWeIgzyFz6w5k/1hirGptCxp1U7XwW3rCskaTTYKgV10v86UiGgNw==
|
||||
|
||||
libsodium-wrappers-sumo@*, libsodium-wrappers-sumo@0.8.4:
|
||||
libsodium-wrappers-sumo@0.8.4:
|
||||
version "0.8.4"
|
||||
resolved "https://registry.yarnpkg.com/libsodium-wrappers-sumo/-/libsodium-wrappers-sumo-0.8.4.tgz#6656a3e7e0551ecce08ddee4bfb501a092eac6fa"
|
||||
integrity sha512-ql7hcgulKZ3ekfa2DGAogcCKsWU0diA/0nArz1CFzh93WQdb46/Kj18ka/Hifq6uA3Ush34Pc6vU/6HXeRwUkg==
|
||||
|
||||
Reference in New Issue
Block a user