15 Commits
Author SHA1 Message Date
sneak 59e65e2a41 Check downloaded originals against their recorded content hash (closes #68)
check / check (push) Successful in 35s
downloadFile, shared by quak get, the content cache and backup, hashes
the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and stores
nothing on a mismatch, failing with an error naming the file ID. A live
photo ZIP is unpacked as it streams with fflate's Unzip, in small
slices so memory stays bounded however far an entry expands, and its
image and video hashed separately as <imageHash>:<videoHash>.
decryptFile reads the older imageHash/videoHash fields for live
photos. A file with no recorded hash is stored unchecked.

Model: opus-5-5
2026-09-23 03:48:53 +00:00
clawbot d05b53d560 Single-source the version from package.json (closes #5)
check / check (push) Successful in 25s
src/index.ts imports package.json for VERSION and bin/quak.ts passes
VERSION to commander, so package.json is the only place the version is
written. tsc copies package.json to dist/package.json, so the import
resolves from the built output; script/build runs the built CLI with
--version to prove it. A test checks VERSION and quak --version against
package.json.

Model: opus-5-5
2026-09-23 05:07:47 +02:00
clawbot 0ca8887f52 Test that close() waits for the originals precache (closes #93)
check / check (push) Successful in 36s
The close() test that holds a precache fetch open now runs twice: once
with only the thumbnail fill and once with only the originals fill, so
removing either wait from Precache.close() fails a test.

Model: opus-5-5
2026-09-23 04:54:45 +02:00
clawbot f1836ced57 Fix two intermittently failing library tests (closes #90)
check / check (push) Successful in 27s
Library.close() now returns a promise that resolves once the work it
started has finished: an in-flight refresh with its cache write, the ML
data fetch, and running precache sweeps. The interval test could see a
refresh's new state, close, and remove the directory while the write was
still running. Every library test and CLI command now awaits close(), and
tests hold each of the three writes open to prove close() waits for it.

The precache test waited for its stub source to be called, but the cache
records a file only after checking it on disk, so status() could lag. It
now waits for both fills to report "done".

Model: opus-5-5
2026-09-23 04:38:00 +02:00
clawbot c75c4f987c Pin three untested guards: download timer, URL fragment, short APP1 (closes #89)
check / check (push) Successful in 24s
The download idle deadline's timer is unref'd so it never holds the
process open, and a test checks no timer is left after a download
completes or fails. A test covers the rejection of "#" in a request
path. The EXIF scan accepts an APP1 segment only when its length is at
least 8, since a shorter one cannot hold the six-byte Exif header;
tests cover lengths 7 and 8.

Model: opus-5-5
2026-09-23 03:44:45 +02:00
clawbot 28a2beeab8 Make the download deadline an idle deadline and cancel failed bodies (closes #24)
check / check (push) Successful in 17s
downloadTimeoutMs now aborts a file or thumbnail download only when no
bytes have arrived for that long (default 60 s, was a 600 s cap on the
whole transfer), so a slow download that keeps making progress completes.
The abort reason is still a TimeoutError, so retry classification is
unchanged. A download that fails before reading the whole response body
cancels it, including when the temp file cannot be opened or the header
is malformed, so a failed file no longer holds its connection.

Model: opus-5-5
2026-09-23 03:14:42 +02:00
clawbot 2b410c3ed6 Build every ApiClient request URL in one place (closes #18)
check / check (push) Successful in 25s
getJSON built its URL with new URL and then overwrote the host and
path, which dropped a base path in a self-hosted apiOrigin; postJSON,
putJSON and the file and thumbnail download URLs joined strings. All of
them now go through one function next to ApiClient that accepts a path
with or without a leading slash and an origin with or without a
trailing slash or base path, and percent-encodes query parameters. A
path containing "?" or "#" is rejected.

Model: opus-5-5
2026-09-23 02:57:42 +02:00
clawbot d07692897b Move CLI commands into testable functions and test them (closes #12)
check / check (push) Successful in 27s
The command bodies in bin/quak.ts become functions in
src/cli-commands.ts that take their options and a context (output
streams, session directory, cache directory, session loader) and return
an exit code. bin/quak.ts wires them to commander and exits with that
code once stdout and stderr have drained; nothing below it calls
process.exit. test/cli/commands.test.ts drives the commands with a fake
client and temp directories. Output is unchanged.

Model: opus-5-5
2026-09-23 02:47:47 +02:00
clawbot ed535be1da Harden the backup tree's atomic copy (closes #22)
check / check (push) Successful in 27s
The backup copy now fsyncs its temp file before the rename and the
directory after it, using the download writer's new fsyncPath helper.
Each backup run deletes .quak-backup-*.tmp files whose process is no
longer running, leaving those of a concurrent backup alone. The rename
sites and the README backup layout state that a symlink at the
destination is replaced and the new file takes the temp file's
permissions, and the README names the temp files. Adds tests for a
missing and an unwritable destination directory for downloadFile and
downloadThumbnail.

Model: opus-5-5
2026-09-23 02:44:46 +02:00
clawbot d545dcd8b1 Harden the retry classifier and pin per-attempt deadlines (closes #80)
check / check (push) Successful in 42s
A POST or PUT is replayed only when every errno in the cause chain is a
connect errno and the walk reached the end of the chain, and
postJSON/putJSON no longer follow redirects, so a redirect is an
ApiError that is not retried. getRetryOptions() returns a copy. New
tests pin every errno the classifier names, the cause-chain depth
limit, a chain deeper than the limit, a two-error cycle, and a fresh
deadline per attempt for every retrying entry point. The README
endpoint list is now the one place naming the requests the replay rule
covers; code comments point to it.

Model: opus-5-5
2026-09-23 02:38:01 +02:00
clawbot 52f58f5d2b Harden the JPEG EXIF scan against malformed input (closes #11)
check / check (push) Successful in 29s
The segment scan behind `backup-metadata --exif` now checks every
segment length against the bytes that remain and stops on lengths
under 2, so a truncated or corrupt original can neither throw nor loop.
A malformed or unparseable EXIF segment is recorded as
`imageMetadata.exifError`, and a failure to read the original as
`imageMetadataError` in the per-file JSON, instead of the field being
silently left out. Tests use short hand-built byte arrays.

Model: opus-5-5
2026-09-23 02:28:15 +02:00
clawbot b44c4ba6d7 Keep make test from collecting tests in nested checkouts (closes #25)
check / check (push) Successful in 33s
vitest does not read .gitignore when finding tests, so a checkout nested
under .claude/ had its whole test/ tree run as part of this suite.
vitest.config.ts adds .claude/** to vitest's default excludes. The new
packaging test plants a nested checkout in a temp directory and fails if
vitest, run with this config, would collect it.

Model: opus-5-5
2026-09-23 02:18:31 +02:00
clawbot d50b296d3a Drop the deprecated @types/libsodium-wrappers-sumo stub (closes #27)
check / check (push) Successful in 46s
The package is an empty stub with no declarations; libsodium-wrappers-sumo
ships its own types. Removed with yarn remove, which regenerated yarn.lock.

Model: opus-5-5
2026-09-23 02:15:08 +02:00
clawbot b7d6ab99f4 Validate session snapshots and wipe keys on logout (closes #10)
check / check (push) Successful in 41s
Client.fromJSON checks every snapshot field and each key's decoded length
and throws an error naming the bad field. toJSON reads the token through a
new ApiClient.getAuthToken and throws when there is none. logout zeroes the
key buffers in place; collectionsSince re-checks for logout after its
request so it never decrypts with zeroed keys. The CLI now reports a
corrupt session file separately from a missing one.

Model: opus-5-5
2026-09-23 02:08:02 +02:00
clawbot 3871d6228e Sanitize file names taken from server metadata (closes #9)
check / check (push) Successful in 37s
A file title or album name decrypted from server data could name a path
outside the chosen directory (`../../.ssh/authorized_keys`). One module,
src/filename.ts, now makes such names safe for `quak get`/`get-thumb`
without `--out`, downloadFile/downloadThumbnail without outPath, and the
backup and metadata backup trees. Originals-cache extensions are limited to
letters and digits. A user-supplied path is still used as is. decryptFile
reads a missing or non-string title as "" and rejects metadata that is not
a JSON object.

Model: opus-5-5
2026-09-23 02:04:31 +02:00
47 changed files with 3655 additions and 791 deletions
+61 -33
View File
@@ -73,7 +73,7 @@ if (photo) {
console.log(`original at ${path}`);
}
lib.close();
await lib.close();
```
The lower-level `Client` (login, session serialization, and the raw
@@ -323,6 +323,7 @@ Endpoints used:
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a
collection; paginate while `hasMore` is true.
- `GET https://files.ente.io/?fileID=<id>`: download encrypted file bytes.
- `POST /files/data/fetch`: fetch encrypted ML data for a batch of files.
- `POST /files/upload-url`: mint a presigned upload URL (for thumbnail repair).
- `PUT /files/thumbnail`: register an uploaded thumbnail's object key.
@@ -362,35 +363,41 @@ and a half seconds of waiting. `sleep` and `random` are injectable through the
same option, which is how the test suite exercises the whole policy without
waiting.
Two deadlines, applied with `AbortSignal.timeout()` and renewed for each
attempt:
Two deadlines, renewed for each attempt:
| Option | Default | Applies to |
| ------------------- | -------- | ------------------------------------------- |
| `requestTimeoutMs` | `30000` | `getJSON`, `postJSON`, `putJSON`, `putFile` |
| `downloadTimeoutMs` | `600000` | file and thumbnail body transfers |
| Option | Default | Applies to | Kind |
| ------------------- | ------- | ------------------------------------------- | ------------------------------------- |
| `requestTimeoutMs` | `30000` | `getJSON`, `postJSON`, `putJSON`, `putFile` | the whole request |
| `downloadTimeoutMs` | `60000` | file and thumbnail downloads | idle: no bytes received for this long |
They are separate because one number cannot serve both: a value short enough to
keep a hung API call from stalling a backup would cancel a legitimate
multi-gigabyte download. The download deadline covers the body, not just the
headers — `getFileStream` returns as soon as headers arrive, so a deadline that
only guarded the initial request would leave the same hang one layer down.
They are different kinds because a download's length depends on the file and the
link: a whole-transfer deadline short enough to catch a hung connection would
cancel a large video on a slow link that is still making progress. The download
deadline restarts every time bytes arrive, so a slow download runs as long as it
keeps moving, and one that stalls is aborted after 60 seconds of silence. It
covers the wait for the headers and the body — `getFileStream` returns as soon
as headers arrive, so a deadline that only guarded the initial request would
leave the same hang one layer down. There is no limit on the total length of a
download.
**Non-idempotent requests are not blindly replayed.** `postJSON` and `putJSON`
reach `/users/srp/create-session`, `/users/two-factor/verify` — which consumes
one of a small number of second-factor attempts — and `/files/thumbnail`. They
are retried only on the three failures that establish no TCP connection to the
server ever existed, so no request byte can have been transmitted: `ENOTFOUND`
and `EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the
peer refused the connection). A 5xx, a mid-flight reset and a deadline are all
left to the caller, because each of them can happen after the server has already
acted. The routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are
excluded for the same reason, despite looking like connect-time failures: on
Linux an ICMP unreachable arriving mid-flight, or a local interface going down
after the request was written, delivers them on an already-established socket.
They stay retryable for the idempotent calls. `putFile` is exempt: a presigned
PUT stores one whole object at one key in one request, so replaying it has no
partial state to damage.
send every `POST` and `PUT` in the endpoint list above; some of them change
server state, and `/users/two-factor/verify` consumes one of a small number of
second-factor attempts. They are retried only when every errno in the error's
`cause` chain is one of the three that establish no TCP connection to the server
ever existed, so no request byte can have been transmitted: `ENOTFOUND` and
`EAI_AGAIN` (name resolution produced no address) and `ECONNREFUSED` (the peer
refused the connection). A 5xx, a mid-flight reset and a deadline are all left
to the caller, because each of them can happen after the server has already
acted. These two do not follow redirects either: a redirect means the server
already received the request, so it is reported as an error and not retried. The
routing errnos `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are excluded for the
same reason, despite looking like connect-time failures: on Linux an ICMP
unreachable arriving mid-flight, or a local interface going down after the
request was written, delivers them on an already-established socket. They stay
retryable for the idempotent calls. `putFile` is exempt: a presigned PUT stores
one whole object at one key in one request, so replaying it has no partial state
to damage.
A download is retried as a whole — request, stream consumption, and decryption —
because a socket reset after the response headers have arrived surfaces in the
@@ -422,13 +429,18 @@ decides how to persist sessions.
`client.toJSON()` returns a `ClientSnapshot` (a plain serializable object with
base64-encoded keys) that the consumer can write to disk, a database, or
whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
working client from that snapshot without re-authenticating.
working client from that snapshot without re-authenticating; it checks every
field and each key's length first, and throws an error naming the bad field.
`client.logout()` clears the token and zeroes the key buffers in place; every
later call on that client throws.
The CLI stores the snapshot at the platform-appropriate data directory via
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
`$XDG_DATA_HOME/quak/session.json` on Linux. The file is written with mode
`0600`. The key material is stored in cleartext in the JSON; treat this file as
you would treat the password itself.
you would treat the password itself. A missing file is reported as "not logged
in"; a file that exists but is corrupt is reported as such, naming the bad
field. Both exit with status 1.
### CLI surface
@@ -485,6 +497,16 @@ appears in. On subsequent runs, existing originals are skipped. If a download
fails, the error is logged and the backup continues with the next file. The exit
code is non-zero if any files failed.
Each original is copied to a temporary file named
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp` in the same directory, synced
to disk, and renamed into place, so an original is either complete or absent,
even after a power cut. A run that is killed can leave one of these temporary
files behind; the next backup deletes those whose process is no longer running.
Downloads and the content cache use the same scheme with `.quak-<random>.tmp`
names. The rename replaces whatever was at the destination rather than writing
through it: a symlink there is replaced, not followed, and the new file has the
temporary file's permissions, not those of the file it replaced.
## TODO
- [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
@@ -541,7 +563,10 @@ and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).
`lib.status()` returns a `LibraryStatus` (collection/file counts, last
refresh/ML times and errors, originals usage and effective limit, precache
progress, and `closed`). `lib.close()` stops the background timer; it is
idempotent, and an in-flight refresh is left to finish.
idempotent, and an in-flight refresh is left to finish. The promise it returns
resolves once that refresh (including its cache write), the ML data fetch and
the precache fetches already running have all finished, so the cache directory
can then be removed.
### Default reads vs. fresh reads
@@ -652,10 +677,13 @@ Under `cacheDirectory`:
```
A stored file appears only via an atomic temp-then-rename, so its presence means
it is complete. The design also calls for a content-hash comparison against
`FileMetadata.hash` on each fetched original; that check is deferred (issue
https://git.eeqj.de/sneak/quak/issues/68) because the exact hash construction
cannot yet be confirmed against the repo's fixtures.
it is complete. Every downloaded original (by `quak get`, the cache, or
`backup`) whose metadata records a content hash (`FileMetadata.hash`) is hashed
as it is written: unkeyed BLAKE2b with a 64-byte output, standard base64. For a
live photo, which is stored as a ZIP, the image and the video are hashed
separately and joined as `<imageHash>:<videoHash>`. A mismatch stores nothing
and fails the download with an error naming the file ID. An original with no
recorded hash, from a very old client, is stored unchecked.
### Key types by source file
+96
View File
@@ -18,6 +18,102 @@ Tag v1.0.0.
# Completed Steps
- 2026-09-23: Checked downloaded originals against their recorded content hash
(issue 68). `downloadFile`, which `quak get`, the content cache and backup all
use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and
stores nothing on a mismatch, failing with an error naming the file ID. A live
photo ZIP is unpacked as it streams with `fflate` and its image and video
hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older
clients' `imageHash` and `videoHash` fields for live photos. A file with no
recorded hash is stored unchecked.
- 2026-09-23: Single-sourced the version string (issue 5). `package.json` is the
only place it is written: `src/index.ts` imports it for `VERSION` and
`bin/quak.ts` passes `VERSION` to commander. tsc copies `package.json` to
`dist/package.json`, so the import resolves from the built output too, and
`script/build` runs the built CLI with `--version` to prove it. A test checks
that `VERSION` and `quak --version` both equal the `package.json` version.
- 2026-09-23: Tested that `Library.close()` waits for the originals precache
(issue 93). The test that holds a precache fetch open while `close()` runs now
runs once with only the thumbnail fill and once with only the originals fill,
so dropping either wait from `Precache.close()` fails a test.
- 2026-09-23: Fixed two intermittently failing library tests (issue 90).
`Library.close()` now returns a promise that resolves once an in-flight
refresh (including its cache write), the ML data fetch and running precache
sweeps have finished; the library tests await it, so `afterEach` no longer
removes the cache directory while something is still writing into it. The
precache test waits for both fills to report "done" instead of for its stub
source to be called, which happened before the cache recorded the file.
- 2026-09-23: Pinned three guards reviewers found untested (issue 89). The
download idle deadline's timer is unref'd, so it can never keep the process
alive, and a test checks no timer is left after a download completes or fails.
A test covers the rejection of `#` in a request path. The EXIF scan compares
the `Exif` header only in an APP1 segment of length 8 or more, so it never
reads the next segment's bytes, with a test for a short one.
- 2026-09-23: Made the download deadline an idle deadline (issue 24).
`downloadTimeoutMs` now aborts a file or thumbnail download only after no
bytes have arrived for that long, default 60 seconds, instead of bounding the
whole transfer at 10 minutes, so a slow download that keeps making progress
completes. A download that fails before reading the whole body cancels it, so
a failed file no longer holds its connection.
- 2026-09-23: Every `ApiClient` request URL is now built by one function next to
the class (issue 18), so a self-hosted `apiOrigin` with a base path keeps it
on every request, a path works with or without a leading slash, and query
parameters are percent-encoded. A path containing `?` or `#` is rejected with
an error instead of being silently cut.
- 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies
moved from `bin/quak.ts` into `src/cli-commands.ts` as functions that take
their options and a context (output streams, session directory, cache
directory, session loader) and return an exit code; `bin/quak.ts` only wires
them to commander and exits with the code once stdout and stderr have drained,
so nothing below it calls `process.exit`. `test/cli/commands.test.ts` drives
them with a fake client: session file modes, logout, the missing and corrupt
session paths, and the output and exit code of `whoami`, `collections`,
`files`, `get`, `get-thumb`, `backup` and `helper list-missing-thumbnails`.
- 2026-09-23: Hardened the backup tree's atomic copy (issue 22). `copyAtomic`
fsyncs its temp file before the rename and the directory after it, through the
download writer's `fsyncPath`; each backup run deletes `.quak-backup-*.tmp`
files whose process is no longer running. The README backup layout names the
temp files and states that the rename replaces a symlink and takes the temp
file's permissions. Added tests for a missing and an unwritable destination
directory for `downloadFile` and `downloadThumbnail`.
- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue
11). Every segment length is checked against the remaining bytes and lengths
under 2 stop the scan, so a truncated or corrupt original can neither throw
nor loop. A malformed or unparseable EXIF segment is recorded as
`imageMetadata.exifError`, and a failure to read the original as
`imageMetadataError` in the per-file JSON, instead of the field being left
out.
- 2026-09-22: Hardened the retry classifier (issue 80). A `POST` or `PUT` is
replayed only when every errno in the cause chain is a connect errno, and it
no longer follows redirects. `getRetryOptions()` returns a copy. Tests pin
every errno the classifier names, the cause-chain depth limit, cycle
termination, and a fresh deadline per attempt for every retrying entry point.
The README's endpoint list is the one place that names the requests the replay
rule covers.
- 2026-09-22: Stopped `make test` collecting tests from checkouts nested under
`.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a
nested checkout ran the whole suite again; `vitest.config.ts` now adds
`.claude/**` to vitest's default excludes, and
`test/packaging/nested-checkout.test.ts` plants a nested checkout in a temp
directory and fails if vitest would collect it.
- 2026-09-22: Dropped the deprecated `@types/libsodium-wrappers-sumo` stub from
`devDependencies` (issue 27). It shipped no declarations; the types come from
`libsodium-wrappers-sumo` itself. `yarn.lock` regenerated by `yarn remove`.
- 2026-09-22: Hardened the client session lifecycle (issue 10).
`Client.fromJSON` checks every snapshot field and each key's decoded length
and names the bad field; `toJSON` reads the token through
`ApiClient.getAuthToken` and throws when there is none; `logout` zeroes the
key buffers, and `collectionsSince` re-checks for logout after its request so
it never decrypts with zeroed keys. The CLI reports a corrupt session file
separately from a missing one (`src/cli-session.ts`).
- 2026-09-22: Sanitized file names taken from server metadata (issue 9). A new
`src/filename.ts` holds the one sanitizer, used by `quak get`/`get-thumb`
without `--out`, `downloadFile`/`downloadThumbnail` without `outPath`, and the
backup and metadata backup trees; it removes separators, control characters,
leading dots and Windows device names, and falls back to a name built from the
ID for an empty title. Originals-cache extensions are letters and digits only,
else `.bin`. A user-supplied path is used as is. `decryptFile` reads a missing
or non-string title as "" and rejects metadata that is not a JSON object.
- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
snippets) to match the shipped cache/API library on `next` (issue 53, issue
13). Documented `Library.open` and its options, the default-read vs `fresh()`
+56 -386
View File
@@ -1,125 +1,62 @@
#!/usr/bin/env node
import { input, password as passwordPrompt } from "@inquirer/prompts";
import { stdout, stderr } from "node:process";
import {
copyFileSync,
existsSync,
mkdirSync,
readFileSync,
writeFileSync,
} from "node:fs";
import { join } from "node:path";
import { Command } from "commander";
import envPaths from "env-paths";
import { Client, type ClientSnapshot } from "../src/client.js";
import { init } from "../src/crypto/index.js";
import { Library, type LibraryClient } from "../src/library/index.js";
import {
fileListRow,
fileListLine,
originalName,
thumbnailName,
} from "../src/cli-output.js";
import { freshCollections, freshFiles, freshFile } from "../src/cli-read.js";
import { runMetadataBackup } from "../src/metadata-backup.js";
import {
listMissingThumbnails,
fixMissingThumbnails,
} from "../src/thumbnails.js";
type CliContext,
loginCommand,
whoamiCommand,
logoutCommand,
collectionsCommand,
filesCommand,
getCommand,
getThumbCommand,
backupMetadataCommand,
backupCommand,
listMissingThumbnailsCommand,
fixMissingThumbnailsCommand,
} from "../src/cli-commands.js";
import { loadSession } from "../src/cli-session.js";
import { VERSION } from "../src/index.js";
const paths = envPaths("quak", { suffix: "" });
const sessionPath = join(paths.data, "session.json");
const loadSession = (): ClientSnapshot | null => {
if (!existsSync(sessionPath)) return null;
try {
return JSON.parse(readFileSync(sessionPath, "utf-8")) as ClientSnapshot;
} catch {
return null;
}
};
const saveSession = (snapshot: ClientSnapshot): void => {
mkdirSync(paths.data, { recursive: true, mode: 0o700 });
writeFileSync(sessionPath, JSON.stringify(snapshot, null, 2), {
mode: 0o600,
});
};
const requireSession = (): Client => {
const snapshot = loadSession();
if (!snapshot) {
stderr.write(
`Not logged in. Run "quak login" first.\nSession file: ${sessionPath}\n`,
);
process.exit(1);
}
return Client.fromJSON(snapshot);
};
const prompt = async (message: string): Promise<string> => input({ message });
const promptSecret = async (message: string): Promise<string> =>
passwordPrompt({ message, mask: true });
const program = new Command();
program
.name("quak")
.description("CLI for the Ente end-to-end encrypted photo service")
.version("0.0.0")
.version(VERSION)
.option(
"--cache-dir <path>",
"Directory for the local metadata/content cache " +
"(default: the per-user cache directory)",
);
// The `--cache-dir` global, or undefined to let the library pick its per-user
// default keyed by the account id.
const cacheDirOption = (): string | undefined =>
program.opts<{ cacheDir?: string }>().cacheDir;
// A library client that omits `fetchMLData`, so the point commands below do not
// kick the library's background ML backfill: they read metadata, or fetch one
// file's content, and exit. `backup` and `backup-metadata` handle ML on their
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
// originals through the on-disk cache.
const readLibraryClient = (client: Client): LibraryClient => ({
whoami: () => client.whoami(),
collectionsSince: (args) => client.collectionsSince(args),
filesSince: (args) => client.filesSince(args),
contentSource: () => client.contentSource(),
const context = (): CliContext => ({
stdout,
stderr,
sessionDir: paths.data,
cacheDir: program.opts<{ cacheDir?: string }>().cacheDir,
loadSession,
});
// Open a library for a single point command: the aggressive background precache
// (issue #48) is off — a one-shot `collections` or `get` must not start
// downloading the whole account — and the refresh interval is long so no second
// refresh fires mid-command.
const openReadLibrary = (client: Client): Promise<Library> =>
Library.open({
client: readLibraryClient(client),
cacheDirectory: cacheDirOption(),
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
});
// Close the library and exit once stdout/stderr have drained. `process.exit`
// alone can truncate buffered piped output, and the library keeps the event
// loop alive with a background refresh, so a plain return could hang; this does
// neither.
const finish = (lib: Library | undefined, code: number): void => {
lib?.close();
// Run a command and exit with its code once stdout/stderr have drained.
// Exiting before the drain can truncate piped output, and the library can keep
// the event loop alive after a command returns, so a plain return could hang.
const run = async (command: Promise<number>): Promise<void> => {
process.exitCode = await command;
const pending = [stdout, stderr].filter((s) => s.writableLength > 0);
if (pending.length === 0) {
process.exit(code);
process.exit();
return;
}
let remaining = pending.length;
for (const s of pending) {
s.once("drain", () => {
if (--remaining === 0) process.exit(code);
if (--remaining === 0) process.exit();
});
}
};
@@ -127,92 +64,25 @@ const finish = (lib: Library | undefined, code: number): void => {
program
.command("login")
.description("Log in to an Ente account and save the session")
.action(async () => {
await init();
const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
const password =
process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
stderr.write("Authenticating...\n");
try {
const client = await Client.login({
email,
password,
totp: async () => prompt("TOTP code: "),
emailOTP: async () => prompt("Email verification code: "),
});
saveSession(client.toJSON());
const info = client.whoami();
stderr.write(`Logged in as ${info.email} (user ${info.userID})\n`);
stderr.write(`Session saved to ${sessionPath}\n`);
} catch (err) {
stderr.write(
`Login failed: ${err instanceof Error ? err.message : err}\n`,
);
process.exit(1);
}
});
.action(() => run(loginCommand(context())));
program
.command("whoami")
.description("Print the logged-in account")
.action(() => {
const client = requireSession();
const info = client.whoami();
stdout.write(JSON.stringify(info) + "\n");
});
.action(() => run(whoamiCommand(context())));
program
.command("logout")
.description("Delete the saved session")
.action(async () => {
if (existsSync(sessionPath)) {
const { unlinkSync } = await import("node:fs");
unlinkSync(sessionPath);
stderr.write("Session deleted.\n");
} else {
stderr.write("No session found.\n");
}
});
.action(() => run(logoutCommand(context())));
program
.command("collections")
.description("List all collections (albums)")
.option("--json", "Output as JSON array")
.action(async (opts: { json?: boolean }) => {
await init();
const client = requireSession();
const lib = await openReadLibrary(client);
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52): the pre-library CLI printed current state in
// this order, not the albums projection's newest-first order.
const collections = await freshCollections(lib);
if (opts.json) {
stdout.write(
JSON.stringify(
collections.map((c) => ({
id: c.id,
name: c.name,
type: c.type,
ownerID: c.ownerID,
isShared: c.isShared,
updationTime: c.updationTime,
})),
null,
2,
) + "\n",
);
} else {
for (const c of collections) {
stdout.write(
`${c.id}\t${c.type}\t${c.name}${c.isShared ? " (shared)" : ""}\n`,
);
}
}
finish(lib, 0);
});
.action((opts: { json?: boolean }) =>
run(collectionsCommand(context(), opts)),
);
program
.command("files")
@@ -222,39 +92,9 @@ program
"Collection ID (from `quak collections`)",
)
.option("--json", "Output as JSON array")
.action(async (opts: { collection: string; json?: boolean }) => {
await init();
const client = requireSession();
const collectionID = Number(opts.collection);
if (!Number.isFinite(collectionID)) {
stderr.write("Invalid collection ID\n");
process.exit(1);
}
const lib = await openReadLibrary(client);
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52). Each file prints from its own decrypted
// metadata (raw title, microsecond creationTime) via cli-output, and in
// the pre-library CLI's enumeration order, not the projection's
// newest-first order.
const files = await freshFiles(lib, collectionID);
if (!files) {
stderr.write(`Collection ${collectionID} not found\n`);
finish(lib, 1);
return;
}
if (opts.json) {
stdout.write(
JSON.stringify(files.map(fileListRow), null, 2) + "\n",
);
} else {
for (const file of files) {
stdout.write(fileListLine(file) + "\n");
}
}
finish(lib, 0);
});
.action((opts: { collection: string; json?: boolean }) =>
run(filesCommand(context(), opts)),
);
program
.command("get")
@@ -262,34 +102,9 @@ program
.argument("<fileID>", "File ID (from `quak files`)")
.option("--out <path>", "Output file path")
.option("--collection <id>", "Accepted for compatibility; ignored")
.action(async (fileIDStr: string, opts: { out?: string }) => {
await init();
const client = requireSession();
const fileID = Number(fileIDStr);
if (!Number.isFinite(fileID)) {
stderr.write("Invalid file ID\n");
process.exit(1);
}
const lib = await openReadLibrary(client);
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
stderr.write(`File ${fileID} not found\n`);
finish(lib, 1);
return;
}
const { photo, file } = resolved;
const result = await photo.original();
// Default name is the file's own title, as the pre-library CLI used
// (not the editedName-preferring projection title) (issue #52).
const outPath = opts.out ?? originalName(file);
copyFileSync(result.path, outPath);
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
finish(lib, 0);
});
.action((fileID: string, opts: { out?: string }) =>
run(getCommand(context(), fileID, opts)),
);
program
.command("get-thumb")
@@ -297,34 +112,9 @@ program
.argument("<fileID>", "File ID (from `quak files`)")
.option("--out <path>", "Output file path")
.option("--collection <id>", "Accepted for compatibility; ignored")
.action(async (fileIDStr: string, opts: { out?: string }) => {
await init();
const client = requireSession();
const fileID = Number(fileIDStr);
if (!Number.isFinite(fileID)) {
stderr.write("Invalid file ID\n");
process.exit(1);
}
const lib = await openReadLibrary(client);
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
stderr.write(`File ${fileID} not found\n`);
finish(lib, 1);
return;
}
const { photo, file } = resolved;
const result = await photo.thumbnail();
// Default name is thumb_<file's own title>, as the pre-library CLI
// used (not the projection title) (issue #52).
const outPath = opts.out ?? thumbnailName(file);
copyFileSync(result.path, outPath);
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
finish(lib, 0);
});
.action((fileID: string, opts: { out?: string }) =>
run(getThumbCommand(context(), fileID, opts)),
);
program
.command("backup-metadata")
@@ -337,16 +127,9 @@ program
"Download each file and extract full EXIF/IPTC/XMP metadata (slow)",
)
.option("--all", "Alias for --exif")
.action(async (dir: string, opts: { exif?: boolean; all?: boolean }) => {
await init();
const client = requireSession();
const lib = await openReadLibrary(client);
await runMetadataBackup(lib, client, dir, {
exif: opts.exif || opts.all,
onProgress: (msg) => stderr.write(msg + "\n"),
});
finish(lib, 0);
});
.action((dir: string, opts: { exif?: boolean; all?: boolean }) =>
run(backupMetadataCommand(context(), dir, opts)),
);
program
.command("backup")
@@ -355,43 +138,9 @@ program
)
.argument("<dir>", "Output directory")
.option("--json", "Print result as JSON instead of human-readable summary")
.action(async (dir: string, opts: { json?: boolean }) => {
await init();
const client = requireSession();
stderr.write("Starting backup...\n");
const lib = await Library.open({
client,
downloadDirectory: dir,
cacheDirectory: cacheDirOption(),
});
const result = await lib.backup({
downloadDirectory: dir,
onProgress: (msg) => {
if (!opts.json) stderr.write(msg + "\n");
},
});
if (opts.json) {
stdout.write(JSON.stringify(result, null, 2) + "\n");
} else {
stderr.write("\n--- Backup complete ---\n");
stderr.write(` Total files: ${result.totalFiles}\n`);
stderr.write(` Downloaded: ${result.downloaded}\n`);
stderr.write(` Skipped: ${result.skipped}\n`);
stderr.write(` Failed: ${result.failed}\n`);
if (result.errors.length > 0) {
stderr.write("\nFailed files:\n");
for (const e of result.errors) {
stderr.write(
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
);
}
}
}
finish(lib, result.failed > 0 ? 1 : 0);
});
.action((dir: string, opts: { json?: boolean }) =>
run(backupCommand(context(), dir, opts)),
);
const helper = program
.command("helper")
@@ -401,32 +150,9 @@ helper
.command("list-missing-thumbnails")
.description("List files whose thumbnails are missing or empty")
.option("--json", "Output as JSON array")
.action(async (opts: { json?: boolean }) => {
await init();
const client = requireSession();
const lib = await openReadLibrary(client);
const missing = await listMissingThumbnails(lib, client, (msg) => {
if (!opts.json) stderr.write(msg + "\n");
});
if (opts.json) {
stdout.write(JSON.stringify(missing, null, 2) + "\n");
} else {
if (missing.length === 0) {
stderr.write("No missing thumbnails found.\n");
} else {
stderr.write(
`\n${missing.length} file(s) with missing thumbnails:\n`,
);
for (const m of missing) {
stdout.write(
`${m.fileID}\t${m.title}\t${m.collection}\t${m.reason}\n`,
);
}
}
}
finish(lib, 0);
});
.action((opts: { json?: boolean }) =>
run(listMissingThumbnailsCommand(context(), opts)),
);
helper
.command("fix-missing-thumbnails")
@@ -438,65 +164,9 @@ helper
"Specific file IDs to fix (default: fix all missing)",
)
.option("--json", "Output as JSON")
.action(async (opts: { file?: string[]; json?: boolean }) => {
await init();
const client = requireSession();
const lib = await openReadLibrary(client);
let fileIDs: number[];
if (opts.file && opts.file.length > 0) {
fileIDs = opts.file.map(Number).filter(Number.isFinite);
} else {
stderr.write("Scanning for missing thumbnails...\n");
const missing = await listMissingThumbnails(lib, client, (msg) => {
if (!opts.json) stderr.write(msg + "\n");
});
fileIDs = missing.map((m) => m.fileID);
if (fileIDs.length === 0) {
stderr.write("No missing thumbnails found.\n");
finish(lib, 0);
return;
}
stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
}
const results = await fixMissingThumbnails(
lib,
client,
fileIDs,
(msg) => {
if (!opts.json) stderr.write(msg + "\n");
},
);
if (opts.json) {
stdout.write(JSON.stringify(results, null, 2) + "\n");
} else {
const fixed = results.filter((r) => r.status === "fixed").length;
const skipped = results.filter(
(r) => r.status === "skipped",
).length;
const failed = results.filter((r) => r.status === "failed").length;
stderr.write(`\n--- Done ---\n`);
stderr.write(` Fixed: ${fixed}\n`);
stderr.write(` Skipped: ${skipped}\n`);
stderr.write(` Failed: ${failed}\n`);
if (skipped > 0) {
stderr.write("\nSkipped (unsupported format):\n");
for (const r of results.filter((r) => r.status === "skipped")) {
stderr.write(` ${r.fileID}\t${r.title}\t${r.reason}\n`);
}
}
if (failed > 0) {
stderr.write("\nFailed files:\n");
for (const r of results.filter((r) => r.status === "failed")) {
stderr.write(` ${r.fileID}\t${r.title}\t${r.reason}\n`);
}
}
}
finish(lib, results.some((r) => r.status === "failed") ? 1 : 0);
});
.action((opts: { file?: string[]; json?: boolean }) =>
run(fixMissingThumbnailsCommand(context(), opts)),
);
await init();
program.parse();
+1 -1
View File
@@ -29,7 +29,6 @@
},
"devDependencies": {
"@eslint/js": "9.38.0",
"@types/libsodium-wrappers-sumo": "0.8.2",
"@types/node": "22.18.13",
"eslint": "9.38.0",
"prettier": "3.8.1",
@@ -43,6 +42,7 @@
"env-paths": "4.0.0",
"exif-reader": "2.0.3",
"fast-srp-hap": "2.0.4",
"fflate": "0.8.3",
"jpeg-js": "0.4.4",
"libsodium-wrappers-sumo": "0.8.4"
}
+14
View File
@@ -45,10 +45,24 @@ for (const bin of bins) {
'
}
# src/index.ts imports ../package.json for the version, which tsc copies to
# dist/package.json. Running the built CLI proves that import resolves from
# dist/ and reports the version package.json declares.
verify_version() {
built="$(node dist/bin/quak.js --version)"
declared="$(node -p 'require("./package.json").version')"
if [ "$built" != "$declared" ]; then
echo "build: dist/bin/quak.js reports $built, package.json declares $declared" >&2
exit 1
fi
echo "build: dist/bin/quak.js reports version $built"
}
main() {
cd "$ROOT"
yarn run tsc
verify_entrypoints
verify_version
}
main "$@"
+110 -60
View File
@@ -19,14 +19,12 @@ const DEFAULT_FILES_ORIGIN = "https://files.ente.io";
const DEFAULT_THUMBS_ORIGIN = "https://thumbnails.ente.io";
const CLIENT_PACKAGE = "berlin.sneak.quak";
// Two deadlines rather than one, because a single number cannot serve both
// jobs. Thirty seconds is generous for a JSON call and short enough that a
// hung API connection cannot stall a backup for long. A file body is a
// different shape of problem: the deadline has to cover the whole transfer,
// which for a large video on a slow link is minutes, so a value sane for JSON
// would cancel legitimate downloads.
// Two deadlines of different kinds. `requestTimeoutMs` bounds a whole JSON
// call. `downloadTimeoutMs` is an idle deadline: a file or thumbnail download
// is aborted only when no bytes have arrived for that long, so a large video on
// a slow link that keeps making progress is never cut off.
export const DEFAULT_REQUEST_TIMEOUT_MS = 30_000;
export const DEFAULT_DOWNLOAD_TIMEOUT_MS = 600_000;
export const DEFAULT_DOWNLOAD_TIMEOUT_MS = 60_000;
export interface ApiClientOptions {
apiOrigin?: string;
@@ -48,18 +46,44 @@ export interface StreamOptions {
retry?: boolean;
}
// Enforce a deadline over a response body, not merely over its headers.
// An abort signal that fires once `ms` pass without a call to `restart`. It
// aborts with a `TimeoutError`, the same reason `AbortSignal.timeout()` gives,
// so the retry classifier treats an idle download exactly as it treats any
// other deadline. `stop` must be called when the download ends. The timer is
// unref'd, so even one left running never keeps the process alive.
const idleDeadline = (ms: number) => {
const controller = new AbortController();
let timer: ReturnType<typeof setTimeout> | undefined;
const stop = (): void => clearTimeout(timer);
const restart = (): void => {
stop();
timer = setTimeout(() => {
controller.abort(
new DOMException(
`download stalled: no bytes received for ${ms} ms`,
"TimeoutError",
),
);
}, ms);
timer.unref();
};
restart();
return { signal: controller.signal, restart, stop };
};
// Enforce the idle deadline over a response body, not merely over its headers.
//
// `getFileStream` returns as soon as headers arrive; the bytes are pulled
// later, in the download layer. Whether the signal passed to `fetch` also
// tears down the body afterwards is up to the fetch implementation, so this
// wrapper makes it a property of quak instead: every read races the signal,
// and an abort errors the stream with the abort reason — which the retry
// classifier recognises.
// each chunk that arrives restarts the deadline, and an abort errors the
// stream with the abort reason — which the retry classifier recognises.
const deadlineStream = (
body: ReadableStream<Uint8Array>,
signal: AbortSignal,
deadline: ReturnType<typeof idleDeadline>,
): ReadableStream<Uint8Array> => {
const { signal } = deadline;
const reader = body.getReader();
let rejectOnAbort: (reason: unknown) => void = () => undefined;
const aborted = new Promise<never>((_resolve, reject) => {
@@ -73,7 +97,10 @@ const deadlineStream = (
const onAbort = (): void => rejectOnAbort(signal.reason);
if (signal.aborted) onAbort();
else signal.addEventListener("abort", onAbort, { once: true });
const release = (): void => signal.removeEventListener("abort", onAbort);
const release = (): void => {
deadline.stop();
signal.removeEventListener("abort", onAbort);
};
return new ReadableStream<Uint8Array>({
async pull(controller) {
@@ -84,6 +111,7 @@ const deadlineStream = (
controller.close();
return;
}
deadline.restart();
controller.enqueue(next.value);
} catch (err) {
release();
@@ -98,6 +126,30 @@ const deadlineStream = (
});
};
// The one place a request URL is built. `origin` may carry a base path (a
// self-hosted server behind a prefix) and may end in a slash; `path` may or
// may not start with one. Query parameters go only through `query`, which
// percent-encodes them: a `?` or `#` in `path` is an error, because
// `new URL` would otherwise quietly treat what follows as something else.
const buildURL = (
origin: string,
path: string,
query?: Record<string, string | number | undefined>,
): string => {
if (path.includes("?") || path.includes("#")) {
throw new Error(
`request path must not contain "?" or "#"; pass query parameters separately: ${path}`,
);
}
const url = new URL(
`${origin.replace(/\/+$/, "")}/${path.replace(/^\/+/, "")}`,
);
for (const [k, v] of Object.entries(query ?? {})) {
if (v !== undefined) url.searchParams.set(k, String(v));
}
return url.href;
};
export class ApiClient {
private readonly apiOrigin: string;
private readonly isCustomOrigin: boolean;
@@ -139,11 +191,16 @@ export class ApiClient {
this.token = undefined;
}
getAuthToken(): string | undefined {
return this.token;
}
// The policy this client was configured with, so that a caller wrapping a
// whole operation in its own `withRetry` — the download layer — runs under
// the same settings rather than under the library defaults.
// A copy, so the caller cannot change this client's settings through it.
getRetryOptions(): ResolvedRetryOptions {
return this.retry;
return { ...this.retry };
}
private headers(extra?: Record<string, string>): Record<string, string> {
@@ -197,22 +254,10 @@ export class ApiClient {
path: string,
query?: Record<string, string | number | undefined>,
): Promise<T> {
const url = new URL(path, this.apiOrigin + "/");
// new URL with a base resolves relative paths; ensure we keep the
// origin from apiOrigin even when path starts with /
url.protocol = new URL(this.apiOrigin).protocol;
url.host = new URL(this.apiOrigin).host;
url.pathname = path;
if (query) {
for (const [k, v] of Object.entries(query)) {
if (v !== undefined) {
url.searchParams.set(k, String(v));
}
}
}
const url = buildURL(this.apiOrigin, path, query);
// A GET changes nothing, so it is retried under the full policy.
return withRetry(async () => {
const resp = await this._fetch(url.href, {
const resp = await this._fetch(url, {
method: "GET",
headers: this.headers(),
signal: AbortSignal.timeout(this.requestTimeoutMs),
@@ -223,16 +268,16 @@ export class ApiClient {
}
async postJSON<T>(path: string, body: unknown): Promise<T> {
const url = `${this.apiOrigin}${path}`;
// Idempotency: this reaches `/users/srp/create-session`,
// `/users/two-factor/verify` and `/users/ott`, all of which change
// server state — verifying a second factor consumes one of a small
// number of attempts. So a POST is replayed only on a failure that
// establishes no TCP connection to the server ever existed: DNS
// produced no address, or the peer refused the connection. A 5xx, a
// mid-flight reset, a routing errno (which Linux also delivers on an
// established socket) and a timeout are all left to the caller,
// because each of them can occur after the server has already acted.
const url = buildURL(this.apiOrigin, path);
// Not idempotent: a POST is replayed only when `isSafeToReplay`
// says no request byte can have reached the server. The endpoints
// this covers are listed in the README under "Endpoints used".
//
// Redirects are not followed. The origin has already received the
// request when it answers with one, so a connection refused by the
// redirect target would look replay-safe when it is not. The API has
// no legitimate redirect, so one surfaces as an `ApiError` with its
// 3xx status, which is not retried.
return withRetry(
async () => {
const resp = await this._fetch(url, {
@@ -241,6 +286,7 @@ export class ApiClient {
"Content-Type": "application/json",
}),
body: JSON.stringify(body),
redirect: "manual",
signal: AbortSignal.timeout(this.requestTimeoutMs),
});
await this.throwIfError(resp);
@@ -255,8 +301,8 @@ export class ApiClient {
opts?: StreamOptions,
): Promise<ReadableStream<Uint8Array>> {
const url = this.isCustomOrigin
? `${this.apiOrigin}/files/download/${fileID}`
: `${this.filesOrigin}/?fileID=${fileID}`;
? buildURL(this.apiOrigin, `/files/download/${fileID}`)
: buildURL(this.filesOrigin, "/", { fileID });
return this.streamRequest(url, opts);
}
@@ -298,10 +344,8 @@ export class ApiClient {
}
async putJSON<T>(path: string, body: unknown): Promise<T> {
const url = `${this.apiOrigin}${path}`;
// Same idempotency rule as `postJSON`, for the same reason: this
// reaches `/files/thumbnail`, which registers an uploaded thumbnail
// against a file.
const url = buildURL(this.apiOrigin, path);
// Same replay and redirect rules as `postJSON`, for the same reasons.
return withRetry(
async () => {
const resp = await this._fetch(url, {
@@ -310,6 +354,7 @@ export class ApiClient {
"Content-Type": "application/json",
}),
body: JSON.stringify(body),
redirect: "manual",
signal: AbortSignal.timeout(this.requestTimeoutMs),
});
await this.throwIfError(resp);
@@ -335,8 +380,8 @@ export class ApiClient {
opts?: StreamOptions,
): Promise<ReadableStream<Uint8Array>> {
const url = this.isCustomOrigin
? `${this.apiOrigin}/files/preview/${fileID}`
: `${this.thumbsOrigin}/?fileID=${fileID}`;
? buildURL(this.apiOrigin, `/files/preview/${fileID}`)
: buildURL(this.thumbsOrigin, "/", { fileID });
return this.streamRequest(url, opts);
}
@@ -345,22 +390,27 @@ export class ApiClient {
opts?: StreamOptions,
): Promise<ReadableStream<Uint8Array>> {
const once = async (): Promise<ReadableStream<Uint8Array>> => {
// A fresh deadline per attempt, so a retry gets the whole budget
// rather than the remainder of the one that just expired.
const signal = AbortSignal.timeout(this.downloadTimeoutMs);
const resp = await this._fetch(url, {
method: "GET",
headers: this.headers(),
signal,
});
await this.throwIfError(resp);
if (!resp.body) {
// Carries the status, and is not retryable: a response that
// arrived without a body is malformed, and asking again
// produces the same malformed response.
throw new ApiError("response body is null", resp.status);
// A fresh deadline per attempt. It also covers the wait for the
// headers, when no bytes have arrived either.
const deadline = idleDeadline(this.downloadTimeoutMs);
try {
const resp = await this._fetch(url, {
method: "GET",
headers: this.headers(),
signal: deadline.signal,
});
await this.throwIfError(resp);
if (!resp.body) {
// Carries the status, and is not retryable: a response
// that arrived without a body is malformed, and asking
// again produces the same malformed response.
throw new ApiError("response body is null", resp.status);
}
return deadlineStream(resp.body, deadline);
} catch (err) {
deadline.stop();
throw err;
}
return deadlineStream(resp.body, signal);
};
return opts?.retry === false ? once() : withRetry(once, this.retry);
}
+59 -20
View File
@@ -29,19 +29,21 @@
// rather than counted forever, which would poison a scheduled backup's exit code.
import {
copyFileSync,
lstatSync,
mkdirSync,
readdirSync,
readFileSync,
readlinkSync,
renameSync,
rmSync,
statSync,
symlinkSync,
writeFileSync,
} from "node:fs";
import { basename, dirname, extname, join, relative } from "node:path";
import { copyFile, rename, rm } from "node:fs/promises";
import { basename, dirname, join, relative } from "node:path";
import { fsyncPath } from "./download/index.js";
import { safeExtension, sanitizeFileName } from "./filename.js";
import type { Collection, EnteFile } from "./model/types.js";
export type ProgressCallback = (message: string) => void;
@@ -108,16 +110,11 @@ interface FailureEntry {
const LEDGER_VERSION = 1;
const sanitizePath = (name: string): string =>
name.replace(/[/\\:*?"<>|]/g, "_").replace(/^\.+/, "_");
// The originals/ filename for a file: `<id><ext>`, the extension taken from the
// title (or `.bin`). Matches the content cache's own naming so a present check
// lines up with what a fetch would write.
const originalName = (file: EnteFile): string => {
const ext = extname(file.metadata.title || "") || ".bin";
return `${file.id}${ext}`;
};
const originalName = (file: EnteFile): string =>
`${file.id}${safeExtension(file.metadata.title)}`;
// A regular file with content is treated as complete. A zero-byte file is not:
// it is the shape an aborted write leaves and must be re-fetched.
@@ -158,8 +155,12 @@ const errorMessage = (err: unknown): string =>
err instanceof Error ? err.message : String(err);
// Copy bytes into `dest` via a temp file in the same directory plus rename, so
// `dest` appears only once it is whole ("present means complete").
const copyAtomic = (src: string, dest: string): void => {
// `dest` appears only once it is whole ("present means complete"). As in the
// download writer, the temp file is fsynced before the rename and the directory
// after it, so a power cut cannot leave a correctly named but short original.
// The temp name carries this process's ID so a later run can tell a leftover
// from a copy still in progress (see `removeLeftoverTempFiles`).
const copyAtomic = async (src: string, dest: string): Promise<void> => {
if (src === dest) return;
const tmp = join(
dirname(dest),
@@ -168,10 +169,45 @@ const copyAtomic = (src: string, dest: string): void => {
.slice(2)}.tmp`,
);
try {
copyFileSync(src, tmp);
renameSync(tmp, dest);
await copyFile(src, tmp);
await fsyncPath(tmp);
// `rename` replaces the destination's directory entry: an existing
// symlink at `dest` is replaced, not followed, and the new file has
// the temp file's permissions (copied from `src`).
await rename(tmp, dest);
await fsyncPath(dirname(dest));
} finally {
rmSync(tmp, { force: true });
await rm(tmp, { force: true });
}
};
// A process-ID check: signal 0 delivers nothing and only reports whether the
// process exists. EPERM means it exists but belongs to another user.
const isRunning = (pid: number): boolean => {
try {
process.kill(pid, 0);
return true;
} catch (err) {
return (err as NodeJS.ErrnoException).code === "EPERM";
}
};
// Delete the temp files `copyAtomic` leaves behind when a backup is killed
// before its rename. Only files whose process is no longer running are
// removed, so a backup running at the same time keeps its own. A reused
// process ID can only keep a leftover a while longer, never remove a live one.
const removeLeftoverTempFiles = (dir: string): void => {
let names: string[];
try {
names = readdirSync(dir);
} catch {
return;
}
for (const name of names) {
const match = /^\.quak-backup-.*-(\d+)-[0-9a-z]*\.tmp$/.exec(name);
if (match && !isRunning(Number(match[1]))) {
rmSync(join(dir, name), { force: true });
}
}
};
@@ -258,6 +294,8 @@ export const runBackup = async (
mkdirSync(originalsDir, { recursive: true });
mkdirSync(collectionsDir, { recursive: true });
if (includeThumbnails) mkdirSync(thumbnailsDir, { recursive: true });
removeLeftoverTempFiles(originalsDir);
removeLeftoverTempFiles(thumbnailsDir);
const ledgerPath = join(downloadDirectory, "failures.json");
const ledger = loadLedger(ledgerPath);
@@ -325,7 +363,7 @@ export const runBackup = async (
try {
log(`Fetching original ${file.metadata.title} (${fileID})...`);
const { path } = await lib.original(fileID);
copyAtomic(path, dest);
await copyAtomic(path, dest);
downloaded++;
} catch (err) {
log(
@@ -346,7 +384,7 @@ export const runBackup = async (
if (isPresent(dest)) continue;
try {
const { path } = await lib.thumbnail(fileID);
copyAtomic(path, dest);
await copyAtomic(path, dest);
} catch (err) {
recordFailure(
file,
@@ -370,7 +408,7 @@ export const runBackup = async (
// Then the per-collection symlink trees and JSON.
for (const c of collections) {
const colDirName = sanitizePath(c.name || `collection-${c.id}`);
const colDirName = sanitizeFileName(c.name, `collection-${c.id}`);
const colDir = join(collectionsDir, colDirName);
mkdirSync(colDir, { recursive: true });
@@ -381,8 +419,9 @@ export const runBackup = async (
if (!includeOriginals) continue;
const orig = join(originalsDir, originalName(file));
if (!isPresent(orig)) continue;
const linkName = sanitizePath(
file.metadata.title || `file-${file.id}`,
const linkName = sanitizeFileName(
file.metadata.title,
`file-${file.id}`,
);
const linkPath = join(colDir, linkName);
try {
+486
View File
@@ -0,0 +1,486 @@
// The CLI's commands as plain functions.
//
// Each command takes its options and a `CliContext` and resolves to the exit
// code; a thrown error is left to the caller. Nothing here calls
// `process.exit`: `bin/quak.ts` wires these to the command line and exits with
// the returned code once output has drained. Output must stay byte-identical
// (see `cli-output.ts`).
import { input, password as passwordPrompt } from "@inquirer/prompts";
import {
copyFileSync,
existsSync,
mkdirSync,
unlinkSync,
writeFileSync,
} from "node:fs";
import { join } from "node:path";
import { Client, type ClientSnapshot } from "./client.js";
import { init } from "./crypto/index.js";
import { Library, type LibraryClient } from "./library/index.js";
import {
fileListRow,
fileListLine,
originalName,
thumbnailName,
} from "./cli-output.js";
import { freshCollections, freshFiles, freshFile } from "./cli-read.js";
import { runMetadataBackup } from "./metadata-backup.js";
import { listMissingThumbnails, fixMissingThumbnails } from "./thumbnails.js";
export interface CliContext {
stdout: { write(text: string): unknown };
stderr: { write(text: string): unknown };
// Directory holding `session.json`.
sessionDir: string;
// The `--cache-dir` global, or undefined to let the library pick its
// per-user default keyed by the account id.
cacheDir?: string;
// Reads the session file into a client, or null when there is none. The
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
loadSession: (path: string) => Client | null;
}
const sessionPath = (ctx: CliContext): string =>
join(ctx.sessionDir, "session.json");
// Write the session readable by its owner only, in a directory only its owner
// can enter.
export const saveSession = (
sessionDir: string,
snapshot: ClientSnapshot,
): void => {
mkdirSync(sessionDir, { recursive: true, mode: 0o700 });
writeFileSync(
join(sessionDir, "session.json"),
JSON.stringify(snapshot, null, 2),
{ mode: 0o600 },
);
};
// The saved client, or undefined after telling the user why there is none.
const requireSession = (ctx: CliContext): Client | undefined => {
let client: Client | null;
try {
client = ctx.loadSession(sessionPath(ctx));
} catch (err) {
ctx.stderr.write(
`${err instanceof Error ? err.message : err}\n` +
`Run "quak logout" and then "quak login" to replace it.\n`,
);
return undefined;
}
if (!client) {
ctx.stderr.write(
`Not logged in. Run "quak login" first.\nSession file: ${sessionPath(ctx)}\n`,
);
return undefined;
}
return client;
};
// A library client that omits `fetchMLData`, so the point commands below do not
// kick the library's background ML backfill: they read metadata, or fetch one
// file's content, and exit. `backup` and `backup-metadata` handle ML on their
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
// originals through the on-disk cache.
const readLibraryClient = (client: Client): LibraryClient => ({
whoami: () => client.whoami(),
collectionsSince: (args) => client.collectionsSince(args),
filesSince: (args) => client.filesSince(args),
contentSource: () => client.contentSource(),
});
// Open a library for a single point command: the aggressive background precache
// (issue #48) is off — a one-shot `collections` or `get` must not start
// downloading the whole account — and the refresh interval is long so no second
// refresh fires mid-command.
const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
Library.open({
client: readLibraryClient(client),
cacheDirectory: ctx.cacheDir,
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
});
const prompt = async (message: string): Promise<string> => input({ message });
const promptSecret = async (message: string): Promise<string> =>
passwordPrompt({ message, mask: true });
export const loginCommand = async (ctx: CliContext): Promise<number> => {
await init();
const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
const password =
process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
ctx.stderr.write("Authenticating...\n");
try {
const client = await Client.login({
email,
password,
totp: async () => prompt("TOTP code: "),
emailOTP: async () => prompt("Email verification code: "),
});
saveSession(ctx.sessionDir, client.toJSON());
const info = client.whoami();
ctx.stderr.write(`Logged in as ${info.email} (user ${info.userID})\n`);
ctx.stderr.write(`Session saved to ${sessionPath(ctx)}\n`);
} catch (err) {
ctx.stderr.write(
`Login failed: ${err instanceof Error ? err.message : err}\n`,
);
return 1;
}
return 0;
};
export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const info = client.whoami();
ctx.stdout.write(JSON.stringify(info) + "\n");
return 0;
};
export const logoutCommand = async (ctx: CliContext): Promise<number> => {
if (existsSync(sessionPath(ctx))) {
unlinkSync(sessionPath(ctx));
ctx.stderr.write("Session deleted.\n");
} else {
ctx.stderr.write("No session found.\n");
}
return 0;
};
export const collectionsCommand = async (
ctx: CliContext,
opts: { json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52): the pre-library CLI printed current state in
// this order, not the albums projection's newest-first order.
const collections = await freshCollections(lib);
if (opts.json) {
ctx.stdout.write(
JSON.stringify(
collections.map((c) => ({
id: c.id,
name: c.name,
type: c.type,
ownerID: c.ownerID,
isShared: c.isShared,
updationTime: c.updationTime,
})),
null,
2,
) + "\n",
);
} else {
for (const c of collections) {
ctx.stdout.write(
`${c.id}\t${c.type}\t${c.name}${c.isShared ? " (shared)" : ""}\n`,
);
}
}
return 0;
} finally {
await lib.close();
}
};
export const filesCommand = async (
ctx: CliContext,
opts: { collection: string; json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const collectionID = Number(opts.collection);
if (!Number.isFinite(collectionID)) {
ctx.stderr.write("Invalid collection ID\n");
return 1;
}
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52). Each file prints from its own decrypted
// metadata (raw title, microsecond creationTime) via cli-output, and in
// the pre-library CLI's enumeration order, not the projection's
// newest-first order.
const files = await freshFiles(lib, collectionID);
if (!files) {
ctx.stderr.write(`Collection ${collectionID} not found\n`);
return 1;
}
if (opts.json) {
ctx.stdout.write(
JSON.stringify(files.map(fileListRow), null, 2) + "\n",
);
} else {
for (const file of files) {
ctx.stdout.write(fileListLine(file) + "\n");
}
}
return 0;
} finally {
await lib.close();
}
};
export const getCommand = async (
ctx: CliContext,
fileIDStr: string,
opts: { out?: string },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const fileID = Number(fileIDStr);
if (!Number.isFinite(fileID)) {
ctx.stderr.write("Invalid file ID\n");
return 1;
}
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
ctx.stderr.write(`File ${fileID} not found\n`);
return 1;
}
const { photo, file } = resolved;
const result = await photo.original();
// Default name is the file's own title, as the pre-library CLI used
// (not the editedName-preferring projection title) (issue #52).
const outPath = opts.out ?? originalName(file);
copyFileSync(result.path, outPath);
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
return 0;
} finally {
await lib.close();
}
};
export const getThumbCommand = async (
ctx: CliContext,
fileIDStr: string,
opts: { out?: string },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const fileID = Number(fileIDStr);
if (!Number.isFinite(fileID)) {
ctx.stderr.write("Invalid file ID\n");
return 1;
}
const lib = await openReadLibrary(ctx, client);
try {
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
ctx.stderr.write(`File ${fileID} not found\n`);
return 1;
}
const { photo, file } = resolved;
const result = await photo.thumbnail();
// Default name is thumb_<file's own title>, as the pre-library CLI
// used (not the projection title) (issue #52).
const outPath = opts.out ?? thumbnailName(file);
copyFileSync(result.path, outPath);
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
return 0;
} finally {
await lib.close();
}
};
export const backupMetadataCommand = async (
ctx: CliContext,
dir: string,
opts: { exif?: boolean; all?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
await runMetadataBackup(lib, client, dir, {
exif: opts.exif || opts.all,
onProgress: (msg) => ctx.stderr.write(msg + "\n"),
});
return 0;
} finally {
await lib.close();
}
};
export const backupCommand = async (
ctx: CliContext,
dir: string,
opts: { json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
ctx.stderr.write("Starting backup...\n");
const lib = await Library.open({
client,
downloadDirectory: dir,
cacheDirectory: ctx.cacheDir,
});
try {
const result = await lib.backup({
downloadDirectory: dir,
onProgress: (msg) => {
if (!opts.json) ctx.stderr.write(msg + "\n");
},
});
if (opts.json) {
ctx.stdout.write(JSON.stringify(result, null, 2) + "\n");
} else {
ctx.stderr.write("\n--- Backup complete ---\n");
ctx.stderr.write(` Total files: ${result.totalFiles}\n`);
ctx.stderr.write(` Downloaded: ${result.downloaded}\n`);
ctx.stderr.write(` Skipped: ${result.skipped}\n`);
ctx.stderr.write(` Failed: ${result.failed}\n`);
if (result.errors.length > 0) {
ctx.stderr.write("\nFailed files:\n");
for (const e of result.errors) {
ctx.stderr.write(
` [${e.collection}] ${e.title} (id ${e.fileID}): ${e.error}\n`,
);
}
}
}
return result.failed > 0 ? 1 : 0;
} finally {
await lib.close();
}
};
export const listMissingThumbnailsCommand = async (
ctx: CliContext,
opts: { json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
const missing = await listMissingThumbnails(lib, client, (msg) => {
if (!opts.json) ctx.stderr.write(msg + "\n");
});
if (opts.json) {
ctx.stdout.write(JSON.stringify(missing, null, 2) + "\n");
} else {
if (missing.length === 0) {
ctx.stderr.write("No missing thumbnails found.\n");
} else {
ctx.stderr.write(
`\n${missing.length} file(s) with missing thumbnails:\n`,
);
for (const m of missing) {
ctx.stdout.write(
`${m.fileID}\t${m.title}\t${m.collection}\t${m.reason}\n`,
);
}
}
}
return 0;
} finally {
await lib.close();
}
};
export const fixMissingThumbnailsCommand = async (
ctx: CliContext,
opts: { file?: string[]; json?: boolean },
): Promise<number> => {
await init();
const client = requireSession(ctx);
if (!client) return 1;
const lib = await openReadLibrary(ctx, client);
try {
let fileIDs: number[];
if (opts.file && opts.file.length > 0) {
fileIDs = opts.file.map(Number).filter(Number.isFinite);
} else {
ctx.stderr.write("Scanning for missing thumbnails...\n");
const missing = await listMissingThumbnails(lib, client, (msg) => {
if (!opts.json) ctx.stderr.write(msg + "\n");
});
fileIDs = missing.map((m) => m.fileID);
if (fileIDs.length === 0) {
ctx.stderr.write("No missing thumbnails found.\n");
return 0;
}
ctx.stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
}
const results = await fixMissingThumbnails(
lib,
client,
fileIDs,
(msg) => {
if (!opts.json) ctx.stderr.write(msg + "\n");
},
);
if (opts.json) {
ctx.stdout.write(JSON.stringify(results, null, 2) + "\n");
} else {
const fixed = results.filter((r) => r.status === "fixed").length;
const skipped = results.filter(
(r) => r.status === "skipped",
).length;
const failed = results.filter((r) => r.status === "failed").length;
ctx.stderr.write(`\n--- Done ---\n`);
ctx.stderr.write(` Fixed: ${fixed}\n`);
ctx.stderr.write(` Skipped: ${skipped}\n`);
ctx.stderr.write(` Failed: ${failed}\n`);
if (skipped > 0) {
ctx.stderr.write("\nSkipped (unsupported format):\n");
for (const r of results.filter((r) => r.status === "skipped")) {
ctx.stderr.write(
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
);
}
}
if (failed > 0) {
ctx.stderr.write("\nFailed files:\n");
for (const r of results.filter((r) => r.status === "failed")) {
ctx.stderr.write(
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
);
}
}
}
return results.some((r) => r.status === "failed") ? 1 : 0;
} finally {
await lib.close();
}
};
+6 -3
View File
@@ -9,6 +9,7 @@
// `metadata.title`, and issue #52 requires that output stay byte-identical, so
// the commands shape their output from the raw `EnteFile` through here.
import { sanitizeFileName } from "./filename.js";
import type { EnteFile, FileType, Microseconds } from "./model/types.js";
// One row of `quak files --json`.
@@ -32,9 +33,11 @@ export const fileListRow = (file: EnteFile): FileListRow => ({
export const fileListLine = (file: EnteFile): string =>
`${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
// Default output path for `quak get` when `--out` is not given.
export const originalName = (file: EnteFile): string => file.metadata.title;
// Default output path for `quak get` when `--out` is not given. The title comes
// from the server, so it is sanitized; `--out` is the user's and is used as is.
export const originalName = (file: EnteFile): string =>
sanitizeFileName(file.metadata.title, `file-${file.id}`);
// Default output path for `quak get-thumb` when `--out` is not given.
export const thumbnailName = (file: EnteFile): string =>
`thumb_${file.metadata.title}`;
`thumb_${originalName(file)}`;
+26
View File
@@ -0,0 +1,26 @@
// How the CLI reads its saved session file back into a `Client`.
//
// A missing file means "not logged in" and returns null. A file that exists but
// cannot be read back into a client (bad JSON, a missing field, a key of the
// wrong length) throws an error saying the session file is corrupt, so the CLI
// can tell the user which of the two it is. Needs `init()` first.
import { existsSync, readFileSync } from "node:fs";
import type { ApiClientOptions } from "./api/client.js";
import { Client } from "./client.js";
export const loadSession = (
path: string,
apiOptions?: ApiClientOptions,
): Client | null => {
if (!existsSync(path)) return null;
try {
return Client.fromJSON(
JSON.parse(readFileSync(path, "utf-8")),
apiOptions,
);
} catch (err) {
const reason = err instanceof Error ? err.message : String(err);
throw new Error(`Session file ${path} is corrupt: ${reason}`);
}
};
+62 -11
View File
@@ -125,18 +125,57 @@ export class Client {
);
}
static fromJSON(
snapshot: ClientSnapshot,
apiOptions?: ApiClientOptions,
): Client {
const api = new ApiClient({ ...apiOptions, authToken: snapshot.token });
// Restore a client from a `toJSON()` snapshot. The snapshot usually comes
// straight from `JSON.parse` of a file on disk, so every field is checked
// before use; a bad one throws an error naming it. Needs `init()` first.
static fromJSON(snapshot: unknown, apiOptions?: ApiClientOptions): Client {
const invalid = (field: string, problem: string): Error =>
new Error(`Invalid session data: ${field} ${problem}`);
if (typeof snapshot !== "object" || snapshot === null) {
throw new Error("Invalid session data: not a JSON object");
}
const s = snapshot as Record<string, unknown>;
for (const field of ["email", "token"]) {
if (typeof s[field] !== "string" || s[field] === "") {
throw invalid(field, "must be a non-empty string");
}
}
if (!Number.isInteger(s.userID)) {
throw invalid("userID", "must be an integer");
}
const key = (field: string): Uint8Array => {
const value = s[field];
if (typeof value !== "string") {
throw invalid(field, "must be a base64 string");
}
let bytes: Uint8Array;
try {
bytes = fromBase64(value);
} catch {
throw invalid(field, "is not valid base64");
}
// The master key (secretbox) and the key pair (box) are all 32 bytes.
if (bytes.length !== 32) {
throw invalid(
field,
`must decode to 32 bytes, got ${bytes.length}`,
);
}
return bytes;
};
const api = new ApiClient({
...apiOptions,
authToken: s.token as string,
});
return new Client(
api,
snapshot.email,
snapshot.userID,
fromBase64(snapshot.masterKey),
fromBase64(snapshot.secretKey),
fromBase64(snapshot.publicKey),
s.email as string,
s.userID as number,
key("masterKey"),
key("secretKey"),
key("publicKey"),
);
}
@@ -164,19 +203,29 @@ export class Client {
toJSON(): ClientSnapshot {
this.assertLoggedIn();
const token = this.api.getAuthToken();
if (!token) {
throw new Error("Cannot serialize client: it has no auth token");
}
return {
email: this.email,
userID: this.userID,
token: this.api["token"]!,
token,
masterKey: toBase64(this.masterKey),
secretKey: toBase64(this.secretKey),
publicKey: toBase64(this.publicKey),
};
}
// Zeroes the key buffers in place, so any copy of the reference held
// elsewhere is wiped too. Every method checks `assertLoggedIn` before
// touching the keys, so nothing decrypts with the zeroed keys.
logout(): void {
this.loggedOut = true;
this.api.clearAuthToken();
this.masterKey.fill(0);
this.secretKey.fill(0);
this.publicKey.fill(0);
}
// Enumerate collections changed since `sinceTime`. Live collections are
@@ -192,6 +241,8 @@ export class Client {
const { collections: raws } = await this.api.getJSON<{
collections: RawCollection[];
}>("/collections/v2", { sinceTime: args.sinceTime });
// logout() may have zeroed the keys while the request was in flight.
this.assertLoggedIn();
const collections: Collection[] = [];
const deleted: number[] = [];
+22
View File
@@ -0,0 +1,22 @@
import sodium, { type StateAddress } from "libsodium-wrappers-sumo";
import { toBase64 } from "./encoding.js";
// The content hash an uploading client records in a file's metadata: unkeyed
// BLAKE2b with a 64-byte output over the original's bytes, fed in chunks, as
// standard base64 with padding. Named after the upstream client's functions.
// The output length is read at call time for the same reason as
// `streamTagFinal` in stream.ts: libsodium sets its constants only once ready.
export const chunkHashInit = (): StateAddress =>
sodium.crypto_generichash_init(null, sodium.crypto_generichash_BYTES_MAX);
export const chunkHashUpdate = (state: StateAddress, chunk: Uint8Array): void =>
sodium.crypto_generichash_update(state, chunk);
export const chunkHashFinal = (state: StateAddress): string =>
toBase64(
sodium.crypto_generichash_final(
state,
sodium.crypto_generichash_BYTES_MAX,
),
);
+1
View File
@@ -7,6 +7,7 @@ export {
} from "./encoding.js";
export { deriveKEK, deriveLoginSubkey } from "./kdf.js";
export { decryptBox, decryptSealed } from "./box.js";
export { chunkHashFinal, chunkHashInit, chunkHashUpdate } from "./hash.js";
export {
decryptBlob,
encryptBlob,
+200 -56
View File
@@ -2,7 +2,11 @@ import { randomUUID } from "node:crypto";
import { open, rename, rm } from "node:fs/promises";
import type { FileHandle } from "node:fs/promises";
import { dirname, join } from "node:path";
import { Unzip, UnzipInflate } from "fflate";
import {
chunkHashFinal,
chunkHashInit,
chunkHashUpdate,
fromBase64,
initStreamPull,
pullStreamChunk,
@@ -11,6 +15,7 @@ import {
streamTagFinal,
} from "../crypto/index.js";
import { TruncatedStreamError } from "../errors.js";
import { sanitizeFileName } from "../filename.js";
import { withRetry } from "../retry.js";
import type { ApiClient } from "../api/client.js";
import type { EnteFile } from "../model/types.js";
@@ -97,47 +102,53 @@ const streamDecrypt = async (
onProgress?.(totalPlain);
};
for (;;) {
const { done, value } = await reader.read();
if (value && value.length > 0) {
pending.push(value);
pendingBytes += value.length;
}
try {
for (;;) {
const { done, value } = await reader.read();
if (value && value.length > 0) {
pending.push(value);
pendingBytes += value.length;
}
while (pendingBytes >= ENC_CHUNK_SIZE) {
const encChunk = takeContiguous(ENC_CHUNK_SIZE);
// A whole chunk that fails to authenticate while the stream carries
// on is corruption, not truncation; that error propagates unchanged.
const { plaintext, tag } = pullStreamChunk(state, encChunk);
await consume(plaintext, tag);
}
while (pendingBytes >= ENC_CHUNK_SIZE) {
const encChunk = takeContiguous(ENC_CHUNK_SIZE);
// A whole chunk that fails to authenticate while the stream
// carries on is corruption, not truncation; that error
// propagates unchanged.
const { plaintext, tag } = pullStreamChunk(state, encChunk);
await consume(plaintext, tag);
}
if (done) {
if (pendingBytes > 0) {
const buffer = takeContiguous(pendingBytes);
// Whatever is left over once every whole chunk has been
// consumed must be the stream's final chunk, and a final
// chunk that actually arrived in full authenticates. If it
// does not, the body stopped part-way through a chunk — the
// ordinary shape of a dropped connection. Poly1305 cannot
// tell a partial chunk from a corrupt one, so this is
// reported as the truncation it almost always is, with the
// authentication failure kept as the error's cause. Only the
// pull is guarded: a sink failure on a chunk that did
// authenticate is a disk error, not a truncation.
let pulled;
try {
pulled = pullStreamChunk(state, buffer);
} catch (err) {
throw new TruncatedStreamError(
`download: stream truncated: response body ended with ${buffer.length} trailing bytes that did not authenticate as a final chunk (transfer stopped mid-chunk, or the data is corrupt)`,
{ cause: err },
);
if (done) {
if (pendingBytes > 0) {
const buffer = takeContiguous(pendingBytes);
// Whatever is left over once every whole chunk has been
// consumed must be the stream's final chunk, and a final
// chunk that actually arrived in full authenticates. If
// it does not, the body stopped part-way through a chunk
// — the ordinary shape of a dropped connection. Poly1305
// cannot tell a partial chunk from a corrupt one, so this
// is reported as the truncation it almost always is, with
// the authentication failure kept as the error's cause.
// Only the pull is guarded: a sink failure on a chunk
// that did authenticate is a disk error, not a
// truncation.
let pulled;
try {
pulled = pullStreamChunk(state, buffer);
} catch (err) {
throw new TruncatedStreamError(
`download: stream truncated: response body ended with ${buffer.length} trailing bytes that did not authenticate as a final chunk (transfer stopped mid-chunk, or the data is corrupt)`,
{ cause: err },
);
}
await consume(pulled.plaintext, pulled.tag);
}
await consume(pulled.plaintext, pulled.tag);
break;
}
break;
}
} finally {
reader.releaseLock();
}
// Only the last chunk of a secretstream carries TAG_FINAL. Everything a
@@ -157,6 +168,18 @@ const streamDecrypt = async (
return totalPlain;
};
// Fsync a file or a directory, so its contents (for a directory, its entries)
// are on stable storage. Exported for the backup tree's copy, which needs the
// same durability as the writer below.
export const fsyncPath = async (path: string): Promise<void> => {
const handle = await open(path, "r");
try {
await handle.sync();
} finally {
await handle.close();
}
};
// Stage a write to `destination` atomically and durably, then rename it into
// place. `fill` writes the contents into the open temp file handle — either the
// whole buffer at once (`writeAtomic`) or chunk by chunk as they decrypt
@@ -192,16 +215,15 @@ const stageAtomic = async (
} finally {
await handle.close();
}
// `rename` replaces the destination's directory entry rather than
// writing through it: an existing symlink at `destination` is
// replaced, not followed, and the new file has the temp file's
// permissions, not those of the file it replaced.
await rename(tmpPath, destination);
// Fsync the directory so the rename itself survives a crash: renaming
// over a synced temp file still leaves the new directory entry in the
// page cache until the directory is synced.
const dirHandle = await open(dir, "r");
try {
await dirHandle.sync();
} finally {
await dirHandle.close();
}
await fsyncPath(dir);
} catch (err) {
// Best-effort cleanup. A failure to remove the temporary file must
// never replace the error that actually explains what went wrong.
@@ -219,31 +241,139 @@ export const writeAtomic = async (
): Promise<void> =>
stageAtomic(destination, (handle) => handle.writeFile(plaintext));
// Hashes an original's bytes as they are decrypted, for comparison with the
// hash its uploader recorded.
interface ContentHasher {
update: (plaintext: Uint8Array) => void;
digest: () => string;
}
const fileHasher = (): ContentHasher => {
const state = chunkHashInit();
return {
update: (plaintext) => chunkHashUpdate(state, plaintext),
digest: () => chunkHashFinal(state),
};
};
// A live photo is stored as a ZIP of its image and its video, and its recorded
// hash is `<imageHash>:<videoHash>`, each over that part's own bytes. Like the
// upstream client's decoder, this takes the first entries whose names start
// with `image` and `video`.
//
// The ZIP is chosen by its uploader and may expand enormously, so entries are
// hashed as they decompress and never held. fflate's `Unzip` inflates each
// push in one piece, and deflate expands at most about 1000-fold, so the ZIP
// is pushed in 4 KiB slices to keep each decompressed piece near 4 MiB, one
// plaintext chunk. Every entry is started, even one that is not hashed,
// because fflate keeps an unstarted entry's data in memory.
const livePhotoHasher = (fileID: number): ContentHasher => {
const sliceSize = 4096;
const fail = (message: string, cause?: unknown): Error =>
new Error(`download: file ${fileID}: ${message}`, { cause });
const claimed = new Set<string>();
const hashes = new Map<string, string>();
const unzip = new Unzip((entry) => {
const part = ["image", "video"].find((p) => entry.name.startsWith(p));
const target =
part === undefined || claimed.has(part)
? undefined
: { part, state: chunkHashInit() };
if (target !== undefined) claimed.add(target.part);
entry.ondata = (err, data, final) => {
if (err) throw err;
if (target === undefined) return;
chunkHashUpdate(target.state, data);
if (final) hashes.set(target.part, chunkHashFinal(target.state));
};
entry.start();
});
unzip.register(UnzipInflate);
// fflate reports a bad ZIP by throwing, sometimes a TypeError, which the
// retry would take for a network failure; a bad ZIP is never retried.
const push = (data: Uint8Array, final: boolean): void => {
try {
unzip.push(data, final);
} catch (err) {
throw fail("live photo is not a readable ZIP", err);
}
};
return {
update: (plaintext) => {
for (let i = 0; i < plaintext.length; i += sliceSize) {
push(plaintext.subarray(i, i + sliceSize), false);
}
},
digest: () => {
push(new Uint8Array(0), true);
const image = hashes.get("image");
const video = hashes.get("video");
if (image === undefined || video === undefined) {
throw fail(
"live photo ZIP does not hold both an image and a video",
);
}
return `${image}:${video}`;
},
};
};
// Decrypt `stream` straight to `destination`, one plaintext chunk at a time,
// under the atomic writer's temp-then-rename discipline. Memory stays bounded
// by the chunk size: each decrypted chunk is written to the temp file and
// dropped. The rename happens only after the stream authenticates as terminated
// on TAG_FINAL; a truncated stream throws and leaves the destination untouched.
// Returns the plaintext length written.
//
// `original` is the file whose original this is (none for a thumbnail, which
// has no recorded hash). When its metadata has a hash, the decrypted bytes
// must match it or nothing is stored. Both a plain file and a live photo's
// parts are hashed as they stream. The mismatch error is not retried.
const decryptToTemp = async (
destination: string,
stream: ReadableStream<Uint8Array>,
header: Uint8Array,
key: Uint8Array,
onProgress?: ProgressCallback,
original?: EnteFile,
): Promise<number> => {
const expected = original?.metadata.hash;
const hasher =
original === undefined || expected === undefined
? undefined
: original.metadata.fileType === "livePhoto"
? livePhotoHasher(original.id)
: fileHasher();
let bytesWritten = 0;
await stageAtomic(destination, async (handle) => {
bytesWritten = await streamDecrypt(
stream,
header,
key,
async (plaintext) => {
await handle.write(plaintext);
},
onProgress,
);
});
try {
await stageAtomic(destination, async (handle) => {
bytesWritten = await streamDecrypt(
stream,
header,
key,
async (plaintext) => {
hasher?.update(plaintext);
await handle.write(plaintext);
},
onProgress,
);
if (original === undefined || hasher === undefined) return;
const actual = hasher.digest();
if (actual !== expected) {
throw new Error(
`download: file ${original.id}: content hash ${actual} does not match the hash its uploader recorded, ${expected}`,
);
}
});
} catch (err) {
// Cancel the body so its connection is closed now rather than held
// until the stream is garbage collected. A backup run carries on past
// a failed file, so without this every failure would hold a socket.
// This covers every failure, including a temp file that cannot be
// opened and a header that is rejected before the body is read.
await stream.cancel(err).catch(() => undefined);
throw err;
}
return bytesWritten;
};
@@ -274,10 +404,18 @@ const fetchAndDecrypt = async (
key: Uint8Array,
destination: string,
onProgress?: ProgressCallback,
original?: EnteFile,
): Promise<number> =>
withRetry(async () => {
const stream = await openStream();
return decryptToTemp(destination, stream, header, key, onProgress);
return decryptToTemp(
destination,
stream,
header,
key,
onProgress,
original,
);
}, api.getRetryOptions());
export const downloadFile = async (
@@ -286,7 +424,10 @@ export const downloadFile = async (
outPath?: string,
onProgress?: ProgressCallback,
): Promise<DownloadResult> => {
const resolvedPath = outPath ?? file.metadata.title;
// `outPath` is the caller's and is used as is; the title is the server's
// and is sanitized so it can only name a file in the current directory.
const resolvedPath =
outPath ?? sanitizeFileName(file.metadata.title, `file-${file.id}`);
const header = fromBase64(file.file.decryptionHeader);
const bytesWritten = await fetchAndDecrypt(
api,
@@ -295,6 +436,7 @@ export const downloadFile = async (
file.key,
resolvedPath,
onProgress,
file,
);
return { path: resolvedPath, bytesWritten };
};
@@ -305,7 +447,9 @@ export const downloadThumbnail = async (
outPath?: string,
onProgress?: ProgressCallback,
): Promise<DownloadResult> => {
const resolvedPath = outPath ?? `thumb_${file.metadata.title}`;
const resolvedPath =
outPath ??
`thumb_${sanitizeFileName(file.metadata.title, `file-${file.id}`)}`;
const header = fromBase64(file.thumbnail.decryptionHeader);
const bytesWritten = await fetchAndDecrypt(
api,
+37
View File
@@ -0,0 +1,37 @@
// File names built from server-supplied metadata.
//
// A file's title and a collection's name are decrypted from data the server
// hands us, and quak does not trust the server. Any name taken from them and
// used on disk goes through here, so it can only ever name one file inside the
// directory the caller chose: never a path, never `..`, never hidden, never a
// Windows device name.
//
// A path the user typed (`--out`, `outPath`) is not passed through here: the
// caller is trusted, the server is not.
import { extname } from "node:path";
// Path separators, characters Windows forbids in file names, and control
// characters (NUL included).
// eslint-disable-next-line no-control-regex
const UNSAFE_CHARACTERS = /[/\\:*?"<>|\x00-\x1f\x7f]/g;
// Names Windows reserves for devices, with or without an extension.
const RESERVED_DEVICE_NAME = /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\.|$)/i;
// `name` made safe to use as a single file name. Each unsafe character becomes
// `_`, a leading run of dots becomes one `_`, and a device name gets a leading
// `_`. A name with none of these comes back unchanged. An empty name becomes
// `fallback`, which the caller derives from the record's ID.
export const sanitizeFileName = (name: string, fallback: string): string => {
if (name === "") return fallback;
const cleaned = name.replace(UNSAFE_CHARACTERS, "_").replace(/^\.+/, "_");
return RESERVED_DEVICE_NAME.test(cleaned) ? `_${cleaned}` : cleaned;
};
// The extension of `title` (".jpg"), or ".bin" when it has none or it holds
// anything but letters and digits.
export const safeExtension = (title: string): string => {
const ext = extname(title);
return /^\.[A-Za-z0-9]+$/.test(ext) ? ext : ".bin";
};
+5 -1
View File
@@ -1,4 +1,8 @@
export const VERSION = "0.0.0";
// package.json is the one place the version is written. tsc copies it to
// dist/package.json, so this path resolves from source and from dist/src/.
import pkg from "../package.json" with { type: "json" };
export const VERSION: string = pkg.version;
export {
Client,
+9 -10
View File
@@ -15,12 +15,12 @@
// Integrity. The reused streaming decrypt is the enforced guarantee: every
// chunk is authenticated and the writer renames the file into place only once
// the stream ends on TAG_FINAL, so a truncated or corrupt fetch throws and
// nothing is stored. On top of that this module refuses to record a stored file
// that came out empty. The design also asks for a content-hash comparison
// against `FileMetadata.hash` (with a `fileSize` fallback); that is deferred —
// see the PR — because the exact hash construction cannot be confirmed against
// the repo's fixtures and `FileBlob.size` is the encrypted object size, not the
// decrypted length this layer has.
// nothing is stored. For an original whose metadata records a content hash
// (`FileMetadata.hash`), the writer also hashes the decrypted bytes and stores
// nothing if they differ, failing the fetch with an error naming the file. An
// original with no recorded hash is stored unchecked, as the upstream client
// does; thumbnails have none. On top of that this module refuses to record a
// stored file that came out empty.
import { existsSync, statSync } from "node:fs";
import {
@@ -40,6 +40,7 @@ import {
downloadThumbnail,
type ProgressCallback,
} from "../download/index.js";
import { safeExtension } from "../filename.js";
import type { EnteFile } from "../model/types.js";
import type { Priority, RequestPools } from "./pools.js";
@@ -209,10 +210,8 @@ class AbortDrop extends Error {
}
}
const originalName = (file: EnteFile): string => {
const ext = extname(file.metadata.title || "") || ".bin";
return `${file.id}${ext}`;
};
const originalName = (file: EnteFile): string =>
`${file.id}${safeExtension(file.metadata.title)}`;
// The fileID a cache filename encodes, or undefined when the name is not one
// the cache writes (`<digits><ext>`).
+17 -10
View File
@@ -266,8 +266,9 @@ export class Library {
// that, a fresh read propagates it.
private cycle?: Promise<void>;
// Guards the ML fetch pass so a slow backfill never runs twice at once; a
// refresh whose pass is still running kicks nothing new.
private mlFetching = false;
// refresh whose pass is still running kicks nothing new. Holds the running
// pass, so `close()` can wait for it.
private mlFetch?: Promise<void>;
private closed = false;
private lastRefreshAt?: number;
private lastError?: string;
@@ -560,14 +561,21 @@ export class Library {
}
// Stop the background timer. Idempotent. An in-flight refresh is left to
// finish; it will not schedule another cycle once closed.
close(): void {
// finish; it will not schedule another cycle once closed. The returned
// promise resolves once that refresh (including its cache write), the ML
// fetch pass and the precache fetches already running have all finished,
// so a caller can then remove the cache directory. A refresh failure is
// reported through `status()`, not thrown here.
async close(): Promise<void> {
this.closed = true;
this.precache?.close();
const precacheClosed = this.precache?.close();
if (this.timer !== undefined) {
clearTimeout(this.timer);
this.timer = undefined;
}
await this.cycle?.catch(() => {});
await this.mlFetch;
await precacheClosed;
}
private scheduleNext(): void {
@@ -631,7 +639,9 @@ export class Library {
// outside the refresh's success/failure so a fetch or disk problem
// there never marks the metadata refresh failed, and it is not
// awaited so it never stalls the refresh interval.
void this.runMLFetch();
this.mlFetch ??= this.runMLFetch().finally(() => {
this.mlFetch = undefined;
});
} catch (err) {
const error = err instanceof Error ? err.message : String(err);
this.lastError = error;
@@ -745,13 +755,12 @@ export class Library {
// Bind so the call keeps the client as its receiver when invoked
// through the pool below.
const fetchMLData = this.client.fetchMLData?.bind(this.client);
if (!mldata || !fetchMLData || this.closed || this.mlFetching) return;
if (!mldata || !fetchMLData || this.closed) return;
const files = this.uniqueFiles();
const needed = mldata.neededFor(files);
if (needed.length === 0) return;
this.mlFetching = true;
this.emit({ operation: "fetchMLData", status: "started" });
try {
const fileKeys = new Map<number, Uint8Array>();
@@ -784,8 +793,6 @@ export class Library {
const error = err instanceof Error ? err.message : String(err);
this.lastMLError = error;
this.emit({ operation: "fetchMLData", status: "failed", error });
} finally {
this.mlFetching = false;
}
}
+16 -12
View File
@@ -92,9 +92,10 @@ export class Precache {
private pinned = new Set<number>();
// A sweep runs at most once per fill at a time; a re-kick while one runs is
// a no-op, and the next refresh re-kicks after it finishes.
private thumbRunning = false;
private originalsRunning = false;
// a no-op, and the next refresh re-kicks after it finishes. Each holds the
// running sweep, so `close()` can wait for it.
private thumbSweep?: Promise<void>;
private originalsSweep?: Promise<void>;
private readonly aborter = new AbortController();
private closed = false;
@@ -191,15 +192,19 @@ export class Precache {
}
// Stop the fills. In-flight fetches are left to settle; queued ones drop.
close(): void {
// Resolves once both sweeps have finished, so nothing is still writing.
async close(): Promise<void> {
this.closed = true;
this.aborter.abort();
await Promise.all([
this.thumbSweep?.catch(() => {}),
this.originalsSweep?.catch(() => {}),
]);
}
private kickThumbnails(): void {
if (this.thumbRunning) return;
this.thumbRunning = true;
void this.sweep(
if (this.thumbSweep) return;
this.thumbSweep = this.sweep(
"precacheThumbnails",
() => this.thumbOrder,
(id) => this.cache!.pathsFor(id).thumbnailPath !== undefined,
@@ -211,14 +216,13 @@ export class Precache {
signal: this.aborter.signal,
}),
).finally(() => {
this.thumbRunning = false;
this.thumbSweep = undefined;
});
}
private kickOriginals(): void {
if (this.originalsRunning) return;
this.originalsRunning = true;
void this.sweep(
if (this.originalsSweep) return;
this.originalsSweep = this.sweep(
"precacheOriginals",
() => this.originalsOrder,
(id) => this.cache!.pathsFor(id).originalPath !== undefined,
@@ -229,7 +233,7 @@ export class Precache {
signal: this.aborter.signal,
}),
).finally(() => {
this.originalsRunning = false;
this.originalsSweep = undefined;
});
}
+95 -71
View File
@@ -4,6 +4,7 @@ import * as jpeg from "jpeg-js";
import exifReader from "exif-reader";
import type { Client } from "./client.js";
import type { Library, Photo } from "./library/index.js";
import { sanitizeFileName } from "./filename.js";
import { fetchMLData } from "./mldata-fetch.js";
import type { EnteFile } from "./model/types.js";
@@ -14,88 +15,110 @@ export interface MetadataBackupOptions {
onProgress?: ProgressCallback;
}
const sanitizePath = (name: string): string =>
name.replace(/[/\\:*?"<>|]/g, "_").replace(/^\.+/, "_");
// Extract the raw EXIF APP1 segment from JPEG bytes. Returns the EXIF
// data buffer (starting after the APP1 length field, at the "Exif\0\0"
// header) or undefined if no APP1 marker is found.
const extractExifFromJpeg = (buf: Uint8Array): Buffer | undefined => {
if (buf[0] !== 0xff || buf[1] !== 0xd8) return undefined;
// Find the raw EXIF APP1 segment in JPEG bytes. Returns `exif` (the segment
// data, starting at the "Exif\0\0" header) when there is one, nothing when the
// bytes are not a JPEG or carry no EXIF, and `error` when the segment layout is
// malformed. Each segment length is checked against the bytes that remain and
// each step moves forward by at least 4 bytes, so the scan ends on any input.
export const extractExifFromJpeg = (
buf: Uint8Array,
): { exif?: Buffer; error?: string } => {
if (buf[0] !== 0xff || buf[1] !== 0xd8) return {};
let offset = 2;
while (offset < buf.length - 1) {
if (buf[offset] !== 0xff) return undefined;
while (offset < buf.length) {
if (offset + 2 > buf.length)
return { error: `truncated segment marker at byte ${offset}` };
if (buf[offset] !== 0xff)
return { error: `no segment marker at byte ${offset}` };
const marker = buf[offset + 1]!;
if (marker === 0xda) break; // start of scan, no more markers
if (offset + 3 >= buf.length) break;
if (marker === 0xda) return {}; // start of scan, no more markers
if (offset + 4 > buf.length)
return { error: `truncated segment length at byte ${offset}` };
const len = (buf[offset + 2]! << 8) | buf[offset + 3]!;
// The length counts its own two bytes, so anything under 2 is invalid.
if (len < 2)
return {
error: `segment length ${len} at byte ${offset} is too small`,
};
if (offset + 2 + len > buf.length)
return {
error: `segment length ${len} at byte ${offset} runs past the end of the file`,
};
if (marker === 0xe1) {
// APP1 — check for "Exif\0\0" header
// APP1 — check for "Exif\0\0" header. A length under 8 cannot hold
// the six-byte header, so the segment is not EXIF; below 6 the
// bytes compared would also lie past the segment.
if (
len >= 8 &&
buf[offset + 4] === 0x45 &&
buf[offset + 5] === 0x78 &&
buf[offset + 6] === 0x69 &&
buf[offset + 7] === 0x66
) {
return Buffer.from(
buf.buffer,
buf.byteOffset + offset + 4,
len - 2,
);
return {
exif: Buffer.from(
buf.buffer,
buf.byteOffset + offset + 4,
len - 2,
),
};
}
}
offset += 2 + len;
}
return undefined;
return { error: "file ends before the image data" };
};
const extractImageMetadata = (
// Extract dimensions, EXIF and XMP from a file's bytes. When the EXIF segment
// is malformed or cannot be parsed, the record carries the reason in
// `exifError`.
export const extractImageMetadata = (
fileBytes: Uint8Array,
): Record<string, unknown> | undefined => {
const result: Record<string, unknown> = {};
// Try to get dimensions from JPEG decode
try {
const result: Record<string, unknown> = {};
// Try to get dimensions from JPEG decode
try {
const decoded = jpeg.decode(fileBytes, {
useTArray: true,
formatAsRGBA: false,
});
result.format = "jpeg";
result.width = decoded.width;
result.height = decoded.height;
} catch {
// Not a JPEG or corrupt; still try EXIF extraction
}
const exifBuf = extractExifFromJpeg(fileBytes);
if (exifBuf) {
try {
result.exif = exifReader(exifBuf);
} catch {
result.exifRaw = exifBuf.toString("base64");
}
}
// Extract XMP (look for "http://ns.adobe.com/xap" in the bytes)
const xmpStart = Buffer.from(fileBytes).indexOf("<?xpacket begin");
if (xmpStart !== -1) {
const xmpEnd = Buffer.from(fileBytes).indexOf(
"<?xpacket end",
xmpStart,
);
if (xmpEnd !== -1) {
const end = Buffer.from(fileBytes).indexOf("?>", xmpEnd);
result.xmp = Buffer.from(fileBytes)
.subarray(xmpStart, end !== -1 ? end + 2 : xmpEnd + 50)
.toString("utf-8");
}
}
return Object.keys(result).length > 0 ? result : undefined;
const decoded = jpeg.decode(fileBytes, {
useTArray: true,
formatAsRGBA: false,
});
result.format = "jpeg";
result.width = decoded.width;
result.height = decoded.height;
} catch {
return undefined;
// Not every original is a JPEG (PNG, HEIC, video), so a failed decode
// is expected and only means no dimensions; a malformed JPEG is still
// reported below through `exifError`.
}
const { exif, error } = extractExifFromJpeg(fileBytes);
if (error) result.exifError = error;
if (exif) {
try {
result.exif = exifReader(exif);
} catch (err) {
result.exifRaw = exif.toString("base64");
result.exifError = err instanceof Error ? err.message : String(err);
}
}
// Extract XMP (look for "http://ns.adobe.com/xap" in the bytes)
const xmpStart = Buffer.from(fileBytes).indexOf("<?xpacket begin");
if (xmpStart !== -1) {
const xmpEnd = Buffer.from(fileBytes).indexOf(
"<?xpacket end",
xmpStart,
);
if (xmpEnd !== -1) {
const end = Buffer.from(fileBytes).indexOf("?>", xmpEnd);
result.xmp = Buffer.from(fileBytes)
.subarray(xmpStart, end !== -1 ? end + 2 : xmpEnd + 50)
.toString("utf-8");
}
}
return Object.keys(result).length > 0 ? result : undefined;
};
// Read a file's original bytes through the library's content cache and extract
@@ -105,13 +128,9 @@ const extractImageMetadata = (
const extractExif = async (
photo: Photo,
): Promise<Record<string, unknown> | undefined> => {
try {
const { path } = await photo.original();
const fileBytes = new Uint8Array(readFileSync(path));
return extractImageMetadata(fileBytes);
} catch {
return undefined;
}
const { path } = await photo.original();
const fileBytes = new Uint8Array(readFileSync(path));
return extractImageMetadata(fileBytes);
};
// Dump every decrypted metadata layer the account holds into a directory tree
@@ -151,7 +170,7 @@ export const runMetadataBackup = async (
const col = lib.getCollection(album.collectionID);
if (!col) continue;
const dirName = `${col.id}-${sanitizePath(col.name || "unnamed")}`;
const dirName = `${col.id}-${sanitizeFileName(col.name, "unnamed")}`;
const colDir = join(outDir, "collections", dirName);
mkdirSync(colDir, { recursive: true });
@@ -217,8 +236,13 @@ export const runMetadataBackup = async (
if (wantExif && !writtenFileIDs.has(file.id)) {
log(`[${file.metadata.title}] Extracting EXIF...`);
const exifData = await extractExif(photo);
if (exifData) fileMeta.imageMetadata = exifData;
try {
const exifData = await extractExif(photo);
if (exifData) fileMeta.imageMetadata = exifData;
} catch (err) {
fileMeta.imageMetadataError =
err instanceof Error ? err.message : String(err);
}
}
writtenFileIDs.add(file.id);
+33 -2
View File
@@ -34,6 +34,28 @@ const FILE_TYPE_MAP: Record<number, FileType> = {
const parseFileType = (n: number): FileType => FILE_TYPE_MAP[n] ?? "unknown";
// The hash the uploading client recorded for the original's bytes, read the
// way the upstream client's `metadataHash` reads it: `hash` if present,
// otherwise, for a live photo from an older client that wrote the two parts
// separately, `<imageHash>:<videoHash>`. A field that is not a non-empty
// string counts as absent, and a file with no hash at all is normal.
const expectedHash = (json: Record<string, unknown>): string | undefined => {
const text = (v: unknown): string | undefined =>
typeof v === "string" && v !== "" ? v : undefined;
const hash = text(json.hash);
if (hash !== undefined) return hash;
const imageHash = text(json.imageHash);
const videoHash = text(json.videoHash);
if (
json.fileType === 2 &&
imageHash !== undefined &&
videoHash !== undefined
) {
return `${imageHash}:${videoHash}`;
}
return undefined;
};
export const decryptCollection = (
raw: RawCollection,
keys: KeyMaterial,
@@ -98,15 +120,24 @@ export const decryptFile = (
key,
);
const metadataJSON = JSON.parse(new TextDecoder().decode(metadataBytes));
if (
typeof metadataJSON !== "object" ||
metadataJSON === null ||
Array.isArray(metadataJSON)
) {
throw new Error(`file ${raw.id}: metadata is not a JSON object`);
}
const metadata: FileMetadata = {
title: metadataJSON.title ?? "",
// The server controls this JSON: a title that is missing or not a
// string becomes "", never an arbitrary value.
title: typeof metadataJSON.title === "string" ? metadataJSON.title : "",
fileType: parseFileType(metadataJSON.fileType ?? -1),
creationTime: metadataJSON.creationTime ?? 0,
modificationTime: metadataJSON.modificationTime ?? 0,
latitude: metadataJSON.latitude,
longitude: metadataJSON.longitude,
hash: metadataJSON.hash,
hash: expectedHash(metadataJSON),
};
const magicMetadata = decryptMagicMetadata(raw.magicMetadata, key);
+3
View File
@@ -29,6 +29,9 @@ export interface FileMetadata {
modificationTime: Microseconds;
latitude?: number;
longitude?: number;
// The content hash the uploader recorded (see `expectedHash` in
// decrypt.ts); `downloadFile` refuses an original that does not match it.
// Absent for files from very old clients.
hash?: string;
}
+25 -12
View File
@@ -88,18 +88,21 @@ const MAX_CAUSE_DEPTH = 8;
// errno on the error it throws — it hangs the underlying socket error off
// `cause`, sometimes more than one level down — so a classifier that only read
// the top-level error would see a bare `Error` and call every dropped
// connection permanent.
const causeCodes = (err: unknown): string[] => {
// connection permanent. `complete` is false when the walk stopped at the
// depth limit with more of the chain still below it.
const causeCodes = (err: unknown): { codes: string[]; complete: boolean } => {
const codes: string[] = [];
let current: unknown = err;
for (let depth = 0; depth < MAX_CAUSE_DEPTH; depth++) {
if (current === null || typeof current !== "object") break;
if (current === null || typeof current !== "object") {
return { codes, complete: true };
}
const { code, cause } = current as { code?: unknown; cause?: unknown };
if (typeof code === "string") codes.push(code);
if (cause === current) break;
if (cause === current) return { codes, complete: true };
current = cause;
}
return codes;
return { codes, complete: current === null || typeof current !== "object" };
};
const isAbort = (err: unknown): boolean => {
@@ -145,15 +148,15 @@ export const isRetryable = (err: unknown): boolean => {
// have succeeded; the cost of the imprecision is bounded by the attempt
// count.
if (err instanceof TypeError) return true;
return causeCodes(err).some((code) => TRANSPORT_CODES.has(code));
return causeCodes(err).codes.some((code) => TRANSPORT_CODES.has(code));
};
// Could the first attempt already have taken effect on the server?
//
// `isRetryable` is the wrong question for a request that changes state.
// quak's non-idempotent calls are `/users/srp/create-session`,
// `/users/two-factor/verify` — which consumes one of a small number of 2FA
// attempts — and `/files/thumbnail`. They are replayed only on the failures in
// `postJSON` and `putJSON` use this for every `POST` and `PUT` listed in the
// README under "Endpoints used"; verifying a second factor, for one, consumes
// one of a small number of attempts. They are replayed only on the failures in
// `CONNECT_CODES`, which establish that no TCP connection to the server ever
// existed: there was no address to connect to, or the peer refused the
// connection outright. A request byte cannot have been transmitted, so the
@@ -162,9 +165,19 @@ export const isRetryable = (err: unknown): boolean => {
// Everything else is ambiguous. A 5xx proves the server did process the
// request. A reset or a broken pipe can arrive after it was fully sent and
// acted on. A routing errno can be delivered on an established socket. A
// deadline says nothing at all about the server's state.
export const isSafeToReplay = (err: unknown): boolean =>
isRetryable(err) && causeCodes(err).some((code) => CONNECT_CODES.has(code));
// deadline says nothing at all about the server's state. So every errno in the
// cause chain must be a connect errno: one other errno anywhere in the chain
// is doubt, and doubt is not replayed. A chain longer than the walk is doubt
// too: the links below the limit were never read.
export const isSafeToReplay = (err: unknown): boolean => {
const { codes, complete } = causeCodes(err);
return (
isRetryable(err) &&
complete &&
codes.length > 0 &&
codes.every((code) => CONNECT_CODES.has(code))
);
};
export interface WithRetryOptions extends RetryOptions {
isRetryable?: (err: unknown) => boolean;
+327 -28
View File
@@ -38,14 +38,18 @@
* the network. The fake records every call for assertion.
*/
import { describe, expect, it } from "vitest";
import { describe, expect, it, vi } from "vitest";
import {
ApiClient,
ApiError,
DEFAULT_DOWNLOAD_TIMEOUT_MS,
DEFAULT_REQUEST_TIMEOUT_MS,
} from "../../src/api/client.js";
import type { RetryOptions } from "../../src/retry.js";
import {
isRetryable,
isSafeToReplay,
type RetryOptions,
} from "../../src/retry.js";
// ---------------------------------------------------------------------------
// Test helpers
@@ -385,6 +389,103 @@ describe("ApiClient custom origins", () => {
});
});
describe("ApiClient request URLs", () => {
it("accepts a path with or without a leading slash", async () => {
const { fetch, calls } = recordingFetch(
jsonResponse({}),
jsonResponse({}),
jsonResponse({}),
);
const client = new ApiClient({ fetch });
await client.getJSON("health");
await client.postJSON("users/ott", {});
await client.putJSON("/files/thumbnail", {});
expect(calls.map((c) => c.url)).toEqual([
"https://api.ente.io/health",
"https://api.ente.io/users/ott",
"https://api.ente.io/files/thumbnail",
]);
});
it("accepts an apiOrigin with a trailing slash", async () => {
const { fetch, calls } = recordingFetch(jsonResponse({}));
const client = new ApiClient({
fetch,
apiOrigin: "https://my-ente.example.com/",
});
await client.getJSON("/health");
expect(calls[0]!.url).toBe("https://my-ente.example.com/health");
});
it("keeps a base path in a self-hosted apiOrigin for every request", async () => {
const body = new Uint8Array([1]);
const { fetch, calls } = recordingFetch(
jsonResponse({}),
jsonResponse({}),
jsonResponse({}),
streamResponse(body),
streamResponse(body),
);
const client = new ApiClient({
fetch,
apiOrigin: "https://example.com/ente/",
});
await client.getJSON("/collections/v2", { sinceTime: 0 });
await client.postJSON("/users/ott", {});
await client.putJSON("/files/thumbnail", {});
await client.getFileStream(99);
await client.getThumbnailStream(77);
expect(calls.map((c) => c.url)).toEqual([
"https://example.com/ente/collections/v2?sinceTime=0",
"https://example.com/ente/users/ott",
"https://example.com/ente/files/thumbnail",
"https://example.com/ente/files/download/99",
"https://example.com/ente/files/preview/77",
]);
});
it("percent-encodes query parameters and skips undefined ones", async () => {
const { fetch, calls } = recordingFetch(jsonResponse({}));
const client = new ApiClient({ fetch });
await client.getJSON("/search", {
q: "a&b=c/d é",
limit: 5,
cursor: undefined,
});
const url = new URL(calls[0]!.url);
expect(url.pathname).toBe("/search");
expect(url.search).toBe("?q=a%26b%3Dc%2Fd+%C3%A9&limit=5");
expect(url.searchParams.get("q")).toBe("a&b=c/d é");
});
it("rejects a path that carries its own query string", async () => {
const { fetch, calls } = recordingFetch();
const client = new ApiClient({ fetch });
await expect(client.getJSON("/diff?sinceTime=0")).rejects.toThrow(
/must not contain "\?"/,
);
await expect(client.postJSON("/users/ott?x=1", {})).rejects.toThrow(
/must not contain "\?"/,
);
expect(calls).toHaveLength(0);
});
it("rejects a path that carries a fragment", async () => {
const { fetch, calls } = recordingFetch();
const client = new ApiClient({ fetch });
await expect(client.getJSON("/diff#top")).rejects.toThrow(
/must not contain "\?" or "#"/,
);
expect(calls).toHaveLength(0);
});
});
describe("ApiError", () => {
it("throws ApiError on 4xx with status, code, requestID", async () => {
const { fetch } = recordingFetch(
@@ -639,17 +740,33 @@ describe("ApiClient retries", () => {
expect(policy.baseDelayMs).toBe(7);
expect(policy.maxDelayMs).toBe(11);
});
it("does not let a caller change its settings through that policy", async () => {
const { fetch, calls } = scriptedFetch(
textResponse("boom", 500),
textResponse("boom", 500),
textResponse("boom", 500),
);
const client = new ApiClient({
fetch,
retry: { ...noWait, attempts: 2 },
});
client.getRetryOptions().attempts = 3;
expect(client.getRetryOptions().attempts).toBe(2);
await expect(client.getJSON("/x")).rejects.toBeInstanceOf(ApiError);
expect(calls).toHaveLength(2);
});
});
describe("ApiClient timeouts", () => {
it("ships bounded default deadlines", () => {
// Asserted here so the README and the code cannot drift. Two numbers
// rather than one, because a deadline that is sane for a JSON call is
// nowhere near enough for a multi-gigabyte body, and a deadline long
// enough for that body would let a hung API call stall a backup for
// ten minutes.
// Asserted here so the README and the code cannot drift. The request
// deadline bounds a whole JSON call; the download deadline is an idle
// one, measured from the last byte that arrived.
expect(DEFAULT_REQUEST_TIMEOUT_MS).toBe(30_000);
expect(DEFAULT_DOWNLOAD_TIMEOUT_MS).toBe(600_000);
expect(DEFAULT_DOWNLOAD_TIMEOUT_MS).toBe(60_000);
});
it("attaches an abort signal to every request", async () => {
@@ -693,6 +810,51 @@ describe("ApiClient timeouts", () => {
expect(new Set(signals).size).toBe(3);
}, 5000);
it("gives every retrying entry point a fresh deadline per attempt", async () => {
// A refused connection is retried by every entry point, the
// non-idempotent ones included. If the deadline were created once,
// outside the retry, both attempts would carry the same signal.
const entryPoints: [
string,
() => Response,
(c: ApiClient) => unknown,
][] = [
["getJSON", () => jsonResponse({}), (c) => c.getJSON("/a")],
["postJSON", () => jsonResponse({}), (c) => c.postJSON("/b", {})],
["putJSON", () => jsonResponse({}), (c) => c.putJSON("/c", {})],
[
"putFile",
() => new Response(null, { status: 200 }),
(c) => c.putFile("https://s3.example/x", new Uint8Array([1])),
],
[
"getFileStream",
() => streamResponse(new Uint8Array([1])),
(c) => c.getFileStream(1),
],
[
"getThumbnailStream",
() => streamResponse(new Uint8Array([1])),
(c) => c.getThumbnailStream(1),
],
];
for (const [name, success, call] of entryPoints) {
const { fetch, calls } = scriptedFetch(
errnoError("ECONNREFUSED", "connect ECONNREFUSED"),
success(),
);
const client = new ApiClient({ fetch, retry: noWait });
await call(client);
expect(calls, name).toHaveLength(2);
const [first, second] = calls.map((c) => c.init?.signal);
expect(first, name).toBeInstanceOf(AbortSignal);
expect(second, name).toBeInstanceOf(AbortSignal);
expect(second, name).not.toBe(first);
}
});
it("recovers when a later attempt answers in time", async () => {
const { fetch, calls } = scriptedFetch(HANG, jsonResponse({ ok: 1 }));
const client = new ApiClient({
@@ -718,25 +880,91 @@ describe("ApiClient timeouts", () => {
// that never produces a chunk and never observes the signal, so the
// only thing that can unblock the read is quak's own enforcement of
// the deadline over the stream it hands out.
const stalling = new Response(
new ReadableStream<Uint8Array>({
pull: () => new Promise<void>(() => {}),
}),
{ status: 200 },
);
const { fetch } = scriptedFetch(stalling);
const client = new ApiClient({
fetch,
downloadTimeoutMs: 20,
retry: { ...noWait, attempts: 1 },
});
//
// The clock is faked, so the test runs under the real default
// deadline and waits for nothing.
vi.useFakeTimers();
try {
const stalling = new Response(
new ReadableStream<Uint8Array>({
pull: () => new Promise<void>(() => {}),
}),
{ status: 200 },
);
const { fetch } = scriptedFetch(stalling);
const client = new ApiClient({
fetch,
retry: { ...noWait, attempts: 1 },
});
const stream = await client.getFileStream(42);
const err: unknown = await readAll(stream).catch((e: unknown) => e);
const stream = await client.getFileStream(42);
let settled = false;
const result = readAll(stream).then(
(n) => n,
(e: unknown) => e,
);
void result.finally(() => {
settled = true;
});
expect(err).toBeInstanceOf(Error);
expect((err as Error).name).toBe("TimeoutError");
}, 5000);
await vi.advanceTimersByTimeAsync(DEFAULT_DOWNLOAD_TIMEOUT_MS - 1);
expect(settled).toBe(false);
await vi.advanceTimersByTimeAsync(1);
const err = await result;
expect(err).toBeInstanceOf(Error);
expect((err as Error).name).toBe("TimeoutError");
// Classified as every deadline is: retried by the idempotent
// downloads, never replayed for a POST or PUT.
expect(isRetryable(err)).toBe(true);
expect(isSafeToReplay(err)).toBe(false);
} finally {
vi.useRealTimers();
}
});
it("does not abort a slow body that keeps making progress", async () => {
// A deadline over the whole transfer would cut off a large video on a
// slow link however steadily it was arriving. The deadline restarts
// with every chunk, so a body that sends one byte every 600 ms for
// well over the 1000 ms deadline completes.
vi.useFakeTimers();
try {
let sent = 0;
const trickling = new Response(
new ReadableStream<Uint8Array>({
async pull(controller) {
await new Promise((resolve) =>
setTimeout(resolve, 600),
);
if (sent === 10) {
controller.close();
return;
}
controller.enqueue(new Uint8Array([sent++]));
},
}),
{ status: 200 },
);
const { fetch } = scriptedFetch(trickling);
const client = new ApiClient({
fetch,
downloadTimeoutMs: 1000,
retry: { ...noWait, attempts: 1 },
});
const stream = await client.getFileStream(42);
const result = readAll(stream).then(
(n) => n,
(e: unknown) => e,
);
await vi.advanceTimersByTimeAsync(11 * 600);
expect(await result).toBe(10);
} finally {
vi.useRealTimers();
}
});
it("lets a body that arrives in time through untouched", async () => {
// The counterpart to the previous test: enforcing the deadline over
@@ -762,6 +990,52 @@ describe("ApiClient timeouts", () => {
}
expect(joined).toEqual(payload);
});
it("leaves no timer pending after a download completes or fails", async () => {
vi.useFakeTimers();
try {
const { fetch } = scriptedFetch(
streamResponse(new Uint8Array([1, 2, 3])),
textResponse("gone", 404),
);
const client = new ApiClient({
fetch,
retry: { ...noWait, attempts: 1 },
});
expect(await readAll(await client.getFileStream(1))).toBe(3);
expect(vi.getTimerCount()).toBe(0);
await expect(client.getFileStream(2)).rejects.toBeInstanceOf(
ApiError,
);
expect(vi.getTimerCount()).toBe(0);
} finally {
vi.useRealTimers();
}
});
it("never lets the download timer keep the process alive", async () => {
const spy = vi.spyOn(globalThis, "setTimeout");
try {
const { fetch } = scriptedFetch(
streamResponse(new Uint8Array([1])),
);
const client = new ApiClient({
fetch,
downloadTimeoutMs: 12_345,
retry: noWait,
});
const stream = await client.getFileStream(1);
const i = spy.mock.calls.findIndex((call) => call[1] === 12_345);
const timer = spy.mock.results[i]!.value as NodeJS.Timeout;
expect(timer.hasRef()).toBe(false);
await stream.cancel();
} finally {
spy.mockRestore();
}
});
});
describe("ApiClient error typing", () => {
@@ -820,9 +1094,8 @@ describe("ApiClient error typing", () => {
describe("ApiClient non-idempotent requests", () => {
/**
* `postJSON` and `putJSON` carry quak's only requests that change server
* state: `/users/srp/create-session`, `/users/two-factor/verify` — which
* consumes one of a small number of 2FA attempts — and `/files/thumbnail`.
* `postJSON` and `putJSON` carry quak's requests that can change server
* state; the README lists them under "Endpoints used".
*
* They are retried only on a failure that establishes no TCP connection to
* the server ever existed — DNS produced no address, or the peer refused
@@ -922,4 +1195,30 @@ describe("ApiClient non-idempotent requests", () => {
await refusedClient.updateThumbnail(1, "key", "header");
expect(refused.calls).toHaveLength(2);
});
it("does not follow or replay a redirect on POST or PUT", async () => {
// The origin has already received a request it answers with a
// redirect, so following it would let a refused connection to the
// redirect target pass for a request that never went out.
for (const send of [
(c: ApiClient) => c.postJSON("/users/ott", {}),
(c: ApiClient) => c.putJSON("/files/thumbnail", {}),
]) {
const { fetch, calls } = scriptedFetch(
new Response(null, {
status: 307,
headers: { location: "https://elsewhere.example/" },
}),
jsonResponse({}),
);
const client = new ApiClient({ fetch, retry: noWait });
const err: unknown = await send(client).catch((e: unknown) => e);
expect(calls[0]?.init?.redirect).toBe("manual");
expect(err).toBeInstanceOf(ApiError);
expect((err as ApiError).status).toBe(307);
expect(calls).toHaveLength(1);
}
});
});
+145 -3
View File
@@ -36,20 +36,50 @@ import {
lstatSync,
mkdirSync,
mkdtempSync,
readdirSync,
readFileSync,
readlinkSync,
rmSync,
writeFileSync,
} from "node:fs";
import { spawnSync } from "node:child_process";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import { Library } from "../../src/library/index.js";
import type { ContentSource } from "../../src/library/content.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
// `open` and `rename` are wrapped to record, in order, every fsync and rename,
// so a test can pin the sequence "fsync the temp file, rename, fsync the
// directory" that makes a copied original survive a power cut. `vi.hoisted`
// because `vi.mock` factories run before module-level constants exist.
const fsEvents = vi.hoisted(() => [] as string[]);
vi.mock("node:fs/promises", async (importOriginal) => {
const actual = await importOriginal<typeof import("node:fs/promises")>();
return {
...actual,
open: async (
...args: Parameters<typeof actual.open>
): Promise<Awaited<ReturnType<typeof actual.open>>> => {
const handle = await actual.open(...args);
const realSync = handle.sync.bind(handle);
handle.sync = async (): Promise<void> => {
fsEvents.push(`sync:${String(args[0])}`);
await realSync();
};
return handle;
},
rename: async (from: string, to: string): Promise<void> => {
fsEvents.push(`rename:${to}`);
await actual.rename(from, to);
},
};
});
const USER_ID = 42;
// Decrypted-byte length each stub original writes, keyed by fileID.
@@ -107,6 +137,29 @@ class MockClient {
}
}
// A server that names an album and a file so as to climb out of the backup
// directory.
class HostileClient extends MockClient {
override async collectionsSince(): Promise<CollectionsPage> {
const page = await super.collectionsSince();
return {
...page,
collections: page.collections.length
? [collection(3, "../escape")]
: [],
};
}
override async filesSince(args: {
collectionID: number;
}): Promise<FilesPage> {
const files =
args.collectionID === 3
? [file(300, 3, "../../.ssh/authorized_keys")]
: [];
return { files, deleted: [], cursor: 1 };
}
}
// A content source that writes byte buffers of the expected length and can be
// told to fail one fileID's original, to exercise per-file resilience.
interface StubSource extends ContentSource {
@@ -136,9 +189,12 @@ const stubSource = (): StubSource => {
let root: string;
const openLibrary = (source: ContentSource): Promise<Library> =>
const openLibrary = (
source: ContentSource,
client: MockClient = new MockClient(),
): Promise<Library> =>
Library.open({
client: new MockClient(),
client,
cacheDirectory: join(root, "cache"),
contentSource: source,
refreshIntervalSeconds: 3600,
@@ -243,6 +299,30 @@ describe("lib.backup", () => {
lib.close();
});
it("keeps server-supplied album and file names inside the backup", async () => {
const lib = await openLibrary(stubSource(), new HostileClient());
const outDir = join(root, "backup");
const result = await lib.backup({ downloadDirectory: outDir });
expect(result.failed).toBe(0);
// The title has no usable extension, so the original is `.bin`.
expect(existsSync(join(outDir, "originals", "300.bin"))).toBe(true);
const link = join(
outDir,
"collections",
"__escape",
"__.._.ssh_authorized_keys",
);
expect(lstatSync(link).isSymbolicLink()).toBe(true);
expect(existsSync(join(outDir, "collections", "__escape.json"))).toBe(
true,
);
// Nothing landed beside or above the backup directory.
expect(readdirSync(root).sort()).toEqual(["backup", "cache"]);
lib.close();
});
it("is an idempotent no-op when every original is already present", async () => {
const source = stubSource();
const lib = await openLibrary(source);
@@ -479,4 +559,66 @@ describe("lib.backup", () => {
expect(readLedger(outDir).files["101"]!.attempts).toBe(1);
lib.close();
});
it("fsyncs a copied original before the rename and its directory after", async () => {
const lib = await openLibrary(stubSource());
const outDir = join(root, "backup");
const originals = join(outDir, "originals");
const dest = join(originals, "100.jpg");
fsEvents.length = 0;
await lib.backup({ downloadDirectory: outDir });
const at = fsEvents.indexOf(`rename:${dest}`);
expect(at).toBeGreaterThan(0);
expect(fsEvents[at - 1]).toMatch(
/^sync:.*\/\.quak-backup-100\.jpg-\d+-[0-9a-z]*\.tmp$/,
);
expect(fsEvents[at + 1]).toBe(`sync:${originals}`);
lib.close();
});
it("removes temp files left by a killed backup but not those of one still running", async () => {
const outDir = join(root, "backup");
const originals = join(outDir, "originals");
mkdirSync(originals, { recursive: true });
// A child that has already exited: its process ID is not running.
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
const leftover = `.quak-backup-100.jpg-${exitedPID}-abc123.tmp`;
// This test's own process stands in for a backup running at the same
// time.
const inProgress = `.quak-backup-101.jpg-${process.pid}-def456.tmp`;
writeFileSync(join(originals, leftover), "partial");
writeFileSync(join(originals, inProgress), "partial");
const lib = await openLibrary(stubSource());
await lib.backup({ downloadDirectory: outDir });
const names = readdirSync(originals);
expect(names).not.toContain(leftover);
expect(names).toContain(inProgress);
lib.close();
});
it("removes leftover temp files in thumbnails/ but not those of a backup still running", async () => {
const outDir = join(root, "backup");
const thumbnails = join(outDir, "thumbnails");
mkdirSync(thumbnails, { recursive: true });
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
const leftover = `.quak-backup-100.jpg-${exitedPID}-abc123.tmp`;
const inProgress = `.quak-backup-101.jpg-${process.pid}-def456.tmp`;
writeFileSync(join(thumbnails, leftover), "partial");
writeFileSync(join(thumbnails, inProgress), "partial");
const lib = await openLibrary(stubSource());
await lib.backup({
downloadDirectory: outDir,
includeThumbnails: true,
});
const names = readdirSync(thumbnails);
expect(names).not.toContain(leftover);
expect(names).toContain(inProgress);
lib.close();
});
});
+397
View File
@@ -0,0 +1,397 @@
/**
* Tests for the CLI commands (`src/cli-commands.ts`, issue #12).
*
* Each command is called directly with a context whose output streams collect
* text, whose session directory is a fresh temp directory, and whose session
* loader hands back a fake client. The fake serves two albums and three files
* from memory, writes stand-in bytes for originals and thumbnails, and makes no
* network calls. The helpers the commands call (`cli-read`, `cli-output`,
* backup, thumbnails) have their own tests; these check what each command
* prints and the exit code it returns.
*/
import {
existsSync,
mkdtempSync,
readFileSync,
rmSync,
statSync,
writeFileSync,
} from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { describe, it, expect, beforeAll, beforeEach, afterEach } from "vitest";
import {
type CliContext,
saveSession,
whoamiCommand,
logoutCommand,
collectionsCommand,
filesCommand,
getCommand,
getThumbCommand,
backupCommand,
listMissingThumbnailsCommand,
} from "../../src/cli-commands.js";
import { loadSession } from "../../src/cli-session.js";
import type { Client, ClientSnapshot } from "../../src/client.js";
import type { ContentSource } from "../../src/library/content.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
import { init } from "../../src/crypto/index.js";
const USER_ID = 42;
const collection = (
id: number,
name: string,
isShared = false,
): Collection => ({
id,
ownerID: USER_ID,
key: new Uint8Array([id]),
name,
type: "album",
updationTime: 1,
isShared,
});
const file = (id: number, collectionID: number, title: string): EnteFile => ({
id,
collectionID,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
metadata: {
title,
fileType: "image",
creationTime: 1000,
modificationTime: 1000,
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime: 1,
});
const COLLECTIONS = [collection(1, "Vacation"), collection(2, "Work", true)];
const FILES: Record<number, EnteFile[]> = {
1: [file(100, 1, "beach.jpg"), file(101, 1, "sunset.jpg")],
2: [file(200, 2, "diagram.png")],
};
// An original is 7 bytes and a thumbnail 3. `failID` makes that file's
// original fail; `emptyThumbID` makes the server report that file's
// thumbnail as empty.
const fakeClient = (opts: { failID?: number; emptyThumbID?: number } = {}) => {
const source: ContentSource = {
original: async ({ file: f, destination }) => {
if (f.id === opts.failID) throw new Error("HTTP 500 from server");
writeFileSync(destination, Buffer.alloc(7, f.id & 0xff));
return { bytesWritten: 7 };
},
thumbnail: async ({ file: f, destination }) => {
writeFileSync(destination, Buffer.alloc(3, f.id & 0xff));
return { bytesWritten: 3 };
},
};
const fake = {
whoami: () => ({ email: "cli@example.com", userID: USER_ID }),
collectionsSince: async () => ({
collections: COLLECTIONS,
deleted: [],
cursor: 1,
}),
filesSince: async (args: { collectionID: number }) => ({
files: FILES[args.collectionID] ?? [],
deleted: [],
cursor: 1,
}),
contentSource: () => source,
getApiClient: () => ({
getThumbnailStream: async (fileID: number) =>
new ReadableStream<Uint8Array>({
start(controller) {
if (fileID !== opts.emptyThumbID) {
controller.enqueue(new Uint8Array(3));
}
controller.close();
},
}),
}),
};
// The commands only call the methods above.
return fake as unknown as Client;
};
// Collects everything written to it.
class Output {
text = "";
write(text: string): void {
this.text += text;
}
}
let root: string;
let stdout: Output;
let stderr: Output;
const context = (client: Client | null = fakeClient()): CliContext => ({
stdout,
stderr,
sessionDir: join(root, "session"),
cacheDir: join(root, "cache"),
loadSession: () => client,
});
beforeAll(async () => {
await init();
});
beforeEach(() => {
root = mkdtempSync(join(tmpdir(), "quak-cli-test-"));
stdout = new Output();
stderr = new Output();
});
afterEach(() => {
rmSync(root, { recursive: true, force: true });
});
describe("session file", () => {
const snapshot: ClientSnapshot = {
email: "cli@example.com",
userID: USER_ID,
token: "token",
masterKey: "a",
secretKey: "b",
publicKey: "c",
};
it("is written with mode 0600 in a directory with mode 0700", () => {
const dir = join(root, "new", "session");
saveSession(dir, snapshot);
expect(statSync(dir).mode & 0o777).toBe(0o700);
const path = join(dir, "session.json");
expect(statSync(path).mode & 0o777).toBe(0o600);
expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot);
});
it("is removed by logout", async () => {
const ctx = context();
saveSession(ctx.sessionDir, snapshot);
expect(await logoutCommand(ctx)).toBe(0);
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
expect(stderr.text).toBe("Session deleted.\n");
});
it("logout without a session says so and exits 0", async () => {
expect(await logoutCommand(context())).toBe(0);
expect(stderr.text).toBe("No session found.\n");
});
it("a missing session exits 1 with 'Not logged in'", async () => {
const ctx = { ...context(), loadSession };
expect(await whoamiCommand(ctx)).toBe(1);
expect(stderr.text).toBe(
`Not logged in. Run "quak login" first.\n` +
`Session file: ${join(ctx.sessionDir, "session.json")}\n`,
);
expect(stdout.text).toBe("");
});
it("a corrupt session exits 1 and says it is corrupt", async () => {
const ctx = { ...context(), loadSession };
saveSession(ctx.sessionDir, snapshot);
expect(await collectionsCommand(ctx, {})).toBe(1);
expect(stderr.text).toContain("is corrupt");
expect(stderr.text).toContain(
`Run "quak logout" and then "quak login" to replace it.\n`,
);
expect(stdout.text).toBe("");
});
});
describe("whoami", () => {
it("prints the account as one line of JSON", async () => {
expect(await whoamiCommand(context())).toBe(0);
expect(stdout.text).toBe(
`{"email":"cli@example.com","userID":${USER_ID}}\n`,
);
});
});
describe("collections", () => {
it("prints one tab-separated line per album", async () => {
expect(await collectionsCommand(context(), {})).toBe(0);
expect(stdout.text).toBe(
"1\talbum\tVacation\n" + "2\talbum\tWork (shared)\n",
);
});
it("prints a JSON array with --json", async () => {
expect(await collectionsCommand(context(), { json: true })).toBe(0);
expect(JSON.parse(stdout.text)).toEqual([
{
id: 1,
name: "Vacation",
type: "album",
ownerID: USER_ID,
isShared: false,
updationTime: 1,
},
{
id: 2,
name: "Work",
type: "album",
ownerID: USER_ID,
isShared: true,
updationTime: 1,
},
]);
});
});
describe("files", () => {
it("prints one tab-separated line per file", async () => {
expect(await filesCommand(context(), { collection: "1" })).toBe(0);
expect(stdout.text).toBe(
"100\timage\tbeach.jpg\n" + "101\timage\tsunset.jpg\n",
);
});
it("prints a JSON array with --json", async () => {
const code = await filesCommand(context(), {
collection: "2",
json: true,
});
expect(code).toBe(0);
expect(JSON.parse(stdout.text)).toEqual([
{
id: 200,
title: "diagram.png",
fileType: "image",
creationTime: 1000,
collectionID: 2,
},
]);
});
it("exits 1 for an unknown collection", async () => {
expect(await filesCommand(context(), { collection: "9" })).toBe(1);
expect(stderr.text).toBe("Collection 9 not found\n");
});
it("exits 1 for a collection ID that is not a number", async () => {
expect(await filesCommand(context(), { collection: "abc" })).toBe(1);
expect(stderr.text).toBe("Invalid collection ID\n");
});
});
describe("get and get-thumb", () => {
it("get finds a file in any album without --collection", async () => {
const out = join(root, "diagram.png");
expect(await getCommand(context(), "200", { out })).toBe(0);
expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
});
it("get-thumb finds a file in any album without --collection", async () => {
const out = join(root, "thumb.jpg");
expect(await getThumbCommand(context(), "200", { out })).toBe(0);
expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
expect(stderr.text).toBe(`3 bytes -> ${out}\n`);
});
it("get exits 1 when no album has the file", async () => {
const out = join(root, "x");
expect(await getCommand(context(), "999", { out })).toBe(1);
expect(stderr.text).toBe("File 999 not found\n");
expect(existsSync(out)).toBe(false);
});
it("get-thumb exits 1 when no album has the file", async () => {
const out = join(root, "x");
expect(await getThumbCommand(context(), "999", { out })).toBe(1);
expect(stderr.text).toBe("File 999 not found\n");
expect(existsSync(out)).toBe(false);
});
it("both exit 1 for a file ID that is not a number", async () => {
expect(await getCommand(context(), "abc", {})).toBe(1);
expect(await getThumbCommand(context(), "abc", {})).toBe(1);
expect(stderr.text).toBe("Invalid file ID\nInvalid file ID\n");
});
});
describe("backup", () => {
it("exits 0 and prints a summary when every file is saved", async () => {
const dir = join(root, "backup");
expect(await backupCommand(context(), dir, {})).toBe(0);
expect(stderr.text).toContain(
"\n--- Backup complete ---\n" +
" Total files: 3\n" +
" Downloaded: 3\n" +
" Skipped: 0\n" +
" Failed: 0\n",
);
expect(stdout.text).toBe("");
});
it("exits 1 and lists the file when one download fails", async () => {
const ctx = context(fakeClient({ failID: 101 }));
expect(await backupCommand(ctx, join(root, "backup"), {})).toBe(1);
expect(stderr.text).toContain(" Failed: 1\n");
expect(stderr.text).toContain(
"\nFailed files:\n" +
" [Vacation] sunset.jpg (id 101): HTTP 500 from server\n",
);
});
it("prints the result as JSON with --json, still exiting 1 on a failure", async () => {
const ctx = context(fakeClient({ failID: 101 }));
const code = await backupCommand(ctx, join(root, "backup"), {
json: true,
});
expect(code).toBe(1);
const result = JSON.parse(stdout.text);
expect(result).toMatchObject({
totalFiles: 3,
downloaded: 2,
skipped: 0,
failed: 1,
});
expect(result.errors[0].fileID).toBe(101);
expect(stderr.text).toBe("Starting backup...\n");
});
});
describe("helper list-missing-thumbnails", () => {
it("prints one line per file with an empty thumbnail", async () => {
const ctx = context(fakeClient({ emptyThumbID: 200 }));
expect(await listMissingThumbnailsCommand(ctx, {})).toBe(0);
expect(stdout.text).toBe(
"200\tdiagram.png\tWork\tempty thumbnail (0 bytes)\n",
);
expect(stderr.text).toContain("\n1 file(s) with missing thumbnails:\n");
});
it("says so when nothing is missing", async () => {
expect(await listMissingThumbnailsCommand(context(), {})).toBe(0);
expect(stdout.text).toBe("");
expect(stderr.text).toContain("No missing thumbnails found.\n");
});
it("prints a JSON array with --json and no progress", async () => {
const ctx = context(fakeClient({ emptyThumbID: 200 }));
expect(await listMissingThumbnailsCommand(ctx, { json: true })).toBe(0);
expect(JSON.parse(stdout.text)).toEqual([
{
fileID: 200,
title: "diagram.png",
collection: "Work",
reason: "empty thumbnail (0 bytes)",
},
]);
expect(stderr.text).toBe("");
});
});
+18 -3
View File
@@ -165,14 +165,15 @@ const buildMetaMock = async (): Promise<MetaMockState> => {
},
};
// Collection 2: "Work" with no magic metadata
// Collection 2: "../Work" with no magic metadata. The server chose a name
// that tries to climb out of the backup directory.
const ck2 = sodium.crypto_secretbox_keygen();
const { ciphertext: encCK2, nonce: ck2N } = encryptSecretbox(
ck2,
masterKey,
);
const { ciphertext: encCN2, nonce: cn2N } = encryptSecretbox(
new TextEncoder().encode("Work"),
new TextEncoder().encode("../Work"),
ck2,
);
const rawColl2 = {
@@ -496,7 +497,8 @@ describe("quak backup-metadata", () => {
await runBackup(outDir);
const collDirs = readdirSync(join(outDir, "collections"));
expect(collDirs.length).toBe(2);
// "../Work" is sanitized into one directory name.
expect(collDirs.sort()).toEqual(["10-Vacation", "20-__Work"]);
// Find the Vacation collection dir (prefixed with ID)
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
@@ -619,5 +621,18 @@ describe("quak backup-metadata", () => {
expect(fileMeta.imageMetadata.format).toBe("jpeg");
expect(fileMeta.imageMetadata.width).toBe(100);
expect(fileMeta.imageMetadata.height).toBe(80);
expect(fileMeta.imageMetadataError).toBeUndefined();
// File 200 has no original on the mock server, so extraction fails
// and the reason is recorded instead of the field being left out.
const workDir = collDirs.find((d) => d.includes("Work"))!;
const failedMeta = JSON.parse(
readFileSync(
join(outDir, "collections", workDir, "200.json"),
"utf-8",
),
);
expect(failedMeta.imageMetadata).toBeUndefined();
expect(failedMeta.imageMetadataError).toEqual(expect.any(String));
});
});
+136
View File
@@ -0,0 +1,136 @@
/**
* Tests for the JPEG EXIF scan behind `quak backup-metadata --exif`.
*
* The originals come from users' libraries, so a truncated or corrupt JPEG
* must neither hang the scan nor throw out of it, and a malformed file must be
* told apart from one that simply has no EXIF: the record carries the reason in
* `exifError`. Each input below is a short hand-built byte array.
*/
import { describe, expect, it } from "vitest";
import {
extractExifFromJpeg,
extractImageMetadata,
} from "../../src/metadata-backup.js";
const SOI = [0xff, 0xd8]; // start of image
const SOS = [0xff, 0xda, 0x00, 0x02]; // start of scan, where the scan stops
const EXIF_HEADER = [0x45, 0x78, 0x69, 0x66, 0x00, 0x00]; // "Exif\0\0"
// A big-endian TIFF block with one IFD entry: Orientation (0x0112), SHORT, 6.
const TIFF_ORIENTATION_6 = [
0x4d, 0x4d, 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08, 0x00, 0x01, 0x01, 0x12,
0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x06, 0x00, 0x00, 0x00, 0x00,
0x00, 0x00,
];
// An APP1 segment whose length field matches its data.
const app1 = (data: number[]): number[] => {
const len = data.length + 2;
return [0xff, 0xe1, len >> 8, len & 0xff, ...data];
};
const bytes = (...parts: number[][]): Uint8Array =>
new Uint8Array(parts.flat());
describe("extractExifFromJpeg", () => {
it("returns the EXIF segment of a valid JPEG", () => {
const data = [...EXIF_HEADER, ...TIFF_ORIENTATION_6];
const scan = extractExifFromJpeg(bytes(SOI, app1(data), SOS));
expect(scan.error).toBeUndefined();
expect([...scan.exif!]).toEqual(data);
});
it("returns nothing for a file that is not a JPEG", () => {
const png = bytes([0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a]);
expect(extractExifFromJpeg(png)).toEqual({});
});
it("returns nothing for a JPEG without EXIF", () => {
const app0 = [0xff, 0xe0, 0x00, 0x04, 0x00, 0x00];
expect(extractExifFromJpeg(bytes(SOI, app0, SOS))).toEqual({});
});
it("ignores an APP1 segment too short to hold the Exif header", () => {
// A length under 8 cannot hold the six-byte "Exif\0\0" header, so the
// segment is not EXIF. This one has length 7 and holds only "Exif\0",
// which the old code, lacking the length check, returned as EXIF.
const short = app1(EXIF_HEADER.slice(0, 5));
expect(extractExifFromJpeg(bytes(SOI, short, SOS))).toEqual({});
});
it("accepts an APP1 segment of length 8 holding just the Exif header", () => {
const scan = extractExifFromJpeg(bytes(SOI, app1(EXIF_HEADER), SOS));
expect(scan.error).toBeUndefined();
expect([...scan.exif!]).toEqual(EXIF_HEADER);
});
it("reports a JPEG truncated inside a segment header", () => {
const scan = extractExifFromJpeg(bytes(SOI, [0xff, 0xe1, 0x00]));
expect(scan.exif).toBeUndefined();
expect(scan.error).toMatch(/truncated segment length/);
});
it("reports a JPEG that ends before the image data", () => {
const app0 = [0xff, 0xe0, 0x00, 0x04, 0x00, 0x00];
const scan = extractExifFromJpeg(bytes(SOI, app0));
expect(scan.error).toMatch(/ends before the image data/);
});
it("stops on a zero-length segment instead of looping", () => {
// A length of 0 would otherwise step the scan by 2 bytes at a time
// through the rest of the file, reading garbage as markers.
const zero = [0xff, 0xe0, 0x00, 0x00];
const scan = extractExifFromJpeg(
bytes(SOI, zero, zero, zero, zero, SOS),
);
expect(scan.error).toMatch(/segment length 0 at byte 2 is too small/);
});
it("stops on a segment length of 1", () => {
const scan = extractExifFromJpeg(
bytes(SOI, [0xff, 0xe0, 0x00, 0x01], SOS),
);
expect(scan.error).toMatch(/segment length 1 at byte 2 is too small/);
});
it("reports a segment length that runs past the end of the file", () => {
// APP1 claims 0x4000 bytes but only the "Exif\0\0" header follows.
const scan = extractExifFromJpeg(
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
);
expect(scan.exif).toBeUndefined();
expect(scan.error).toMatch(/runs past the end of the file/);
});
});
describe("extractImageMetadata", () => {
it("parses EXIF from a valid JPEG", () => {
const meta = extractImageMetadata(
bytes(SOI, app1([...EXIF_HEADER, ...TIFF_ORIENTATION_6]), SOS),
);
expect(meta?.exifError).toBeUndefined();
expect(meta?.exif).toMatchObject({ Image: { Orientation: 6 } });
});
it("returns nothing for a file that is not a JPEG", () => {
const text = new TextEncoder().encode("just some text, not an image");
expect(extractImageMetadata(text)).toBeUndefined();
});
it("records the reason when the JPEG is malformed", () => {
const meta = extractImageMetadata(
bytes(SOI, [0xff, 0xe1, 0x40, 0x00], EXIF_HEADER),
);
expect(meta?.exif).toBeUndefined();
expect(meta?.exifError).toMatch(/runs past the end of the file/);
});
it("keeps the raw bytes and the reason when EXIF cannot be parsed", () => {
const data = [...EXIF_HEADER, 0x58, 0x58];
const meta = extractImageMetadata(bytes(SOI, app1(data), SOS));
expect(meta?.exif).toBeUndefined();
expect(meta?.exifRaw).toBe(Buffer.from(data).toString("base64"));
expect(meta?.exifError).toEqual(expect.any(String));
});
});
+18
View File
@@ -64,6 +64,24 @@ describe("CLI file output (issue #52)", () => {
expect(thumbnailName(renamedFile)).toBe(`thumb_${RAW_TITLE}`);
});
it("sanitizes the title when naming `quak get` downloads", () => {
// Without `--out`, the server-supplied title names the file, so it must
// not be able to point outside the working directory.
const hostile = {
...renamedFile,
metadata: { ...renamedFile.metadata, title: "../../.bashrc" },
};
expect(originalName(hostile)).toBe("__.._.bashrc");
expect(thumbnailName(hostile)).toBe("thumb___.._.bashrc");
const untitled = {
...renamedFile,
metadata: { ...renamedFile.metadata, title: "" },
};
expect(originalName(untitled)).toBe("file-100");
expect(thumbnailName(untitled)).toBe("thumb_file-100");
});
it("does not use the editedName/editedTime projection", () => {
const record = deriveRecords([], [renamedFile]).photos.get(100);
// The projection prefers the edits and reports milliseconds; the CLI
+197
View File
@@ -0,0 +1,197 @@
/**
* Tests for the client session lifecycle: `toJSON`, `fromJSON`, `logout`, and
* the CLI's `loadSession`, which reads the saved session file back into a
* client.
*/
import { mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import sodium from "libsodium-wrappers-sumo";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import { init, toBase64 } from "../../src/crypto/index.js";
import { Client, type ClientSnapshot } from "../../src/client.js";
import { loadSession } from "../../src/cli-session.js";
const validSnapshot = (): ClientSnapshot => {
const kp = sodium.crypto_box_keypair();
return {
email: "user@example.com",
userID: 42,
token: "test-token",
masterKey: toBase64(sodium.crypto_secretbox_keygen()),
secretKey: toBase64(kp.privateKey),
publicKey: toBase64(kp.publicKey),
};
};
// The client's key buffers are private; the tests read them to prove that
// logout wipes them.
const keyBuffers = (client: Client): Uint8Array[] => [
client["masterKey"],
client["secretKey"],
client["publicKey"],
];
beforeAll(async () => {
await init();
});
describe("Client.toJSON", () => {
it("round-trips through fromJSON unchanged", () => {
const snapshot = validSnapshot();
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
});
it("throws instead of emitting a snapshot without a token", () => {
const client = Client.fromJSON(validSnapshot());
client.getApiClient().clearAuthToken();
expect(() => client.toJSON()).toThrow(/no auth token/);
});
});
describe("Client.fromJSON", () => {
const shortKey = toBase64(new Uint8Array(16));
it.each([
["email", undefined],
["email", 7],
["email", ""],
["token", undefined],
["token", null],
["token", ""],
["userID", undefined],
["userID", "42"],
["userID", 4.2],
["masterKey", undefined],
["masterKey", 7],
["masterKey", "not base64!"],
["masterKey", shortKey],
["secretKey", undefined],
["secretKey", "not base64!"],
["secretKey", shortKey],
["publicKey", undefined],
["publicKey", "not base64!"],
["publicKey", shortKey],
])("rejects %s = %j, naming the field", (field, value) => {
const snapshot: Record<string, unknown> = { ...validSnapshot() };
snapshot[field] = value;
expect(() => Client.fromJSON(snapshot)).toThrow(
new RegExp(`^Invalid session data: ${field} `),
);
});
it.each([null, "a string", 42])("rejects a non-object %j", (value) => {
expect(() => Client.fromJSON(value)).toThrow(
/^Invalid session data: not a JSON object/,
);
});
});
describe("Client.logout", () => {
it("zeroes the key buffers and clears the token", () => {
const client = Client.fromJSON(validSnapshot());
const api = client.getApiClient();
const keys = keyBuffers(client);
client.logout();
for (const key of keys) {
expect(key.length).toBe(32);
expect(key.every((b) => b === 0)).toBe(true);
}
expect(api.getAuthToken()).toBeUndefined();
});
it("makes every later operation throw", async () => {
const client = Client.fromJSON(validSnapshot());
client.logout();
expect(() => client.whoami()).toThrow(/logged out/);
expect(() => client.toJSON()).toThrow(/logged out/);
expect(() => client.getApiClient()).toThrow(/logged out/);
expect(() => client.contentSource()).toThrow(/logged out/);
await expect(client.listCollections()).rejects.toThrow(/logged out/);
await expect(client.collectionsSince({ sinceTime: 0 })).rejects.toThrow(
/logged out/,
);
await expect(
client.filesSince({
collectionID: 1,
collectionKey: new Uint8Array(32),
sinceTime: 0,
}),
).rejects.toThrow(/logged out/);
await expect(
client.fetchMLData({ fileIDs: [1], fileKeys: new Map() }),
).rejects.toThrow(/logged out/);
});
it("stops a listing in flight from decrypting with the zeroed keys", async () => {
// The server answers only after the client has logged out. If the
// listing went on to decrypt this row with all-zero keys it would fail
// with a decryption error, not the logged-out one.
const row = {
id: 1,
owner: { id: 42 },
encryptedKey: toBase64(new Uint8Array(48)),
keyDecryptionNonce: toBase64(new Uint8Array(24)),
updationTime: 1,
};
const client: Client = Client.fromJSON(validSnapshot(), {
fetch: async () => {
client.logout();
return new Response(JSON.stringify({ collections: [row] }), {
status: 200,
headers: { "content-type": "application/json" },
});
},
});
await expect(client.listCollections()).rejects.toThrow(/logged out/);
});
});
describe("loadSession", () => {
let dir: string;
beforeAll(() => {
dir = mkdtempSync(join(tmpdir(), "quak-session-test-"));
});
afterAll(() => {
rmSync(dir, { recursive: true, force: true });
});
it("returns null when there is no session file", () => {
expect(loadSession(join(dir, "missing.json"))).toBeNull();
});
it("restores a client from a valid session file", () => {
const path = join(dir, "valid.json");
writeFileSync(path, JSON.stringify(validSnapshot()));
expect(loadSession(path)!.whoami()).toEqual({
email: "user@example.com",
userID: 42,
});
});
it("says the file is corrupt when it is not JSON", () => {
const path = join(dir, "truncated.json");
writeFileSync(path, '{"email": "user@exa');
expect(() => loadSession(path)).toThrow(
`Session file ${path} is corrupt`,
);
});
it("says the file is corrupt and names the bad field", () => {
const path = join(dir, "bad-key.json");
writeFileSync(
path,
JSON.stringify({ ...validSnapshot(), secretKey: "AAAA" }),
);
expect(() => loadSession(path)).toThrow(
new RegExp(`^Session file ${path} is corrupt: .*secretKey`),
);
});
});
+33
View File
@@ -0,0 +1,33 @@
import { beforeAll, describe, expect, it } from "vitest";
import {
chunkHashFinal,
chunkHashInit,
chunkHashUpdate,
init,
} from "../../src/crypto/index.js";
beforeAll(async () => {
await init();
});
describe("content hash", () => {
// RFC 7693 Appendix A: BLAKE2b-512 of "abc".
const abc = Buffer.from(
"ba80a53f981c4d0d6a2797b69f12f6e94c212f14685ac4b74b12bb6fdbffa2d1" +
"7d87c5392aab792dc252d5de4533cc9518d38aa8dbf1925ab92386edd4009923",
"hex",
).toString("base64");
it("is unkeyed BLAKE2b-512 in standard base64", () => {
const state = chunkHashInit();
chunkHashUpdate(state, new TextEncoder().encode("abc"));
expect(chunkHashFinal(state)).toBe(abc);
});
it("gives the same hash when the input arrives in chunks", () => {
const state = chunkHashInit();
chunkHashUpdate(state, new TextEncoder().encode("a"));
chunkHashUpdate(state, new TextEncoder().encode("bc"));
expect(chunkHashFinal(state)).toBe(abc);
});
});
+355 -11
View File
@@ -48,7 +48,9 @@
*/
import {
chmodSync,
existsSync,
mkdirSync,
readdirSync,
readFileSync,
rmSync,
@@ -59,6 +61,7 @@ import { dirname, join } from "node:path";
import { tmpdir } from "node:os";
import { createHash } from "node:crypto";
import sodium from "libsodium-wrappers-sumo";
import { zipSync } from "fflate";
import {
beforeAll,
beforeEach,
@@ -186,7 +189,31 @@ vi.mock("node:fs/promises", async (importOriginal) => {
};
});
/**
* `chunkHashUpdate` is wrapped to record the length of every piece hashed, so
* a test can show that a live photo entry reaches the hash in pieces far
* smaller than the entry, rather than decompressed whole first.
*/
const hashHook = vi.hoisted(() => ({
lengths: [] as number[],
}));
vi.mock("../../src/crypto/index.js", async (importOriginal) => {
const actual =
await importOriginal<typeof import("../../src/crypto/index.js")>();
return {
...actual,
chunkHashUpdate: (
...args: Parameters<typeof actual.chunkHashUpdate>
): void => {
hashHook.lengths.push(args[1].length);
actual.chunkHashUpdate(...args);
},
};
});
beforeEach(() => {
hashHook.lengths.length = 0;
renameHook.calls.length = 0;
renameHook.failWith = null;
durabilityHook.events.length = 0;
@@ -501,6 +528,22 @@ const entryPoints = [
{ name: "downloadThumbnail", download: downloadThumbnail },
];
// With no `outPath`, the destination is named after `metadata.title`, relative
// to the working directory. Such tests run inside a temporary directory:
// `make check` must not create files in the repo root.
const inDirectory = async <T>(
dir: string,
run: () => Promise<T>,
): Promise<T> => {
const previous = process.cwd();
process.chdir(dir);
try {
return await run();
} finally {
process.chdir(previous);
}
};
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
@@ -536,26 +579,59 @@ describe("downloadFile", () => {
});
it("uses metadata.title as filename when outPath is omitted", async () => {
// With no `outPath`, the destination is `metadata.title`, used
// verbatim as a path. The title here is therefore given inside the
// test's temporary directory: a bare relative name would resolve
// against the process working directory, i.e. the repo root, and
// `make check` must not create files in the repo — a failure between
// the write and any cleanup would leave one behind.
const plaintext = new Uint8Array([1, 2, 3]);
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(plaintext, key);
const thumbPush =
sodium.crypto_secretstream_xchacha20poly1305_init_push(key);
const file = buildMockEnteFile(key, header, thumbPush.header);
const titlePath = join(testDir, "fallback-name.png");
file.metadata.title = titlePath;
file.metadata.title = "fallback-name.png";
const dir = mkdtempSync(join(testDir, "title-"));
const api = new ApiClient({ fetch: mockFetchForBody(ciphertext) });
const result = await downloadFile(api, file);
const result = await inDirectory(dir, () => downloadFile(api, file));
expect(result.path).toBe(titlePath);
expect(readFileSync(result.path)).toEqual(Buffer.from(plaintext));
expect(result.path).toBe("fallback-name.png");
expect(readFileSync(join(dir, "fallback-name.png"))).toEqual(
Buffer.from(plaintext),
);
});
it("keeps a hostile title inside the working directory", async () => {
// The server controls the title. `../escaped.png` must not write to
// the parent directory; it becomes one file name in the current one.
const { api, file } = fixtureFor(
multiChunkKey,
multiChunk.header,
multiChunk.body,
);
file.metadata.title = "../escaped.png";
const parent = mkdtempSync(join(testDir, "hostile-"));
const dir = join(parent, "cwd");
mkdirSync(dir);
const result = await inDirectory(dir, () => downloadFile(api, file));
expect(result.path).toBe("__escaped.png");
expect(readdirSync(dir)).toEqual(["__escaped.png"]);
expect(readdirSync(parent)).toEqual(["cwd"]);
});
it("uses an explicit outPath verbatim, even one with ..", async () => {
// The caller is trusted: its path is not sanitized.
const { api, file } = fixtureFor(
multiChunkKey,
multiChunk.header,
multiChunk.body,
);
const dir = mkdtempSync(join(testDir, "explicit-"));
mkdirSync(join(dir, "sub"));
const outPath = join(dir, "sub", "..", "explicit.bin");
const result = await downloadFile(api, file, outPath);
expect(result.path).toBe(outPath);
expect(existsSync(join(dir, "explicit.bin"))).toBe(true);
});
it("handles a larger single-chunk file (random binary payload)", async () => {
@@ -617,6 +693,23 @@ describe("downloadThumbnail", () => {
expect(result).toEqual({ path: outPath, bytesWritten: 4 });
expect(readFileSync(outPath)).toEqual(Buffer.from(plaintext));
});
it("names the thumbnail thumb_ plus the sanitized title", async () => {
const { api, file } = fixtureFor(
multiChunkKey,
multiChunk.header,
multiChunk.body,
);
file.metadata.title = "/etc/passwd";
const dir = mkdtempSync(join(testDir, "thumb-title-"));
const result = await inDirectory(dir, () =>
downloadThumbnail(api, file),
);
expect(result.path).toBe("thumb__etc_passwd");
expect(readdirSync(dir)).toEqual(["thumb__etc_passwd"]);
});
});
// ---------------------------------------------------------------------------
@@ -922,6 +1015,51 @@ describe.each(entryPoints)(
expect(readFileSync(outPath)).toEqual(Buffer.from(existing));
expect(readdirSync(dir)).toEqual(["rename-fails.bin"]);
});
it("fails without creating anything when the destination directory does not exist", async () => {
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(
patternBytes(64, 33),
key,
);
const { api, file } = fixtureFor(key, header, ciphertext);
const dir = freshDir();
const outPath = join(dir, "missing", "never.bin");
await expect(download(api, file, outPath)).rejects.toMatchObject({
code: "ENOENT",
});
// The missing directory is not created on the caller's behalf.
expect(readdirSync(dir)).toEqual([]);
});
// Root ignores directory permissions, so this cannot fail as root
// (the Docker test image runs as root).
it.skipIf(process.getuid?.() === 0)(
"fails without creating anything when the destination directory is not writable",
async () => {
const key =
sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(
patternBytes(64, 34),
key,
);
const { api, file } = fixtureFor(key, header, ciphertext);
const dir = freshDir();
const outPath = join(dir, "never.bin");
chmodSync(dir, 0o500);
try {
await expect(
download(api, file, outPath),
).rejects.toMatchObject({ code: "EACCES" });
} finally {
chmodSync(dir, 0o700);
}
expect(readdirSync(dir)).toEqual([]);
},
);
},
);
@@ -1224,6 +1362,102 @@ describe("download retries: corruption is not retried", () => {
expect(requests()).toBe(1);
});
it("cancels the response body when decryption fails", async () => {
// A backup run carries on past a failed file, so a body left open on
// failure would hold its connection until garbage collection, once
// per failed file. This body delivers a corrupt chunk and then stays
// open, so only a cancel from the downloader can close it.
const corrupted = Uint8Array.from(multiChunk.body);
corrupted[10] ^= 0xff;
let cancelled = false;
const fetch = (async () =>
new Response(
new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(corrupted);
},
cancel() {
cancelled = true;
},
}),
{ status: 200 },
)) as typeof globalThis.fetch;
const api = new ApiClient({ fetch, retry: { ...noWait, attempts: 1 } });
const file = buildMockEnteFile(
multiChunkKey,
multiChunk.header,
multiChunk.header,
);
const outPath = join(mkdtempSync(join(testDir, "cancel-")), "c.bin");
await expect(downloadFile(api, file, outPath)).rejects.toThrow(
/authentication failed/i,
);
expect(cancelled).toBe(true);
});
it("cancels the response body when the temp file cannot be opened", async () => {
// The download fails before a byte of the body is read, so the body
// is still open and only a cancel from the downloader can close it.
let cancelled = false;
const fetch = (async () =>
new Response(
new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(multiChunk.body);
},
cancel() {
cancelled = true;
},
}),
{ status: 200 },
)) as typeof globalThis.fetch;
const api = new ApiClient({ fetch, retry: { ...noWait, attempts: 1 } });
const file = buildMockEnteFile(
multiChunkKey,
multiChunk.header,
multiChunk.header,
);
const outPath = join(
mkdtempSync(join(testDir, "cancel-")),
"missing",
"c.bin",
);
await expect(downloadFile(api, file, outPath)).rejects.toMatchObject({
code: "ENOENT",
});
expect(cancelled).toBe(true);
});
it("cancels the response body when the header is malformed", async () => {
// The header is rejected before the body is read, so the body is
// still open and only a cancel from the downloader can close it.
let cancelled = false;
const fetch = (async () =>
new Response(
new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(multiChunk.body);
},
cancel() {
cancelled = true;
},
}),
{ status: 200 },
)) as typeof globalThis.fetch;
const api = new ApiClient({ fetch, retry: { ...noWait, attempts: 1 } });
const shortHeader = multiChunk.header.subarray(0, 5);
const file = buildMockEnteFile(multiChunkKey, shortHeader, shortHeader);
const outPath = join(mkdtempSync(join(testDir, "cancel-")), "c.bin");
await expect(downloadFile(api, file, outPath)).rejects.toThrow();
expect(cancelled).toBe(true);
});
});
// ---------------------------------------------------------------------------
@@ -1404,3 +1638,113 @@ describe.each(entryPoints)("$name progress", ({ name, download }) => {
expectSameBytes(readFileSync(outPath), plaintext);
});
});
describe("downloadFile content hash", () => {
// Node's own BLAKE2b-512 is the reference, so these tests do not depend
// on the code under test to compute what they expect.
const blake2b = (bytes: Uint8Array): string =>
createHash("blake2b512").update(bytes).digest("base64");
// Serve `plaintext` encrypted as file 999 with the given metadata. Four
// responses are scripted so a retried mismatch would show in `requests`.
const setup = (plaintext: Uint8Array, metadata: Partial<FileMetadata>) => {
const key = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const { header, ciphertext } = encryptFileBody(plaintext, key);
const file = buildMockEnteFile(key, header, header);
file.metadata = { ...file.metadata, ...metadata };
const body = { kind: "body", bytes: ciphertext } as const;
const { fetch, requests } = scriptedCdnFetch(body, body, body, body);
const api = new ApiClient({ fetch, retry: { ...noWait, attempts: 4 } });
const dir = mkdtempSync(join(testDir, "hash-"));
const outPath = join(dir, "f.bin");
return {
run: () => downloadFile(api, file, outPath),
dir,
outPath,
requests,
};
};
const livePhotoZip = zipSync({
"image.heic": patternBytes(500, 81),
"video.mov": patternBytes(900, 82),
});
const livePhotoHash = `${blake2b(patternBytes(500, 81))}:${blake2b(patternBytes(900, 82))}`;
it("stores a file whose hash matches", async () => {
const plaintext = patternBytes(700, 80);
const t = setup(plaintext, { hash: blake2b(plaintext) });
await t.run();
expectSameBytes(readFileSync(t.outPath), plaintext);
});
it("rejects a mismatch, stores nothing, names the file and does not retry", async () => {
const t = setup(patternBytes(700, 80), {
hash: blake2b(patternBytes(700, 79)),
});
await expect(t.run()).rejects.toThrow(
/file 999: content hash .* does not match/,
);
expect(readdirSync(t.dir)).toEqual([]);
expect(t.requests()).toBe(1);
});
it("stores a file with no recorded hash unchecked", async () => {
const plaintext = patternBytes(700, 80);
const t = setup(plaintext, { hash: undefined });
await t.run();
expectSameBytes(readFileSync(t.outPath), plaintext);
});
it("stores a live photo whose image and video hashes match", async () => {
const t = setup(livePhotoZip, {
fileType: "livePhoto",
hash: livePhotoHash,
});
await t.run();
expectSameBytes(readFileSync(t.outPath), livePhotoZip);
});
it("hashes a large live photo entry as it decompresses, never whole", async () => {
// 64 MiB of zeros deflates to a few kilobytes, the shape of a ZIP
// that would exhaust memory if expanded whole.
const image = new Uint8Array(64 * 1024 * 1024);
const video = patternBytes(900, 83);
const zip = zipSync({ "image.heic": image, "video.mov": video });
const t = setup(zip, {
fileType: "livePhoto",
hash: `${blake2b(image)}:${blake2b(video)}`,
});
await t.run();
expectSameBytes(readFileSync(t.outPath), zip);
const hashed = hashHook.lengths.reduce((a, b) => a + b, 0);
expect(hashed).toBe(image.length + video.length);
expect(Math.max(...hashHook.lengths)).toBeLessThanOrEqual(
2 * STREAM_CHUNK_SIZE,
);
});
it("rejects a live photo whose hash does not match", async () => {
// The whole ZIP's hash is not the recorded one: each part is hashed.
const t = setup(livePhotoZip, {
fileType: "livePhoto",
hash: blake2b(livePhotoZip),
});
await expect(t.run()).rejects.toThrow(
/file 999: content hash .* does not match/,
);
expect(readdirSync(t.dir)).toEqual([]);
});
});
+93
View File
@@ -0,0 +1,93 @@
// File names built from server-supplied metadata.
//
// quak does not trust the server. A file's title and a collection's name are
// decrypted from data the server hands us, and a hostile server (or a
// compromised account) can set them to anything. quak uses them to name files
// on disk: `quak get` without `--out`, `downloadFile` without `outPath`, the
// backup's symlink and collection directories, and the extension of every file
// in the originals cache. Each of those goes through `sanitizeFileName` or
// `safeExtension`, so a title can only ever name one file inside the directory
// the caller chose.
//
// A path the user supplies (`--out`, `outPath`) is never sanitized: the caller
// is trusted, the server is not.
import { describe, expect, it } from "vitest";
import { safeExtension, sanitizeFileName } from "../../src/filename.js";
const FALLBACK = "file-42";
describe("sanitizeFileName", () => {
it("passes a normal title through unchanged", () => {
expect(sanitizeFileName("IMG_0001.HEIC", FALLBACK)).toBe(
"IMG_0001.HEIC",
);
expect(sanitizeFileName("Holiday 2024 (1).jpg", FALLBACK)).toBe(
"Holiday 2024 (1).jpg",
);
expect(sanitizeFileName("café.jpg", FALLBACK)).toBe("café.jpg");
});
it("cannot climb out of the directory with ../", () => {
// Without sanitizing, this would overwrite the user's SSH keys.
expect(sanitizeFileName("../../.ssh/authorized_keys", FALLBACK)).toBe(
"__.._.ssh_authorized_keys",
);
expect(sanitizeFileName("..", FALLBACK)).toBe("_");
expect(sanitizeFileName("..\\..\\x", FALLBACK)).toBe("__.._x");
});
it("cannot name an absolute path", () => {
expect(sanitizeFileName("/etc/passwd", FALLBACK)).toBe("_etc_passwd");
expect(sanitizeFileName("C:\\Windows\\x.dll", FALLBACK)).toBe(
"C__Windows_x.dll",
);
});
it("replaces embedded separators, so the name stays one file", () => {
expect(sanitizeFileName("a/b\\c.jpg", FALLBACK)).toBe("a_b_c.jpg");
});
it("replaces NUL and other control characters", () => {
// A NUL truncates the path in C code and makes Node's fs throw.
expect(sanitizeFileName("evil\0.jpg", FALLBACK)).toBe("evil_.jpg");
expect(sanitizeFileName("line\nbreak\x7f.jpg", FALLBACK)).toBe(
"line_break_.jpg",
);
});
it("does not produce a hidden file", () => {
expect(sanitizeFileName(".bashrc", FALLBACK)).toBe("_bashrc");
});
it("does not produce a Windows device name", () => {
expect(sanitizeFileName("CON", FALLBACK)).toBe("_CON");
expect(sanitizeFileName("nul.txt", FALLBACK)).toBe("_nul.txt");
expect(sanitizeFileName("LPT1", FALLBACK)).toBe("_LPT1");
// Only the exact names are reserved.
expect(sanitizeFileName("console.jpg", FALLBACK)).toBe("console.jpg");
});
it("falls back to the given name for an empty title", () => {
expect(sanitizeFileName("", FALLBACK)).toBe(FALLBACK);
});
});
describe("safeExtension", () => {
it("keeps a normal extension", () => {
expect(safeExtension("IMG_0001.HEIC")).toBe(".HEIC");
expect(safeExtension("clip.mp4")).toBe(".mp4");
});
it("uses .bin when there is no extension", () => {
expect(safeExtension("")).toBe(".bin");
expect(safeExtension("README")).toBe(".bin");
});
it("uses .bin when the extension holds anything but letters and digits", () => {
expect(safeExtension("x.j\\..\\pg")).toBe(".bin");
expect(safeExtension("x.jp g")).toBe(".bin");
expect(safeExtension("x.jpg\0")).toBe(".bin");
});
});
+3 -3
View File
@@ -116,7 +116,7 @@ describe("Library content wiring", () => {
expect(lib.photos.byID({ fileID: 1 })!.record().thumbnailPath).toBe(
result.path,
);
lib.close();
await lib.close();
});
it("drives thumbnails.ensure through the cache", async () => {
@@ -139,7 +139,7 @@ describe("Library content wiring", () => {
expect(results).toEqual([
{ fileID: 1, path: join(root, "cache", "thumbnails", "1.jpg") },
]);
lib.close();
await lib.close();
});
it("throws from content methods when opened without a content source", async () => {
@@ -155,6 +155,6 @@ describe("Library content wiring", () => {
await expect(
lib.thumbnails.ensure({ fileIDs: [1], priority: "visible" }),
).rejects.toThrow(/content cache/i);
lib.close();
await lib.close();
});
});
+19
View File
@@ -226,6 +226,25 @@ describe("ContentCache.original / thumbnail", () => {
expect(skips).toEqual(["skipped"]);
});
it("takes only a letters-and-digits extension from the title", async () => {
// The title comes from the server; an extension such as `.\..\x`
// must not reach the cache file name, so it becomes `.bin`.
const { cache } = buildCache({
files: [file(1, "a.jpg"), file(2, "b.\\..\\x"), file(3, "")],
});
await cache.open();
expect((await cache.original(1)).path).toBe(
join(cacheDir, "originals", "1.jpg"),
);
expect((await cache.original(2)).path).toBe(
join(cacheDir, "originals", "2.bin"),
);
expect((await cache.original(3)).path).toBe(
join(cacheDir, "originals", "3.bin"),
);
});
it("serves a file already present in the download directory without fetching", async () => {
const downloadDirectory = join(root, "backup");
mkdirSync(join(downloadDirectory, "originals"), { recursive: true });
+3 -3
View File
@@ -179,7 +179,7 @@ describe("Library.fresh", () => {
// And the change is now live for the default namespaces too.
expect(lib.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
} finally {
lib.close();
await lib.close();
}
});
@@ -228,7 +228,7 @@ describe("Library.fresh", () => {
expect(reads.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
}
} finally {
lib.close();
await lib.close();
}
});
@@ -266,7 +266,7 @@ describe("Library.fresh", () => {
);
expect(lib.status().lastError).toBeUndefined();
} finally {
lib.close();
await lib.close();
}
});
});
+89 -17
View File
@@ -17,7 +17,8 @@
* failure surfaces via `onProgress` ("failed") and `status()`, and a later
* success clears the error. `open()` itself resolves even when the first
* refresh fails (offline start from cache).
* 5. `close()` stops the timer and is idempotent.
* 5. `close()` stops the timer and is idempotent, and its promise resolves
* only once an in-flight refresh has written the cache file.
* 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id.
* 7. `open()` branches on the cache: an empty cache awaits the first refresh
* (it has nothing to serve yet); an existing cache serves its copy at once
@@ -37,6 +38,11 @@
* short interval and `vi.waitFor`: a fake clock cannot settle the real
* fsync-and-rename cache write, and empty diffs never write, so the eventual
* state is stable to poll for.
*
* A refresh changes RAM before it writes the cache file, so a polled state can
* be visible while that write is still running. Every test therefore awaits
* `close()`, which waits for the in-flight refresh, before `afterEach` removes
* the directory.
*/
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
@@ -193,7 +199,7 @@ describe("Library.open and background refresh", () => {
expect(reloaded.getFile(1, 1001)?.id).toBe(1001);
expect(reloaded.collectionsSinceTime).toBe(100);
} finally {
lib.close();
await lib.close();
}
});
@@ -224,7 +230,7 @@ describe("Library.open and background refresh", () => {
expect(client.collectionsSinceTimes.length).toBe(collectionCalls);
expect(client.filesCalls.length).toBe(fileCalls);
} finally {
lib.close();
await lib.close();
}
});
@@ -271,7 +277,7 @@ describe("Library.open and background refresh", () => {
{ timeout: 2000, interval: 5 },
);
} finally {
lib.close();
await lib.close();
}
});
@@ -303,7 +309,7 @@ describe("Library.open and background refresh", () => {
// never re-fetched.
expect(client.filesCalls).toEqual([]);
} finally {
lib.close();
await lib.close();
}
});
@@ -357,7 +363,7 @@ describe("Library.open and background refresh", () => {
{ timeout: 2000, interval: 5 },
);
} finally {
lib.close();
await lib.close();
}
});
@@ -402,7 +408,7 @@ describe("Library.open and background refresh", () => {
await new Promise((r) => setTimeout(r, FAST_INTERVAL * 1000 * 4));
expect(saveSpy).toHaveBeenCalledTimes(2);
} finally {
lib.close();
await lib.close();
saveSpy.mockRestore();
}
});
@@ -462,7 +468,7 @@ describe("Library.open and background refresh", () => {
{ timeout: 2000, interval: 5 },
);
} finally {
lib.close();
await lib.close();
}
});
@@ -488,7 +494,7 @@ describe("Library.open and background refresh", () => {
),
).toBe(true);
} finally {
lib.close();
await lib.close();
}
});
@@ -502,9 +508,14 @@ describe("Library.open and background refresh", () => {
seed.putFile(file(1001, 1, 400));
await seed.save();
// The server never answers this run's first refresh.
// The server does not answer this run's first refresh until the test
// is done with it.
let answerFirstFetch: (page: CollectionsPage) => void = () => {};
const client = new MockClient();
client.collectionsSince = () => new Promise<CollectionsPage>(() => {});
client.collectionsSince = () =>
new Promise<CollectionsPage>((resolve) => {
answerFirstFetch = resolve;
});
// open() must resolve from the cache without blocking on the network,
// and reads must serve the seeded copy.
@@ -517,7 +528,9 @@ describe("Library.open and background refresh", () => {
expect(lib.status().lastRefreshAt).toBeUndefined();
expect(lib.status().lastError).toBeUndefined();
} finally {
lib.close();
// close() waits for the outstanding refresh, so let it finish.
answerFirstFetch({ collections: [], deleted: [], cursor: 500 });
await lib.close();
}
});
@@ -564,7 +577,7 @@ describe("Library.open and background refresh", () => {
expect(lib.listFiles(1).map((f) => f.id)).toEqual([1001]);
expect(lib.status().lastRefreshAt).toBeGreaterThan(0);
} finally {
lib.close();
await lib.close();
}
});
@@ -619,7 +632,7 @@ describe("Library.open and background refresh", () => {
);
expect(reloaded.getFile(1, 1001)?.id).toBe(1001);
} finally {
lib.close();
await lib.close();
saveSpy.mockRestore();
}
});
@@ -634,8 +647,8 @@ describe("Library.open and background refresh", () => {
});
const callsAfterOpen = client.collectionsSinceTimes.length;
lib.close();
lib.close(); // second close must not throw
await lib.close();
await lib.close(); // second close must not throw
expect(lib.status().closed).toBe(true);
// No further refreshes fire once closed.
@@ -643,6 +656,65 @@ describe("Library.open and background refresh", () => {
expect(client.collectionsSinceTimes.length).toBe(callsAfterOpen);
});
it("close() resolves only after an in-flight refresh has written the cache", async () => {
const path = join(cacheDirectory, "metadata.json");
const seed = await MetadataStore.load(path);
seed.userID = USER_ID;
seed.collectionsSinceTime = 500;
seed.putCollection(collection(1, 400));
seed.putFile(file(1001, 1, 400));
await seed.save();
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 600)],
deleted: [],
cursor: 600,
});
client.filesFor(1, {
files: [file(1002, 1, 600)],
deleted: [],
cursor: 600,
});
// Hold the refresh's cache write until the test releases it.
const realSave = MetadataStore.prototype.save;
let releaseSave: () => void = () => {};
const saveHeld = new Promise<void>((resolve) => {
releaseSave = resolve;
});
const saveSpy = vi
.spyOn(MetadataStore.prototype, "save")
.mockImplementation(async function (this: MetadataStore) {
await saveHeld;
return realSave.call(this);
});
const lib = await Library.open({ client, cacheDirectory });
try {
await vi.waitFor(() => expect(saveSpy).toHaveBeenCalled(), {
timeout: 2000,
interval: 5,
});
let closed = false;
const closing = lib.close().then(() => {
closed = true;
});
await new Promise((r) => setTimeout(r, 50));
expect(closed).toBe(false);
releaseSave();
await closing;
const reloaded = await MetadataStore.load(path);
expect(reloaded.getFile(1, 1002)?.id).toBe(1002);
} finally {
releaseSave();
await lib.close();
saveSpy.mockRestore();
}
});
it("defaults cacheDirectory to the env-paths cache dir plus user id", async () => {
const xdg = join(dir, "xdg-cache");
const prev = process.env.XDG_CACHE_HOME;
@@ -659,7 +731,7 @@ describe("Library.open and background refresh", () => {
expect(lib.cacheDirectory.startsWith(xdg)).toBe(true);
expect(lib.cacheDirectory.endsWith(String(USER_ID))).toBe(true);
} finally {
lib.close();
await lib.close();
}
} finally {
if (prev === undefined) delete process.env.XDG_CACHE_HOME;
+54 -2
View File
@@ -374,7 +374,7 @@ describe("Library ML-data fetch on refresh", () => {
await new Promise((r) => setTimeout(r, FAST_INTERVAL * 1000 * 5));
expect(client.mlFetchCalls.length).toBe(callsAfterFirst);
} finally {
lib.close();
await lib.close();
}
});
@@ -443,7 +443,59 @@ describe("Library ML-data fetch on refresh", () => {
expect(client.mlFetchCalls.length).toBeGreaterThan(callsBefore);
expect(client.mlFetchCalls.flat()).toContain(1001);
} finally {
lib.close();
await lib.close();
}
});
it("close() resolves only after a running ML data fetch has stored its payloads", async () => {
const client = new MLMockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
client.mlByFile.set(1001, payload([0.5, 0.25, 0.75]));
// Hold the ML data fetch open until the test releases it.
let release!: () => void;
const held = new Promise<void>((r) => (release = r));
let fetchStarted!: () => void;
const started = new Promise<void>((r) => (fetchStarted = r));
const realFetch = client.fetchMLData.bind(client);
client.fetchMLData = async (args) => {
fetchStarted();
await held;
return realFetch(args);
};
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: 3600,
});
try {
await started;
let closed = false;
const closing = lib.close().then(() => {
closed = true;
});
await new Promise((r) => setTimeout(r, 20));
expect(closed).toBe(false);
release();
await closing;
expect(
existsSync(join(cacheDirectory, "mldata", "1001.json")),
).toBe(true);
} finally {
release();
await lib.close();
}
});
});
+70 -8
View File
@@ -402,35 +402,97 @@ describe("Precache through Library.open", () => {
}
it("starts both precaches from open() and reports them in status()", async () => {
const thumbFetched = new Set<number>();
const origFetched = new Set<number>();
const source: ContentSource = {
original: async ({ file: f, destination }) => {
origFetched.add(f.id);
original: async ({ destination }) => {
await writeFile(destination, Buffer.alloc(10, 1));
return { bytesWritten: 10 };
},
thumbnail: async ({ file: f, destination }) => {
thumbFetched.add(f.id);
thumbnail: async ({ destination }) => {
await writeFile(destination, Buffer.alloc(10, 1));
return { bytesWritten: 10 };
},
};
// Each fill reports "done" once the cache has recorded its files. The
// source returning is not enough: the cache records a file only after
// it has checked it on disk.
const finished = new Set<string>();
let bothFinished!: () => void;
const precached = new Promise<void>((r) => (bothFinished = r));
const lib = await Library.open({
client: new MockClient(),
cacheDirectory: join(root, "cache"),
contentSource: source,
refreshIntervalSeconds: 3600,
onProgress: (e) => {
if (
e.status === "done" &&
(e.operation === "precacheThumbnails" ||
e.operation === "precacheOriginals")
) {
finished.add(e.operation);
if (finished.size === 2) bothFinished();
}
},
});
// Every file's thumbnail is precached; the favorite (file 3) and the
// week's files (1, 2) all have their originals precached.
await until(() => thumbFetched.size === 3 && origFetched.size === 3);
await precached;
const status = lib.status();
expect(status.thumbnailsTotal).toBe(3);
expect(status.thumbnailsCached).toBe(3);
expect(status.originalsPinned).toBe(3);
expect(status.originalsCached).toBe(3);
lib.close();
await lib.close();
});
it.each(["thumbnail", "original"] as const)(
"close() resolves only after a running %s precache fetch has written its file",
async (kind) => {
// Only the fill under test runs, and each of its fetches waits
// until the test releases it.
let release!: () => void;
const held = new Promise<void>((r) => (release = r));
let fetchStarted!: (destination: string) => void;
const started = new Promise<string>((r) => (fetchStarted = r));
const fetch = async ({ destination }: { destination: string }) => {
fetchStarted(destination);
await held;
await writeFile(destination, Buffer.alloc(10, 1));
return { bytesWritten: 10 };
};
const unused = async () => {
throw new Error("this fill is turned off");
};
const source: ContentSource =
kind === "thumbnail"
? { original: unused, thumbnail: fetch }
: { original: fetch, thumbnail: unused };
const lib = await Library.open({
client: new MockClient(),
cacheDirectory: join(root, "cache"),
contentSource: source,
refreshIntervalSeconds: 3600,
precacheThumbnails: kind === "thumbnail",
precacheOriginals: kind === "original",
});
try {
const destination = await started;
let closed = false;
const closing = lib.close().then(() => {
closed = true;
});
await new Promise((r) => setTimeout(r, 20));
expect(closed).toBe(false);
release();
await closing;
expect(existsSync(destination)).toBe(true);
} finally {
release();
await lib.close();
}
},
);
});
+1 -1
View File
@@ -531,7 +531,7 @@ describe("Library exposes the read surface over its live store", () => {
expect(client.collectionsCalls).toBe(collectionsBefore);
expect(client.filesCalls).toBe(filesBefore);
} finally {
lib.close();
await lib.close();
}
});
});
+4 -4
View File
@@ -159,7 +159,7 @@ describe("Library.snapshot and Library.subscribe", () => {
for (const p of snap.photos) expect("key" in p).toBe(false);
for (const a of snap.albums) expect("key" in a).toBe(false);
} finally {
lib.close();
await lib.close();
}
});
@@ -219,7 +219,7 @@ describe("Library.snapshot and Library.subscribe", () => {
expect(change.refreshedAt).toBeGreaterThan(0);
} finally {
unsubscribe();
lib.close();
await lib.close();
}
});
@@ -251,7 +251,7 @@ describe("Library.snapshot and Library.subscribe", () => {
expect(changes).toEqual([]);
} finally {
unsubscribe();
lib.close();
await lib.close();
}
});
@@ -297,7 +297,7 @@ describe("Library.snapshot and Library.subscribe", () => {
);
expect(changes).toEqual([]);
} finally {
lib.close();
await lib.close();
}
});
});
+73 -3
View File
@@ -146,10 +146,13 @@ const buildSharedRawCollection = (
const buildRawFile = (
collectionKey: Uint8Array,
opts?: {
title?: string;
// Any JSON value; `undefined` leaves the title out of the metadata.
title?: unknown;
fileType?: number;
creationTime?: number;
info?: { fileSize?: number; thumbSize?: number };
// Replaces the whole metadata JSON value.
metadata?: unknown;
},
): RawEnteFile => {
const fileKey = sodium.crypto_secretbox_keygen();
@@ -158,8 +161,8 @@ const buildRawFile = (
collectionKey,
);
const metadata = {
title: opts?.title ?? "IMG_0001.jpg",
const defaultMetadata = {
title: opts && "title" in opts ? opts.title : "IMG_0001.jpg",
fileType: opts?.fileType ?? 0,
creationTime: opts?.creationTime ?? 1700000000000000,
modificationTime: 1700000000000000,
@@ -167,6 +170,8 @@ const buildRawFile = (
longitude: 2.3522,
hash: "abcdef1234567890",
};
const metadata =
opts && "metadata" in opts ? opts.metadata : defaultMetadata;
// File metadata is encrypted as a single-chunk secretstream blob
// (not secretbox). The decryptionHeader is the secretstream init header.
const metadataBytes = new TextEncoder().encode(JSON.stringify(metadata));
@@ -321,6 +326,71 @@ describe("model.decryptFile", () => {
expect(file.metadata.longitude).toBeCloseTo(2.3522);
});
it("reads a missing or non-string title as an empty string", () => {
// The server controls the metadata JSON. A title that is not a
// string must not reach code that builds file names from it.
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
for (const title of [undefined, null, 42, ["a"], { x: "../y" }]) {
const file = decryptFile(
buildRawFile(collectionKey, { title }),
collectionKey,
);
expect(file.metadata.title).toBe("");
}
});
it("rejects metadata that is not a JSON object", () => {
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
for (const metadata of [null, "IMG_0001.jpg", 7, []]) {
const raw = buildRawFile(collectionKey, { metadata });
expect(() => decryptFile(raw, collectionKey)).toThrow(
"file 200: metadata is not a JSON object",
);
}
});
it("reads the recorded content hash, joining an older live photo's two parts", () => {
const masterKey = sodium.crypto_secretbox_keygen();
const { collectionKey } = buildRawCollection(masterKey);
const hashOf = (metadata: Record<string, unknown>) =>
decryptFile(
buildRawFile(collectionKey, {
metadata: { title: "x", ...metadata },
}),
collectionKey,
).metadata.hash;
expect(hashOf({ fileType: 0, hash: "H" })).toBe("H");
expect(
hashOf({ fileType: 2, hash: "H", imageHash: "I", videoHash: "V" }),
).toBe("H");
expect(hashOf({ fileType: 2, imageHash: "I", videoHash: "V" })).toBe(
"I:V",
);
expect(hashOf({ fileType: 2, imageHash: "I" })).toBeUndefined();
expect(
hashOf({ fileType: 0, imageHash: "I", videoHash: "V" }),
).toBeUndefined();
expect(hashOf({ fileType: 0 })).toBeUndefined();
expect(hashOf({ fileType: 0, hash: 42 })).toBeUndefined();
expect(
hashOf({ fileType: 2, imageHash: "I", videoHash: 7 }),
).toBeUndefined();
// An empty string counts as absent, not as a hash to match.
expect(hashOf({ fileType: 0, hash: "" })).toBeUndefined();
expect(
hashOf({ fileType: 2, hash: "", imageHash: "I", videoHash: "V" }),
).toBe("I:V");
expect(
hashOf({ fileType: 2, imageHash: "", videoHash: "V" }),
).toBeUndefined();
expect(
hashOf({ fileType: 2, imageHash: "I", videoHash: "" }),
).toBeUndefined();
});
it("maps fileType numbers to FileType strings", () => {
// Ente uses: 0=image, 1=video, 2=livePhoto
const masterKey = sodium.crypto_secretbox_keygen();
+50
View File
@@ -0,0 +1,50 @@
// A checkout nested under `.claude/` must not add its tests to this suite.
// The test plants one in a temporary directory next to a real test file and
// asks vitest, with this repo's config, which test files it would run.
import { afterEach, describe, expect, it } from "vitest";
import { execFileSync } from "node:child_process";
import { mkdirSync, mkdtempSync, rmSync, writeFileSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
let root = "";
afterEach(() => {
rmSync(root, { recursive: true, force: true });
});
const writeTest = (path: string): void => {
mkdirSync(join(root, path, ".."), { recursive: true });
writeFileSync(
join(root, path),
'import { it } from "vitest";\nit("runs", () => {});\n',
);
};
describe("vitest.config.ts", () => {
it("does not collect tests from a checkout nested under .claude/", () => {
root = mkdtempSync(join(tmpdir(), "quak-nested-checkout-"));
writeTest("test/real.test.ts");
writeTest(".claude/worktrees/other/test/real.test.ts");
const output = execFileSync(
process.execPath,
[
join(repoRoot, "node_modules/vitest/vitest.mjs"),
"list",
"--filesOnly",
"--config",
join(repoRoot, "vitest.config.ts"),
"--root",
root,
],
{ cwd: root, encoding: "utf-8" },
);
const files = output.split("\n").filter((line) => line !== "");
expect(files).toEqual(["test/real.test.ts"]);
});
});
+75 -4
View File
@@ -149,8 +149,11 @@ describe("isRetryable: transport failures", () => {
});
it("retries an errno carried on the error itself", () => {
// Every errno the classifier names, so none can be reclassified
// unnoticed.
for (const code of [
"ECONNRESET",
"ECONNABORTED",
"ETIMEDOUT",
"EPIPE",
"ENOTFOUND",
@@ -158,6 +161,8 @@ describe("isRetryable: transport failures", () => {
"ECONNREFUSED",
"EHOSTUNREACH",
"ENETUNREACH",
"ENETRESET",
"ENETDOWN",
]) {
expect(isRetryable(errnoError(code))).toBe(true);
}
@@ -215,6 +220,27 @@ describe("isRetryable: transport failures", () => {
looped.cause = looped;
expect(isRetryable(looped)).toBe(false);
});
it("terminates on a cause chain that loops through two errors", () => {
const first: Error & { cause?: unknown } = new Error("first");
const second = new Error("second", { cause: first });
first.cause = second;
expect(isRetryable(first)).toBe(false);
});
it("reads the error and at most seven causes below it", () => {
// The walk is bounded at eight links. An errno at the eighth link is
// found; one at the ninth is not.
const buried = (causes: number): Error => {
let err = errnoError("ECONNRESET");
for (let i = 0; i < causes; i++) {
err = new Error(`wrapper ${i}`, { cause: err });
}
return err;
};
expect(isRetryable(buried(7))).toBe(true);
expect(isRetryable(buried(8))).toBe(false);
});
});
describe("isRetryable: stream truncation versus corruption", () => {
@@ -279,10 +305,9 @@ describe("isSafeToReplay", () => {
* that is not the whole question: the other half is "could the first
* attempt already have taken effect on the server?".
*
* quak's non-idempotent calls are `/users/srp/create-session`,
* `/users/two-factor/verify` (which consumes one of a limited number of
* 2FA attempts) and `/files/thumbnail`. A blind replay of any of them can
* do real damage, so they retry only on the failures that establish no TCP
* The calls this guards are the `POST` and `PUT` requests listed in the
* README under "Endpoints used". A blind replay of some of them can do
* real damage, so they retry only on the failures that establish no TCP
* connection to the server ever existed DNS produced no address, or the
* peer refused the connection and therefore that no request byte can
* have been transmitted.
@@ -333,6 +358,52 @@ describe("isSafeToReplay", () => {
).toBe(false);
expect(isSafeToReplay(new TypeError("fetch failed"))).toBe(false);
});
it("does not replay any other errno the classifier names", () => {
for (const code of [
"ECONNRESET",
"ECONNABORTED",
"ETIMEDOUT",
"EPIPE",
"EHOSTUNREACH",
"ENETUNREACH",
"ENETRESET",
"ENETDOWN",
]) {
expect(isSafeToReplay(errnoError(code))).toBe(false);
}
});
it("does not replay a chain that also shows the request may have gone out", () => {
// A connect errno somewhere in the chain is not enough: any other
// errno beside it is doubt, and doubt is not replayed.
const reset = Object.assign(
new Error("read ECONNRESET", { cause: errnoError("ECONNREFUSED") }),
{ code: "ECONNRESET" },
);
const mixed = new TypeError("fetch failed", { cause: reset });
expect(isRetryable(mixed)).toBe(true);
expect(isSafeToReplay(mixed)).toBe(false);
});
it("does not replay a chain longer than the walk reads", () => {
// Eight connect errnos, then a reset at the ninth link, below the
// limit. The walk never sees the reset, so it cannot rule it out.
const refusedChain = (below: Error | undefined): Error => {
let err = below;
for (let i = 0; i < 8; i++) {
err = Object.assign(new Error(`refused ${i}`, { cause: err }), {
code: "ECONNREFUSED",
});
}
return err as Error;
};
expect(isSafeToReplay(refusedChain(errnoError("ECONNRESET")))).toBe(
false,
);
// The same eight links with nothing below them are replayable.
expect(isSafeToReplay(refusedChain(undefined))).toBe(true);
});
});
// ---------------------------------------------------------------------------
+39 -5
View File
@@ -1,9 +1,43 @@
import { describe, expect, it } from "vitest";
import { afterEach, describe, expect, it, vi } from "vitest";
import { readFileSync } from "node:fs";
import { VERSION } from "../src/index.js";
describe("quak", () => {
it("exports a version string", () => {
expect(typeof VERSION).toBe("string");
expect(VERSION.length).toBeGreaterThan(0);
const packageVersion = (
JSON.parse(
readFileSync(new URL("../package.json", import.meta.url), "utf-8"),
) as { version: string }
).version;
class ExitCalled extends Error {}
describe("version", () => {
const argv = process.argv;
afterEach(() => {
process.argv = argv;
vi.restoreAllMocks();
});
it("exports the version from package.json", () => {
expect(VERSION).toBe(packageVersion);
});
// Runs bin/quak.ts with --version. commander prints the version and then
// calls process.exit, which is stubbed to throw so the test survives.
it("reports the version from package.json in quak --version", async () => {
const printed: string[] = [];
vi.spyOn(process.stdout, "write").mockImplementation((chunk) => {
printed.push(String(chunk));
return true;
});
vi.spyOn(process, "exit").mockImplementation(() => {
throw new ExitCalled();
});
process.argv = ["node", "quak", "--version"];
await expect(import("../bin/quak.js")).rejects.toBeInstanceOf(
ExitCalled,
);
expect(printed.join("").trim()).toBe(packageVersion);
});
});
+10
View File
@@ -0,0 +1,10 @@
import { configDefaults, defineConfig } from "vitest/config";
// vitest does not read .gitignore when looking for tests. A checkout nested
// under .claude/ has its own test/ tree, and without this exclude the suite
// runs once per nested checkout and still reports success.
export default defineConfig({
test: {
exclude: [...configDefaults.exclude, ".claude/**"],
},
});
+6 -8
View File
@@ -528,13 +528,6 @@
resolved "https://registry.yarnpkg.com/@types/json-schema/-/json-schema-7.0.15.tgz#596a1747233694d50f6ad8a7869fcb6f56cf5841"
integrity sha512-5+fP8P8MFNC+AyZCDxrB2pkZFPGzqQWUzpSeuuVLvm8VMcorNYavBqoFcxK8bQz4Qsbn4oUEEem4wDLfcysGHA==
"@types/libsodium-wrappers-sumo@0.8.2":
version "0.8.2"
resolved "https://registry.yarnpkg.com/@types/libsodium-wrappers-sumo/-/libsodium-wrappers-sumo-0.8.2.tgz#488e8747fbb982fe901020b5afeaddfa63da6830"
integrity sha512-uFOBpg/r21hExVlh2ty8YpDfSR+Yy3Jn8XS4+SSjitbhTxdYq+pBz/49XRxyUFe8SzqujHf/Wu0/O4d+FUtNfQ==
dependencies:
libsodium-wrappers-sumo "*"
"@types/node@22.18.13":
version "22.18.13"
resolved "https://registry.yarnpkg.com/@types/node/-/node-22.18.13.tgz#a037c4f474b860be660e05dbe92a9ef945472e28"
@@ -1076,6 +1069,11 @@ fastq@^1.6.0:
dependencies:
reusify "^1.0.4"
fflate@0.8.3:
version "0.8.3"
resolved "https://registry.yarnpkg.com/fflate/-/fflate-0.8.3.tgz#bc27d8eb30343d4d512abb03480202ce65d825fc"
integrity sha512-tbZNuJrLwGUp3zshBtdy4W+ORxZuIh8a5ilyIEQDC5rY1f3U20JMry0Ll3WBzU58EZKsEuJFXhb5gwv8CsPvgA==
file-entry-cache@^8.0.0:
version "8.0.0"
resolved "https://registry.yarnpkg.com/file-entry-cache/-/file-entry-cache-8.0.0.tgz#7787bddcf1131bffb92636c69457bbc0edd6d81f"
@@ -1249,7 +1247,7 @@ libsodium-sumo@^0.8.0:
resolved "https://registry.yarnpkg.com/libsodium-sumo/-/libsodium-sumo-0.8.4.tgz#6d4687781fa0ad398af14a7df872d5c27cf8cd31"
integrity sha512-TMtHShQfVVsaxDygyapvUC3o7YsPgXa/hRWeIgzyFz6w5k/1hirGptCxp1U7XwW3rCskaTTYKgV10v86UiGgNw==
libsodium-wrappers-sumo@*, libsodium-wrappers-sumo@0.8.4:
libsodium-wrappers-sumo@0.8.4:
version "0.8.4"
resolved "https://registry.yarnpkg.com/libsodium-wrappers-sumo/-/libsodium-wrappers-sumo-0.8.4.tgz#6656a3e7e0551ecce08ddee4bfb501a092eac6fa"
integrity sha512-ql7hcgulKZ3ekfa2DGAogcCKsWU0diA/0nArz1CFzh93WQdb46/Kj18ka/Hifq6uA3Ush34Pc6vU/6HXeRwUkg==