diff --git a/README.md b/README.md index b80ae90..58c2bb3 100644 --- a/README.md +++ b/README.md @@ -229,7 +229,7 @@ below belong to quak itself, and go before or after the command's name. ### The commands ``` -quak login +quak login [--server ] quak whoami quak logout quak collections [--json] @@ -246,13 +246,26 @@ Every command but `login` and `logout` needs a saved session. Every command but `login`, `whoami` and `logout` opens the library. How each one gets current data from the server is described under "How a command gets current data". -#### `quak login` +#### `quak login [--server ]` Logs in to an account and saves the session. It reads the email from `QUAK_EMAIL` and the password from `QUAK_PASSWORD`. For each one that is not set, it asks on the terminal. The password shows as `*` characters while it is typed. +quak logs in to Ente's hosted service unless you name a self-hosted Ente server. +Name it with `--server`, or with `QUAK_SERVER` when `--server` is not given. +Give the address of the server's API, for example +`quak login --server https://ente.example.com`. The address is an http or https +URL. It may include a path, but no query or fragment. quak refuses any other +address before it asks for anything. It logs +`Invalid server address "": it must be an http or https URL with no query or fragment`, +and the exit status is 1. + +The session file keeps the server's address. Every later command, cron runs +included, uses that server without further setting. To change server, log in +again. + When the account has a second factor, quak asks for the code. It asks `TOTP code: ` for a time-based code from an authenticator app. An account whose only second factor is a passkey cannot log in with quak. When the account has @@ -597,9 +610,11 @@ Linux that is `$XDG_DATA_HOME/quak/session.json`, or mode `0600`, in a directory it creates with mode `0700`. The file holds the account's email, its user ID, the session token and the -account's three keys, in base64 and unencrypted. Treat it as you would treat the -password. A missing file means "not logged in". A file that cannot be read back -is reported as corrupt, with the reason. Run `quak logout`, then `quak login`. +account's three keys, in base64 and unencrypted. After a login to a self-hosted +server, it also holds the server's address as `apiOrigin`. Treat it as you would +treat the password. A missing file means "not logged in". A file that cannot be +read back is reported as corrupt, with the reason. Run `quak logout`, then +`quak login`. A command that needs a session exits 3 when the file is missing or corrupt. It also exits 3 when the server no longer accepts the session, and prints @@ -617,6 +632,11 @@ Without `--cache-dir`, each account has its own cache directory: the variable is not set; - on macOS, `~/Library/Caches/quak/`. +Two servers can give two accounts the same user ID. So an account on a +self-hosted server has the server's address, URL-encoded, between `quak` and the +user ID. For `https://ente.example.com`, that is +`quak/https%3A%2F%2Fente.example.com/`. + The directory is readable by its owner only (mode `0700`), because it holds decrypted data, file keys included. @@ -641,6 +661,13 @@ the library deletes it and `mldata/` and starts from an empty cache. Cached originals and thumbnails are kept. They are reached only through the records of the current account, so another account's files are never served. +When `metadata.json` belongs to an account on another server, opening the +library also deletes the originals and thumbnails. Two servers can give two +files the same ID, so one server's cached original could be served for the +other's file. quak records the account in `metadata.json` before it stores any +original or thumbnail. It does so also when `quak get` finds a file on an empty +cache. + A file appears in the cache only once it is complete: each one is written to a temporary file and renamed into place. Ente's clients record a hash of each file's bytes at upload, the content hash. Every original whose metadata records @@ -1044,7 +1071,8 @@ onto what the cache holds, empty or not. A refresh then starts only when - `client`, required: the account's `Client`, or anything that satisfies `LibraryClient`. - `cacheDirectory`: where `metadata.json` and the content cache live. Default: - `quak/` under the per-user cache directory (see "The cache"). + `quak/` under the per-user cache directory, with the URL-encoded + `apiOrigin` before `` for a client that has one (see "The cache"). - `downloadDirectory`: the root of the save paths. Default: `photos` in the working directory, resolved once when the library opens; `lib.downloadDirectory` holds the result. An original stored at its save path @@ -1241,7 +1269,8 @@ do. fetch and cache as `photo.original()` and `photo.thumbnail()` do. `file` is an `EnteFile` the caller holds, and the library need not hold it. `quak get` and `quak get-thumb` use them; on an empty cache they pass a file that -`lib.freshFiles()` returned. +`lib.freshFiles()` returned. In a cache that names no account yet, they first +save the client's user ID and `apiOrigin` in `metadata.json`. Lower-level reads return the decrypted model objects, which hold key material: `lib.listCollections()`, `lib.getCollection(id)`, `lib.listFiles(collectionID)`, @@ -1477,6 +1506,14 @@ write wherever it likes. `Client.fromJSON(snapshot, apiOptions?)` restores a working client from that snapshot without logging in again. It checks every field and each key's length first, and throws an error naming the bad field. +A client whose `apiOptions` named an `apiOrigin` puts it in its snapshot. The +restored client then uses that server, even when the `apiOptions` given to +`Client.fromJSON` name another. A snapshot without `apiOrigin` uses the +`apiOrigin` of those `apiOptions`, or the default server. `client.apiOrigin` +holds the one in use, or undefined when none was named. `Client.fromJSON` +refuses an `apiOrigin` that is not an http or https URL, or that has a query or +fragment, as `quak login --server` does. + `Client.fromJSON` works only once libsodium, the cryptography library quak uses, is loaded. `Client.login` loads it, and `Client.fromJSON` does not. A program that restores a session in a new process first runs `await sodium.ready`, with diff --git a/bin/quak.ts b/bin/quak.ts index 9c2a588..cc37e37 100644 --- a/bin/quak.ts +++ b/bin/quak.ts @@ -74,7 +74,11 @@ const context = (): CliContext => ({ sessionDir: paths.data, cacheDir: program.opts().cacheDir, loadSession: (path) => loadSession(path, logRequests(log())), - login: (opts) => Client.login({ ...opts, apiOptions: logRequests(log()) }), + login: (opts) => + Client.login({ + ...opts, + apiOptions: { ...opts.apiOptions, ...logRequests(log()) }, + }), prompt: (message) => input({ message }), promptSecret: (message) => password({ message, mask: true }), }); @@ -85,7 +89,12 @@ const run = (command: Promise): Promise => program .command("login") .description("Log in to an Ente account and save the session") - .action(() => run(loginCommand(context()))); + .option( + "--server ", + "Address of a self-hosted Ente server's API, saved with the session " + + "(default: QUAK_SERVER, then Ente's hosted service)", + ) + .action((opts: { server?: string }) => run(loginCommand(context(), opts))); program .command("whoami") diff --git a/src/cli-commands.ts b/src/cli-commands.ts index 6089b9b..b123ce1 100644 --- a/src/cli-commands.ts +++ b/src/cli-commands.ts @@ -21,6 +21,7 @@ import { extname, join } from "node:path"; import { type Client, type ClientSnapshot, + isServerAddress, type LoginOptions, } from "./client.js"; import { lockBackupDirectory, type BackupLock } from "./backup.js"; @@ -56,7 +57,7 @@ export interface CliContext { // Directory holding `session.json`. sessionDir: string; // The `--cache-dir` global, or undefined to let the library pick its - // per-user default keyed by the account id. + // per-user default keyed by the account's server and user ID. cacheDir?: string; // Reads the session file into a client, or null when there is none. The // CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client. @@ -129,6 +130,7 @@ const parseID = ( // own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch // originals through the on-disk cache. const readLibraryClient = (client: Client): LibraryClient => ({ + apiOrigin: client.apiOrigin, whoami: () => client.whoami(), collectionsSince: (args) => client.collectionsSince(args), filesSince: (args) => client.filesSince(args), @@ -155,8 +157,23 @@ const openReadLibrary = (ctx: CliContext, client: Client): Promise => onProgress: logRefresh(ctx.log), }); -export const loginCommand = async (ctx: CliContext): Promise => { +// Logs in to the server `--server` or `QUAK_SERVER` names, the flag first, or +// to Ente's hosted service when neither is set. The saved session keeps the +// server, so every later command uses it. +export const loginCommand = async ( + ctx: CliContext, + opts: { server?: string }, +): Promise => { await init(); + const server = opts.server ?? process.env.QUAK_SERVER; + if (server !== undefined && !isServerAddress(server)) { + ctx.log( + "fatal", + `Invalid server address "${server}": it must be an http or ` + + `https URL with no query or fragment`, + ); + return 1; + } const email = process.env.QUAK_EMAIL ?? (await ctx.prompt("Email")); const password = process.env.QUAK_PASSWORD ?? (await ctx.promptSecret("Password")); @@ -168,6 +185,8 @@ export const loginCommand = async (ctx: CliContext): Promise => { password, totp: async () => ctx.prompt("TOTP code: "), emailOTP: async () => ctx.prompt("Email verification code: "), + apiOptions: + server === undefined ? undefined : { apiOrigin: server }, }); saveSession(ctx.sessionDir, client.toJSON()); @@ -208,7 +227,10 @@ export const logoutCommand = async (ctx: CliContext): Promise => { try { const client = ctx.loadSession(path); if (client) { - cacheDir ??= defaultCacheDirectory(client.whoami().userID); + cacheDir ??= defaultCacheDirectory( + client.whoami().userID, + client.apiOrigin, + ); await client.logoutOnServer(); client.logout(); } diff --git a/src/client.ts b/src/client.ts index 2f5ea8d..6192e71 100644 --- a/src/client.ts +++ b/src/client.ts @@ -40,6 +40,9 @@ export interface ClientSnapshot { masterKey: string; secretKey: string; publicKey: string; + // The `apiOrigin` the client was given, so a restored client talks to + // the server that issued its token. Absent means the default server. + apiOrigin?: string; } // The result of a resumable enumeration. Live decrypted records and deleted @@ -66,7 +69,19 @@ export interface FilesPage { const errorMessage = (err: unknown): string => err instanceof Error ? err.message : String(err); +// Whether `text` can be an Ente server's address: an http or https URL, which +// may have a path, and no query or fragment, which request URLs could not +// carry. `quak login --server` and `Client.fromJSON` both check with it. +export const isServerAddress = (text: string): boolean => + URL.canParse(text) && + ["http:", "https:"].includes(new URL(text).protocol) && + !/[?#]/.test(text); + export class Client { + // The `apiOrigin` the client was given, or undefined when it was given + // none, which means Ente's hosted service. The library keeps a cache per + // server as well as per user ID. + readonly apiOrigin: string | undefined; private readonly api: ApiClient; private readonly email: string; private readonly userID: number; @@ -82,6 +97,7 @@ export class Client { masterKey: Uint8Array, secretKey: Uint8Array, publicKey: Uint8Array, + apiOrigin: string | undefined, ) { this.api = api; this.email = email; @@ -89,6 +105,7 @@ export class Client { this.masterKey = masterKey; this.secretKey = secretKey; this.publicKey = publicKey; + this.apiOrigin = apiOrigin; } static async login(opts: LoginOptions): Promise { @@ -130,12 +147,15 @@ export class Client { unwrapped.masterKey, unwrapped.secretKey, unwrapped.publicKey, + opts.apiOptions?.apiOrigin, ); } // Restore a client from a `toJSON()` snapshot. The snapshot usually comes // straight from `JSON.parse` of a file on disk, so every field is checked - // before use; a bad one throws an error naming it. Needs `init()` first. + // before use; a bad one throws an error naming it. The snapshot's + // `apiOrigin`, when it has one, is used instead of `apiOptions.apiOrigin`. + // Needs `init()` first. static fromJSON(snapshot: unknown, apiOptions?: ApiClientOptions): Client { const invalid = (field: string, problem: string): Error => new Error(`Invalid session data: ${field} ${problem}`); @@ -152,6 +172,17 @@ export class Client { if (!Number.isInteger(s.userID)) { throw invalid("userID", "must be an integer"); } + if ( + s.apiOrigin !== undefined && + (typeof s.apiOrigin !== "string" || !isServerAddress(s.apiOrigin)) + ) { + throw invalid( + "apiOrigin", + "must be an http or https URL with no query or fragment", + ); + } + const apiOrigin = + (s.apiOrigin as string | undefined) ?? apiOptions?.apiOrigin; const key = (field: string): Uint8Array => { const value = s[field]; if (typeof value !== "string") { @@ -175,6 +206,7 @@ export class Client { const api = new ApiClient({ ...apiOptions, + apiOrigin, authToken: s.token as string, }); return new Client( @@ -184,6 +216,7 @@ export class Client { key("masterKey"), key("secretKey"), key("publicKey"), + apiOrigin, ); } @@ -222,6 +255,9 @@ export class Client { masterKey: toBase64(this.masterKey), secretKey: toBase64(this.secretKey), publicKey: toBase64(this.publicKey), + ...(this.apiOrigin === undefined + ? {} + : { apiOrigin: this.apiOrigin }), }; } diff --git a/src/library/index.ts b/src/library/index.ts index 23239e9..ac068d9 100644 --- a/src/library/index.ts +++ b/src/library/index.ts @@ -119,9 +119,18 @@ export { export const DEFAULT_REFRESH_INTERVAL_SECONDS = 3; // The account's cache directory when `cacheDirectory` is not given: the -// env-paths cache directory plus the user id, so each account has its own. -export const defaultCacheDirectory = (userID: number): string => - join(envPaths("quak", { suffix: "" }).cache, String(userID)); +// env-paths cache directory plus the user id, so each account has its own. An +// account on a self-hosted server has the server's address, URL-encoded, in +// between, because two servers can give two accounts the same user id. +export const defaultCacheDirectory = ( + userID: number, + apiOrigin?: string, +): string => { + const cache = envPaths("quak", { suffix: "" }).cache; + return apiOrigin === undefined + ? join(cache, String(userID)) + : join(cache, encodeURIComponent(apiOrigin), String(userID)); +}; // Project a metadata store into by-id records, filling each record's cache // paths from the content cache when one is given. Shared by the live read @@ -144,6 +153,9 @@ const deriveRecordsFromStore = ( // tests drive a mock with no crypto or network; the real `Client` satisfies it // structurally. export interface LibraryClient { + // The server the client talks to, as `Client.apiOrigin` names it. Absent + // means Ente's hosted service. + readonly apiOrigin?: string; whoami(): { email: string; userID: number }; collectionsSince(args: { sinceTime: number }): Promise; filesSince(args: { @@ -196,8 +208,8 @@ export type RefreshProgressCallback = (event: RefreshEvent) => void; export interface LibraryOptions { client: LibraryClient; - // Where `metadata.json` lives. Defaults to the env-paths cache directory - // plus the user id, so each account has its own cache. + // Where `metadata.json` lives. Defaults to `defaultCacheDirectory`, so + // each account has its own cache. cacheDirectory?: string; // The root of every photo's save path, where `Photo.download()` and // `lib.backup()` put originals. Defaults to `photos` in the working @@ -300,6 +312,7 @@ export class Library { // available (a metadata-only client with no explicit source). private readonly cache?: ContentCache; private readonly userID: number; + private readonly apiOrigin: string | undefined; private readonly intervalMs: number; private readonly onProgress?: RefreshProgressCallback; private readonly pools: RequestPools; @@ -346,6 +359,7 @@ export class Library { client: LibraryClient; store: MetadataStore; userID: number; + apiOrigin: string | undefined; cacheDirectory: string; downloadDirectory: string; intervalMs: number; @@ -358,6 +372,7 @@ export class Library { this.client = args.client; this.store = args.store; this.userID = args.userID; + this.apiOrigin = args.apiOrigin; this.cacheDirectory = args.cacheDirectory; this.downloadDirectory = args.downloadDirectory; this.intervalMs = args.intervalMs; @@ -406,8 +421,9 @@ export class Library { // refresh: `open()` returns onto the cache as it is. static async open(opts: LibraryOptions): Promise { const { userID } = opts.client.whoami(); + const { apiOrigin } = opts.client; const cacheDirectory = - opts.cacheDirectory ?? defaultCacheDirectory(userID); + opts.cacheDirectory ?? defaultCacheDirectory(userID, apiOrigin); if (opts.downloadDirectory === "") { throw new Error( "library: downloadDirectory is empty (leave it out to save " + @@ -419,14 +435,24 @@ export class Library { let store = await MetadataStore.load(metadataPath); // A cache directory given explicitly can hold another account's cache. // Its records and cursor are not this account's, so delete it and the - // ML data beside it and start empty. A user ID of 0 means the cache - // was never refreshed and so holds nothing to discard. - if (store.userID !== 0 && store.userID !== userID) { - await rm(metadataPath, { force: true }); - await rm(join(cacheDirectory, "mldata"), { - recursive: true, - force: true, - }); + // ML data beside it and start empty. A cache from another server goes + // whole: two servers can give two files the same ID, so its originals + // and thumbnails could be served for this account's files. A user ID + // of 0 means the cache names no account, so nothing is stored in it. + if ( + store.userID !== 0 && + (store.userID !== userID || store.apiOrigin !== apiOrigin) + ) { + const discard = ["metadata.json", "mldata"]; + if (store.apiOrigin !== apiOrigin) { + discard.push("originals", "thumbnails"); + } + for (const name of discard) { + await rm(join(cacheDirectory, name), { + recursive: true, + force: true, + }); + } store = await MetadataStore.load(metadataPath); } const intervalMs = @@ -490,6 +516,7 @@ export class Library { client: opts.client, store, userID, + apiOrigin, cacheDirectory, downloadDirectory, intervalMs, @@ -509,9 +536,11 @@ export class Library { if (opts.refreshOnOpen === false) { // The caller asks for each refresh it needs. lib.scheduleNext(); - } else if (store.loadedFromDisk) { + } else if (store.listCollections().length > 0) { // An existing copy already answers reads; refresh in the background - // and start the interval once that first cycle settles. + // and start the interval once that first cycle settles. A cache + // that names the account but holds no collection has nothing to + // answer with, such as after `originalOf` on an empty cache. void lib.runRefresh().then(() => lib.scheduleNext()); } else { // Nothing was cached: wait for the first refresh to fill the store @@ -658,6 +687,7 @@ export class Library { "originalOf requires a library opened with a content cache", ); } + await this.saveAccountInEmptyCache(); return this.cache.originalOf(file, opts); } @@ -673,9 +703,23 @@ export class Library { "thumbnailOf requires a library opened with a content cache", ); } + await this.saveAccountInEmptyCache(); return this.cache.thumbnailOf(file, opts); } + // Saves the account's user ID and server in a cache that names no account + // yet, before `originalOf` or `thumbnailOf` stores a file there. A later + // `open()` for an account on another server then deletes that file (see + // `open()`), although no refresh has stored any records. + private async saveAccountInEmptyCache(): Promise { + if (this.store.userID !== 0) return; + this.store.userID = this.userID; + this.store.apiOrigin = this.apiOrigin; + this.unsaved = true; + await this.store.save(); + this.unsaved = false; + } + // Back up every in-scope file to `opts.downloadDirectory`, or else the // library's, each original at its save path, with a durable failure // ledger (issue #51). Takes the lock in that directory first, unless the @@ -857,8 +901,12 @@ export class Library { // actually changed. let changed = false; - if (this.store.userID !== this.userID) { + if ( + this.store.userID !== this.userID || + this.store.apiOrigin !== this.apiOrigin + ) { this.store.userID = this.userID; + this.store.apiOrigin = this.apiOrigin; changed = true; } diff --git a/src/library/store.ts b/src/library/store.ts index fc59b78..068cf37 100644 --- a/src/library/store.ts +++ b/src/library/store.ts @@ -1,11 +1,12 @@ // On-disk JSON metadata store for the local cache. // // The store keeps one `metadata.json` file holding the account's server -// state: the user id, a schema version, the cursor for the incremental -// collections listing, the decrypted collection and file records, and the -// collections and files the refresh could not decrypt. The whole file is read -// into RAM on load and rewritten as a whole on save; there is no partial -// update and no lock file. A separate refresh unit populates the store from +// state: the user id, the server's address for an account on a self-hosted +// server, a schema version, the cursor for the incremental collections +// listing, the decrypted collection and file records, and the collections and +// files the refresh could not decrypt. The whole file is read into RAM on +// load and rewritten as a whole on save; there is no partial update and no +// lock file. A separate refresh unit populates the store from // the server — this module only stores what it is given. // // The file is a cache, so it is never trusted to exist or to be intact: a @@ -46,6 +47,7 @@ export interface Undecryptable { interface StoredMetadata { schemaVersion: number; userID: number; + apiOrigin?: string; collectionsSinceTime: Microseconds; collections: StoredCollection[]; files: StoredFile[]; @@ -68,14 +70,10 @@ export class MetadataStore { readonly path: string; readonly schemaVersion = METADATA_SCHEMA_VERSION; userID = 0; + // The account's `apiOrigin`, or undefined for Ente's hosted service. + apiOrigin: string | undefined; collectionsSinceTime: Microseconds = 0; - // True when `load` populated this store from a valid existing file; false - // on a first run or a missing/corrupt/wrong-version file that loaded empty. - // `Library.open` reads it to decide whether the first refresh may run in - // the background (an existing copy already serves reads) or must be awaited. - loadedFromDisk = false; - private readonly collections = new Map(); private readonly files = new Map(); private undecryptable: Undecryptable[] = []; @@ -102,8 +100,8 @@ export class MetadataStore { if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) { return store; } - store.loadedFromDisk = true; store.userID = parsed.userID ?? 0; + store.apiOrigin = parsed.apiOrigin; store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0; for (const stored of parsed.collections ?? []) { const collection: Collection = { @@ -143,6 +141,7 @@ export class MetadataStore { const model: StoredMetadata = { schemaVersion: METADATA_SCHEMA_VERSION, userID: this.userID, + apiOrigin: this.apiOrigin, collectionsSinceTime: this.collectionsSinceTime, collections: [...this.collections.values()].map((c) => ({ ...c, diff --git a/test/cli/bin.test.ts b/test/cli/bin.test.ts index 4f0fd4a..98464e9 100644 --- a/test/cli/bin.test.ts +++ b/test/cli/bin.test.ts @@ -2,8 +2,9 @@ * Tests for the retry options `bin/quak.ts` loads the saved session with * (`quak backup` gets the unattended ones, every other command the default), * each with the callbacks that log its requests and retries, for `quak files` - * taking its collection ID as an optional argument, and for `quak get` and - * `quak get-thumb` refusing `--collection`. + * taking its collection ID as an optional argument, for `quak get` and + * `quak get-thumb` refusing `--collection`, and for `quak login` taking + * `--server`. * * Each test runs `bin/quak.ts`, as the smoke test does, with its session loader * replaced by one that records the options it is given and reports no session, @@ -117,3 +118,13 @@ describe("bin/quak.ts get and get-thumb", () => { }, ); }); + +// The command refuses an address that is not a URL before it asks for anything +// or sends any request. +describe("bin/quak.ts login", () => { + it("hands --server to the command", async () => { + expect(await quak("login", "--server", "ente.example.com")).toMatch( + /^Invalid server address "ente\.example\.com"/, + ); + }); +}); diff --git a/test/cli/commands.test.ts b/test/cli/commands.test.ts index d3c2530..044b694 100644 --- a/test/cli/commands.test.ts +++ b/test/cli/commands.test.ts @@ -328,6 +328,7 @@ describe("login", () => { prompts = []; vi.stubEnv("QUAK_EMAIL", "cli@example.com"); vi.stubEnv("QUAK_PASSWORD", "hunter2"); + vi.stubEnv("QUAK_SERVER", undefined); }); afterEach(() => { @@ -340,12 +341,14 @@ describe("login", () => { calls.push(opts); return loggedIn; }); - expect(await loginCommand(ctx)).toBe(0); + expect(await loginCommand(ctx, {})).toBe(0); expect(prompts).toEqual([]); expect(calls).toHaveLength(1); expect(calls[0]!.email).toBe("cli@example.com"); expect(calls[0]!.password).toBe("hunter2"); + // No server named: the client uses Ente's hosted service. + expect(calls[0]!.apiOptions).toBeUndefined(); const path = join(ctx.sessionDir, "session.json"); expect(stderr.text).toBe( "Authenticating...\n" + @@ -356,7 +359,7 @@ describe("login", () => { it("saves the session with mode 0600 in a directory with mode 0700", async () => { const ctx = loginContext(async () => loggedIn); - expect(await loginCommand(ctx)).toBe(0); + expect(await loginCommand(ctx, {})).toBe(0); expect(statSync(ctx.sessionDir).mode & 0o777).toBe(0o700); const path = join(ctx.sessionDir, "session.json"); @@ -370,7 +373,7 @@ describe("login", () => { code = await opts.totp!(); return loggedIn; }); - expect(await loginCommand(ctx)).toBe(0); + expect(await loginCommand(ctx, {})).toBe(0); expect(prompts).toEqual(["TOTP code: "]); expect(code).toBe("123456"); @@ -380,13 +383,68 @@ describe("login", () => { const ctx = loginContext(async () => { throw new Error("HTTP 401 from server"); }); - expect(await loginCommand(ctx)).toBe(1); + expect(await loginCommand(ctx, {})).toBe(1); expect(stderr.text).toBe( "Authenticating...\nLogin failed: HTTP 401 from server\n", ); expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false); }); + + it("logs in to the server --server names, ahead of QUAK_SERVER", async () => { + vi.stubEnv("QUAK_SERVER", "https://other.example.com"); + const calls: LoginOptions[] = []; + const ctx = loginContext(async (opts) => { + calls.push(opts); + return loggedIn; + }); + expect( + await loginCommand(ctx, { server: "https://ente.example.com" }), + ).toBe(0); + + expect(calls[0]!.apiOptions).toEqual({ + apiOrigin: "https://ente.example.com", + }); + }); + + it("logs in to the server QUAK_SERVER names without --server", async () => { + vi.stubEnv("QUAK_SERVER", "http://192.0.2.1:8080/ente"); + const calls: LoginOptions[] = []; + const ctx = loginContext(async (opts) => { + calls.push(opts); + return loggedIn; + }); + expect(await loginCommand(ctx, {})).toBe(0); + + expect(calls[0]!.apiOptions).toEqual({ + apiOrigin: "http://192.0.2.1:8080/ente", + }); + }); + + it.each([ + "", + "ente.example.com", + "ftp://ente.example.com", + "https://ente.example.com/?region=eu", + "https://ente.example.com/#top", + ])( + "refuses the server address %j before asking for anything", + async (server) => { + const ctx = loginContext(async () => { + throw new Error("login must not be called"); + }); + expect(await loginCommand(ctx, { server })).toBe(1); + + expect(prompts).toEqual([]); + expect(stderr.text).toBe( + `Invalid server address "${server}": it must be an http or ` + + `https URL with no query or fragment\n`, + ); + expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe( + false, + ); + }, + ); }); // These use a real client read from the session file, over a fake API that @@ -461,6 +519,16 @@ describe("logout", () => { ); }); + it("names a self-hosted account's default cache directory, under its server's address", async () => { + const ctx = { ...logoutContext(200), cacheDir: undefined }; + const apiOrigin = "https://ente.example.com"; + saveSession(ctx.sessionDir, { ...snapshot, apiOrigin }); + expect(await logoutCommand(ctx)).toBe(0); + expect(stderr.text).toContain( + `Cache directory ${defaultCacheDirectory(USER_ID, apiOrigin)} still holds decrypted data`, + ); + }); + it("without a session says so, calls nothing and exits 0", async () => { expect(await logoutCommand(logoutContext(200))).toBe(0); expect(requests).toHaveLength(0); @@ -1618,18 +1686,19 @@ describe("an account with many albums", () => { // encrypted as the server stores it. The original's bytes are the title. // These tests fetch no thumbnail. The server records a thumbnail size, so // `helper fix-missing-thumbnails` downloads the original before it skips - // the photo as not a JPEG. - const photoRow = (photo: { - id: number; - collectionID: number; - title: string; - updationTime: number; - }): { row: Record; original: Uint8Array } => { + // the photo as not a JPEG. `albumKey` is that album's key, by default the + // one in `albumKeys`. + const photoRow = ( + photo: { + id: number; + collectionID: number; + title: string; + updationTime: number; + }, + albumKey = albumKeys.get(photo.collectionID)!, + ): { row: Record; original: Uint8Array } => { const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen(); - const photoKeyBox = encryptSecretbox( - photoKey, - albumKeys.get(photo.collectionID)!, - ); + const photoKeyBox = encryptSecretbox(photoKey, albumKey); const metadata = encryptStreamBlob( new TextEncoder().encode( JSON.stringify({ @@ -1742,6 +1811,77 @@ describe("an account with many albums", () => { }; }; + // The albums in the cache's `metadata.json`. A search on an empty cache + // stores none there, only the account whose original or thumbnail it + // stored. + const cachedAlbums = (): unknown[] => + JSON.parse(readFileSync(join(root, "cache", "metadata.json"), "utf-8")) + .collections; + + // A context whose login and saved session send each request to the server + // its host names: `ente.example.com` is the fake server above, and + // `photos.example.org` a second one. Every account the fake server logs in + // has user ID 42, so the second server's account has the first one's user + // ID. Its one album holds photo 200, `other-200.jpg`; the first server + // gives ID 200 to `photo-2.jpg`. + const twoServerContext = async (): Promise => { + const account = await buildAccount(PASSWORD); + const albumKey = sodium.crypto_secretbox_keygen(); + const albumKeyBox = encryptSecretbox(albumKey, account.masterKey); + const name = encryptSecretbox( + new TextEncoder().encode("Other album"), + albumKey, + ); + const { row, original } = photoRow( + { + id: 200, + collectionID: 1, + title: "other-200.jpg", + updationTime: 1, + }, + albumKey, + ); + const other = buildMetaFetch({ + verifier: account.verifier, + srpAttributes: account.srpAttributes, + keyAttributes: account.keyAttributes, + encryptedToken: account.encryptedToken, + collections: [ + { + id: 1, + owner: { id: USER_ID }, + encryptedKey: toBase64(albumKeyBox.ciphertext), + keyDecryptionNonce: toBase64(albumKeyBox.nonce), + encryptedName: toBase64(name.ciphertext), + nameDecryptionNonce: toBase64(name.nonce), + type: "album", + updationTime: 1, + }, + ], + filesByCollection: { 1: [row] }, + encryptedMLData: {}, + fileCiphertexts: { 200: original }, + }); + const fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + const { host } = new URL( + input instanceof Request ? input.url : input, + ); + return (host === "photos.example.org" ? other : server)( + input, + init, + ); + }) as typeof globalThis.fetch; + return { + ...context(), + login: (opts) => + Client.login({ + ...opts, + apiOptions: { ...opts.apiOptions, fetch }, + }), + loadSession: (path) => loadSession(path, { fetch }), + }; + }; + it("collections lists every album from one request, fetching no album's files", async () => { const ctx = accountContext(); expect(await collectionsCommand(ctx, {})).toBe(0); @@ -1848,7 +1988,100 @@ describe("an account with many albums", () => { "/collections/v2/diff?collectionID=2&sinceTime=0", "/?fileID=200", ]); - expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false); + expect(cachedAlbums()).toEqual([]); + }); + + // A self-hosted server serves downloads under its own address, at + // `/files/download/`, so no request goes to Ente's hosted service. + it("login --server saves the server, and a later get sends every request to it", async () => { + vi.stubEnv("QUAK_EMAIL", "many@example.com"); + vi.stubEnv("QUAK_PASSWORD", PASSWORD); + vi.stubEnv("QUAK_SERVER", undefined); + const hosts = new Set(); + const fetch = (async (input: RequestInfo | URL, init?: RequestInit) => { + hosts.add( + new URL(input instanceof Request ? input.url : input).host, + ); + return server(input, init); + }) as typeof globalThis.fetch; + const ctx: CliContext = { + ...context(), + login: (opts) => + Client.login({ + ...opts, + apiOptions: { ...opts.apiOptions, fetch }, + }), + loadSession: (path) => loadSession(path, { fetch }), + }; + const out = join(root, "photo.jpg"); + try { + const opts = { server: "https://ente.example.com" }; + expect(await loginCommand(ctx, opts)).toBe(0); + requests.length = 0; + expect(await getCommand(ctx, "200", { out })).toBe(0); + } finally { + vi.unstubAllEnvs(); + } + + const path = join(ctx.sessionDir, "session.json"); + expect(JSON.parse(readFileSync(path, "utf-8")).apiOrigin).toBe( + "https://ente.example.com", + ); + expect(readFileSync(out, "utf-8")).toBe("photo-2.jpg"); + expect(requests.at(-1)).toBe("/files/download/200"); + expect([...hosts]).toEqual(["ente.example.com"]); + }); + + // The two servers' accounts share the cache directory `--cache-dir` names. + it("after a login to another server whose account has the same user ID, files and get see none of the first server's albums and originals", async () => { + const ctx = await twoServerContext(); + const first = join(root, "first.jpg"); + const second = join(root, "second.jpg"); + vi.stubEnv("QUAK_EMAIL", "many@example.com"); + vi.stubEnv("QUAK_PASSWORD", PASSWORD); + vi.stubEnv("QUAK_SERVER", undefined); + try { + const opts = { server: "https://ente.example.com" }; + expect(await loginCommand(ctx, opts)).toBe(0); + expect(await filesCommand(ctx, undefined, {})).toBe(0); + expect(await getCommand(ctx, "200", { out: first })).toBe(0); + + const otherOpts = { server: "https://photos.example.org" }; + expect(await loginCommand(ctx, otherOpts)).toBe(0); + stdout.text = ""; + expect(await filesCommand(ctx, undefined, {})).toBe(0); + expect(await getCommand(ctx, "200", { out: second })).toBe(0); + } finally { + vi.unstubAllEnvs(); + } + + expect(stdout.text).toBe("200\timage\tother-200.jpg\n"); + expect(readFileSync(first, "utf-8")).toBe("photo-2.jpg"); + expect(readFileSync(second, "utf-8")).toBe("other-200.jpg"); + }); + + // `get` on an empty cache stores the original but no album. + it("get on an empty cache, after the same on another server whose account has the same user ID, saves this server's photo", async () => { + const ctx = await twoServerContext(); + const first = join(root, "first.jpg"); + const second = join(root, "second.jpg"); + vi.stubEnv("QUAK_EMAIL", "many@example.com"); + vi.stubEnv("QUAK_PASSWORD", PASSWORD); + vi.stubEnv("QUAK_SERVER", undefined); + try { + const opts = { server: "https://ente.example.com" }; + expect(await loginCommand(ctx, opts)).toBe(0); + expect(await getCommand(ctx, "200", { out: first })).toBe(0); + + const otherOpts = { server: "https://photos.example.org" }; + expect(await loginCommand(ctx, otherOpts)).toBe(0); + expect(await getCommand(ctx, "200", { out: second })).toBe(0); + } finally { + vi.unstubAllEnvs(); + } + + expect(readFileSync(first, "utf-8")).toBe("photo-2.jpg"); + expect(readFileSync(second, "utf-8")).toBe("other-200.jpg"); }); // `files` first fills the cache, so `get` waits for a refresh. No album has @@ -1906,7 +2139,7 @@ describe("an account with many albums", () => { "/collections/v2/diff?collectionID=39&sinceTime=0", "/?fileID=3900", ]); - expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false); + expect(cachedAlbums()).toEqual([]); }); // `files` first fills the cache, so the helper waits for a refresh. No diff --git a/test/client/session.test.ts b/test/client/session.test.ts index 5a14775..8d4b124 100644 --- a/test/client/session.test.ts +++ b/test/client/session.test.ts @@ -47,6 +47,20 @@ describe("Client.toJSON", () => { expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot); }); + it("round-trips a snapshot that names a server", () => { + const snapshot = { + ...validSnapshot(), + apiOrigin: "https://ente.example.com", + }; + expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot); + }); + + it("names the server apiOptions gave fromJSON", () => { + const apiOrigin = "https://ente.example.com"; + const client = Client.fromJSON(validSnapshot(), { apiOrigin }); + expect(client.toJSON().apiOrigin).toBe(apiOrigin); + }); + it("throws instead of emitting a snapshot without a token", () => { const client = Client.fromJSON(validSnapshot()); client.getApiClient().clearAuthToken(); @@ -77,6 +91,10 @@ describe("Client.fromJSON", () => { ["publicKey", undefined], ["publicKey", "not base64!"], ["publicKey", shortKey], + ["apiOrigin", 7], + ["apiOrigin", ""], + ["apiOrigin", "ente.example.com"], + ["apiOrigin", "ftp://ente.example.com"], ])("rejects %s = %j, naming the field", (field, value) => { const snapshot: Record = { ...validSnapshot() }; snapshot[field] = value; @@ -85,6 +103,30 @@ describe("Client.fromJSON", () => { ); }); + // The token belongs to the snapshot's server, so that server wins over + // the one in `apiOptions`. + it("sends requests to the snapshot's server", async () => { + const urls: string[] = []; + const client = Client.fromJSON( + { ...validSnapshot(), apiOrigin: "https://ente.example.com/" }, + { + apiOrigin: "https://other.example.com", + fetch: async (input) => { + urls.push(String(input)); + return new Response(JSON.stringify({ collections: [] }), { + status: 200, + headers: { "content-type": "application/json" }, + }); + }, + }, + ); + + await client.listCollections(); + expect(urls).toEqual([ + "https://ente.example.com/collections/v2?sinceTime=0", + ]); + }); + it.each([null, "a string", 42])("rejects a non-object %j", (value) => { expect(() => Client.fromJSON(value)).toThrow( /^Invalid session data: not a JSON object/, diff --git a/test/library/library.test.ts b/test/library/library.test.ts index def8c33..04f313f 100644 --- a/test/library/library.test.ts +++ b/test/library/library.test.ts @@ -19,16 +19,20 @@ * refresh fails (offline start from cache). * 5. `close()` stops the timer and is idempotent, and its promise resolves * only once an in-flight refresh has written the cache file. - * 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id. - * 7. `open()` branches on the cache: an empty cache awaits the first refresh - * (it has nothing to serve yet); an existing cache serves its copy at once - * and refreshes in the background, so a slow or dead server never stalls - * opening. + * 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id, + * with a self-hosted server's address before the user id. + * 7. `open()` branches on the cache: a cache that holds no collection awaits + * the first refresh (it has nothing to serve yet); an existing cache serves + * its copy at once and refreshes in the background, so a slow or dead + * server never stalls opening. * 8. A save failure that leaves RAM ahead of disk keeps `status().lastError` * set and keeps retrying the write; a later empty refresh does not clear it. * 9. With `refreshOnOpen: false`, `open()` starts no refresh at all, and a * refresh reports through `onProgress` each collection whose files it * fetched. + * 10. A cache another account filled starts empty. One filled from another + * server also loses its originals and thumbnails, since two servers can + * give two files the same ID. * * The client is a mock: no crypto, no network. It serves scripted pages and * records the `sinceTime` each call carried so cursor threading is provable. @@ -49,7 +53,13 @@ */ import { describe, it, expect, beforeEach, afterEach, vi } from "vitest"; -import { mkdtempSync, rmSync } from "node:fs"; +import { + existsSync, + mkdirSync, + mkdtempSync, + rmSync, + writeFileSync, +} from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import envPaths from "env-paths"; @@ -107,6 +117,7 @@ const file = ( */ class MockClient { userID = USER_ID; + apiOrigin?: string; failCollections = false; collectionsQueue: CollectionsPage[] = []; filesByCollection = new Map(); @@ -584,6 +595,44 @@ describe("Library.open and background refresh", () => { } }); + // `originalOf` on an empty cache leaves such a cache behind: it names the + // account before it stores the original. + it("awaits the first refresh on a cache that names the account but holds no collection", async () => { + const seed = await MetadataStore.load( + join(cacheDirectory, "metadata.json"), + ); + seed.userID = USER_ID; + await seed.save(); + let releaseFirstFetch: (page: CollectionsPage) => void = () => {}; + const gate = new Promise((resolve) => { + releaseFirstFetch = resolve; + }); + const client = new MockClient(); + client.collectionsSince = async () => gate; + + let opened = false; + const openPromise = Library.open({ client, cacheDirectory }).then( + (l) => { + opened = true; + return l; + }, + ); + await new Promise((r) => setTimeout(r, 20)); + expect(opened).toBe(false); + + releaseFirstFetch({ + collections: [collection(1, 100)], + deleted: [], + cursor: 100, + }); + const lib = await openPromise; + try { + expect(lib.listCollections().map((c) => c.id)).toEqual([1]); + } finally { + await lib.close(); + } + }); + // A caller that asks for each refresh it needs, as the CLI's read commands // do, opens without one: on an empty cache the refresh `open()` would // await fetches the files of every collection, which `quak collections` @@ -842,4 +891,94 @@ describe("Library.open and background refresh", () => { else process.env.XDG_CACHE_HOME = prev; } }); + + it("puts a self-hosted server's address, URL-encoded, before the user id in the default cacheDirectory", async () => { + const xdg = join(dir, "xdg-cache"); + const prev = process.env.XDG_CACHE_HOME; + process.env.XDG_CACHE_HOME = xdg; + try { + const client = new MockClient(); + client.apiOrigin = "https://ente.example.com:8443/api"; + const lib = await Library.open({ client }); + try { + expect(lib.cacheDirectory).toBe( + join( + envPaths("quak", { suffix: "" }).cache, + "https%3A%2F%2Fente.example.com%3A8443%2Fapi", + String(USER_ID), + ), + ); + expect(lib.cacheDirectory.startsWith(xdg)).toBe(true); + } finally { + await lib.close(); + } + } finally { + if (prev === undefined) delete process.env.XDG_CACHE_HOME; + else process.env.XDG_CACHE_HOME = prev; + } + }); + + // A cache that account USER_ID on `apiOrigin` filled: its records, and an + // original and a thumbnail of file 1001. + const fillCache = async (apiOrigin?: string): Promise => { + const seed = await MetadataStore.load( + join(cacheDirectory, "metadata.json"), + ); + seed.userID = USER_ID; + seed.apiOrigin = apiOrigin; + seed.collectionsSinceTime = 500; + seed.putCollection(collection(1, 400)); + seed.putFile(file(1001, 1, 400)); + await seed.save(); + for (const name of ["originals", "thumbnails"]) { + mkdirSync(join(cacheDirectory, name)); + writeFileSync(join(cacheDirectory, name, "1001.jpg"), "1001"); + } + }; + + it("deletes the whole cache an account on another server filled", async () => { + await fillCache("https://ente.example.com"); + const client = new MockClient(); + client.apiOrigin = "https://photos.example.org"; + const lib = await Library.open({ + client, + cacheDirectory, + refreshIntervalSeconds: 3600, + }); + try { + expect(client.collectionsSinceTimes).toEqual([0]); + expect(lib.listCollections()).toEqual([]); + for (const name of ["originals", "thumbnails"]) { + expect(existsSync(join(cacheDirectory, name, "1001.jpg"))).toBe( + false, + ); + } + } finally { + await lib.close(); + } + }); + + // A server gives each file its own ID, so another account's cached file + // is never served for this account's. + it("keeps the originals and thumbnails another account on the same server cached", async () => { + await fillCache(); + const client = new MockClient(); + client.userID = USER_ID + 1; + const lib = await Library.open({ + client, + cacheDirectory, + refreshIntervalSeconds: 3600, + }); + try { + expect(client.collectionsSinceTimes).toEqual([0]); + expect(lib.listCollections()).toEqual([]); + for (const name of ["originals", "thumbnails"]) { + expect(existsSync(join(cacheDirectory, name, "1001.jpg"))).toBe( + true, + ); + } + } finally { + await lib.close(); + } + }); });