Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
6f8f85fff5 |
No files matched your search
@@ -632,6 +632,11 @@ Without `--cache-dir`, each account has its own cache directory:
|
|||||||
the variable is not set;
|
the variable is not set;
|
||||||
- on macOS, `~/Library/Caches/quak/<userID>`.
|
- on macOS, `~/Library/Caches/quak/<userID>`.
|
||||||
|
|
||||||
|
Two servers can give two accounts the same user ID. So an account on a
|
||||||
|
self-hosted server has the server's address, URL-encoded, between `quak` and the
|
||||||
|
user ID. For `https://ente.example.com`, that is
|
||||||
|
`quak/https%3A%2F%2Fente.example.com/<userID>`.
|
||||||
|
|
||||||
The directory is readable by its owner only (mode `0700`), because it holds
|
The directory is readable by its owner only (mode `0700`), because it holds
|
||||||
decrypted data, file keys included.
|
decrypted data, file keys included.
|
||||||
|
|
||||||
@@ -656,6 +661,13 @@ the library deletes it and `mldata/` and starts from an empty cache. Cached
|
|||||||
originals and thumbnails are kept. They are reached only through the records of
|
originals and thumbnails are kept. They are reached only through the records of
|
||||||
the current account, so another account's files are never served.
|
the current account, so another account's files are never served.
|
||||||
|
|
||||||
|
When `metadata.json` belongs to an account on another server, opening the
|
||||||
|
library also deletes the originals and thumbnails. Two servers can give two
|
||||||
|
files the same ID, so one server's cached original could be served for the
|
||||||
|
other's file. quak records the account in `metadata.json` before it stores any
|
||||||
|
original or thumbnail. It does so also when `quak get` finds a file on an empty
|
||||||
|
cache.
|
||||||
|
|
||||||
A file appears in the cache only once it is complete: each one is written to a
|
A file appears in the cache only once it is complete: each one is written to a
|
||||||
temporary file and renamed into place. Ente's clients record a hash of each
|
temporary file and renamed into place. Ente's clients record a hash of each
|
||||||
file's bytes at upload, the content hash. Every original whose metadata records
|
file's bytes at upload, the content hash. Every original whose metadata records
|
||||||
@@ -1059,7 +1071,8 @@ onto what the cache holds, empty or not. A refresh then starts only when
|
|||||||
- `client`, required: the account's `Client`, or anything that satisfies
|
- `client`, required: the account's `Client`, or anything that satisfies
|
||||||
`LibraryClient`.
|
`LibraryClient`.
|
||||||
- `cacheDirectory`: where `metadata.json` and the content cache live. Default:
|
- `cacheDirectory`: where `metadata.json` and the content cache live. Default:
|
||||||
`quak/<userID>` under the per-user cache directory (see "The cache").
|
`quak/<userID>` under the per-user cache directory, with the URL-encoded
|
||||||
|
`apiOrigin` before `<userID>` for a client that has one (see "The cache").
|
||||||
- `downloadDirectory`: the root of the save paths. Default: `photos` in the
|
- `downloadDirectory`: the root of the save paths. Default: `photos` in the
|
||||||
working directory, resolved once when the library opens;
|
working directory, resolved once when the library opens;
|
||||||
`lib.downloadDirectory` holds the result. An original stored at its save path
|
`lib.downloadDirectory` holds the result. An original stored at its save path
|
||||||
@@ -1256,7 +1269,8 @@ do.
|
|||||||
fetch and cache as `photo.original()` and `photo.thumbnail()` do. `file` is an
|
fetch and cache as `photo.original()` and `photo.thumbnail()` do. `file` is an
|
||||||
`EnteFile` the caller holds, and the library need not hold it. `quak get` and
|
`EnteFile` the caller holds, and the library need not hold it. `quak get` and
|
||||||
`quak get-thumb` use them; on an empty cache they pass a file that
|
`quak get-thumb` use them; on an empty cache they pass a file that
|
||||||
`lib.freshFiles()` returned.
|
`lib.freshFiles()` returned. In a cache that names no account yet, they first
|
||||||
|
save the client's user ID and `apiOrigin` in `metadata.json`.
|
||||||
|
|
||||||
Lower-level reads return the decrypted model objects, which hold key material:
|
Lower-level reads return the decrypted model objects, which hold key material:
|
||||||
`lib.listCollections()`, `lib.getCollection(id)`, `lib.listFiles(collectionID)`,
|
`lib.listCollections()`, `lib.getCollection(id)`, `lib.listFiles(collectionID)`,
|
||||||
@@ -1495,7 +1509,10 @@ field and each key's length first, and throws an error naming the bad field.
|
|||||||
A client whose `apiOptions` named an `apiOrigin` puts it in its snapshot. The
|
A client whose `apiOptions` named an `apiOrigin` puts it in its snapshot. The
|
||||||
restored client then uses that server, even when the `apiOptions` given to
|
restored client then uses that server, even when the `apiOptions` given to
|
||||||
`Client.fromJSON` name another. A snapshot without `apiOrigin` uses the
|
`Client.fromJSON` name another. A snapshot without `apiOrigin` uses the
|
||||||
`apiOrigin` of those `apiOptions`, or the default server.
|
`apiOrigin` of those `apiOptions`, or the default server. `client.apiOrigin`
|
||||||
|
holds the one in use, or undefined when none was named. `Client.fromJSON`
|
||||||
|
refuses an `apiOrigin` that is not an http or https URL, or that has a query or
|
||||||
|
fragment, as `quak login --server` does.
|
||||||
|
|
||||||
`Client.fromJSON` works only once libsodium, the cryptography library quak uses,
|
`Client.fromJSON` works only once libsodium, the cryptography library quak uses,
|
||||||
is loaded. `Client.login` loads it, and `Client.fromJSON` does not. A program
|
is loaded. `Client.login` loads it, and `Client.fromJSON` does not. A program
|
||||||
|
|||||||
+7
-10
@@ -21,6 +21,7 @@ import { extname, join } from "node:path";
|
|||||||
import {
|
import {
|
||||||
type Client,
|
type Client,
|
||||||
type ClientSnapshot,
|
type ClientSnapshot,
|
||||||
|
isServerAddress,
|
||||||
type LoginOptions,
|
type LoginOptions,
|
||||||
} from "./client.js";
|
} from "./client.js";
|
||||||
import { lockBackupDirectory, type BackupLock } from "./backup.js";
|
import { lockBackupDirectory, type BackupLock } from "./backup.js";
|
||||||
@@ -56,7 +57,7 @@ export interface CliContext {
|
|||||||
// Directory holding `session.json`.
|
// Directory holding `session.json`.
|
||||||
sessionDir: string;
|
sessionDir: string;
|
||||||
// The `--cache-dir` global, or undefined to let the library pick its
|
// The `--cache-dir` global, or undefined to let the library pick its
|
||||||
// per-user default keyed by the account id.
|
// per-user default keyed by the account's server and user ID.
|
||||||
cacheDir?: string;
|
cacheDir?: string;
|
||||||
// Reads the session file into a client, or null when there is none. The
|
// Reads the session file into a client, or null when there is none. The
|
||||||
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
|
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
|
||||||
@@ -129,6 +130,7 @@ const parseID = (
|
|||||||
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
|
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
|
||||||
// originals through the on-disk cache.
|
// originals through the on-disk cache.
|
||||||
const readLibraryClient = (client: Client): LibraryClient => ({
|
const readLibraryClient = (client: Client): LibraryClient => ({
|
||||||
|
apiOrigin: client.apiOrigin,
|
||||||
whoami: () => client.whoami(),
|
whoami: () => client.whoami(),
|
||||||
collectionsSince: (args) => client.collectionsSince(args),
|
collectionsSince: (args) => client.collectionsSince(args),
|
||||||
filesSince: (args) => client.filesSince(args),
|
filesSince: (args) => client.filesSince(args),
|
||||||
@@ -158,14 +160,6 @@ const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
|
|||||||
// Logs in to the server `--server` or `QUAK_SERVER` names, the flag first, or
|
// Logs in to the server `--server` or `QUAK_SERVER` names, the flag first, or
|
||||||
// to Ente's hosted service when neither is set. The saved session keeps the
|
// to Ente's hosted service when neither is set. The saved session keeps the
|
||||||
// server, so every later command uses it.
|
// server, so every later command uses it.
|
||||||
// Whether `text` can be an Ente server's address: an http or https URL, which
|
|
||||||
// may have a path, and no query or fragment, which request URLs could not
|
|
||||||
// carry.
|
|
||||||
const isServerAddress = (text: string): boolean =>
|
|
||||||
URL.canParse(text) &&
|
|
||||||
["http:", "https:"].includes(new URL(text).protocol) &&
|
|
||||||
!/[?#]/.test(text);
|
|
||||||
|
|
||||||
export const loginCommand = async (
|
export const loginCommand = async (
|
||||||
ctx: CliContext,
|
ctx: CliContext,
|
||||||
opts: { server?: string },
|
opts: { server?: string },
|
||||||
@@ -233,7 +227,10 @@ export const logoutCommand = async (ctx: CliContext): Promise<number> => {
|
|||||||
try {
|
try {
|
||||||
const client = ctx.loadSession(path);
|
const client = ctx.loadSession(path);
|
||||||
if (client) {
|
if (client) {
|
||||||
cacheDir ??= defaultCacheDirectory(client.whoami().userID);
|
cacheDir ??= defaultCacheDirectory(
|
||||||
|
client.whoami().userID,
|
||||||
|
client.apiOrigin,
|
||||||
|
);
|
||||||
await client.logoutOnServer();
|
await client.logoutOnServer();
|
||||||
client.logout();
|
client.logout();
|
||||||
}
|
}
|
||||||
|
|||||||
+17
-3
@@ -69,14 +69,25 @@ export interface FilesPage {
|
|||||||
const errorMessage = (err: unknown): string =>
|
const errorMessage = (err: unknown): string =>
|
||||||
err instanceof Error ? err.message : String(err);
|
err instanceof Error ? err.message : String(err);
|
||||||
|
|
||||||
|
// Whether `text` can be an Ente server's address: an http or https URL, which
|
||||||
|
// may have a path, and no query or fragment, which request URLs could not
|
||||||
|
// carry. `quak login --server` and `Client.fromJSON` both check with it.
|
||||||
|
export const isServerAddress = (text: string): boolean =>
|
||||||
|
URL.canParse(text) &&
|
||||||
|
["http:", "https:"].includes(new URL(text).protocol) &&
|
||||||
|
!/[?#]/.test(text);
|
||||||
|
|
||||||
export class Client {
|
export class Client {
|
||||||
|
// The `apiOrigin` the client was given, or undefined when it was given
|
||||||
|
// none, which means Ente's hosted service. The library keeps a cache per
|
||||||
|
// server as well as per user ID.
|
||||||
|
readonly apiOrigin: string | undefined;
|
||||||
private readonly api: ApiClient;
|
private readonly api: ApiClient;
|
||||||
private readonly email: string;
|
private readonly email: string;
|
||||||
private readonly userID: number;
|
private readonly userID: number;
|
||||||
private readonly masterKey: Uint8Array;
|
private readonly masterKey: Uint8Array;
|
||||||
private readonly secretKey: Uint8Array;
|
private readonly secretKey: Uint8Array;
|
||||||
private readonly publicKey: Uint8Array;
|
private readonly publicKey: Uint8Array;
|
||||||
private readonly apiOrigin: string | undefined;
|
|
||||||
private loggedOut = false;
|
private loggedOut = false;
|
||||||
|
|
||||||
private constructor(
|
private constructor(
|
||||||
@@ -163,9 +174,12 @@ export class Client {
|
|||||||
}
|
}
|
||||||
if (
|
if (
|
||||||
s.apiOrigin !== undefined &&
|
s.apiOrigin !== undefined &&
|
||||||
(typeof s.apiOrigin !== "string" || s.apiOrigin === "")
|
(typeof s.apiOrigin !== "string" || !isServerAddress(s.apiOrigin))
|
||||||
) {
|
) {
|
||||||
throw invalid("apiOrigin", "must be a non-empty string");
|
throw invalid(
|
||||||
|
"apiOrigin",
|
||||||
|
"must be an http or https URL with no query or fragment",
|
||||||
|
);
|
||||||
}
|
}
|
||||||
const apiOrigin =
|
const apiOrigin =
|
||||||
(s.apiOrigin as string | undefined) ?? apiOptions?.apiOrigin;
|
(s.apiOrigin as string | undefined) ?? apiOptions?.apiOrigin;
|
||||||
|
|||||||
+65
-17
@@ -119,9 +119,18 @@ export {
|
|||||||
export const DEFAULT_REFRESH_INTERVAL_SECONDS = 3;
|
export const DEFAULT_REFRESH_INTERVAL_SECONDS = 3;
|
||||||
|
|
||||||
// The account's cache directory when `cacheDirectory` is not given: the
|
// The account's cache directory when `cacheDirectory` is not given: the
|
||||||
// env-paths cache directory plus the user id, so each account has its own.
|
// env-paths cache directory plus the user id, so each account has its own. An
|
||||||
export const defaultCacheDirectory = (userID: number): string =>
|
// account on a self-hosted server has the server's address, URL-encoded, in
|
||||||
join(envPaths("quak", { suffix: "" }).cache, String(userID));
|
// between, because two servers can give two accounts the same user id.
|
||||||
|
export const defaultCacheDirectory = (
|
||||||
|
userID: number,
|
||||||
|
apiOrigin?: string,
|
||||||
|
): string => {
|
||||||
|
const cache = envPaths("quak", { suffix: "" }).cache;
|
||||||
|
return apiOrigin === undefined
|
||||||
|
? join(cache, String(userID))
|
||||||
|
: join(cache, encodeURIComponent(apiOrigin), String(userID));
|
||||||
|
};
|
||||||
|
|
||||||
// Project a metadata store into by-id records, filling each record's cache
|
// Project a metadata store into by-id records, filling each record's cache
|
||||||
// paths from the content cache when one is given. Shared by the live read
|
// paths from the content cache when one is given. Shared by the live read
|
||||||
@@ -144,6 +153,9 @@ const deriveRecordsFromStore = (
|
|||||||
// tests drive a mock with no crypto or network; the real `Client` satisfies it
|
// tests drive a mock with no crypto or network; the real `Client` satisfies it
|
||||||
// structurally.
|
// structurally.
|
||||||
export interface LibraryClient {
|
export interface LibraryClient {
|
||||||
|
// The server the client talks to, as `Client.apiOrigin` names it. Absent
|
||||||
|
// means Ente's hosted service.
|
||||||
|
readonly apiOrigin?: string;
|
||||||
whoami(): { email: string; userID: number };
|
whoami(): { email: string; userID: number };
|
||||||
collectionsSince(args: { sinceTime: number }): Promise<CollectionsPage>;
|
collectionsSince(args: { sinceTime: number }): Promise<CollectionsPage>;
|
||||||
filesSince(args: {
|
filesSince(args: {
|
||||||
@@ -196,8 +208,8 @@ export type RefreshProgressCallback = (event: RefreshEvent) => void;
|
|||||||
|
|
||||||
export interface LibraryOptions {
|
export interface LibraryOptions {
|
||||||
client: LibraryClient;
|
client: LibraryClient;
|
||||||
// Where `metadata.json` lives. Defaults to the env-paths cache directory
|
// Where `metadata.json` lives. Defaults to `defaultCacheDirectory`, so
|
||||||
// plus the user id, so each account has its own cache.
|
// each account has its own cache.
|
||||||
cacheDirectory?: string;
|
cacheDirectory?: string;
|
||||||
// The root of every photo's save path, where `Photo.download()` and
|
// The root of every photo's save path, where `Photo.download()` and
|
||||||
// `lib.backup()` put originals. Defaults to `photos` in the working
|
// `lib.backup()` put originals. Defaults to `photos` in the working
|
||||||
@@ -300,6 +312,7 @@ export class Library {
|
|||||||
// available (a metadata-only client with no explicit source).
|
// available (a metadata-only client with no explicit source).
|
||||||
private readonly cache?: ContentCache;
|
private readonly cache?: ContentCache;
|
||||||
private readonly userID: number;
|
private readonly userID: number;
|
||||||
|
private readonly apiOrigin: string | undefined;
|
||||||
private readonly intervalMs: number;
|
private readonly intervalMs: number;
|
||||||
private readonly onProgress?: RefreshProgressCallback;
|
private readonly onProgress?: RefreshProgressCallback;
|
||||||
private readonly pools: RequestPools;
|
private readonly pools: RequestPools;
|
||||||
@@ -346,6 +359,7 @@ export class Library {
|
|||||||
client: LibraryClient;
|
client: LibraryClient;
|
||||||
store: MetadataStore;
|
store: MetadataStore;
|
||||||
userID: number;
|
userID: number;
|
||||||
|
apiOrigin: string | undefined;
|
||||||
cacheDirectory: string;
|
cacheDirectory: string;
|
||||||
downloadDirectory: string;
|
downloadDirectory: string;
|
||||||
intervalMs: number;
|
intervalMs: number;
|
||||||
@@ -358,6 +372,7 @@ export class Library {
|
|||||||
this.client = args.client;
|
this.client = args.client;
|
||||||
this.store = args.store;
|
this.store = args.store;
|
||||||
this.userID = args.userID;
|
this.userID = args.userID;
|
||||||
|
this.apiOrigin = args.apiOrigin;
|
||||||
this.cacheDirectory = args.cacheDirectory;
|
this.cacheDirectory = args.cacheDirectory;
|
||||||
this.downloadDirectory = args.downloadDirectory;
|
this.downloadDirectory = args.downloadDirectory;
|
||||||
this.intervalMs = args.intervalMs;
|
this.intervalMs = args.intervalMs;
|
||||||
@@ -406,8 +421,9 @@ export class Library {
|
|||||||
// refresh: `open()` returns onto the cache as it is.
|
// refresh: `open()` returns onto the cache as it is.
|
||||||
static async open(opts: LibraryOptions): Promise<Library> {
|
static async open(opts: LibraryOptions): Promise<Library> {
|
||||||
const { userID } = opts.client.whoami();
|
const { userID } = opts.client.whoami();
|
||||||
|
const { apiOrigin } = opts.client;
|
||||||
const cacheDirectory =
|
const cacheDirectory =
|
||||||
opts.cacheDirectory ?? defaultCacheDirectory(userID);
|
opts.cacheDirectory ?? defaultCacheDirectory(userID, apiOrigin);
|
||||||
if (opts.downloadDirectory === "") {
|
if (opts.downloadDirectory === "") {
|
||||||
throw new Error(
|
throw new Error(
|
||||||
"library: downloadDirectory is empty (leave it out to save " +
|
"library: downloadDirectory is empty (leave it out to save " +
|
||||||
@@ -419,14 +435,24 @@ export class Library {
|
|||||||
let store = await MetadataStore.load(metadataPath);
|
let store = await MetadataStore.load(metadataPath);
|
||||||
// A cache directory given explicitly can hold another account's cache.
|
// A cache directory given explicitly can hold another account's cache.
|
||||||
// Its records and cursor are not this account's, so delete it and the
|
// Its records and cursor are not this account's, so delete it and the
|
||||||
// ML data beside it and start empty. A user ID of 0 means the cache
|
// ML data beside it and start empty. A cache from another server goes
|
||||||
// was never refreshed and so holds nothing to discard.
|
// whole: two servers can give two files the same ID, so its originals
|
||||||
if (store.userID !== 0 && store.userID !== userID) {
|
// and thumbnails could be served for this account's files. A user ID
|
||||||
await rm(metadataPath, { force: true });
|
// of 0 means the cache names no account, so nothing is stored in it.
|
||||||
await rm(join(cacheDirectory, "mldata"), {
|
if (
|
||||||
recursive: true,
|
store.userID !== 0 &&
|
||||||
force: true,
|
(store.userID !== userID || store.apiOrigin !== apiOrigin)
|
||||||
});
|
) {
|
||||||
|
const discard = ["metadata.json", "mldata"];
|
||||||
|
if (store.apiOrigin !== apiOrigin) {
|
||||||
|
discard.push("originals", "thumbnails");
|
||||||
|
}
|
||||||
|
for (const name of discard) {
|
||||||
|
await rm(join(cacheDirectory, name), {
|
||||||
|
recursive: true,
|
||||||
|
force: true,
|
||||||
|
});
|
||||||
|
}
|
||||||
store = await MetadataStore.load(metadataPath);
|
store = await MetadataStore.load(metadataPath);
|
||||||
}
|
}
|
||||||
const intervalMs =
|
const intervalMs =
|
||||||
@@ -490,6 +516,7 @@ export class Library {
|
|||||||
client: opts.client,
|
client: opts.client,
|
||||||
store,
|
store,
|
||||||
userID,
|
userID,
|
||||||
|
apiOrigin,
|
||||||
cacheDirectory,
|
cacheDirectory,
|
||||||
downloadDirectory,
|
downloadDirectory,
|
||||||
intervalMs,
|
intervalMs,
|
||||||
@@ -509,9 +536,11 @@ export class Library {
|
|||||||
if (opts.refreshOnOpen === false) {
|
if (opts.refreshOnOpen === false) {
|
||||||
// The caller asks for each refresh it needs.
|
// The caller asks for each refresh it needs.
|
||||||
lib.scheduleNext();
|
lib.scheduleNext();
|
||||||
} else if (store.loadedFromDisk) {
|
} else if (store.listCollections().length > 0) {
|
||||||
// An existing copy already answers reads; refresh in the background
|
// An existing copy already answers reads; refresh in the background
|
||||||
// and start the interval once that first cycle settles.
|
// and start the interval once that first cycle settles. A cache
|
||||||
|
// that names the account but holds no collection has nothing to
|
||||||
|
// answer with, such as after `originalOf` on an empty cache.
|
||||||
void lib.runRefresh().then(() => lib.scheduleNext());
|
void lib.runRefresh().then(() => lib.scheduleNext());
|
||||||
} else {
|
} else {
|
||||||
// Nothing was cached: wait for the first refresh to fill the store
|
// Nothing was cached: wait for the first refresh to fill the store
|
||||||
@@ -658,6 +687,7 @@ export class Library {
|
|||||||
"originalOf requires a library opened with a content cache",
|
"originalOf requires a library opened with a content cache",
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
await this.saveAccountInEmptyCache();
|
||||||
return this.cache.originalOf(file, opts);
|
return this.cache.originalOf(file, opts);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -673,9 +703,23 @@ export class Library {
|
|||||||
"thumbnailOf requires a library opened with a content cache",
|
"thumbnailOf requires a library opened with a content cache",
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
await this.saveAccountInEmptyCache();
|
||||||
return this.cache.thumbnailOf(file, opts);
|
return this.cache.thumbnailOf(file, opts);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Saves the account's user ID and server in a cache that names no account
|
||||||
|
// yet, before `originalOf` or `thumbnailOf` stores a file there. A later
|
||||||
|
// `open()` for an account on another server then deletes that file (see
|
||||||
|
// `open()`), although no refresh has stored any records.
|
||||||
|
private async saveAccountInEmptyCache(): Promise<void> {
|
||||||
|
if (this.store.userID !== 0) return;
|
||||||
|
this.store.userID = this.userID;
|
||||||
|
this.store.apiOrigin = this.apiOrigin;
|
||||||
|
this.unsaved = true;
|
||||||
|
await this.store.save();
|
||||||
|
this.unsaved = false;
|
||||||
|
}
|
||||||
|
|
||||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||||
// library's, each original at its save path, with a durable failure
|
// library's, each original at its save path, with a durable failure
|
||||||
// ledger (issue #51). Takes the lock in that directory first, unless the
|
// ledger (issue #51). Takes the lock in that directory first, unless the
|
||||||
@@ -857,8 +901,12 @@ export class Library {
|
|||||||
// actually changed.
|
// actually changed.
|
||||||
let changed = false;
|
let changed = false;
|
||||||
|
|
||||||
if (this.store.userID !== this.userID) {
|
if (
|
||||||
|
this.store.userID !== this.userID ||
|
||||||
|
this.store.apiOrigin !== this.apiOrigin
|
||||||
|
) {
|
||||||
this.store.userID = this.userID;
|
this.store.userID = this.userID;
|
||||||
|
this.store.apiOrigin = this.apiOrigin;
|
||||||
changed = true;
|
changed = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+11
-12
@@ -1,11 +1,12 @@
|
|||||||
// On-disk JSON metadata store for the local cache.
|
// On-disk JSON metadata store for the local cache.
|
||||||
//
|
//
|
||||||
// The store keeps one `metadata.json` file holding the account's server
|
// The store keeps one `metadata.json` file holding the account's server
|
||||||
// state: the user id, a schema version, the cursor for the incremental
|
// state: the user id, the server's address for an account on a self-hosted
|
||||||
// collections listing, the decrypted collection and file records, and the
|
// server, a schema version, the cursor for the incremental collections
|
||||||
// collections and files the refresh could not decrypt. The whole file is read
|
// listing, the decrypted collection and file records, and the collections and
|
||||||
// into RAM on load and rewritten as a whole on save; there is no partial
|
// files the refresh could not decrypt. The whole file is read into RAM on
|
||||||
// update and no lock file. A separate refresh unit populates the store from
|
// load and rewritten as a whole on save; there is no partial update and no
|
||||||
|
// lock file. A separate refresh unit populates the store from
|
||||||
// the server — this module only stores what it is given.
|
// the server — this module only stores what it is given.
|
||||||
//
|
//
|
||||||
// The file is a cache, so it is never trusted to exist or to be intact: a
|
// The file is a cache, so it is never trusted to exist or to be intact: a
|
||||||
@@ -46,6 +47,7 @@ export interface Undecryptable {
|
|||||||
interface StoredMetadata {
|
interface StoredMetadata {
|
||||||
schemaVersion: number;
|
schemaVersion: number;
|
||||||
userID: number;
|
userID: number;
|
||||||
|
apiOrigin?: string;
|
||||||
collectionsSinceTime: Microseconds;
|
collectionsSinceTime: Microseconds;
|
||||||
collections: StoredCollection[];
|
collections: StoredCollection[];
|
||||||
files: StoredFile[];
|
files: StoredFile[];
|
||||||
@@ -68,14 +70,10 @@ export class MetadataStore {
|
|||||||
readonly path: string;
|
readonly path: string;
|
||||||
readonly schemaVersion = METADATA_SCHEMA_VERSION;
|
readonly schemaVersion = METADATA_SCHEMA_VERSION;
|
||||||
userID = 0;
|
userID = 0;
|
||||||
|
// The account's `apiOrigin`, or undefined for Ente's hosted service.
|
||||||
|
apiOrigin: string | undefined;
|
||||||
collectionsSinceTime: Microseconds = 0;
|
collectionsSinceTime: Microseconds = 0;
|
||||||
|
|
||||||
// True when `load` populated this store from a valid existing file; false
|
|
||||||
// on a first run or a missing/corrupt/wrong-version file that loaded empty.
|
|
||||||
// `Library.open` reads it to decide whether the first refresh may run in
|
|
||||||
// the background (an existing copy already serves reads) or must be awaited.
|
|
||||||
loadedFromDisk = false;
|
|
||||||
|
|
||||||
private readonly collections = new Map<number, Collection>();
|
private readonly collections = new Map<number, Collection>();
|
||||||
private readonly files = new Map<string, EnteFile>();
|
private readonly files = new Map<string, EnteFile>();
|
||||||
private undecryptable: Undecryptable[] = [];
|
private undecryptable: Undecryptable[] = [];
|
||||||
@@ -102,8 +100,8 @@ export class MetadataStore {
|
|||||||
if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) {
|
if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) {
|
||||||
return store;
|
return store;
|
||||||
}
|
}
|
||||||
store.loadedFromDisk = true;
|
|
||||||
store.userID = parsed.userID ?? 0;
|
store.userID = parsed.userID ?? 0;
|
||||||
|
store.apiOrigin = parsed.apiOrigin;
|
||||||
store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0;
|
store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0;
|
||||||
for (const stored of parsed.collections ?? []) {
|
for (const stored of parsed.collections ?? []) {
|
||||||
const collection: Collection = {
|
const collection: Collection = {
|
||||||
@@ -143,6 +141,7 @@ export class MetadataStore {
|
|||||||
const model: StoredMetadata = {
|
const model: StoredMetadata = {
|
||||||
schemaVersion: METADATA_SCHEMA_VERSION,
|
schemaVersion: METADATA_SCHEMA_VERSION,
|
||||||
userID: this.userID,
|
userID: this.userID,
|
||||||
|
apiOrigin: this.apiOrigin,
|
||||||
collectionsSinceTime: this.collectionsSinceTime,
|
collectionsSinceTime: this.collectionsSinceTime,
|
||||||
collections: [...this.collections.values()].map((c) => ({
|
collections: [...this.collections.values()].map((c) => ({
|
||||||
...c,
|
...c,
|
||||||
|
|||||||
+25
-4
@@ -4,11 +4,12 @@
|
|||||||
* each with the callbacks that log its requests and retries, for `quak files`
|
* each with the callbacks that log its requests and retries, for `quak files`
|
||||||
* taking its collection ID as an optional argument, for `quak get` and
|
* taking its collection ID as an optional argument, for `quak get` and
|
||||||
* `quak get-thumb` refusing `--collection`, and for `quak login` taking
|
* `quak get-thumb` refusing `--collection`, and for `quak login` taking
|
||||||
* `--server`.
|
* `--server` and sending its first request to the server it names.
|
||||||
*
|
*
|
||||||
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
|
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
|
||||||
* replaced by one that records the options it is given and reports no session,
|
* replaced by one that records the options it is given and reports no session,
|
||||||
* so the command stops before it makes any request.
|
* so the command stops before it makes any request. `quak login` reads no
|
||||||
|
* session, so its test stubs `fetch` instead and records each request.
|
||||||
*
|
*
|
||||||
* The first load of the sources `bin/quak.ts` imports compiles them, and the
|
* The first load of the sources `bin/quak.ts` imports compiles them, and the
|
||||||
* first `init()` starts libsodium. On a loaded host that takes longer than a
|
* first `init()` starts libsodium. On a loaded host that takes longer than a
|
||||||
@@ -59,6 +60,8 @@ afterEach(() => {
|
|||||||
process.argv = argv;
|
process.argv = argv;
|
||||||
loaded.length = 0;
|
loaded.length = 0;
|
||||||
vi.restoreAllMocks();
|
vi.restoreAllMocks();
|
||||||
|
vi.unstubAllGlobals();
|
||||||
|
vi.unstubAllEnvs();
|
||||||
});
|
});
|
||||||
|
|
||||||
afterAll(() => {
|
afterAll(() => {
|
||||||
@@ -119,12 +122,30 @@ describe("bin/quak.ts get and get-thumb", () => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
// The command refuses an address that is not a URL before it asks for anything
|
|
||||||
// or sends any request.
|
|
||||||
describe("bin/quak.ts login", () => {
|
describe("bin/quak.ts login", () => {
|
||||||
|
// The command refuses an address that is not a URL before it asks for
|
||||||
|
// anything or sends any request.
|
||||||
it("hands --server to the command", async () => {
|
it("hands --server to the command", async () => {
|
||||||
expect(await quak("login", "--server", "ente.example.com")).toMatch(
|
expect(await quak("login", "--server", "ente.example.com")).toMatch(
|
||||||
/^Invalid server address "ente\.example\.com"/,
|
/^Invalid server address "ente\.example\.com"/,
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// `fetch` answers 404, so the login fails at its first request. A login
|
||||||
|
// that succeeded would save its session in this user's real data
|
||||||
|
// directory, since env-paths is not mocked here.
|
||||||
|
it("sends its first request to the server --server names", async () => {
|
||||||
|
const requests: string[] = [];
|
||||||
|
vi.stubGlobal("fetch", async (input: string) => {
|
||||||
|
const { origin, pathname } = new URL(input);
|
||||||
|
requests.push(origin + pathname);
|
||||||
|
return new Response("not found", { status: 404 });
|
||||||
|
});
|
||||||
|
vi.stubEnv("QUAK_EMAIL", "login@example.com");
|
||||||
|
vi.stubEnv("QUAK_PASSWORD", "password");
|
||||||
|
await quak("login", "--server", "https://ente.example.com");
|
||||||
|
expect(requests).toEqual([
|
||||||
|
"https://ente.example.com/users/srp/attributes",
|
||||||
|
]);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
+162
-13
@@ -445,6 +445,21 @@ describe("login", () => {
|
|||||||
);
|
);
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|
||||||
|
it("refuses an empty QUAK_SERVER before asking for anything", async () => {
|
||||||
|
vi.stubEnv("QUAK_SERVER", "");
|
||||||
|
const ctx = loginContext(async () => {
|
||||||
|
throw new Error("login must not be called");
|
||||||
|
});
|
||||||
|
expect(await loginCommand(ctx, {})).toBe(1);
|
||||||
|
|
||||||
|
expect(prompts).toEqual([]);
|
||||||
|
expect(stderr.text).toBe(
|
||||||
|
`Invalid server address "": it must be an http or https URL ` +
|
||||||
|
`with no query or fragment\n`,
|
||||||
|
);
|
||||||
|
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// These use a real client read from the session file, over a fake API that
|
// These use a real client read from the session file, over a fake API that
|
||||||
@@ -519,6 +534,16 @@ describe("logout", () => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("names a self-hosted account's default cache directory, under its server's address", async () => {
|
||||||
|
const ctx = { ...logoutContext(200), cacheDir: undefined };
|
||||||
|
const apiOrigin = "https://ente.example.com";
|
||||||
|
saveSession(ctx.sessionDir, { ...snapshot, apiOrigin });
|
||||||
|
expect(await logoutCommand(ctx)).toBe(0);
|
||||||
|
expect(stderr.text).toContain(
|
||||||
|
`Cache directory ${defaultCacheDirectory(USER_ID, apiOrigin)} still holds decrypted data`,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
it("without a session says so, calls nothing and exits 0", async () => {
|
it("without a session says so, calls nothing and exits 0", async () => {
|
||||||
expect(await logoutCommand(logoutContext(200))).toBe(0);
|
expect(await logoutCommand(logoutContext(200))).toBe(0);
|
||||||
expect(requests).toHaveLength(0);
|
expect(requests).toHaveLength(0);
|
||||||
@@ -1676,18 +1701,19 @@ describe("an account with many albums", () => {
|
|||||||
// encrypted as the server stores it. The original's bytes are the title.
|
// encrypted as the server stores it. The original's bytes are the title.
|
||||||
// These tests fetch no thumbnail. The server records a thumbnail size, so
|
// These tests fetch no thumbnail. The server records a thumbnail size, so
|
||||||
// `helper fix-missing-thumbnails` downloads the original before it skips
|
// `helper fix-missing-thumbnails` downloads the original before it skips
|
||||||
// the photo as not a JPEG.
|
// the photo as not a JPEG. `albumKey` is that album's key, by default the
|
||||||
const photoRow = (photo: {
|
// one in `albumKeys`.
|
||||||
id: number;
|
const photoRow = (
|
||||||
collectionID: number;
|
photo: {
|
||||||
title: string;
|
id: number;
|
||||||
updationTime: number;
|
collectionID: number;
|
||||||
}): { row: Record<string, unknown>; original: Uint8Array } => {
|
title: string;
|
||||||
|
updationTime: number;
|
||||||
|
},
|
||||||
|
albumKey = albumKeys.get(photo.collectionID)!,
|
||||||
|
): { row: Record<string, unknown>; original: Uint8Array } => {
|
||||||
const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||||
const photoKeyBox = encryptSecretbox(
|
const photoKeyBox = encryptSecretbox(photoKey, albumKey);
|
||||||
photoKey,
|
|
||||||
albumKeys.get(photo.collectionID)!,
|
|
||||||
);
|
|
||||||
const metadata = encryptStreamBlob(
|
const metadata = encryptStreamBlob(
|
||||||
new TextEncoder().encode(
|
new TextEncoder().encode(
|
||||||
JSON.stringify({
|
JSON.stringify({
|
||||||
@@ -1800,6 +1826,77 @@ describe("an account with many albums", () => {
|
|||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// The albums in the cache's `metadata.json`. A search on an empty cache
|
||||||
|
// stores none there, only the account whose original or thumbnail it
|
||||||
|
// stored.
|
||||||
|
const cachedAlbums = (): unknown[] =>
|
||||||
|
JSON.parse(readFileSync(join(root, "cache", "metadata.json"), "utf-8"))
|
||||||
|
.collections;
|
||||||
|
|
||||||
|
// A context whose login and saved session send each request to the server
|
||||||
|
// its host names: `ente.example.com` is the fake server above, and
|
||||||
|
// `photos.example.org` a second one. Every account the fake server logs in
|
||||||
|
// has user ID 42, so the second server's account has the first one's user
|
||||||
|
// ID. Its one album holds photo 200, `other-200.jpg`; the first server
|
||||||
|
// gives ID 200 to `photo-2.jpg`.
|
||||||
|
const twoServerContext = async (): Promise<CliContext> => {
|
||||||
|
const account = await buildAccount(PASSWORD);
|
||||||
|
const albumKey = sodium.crypto_secretbox_keygen();
|
||||||
|
const albumKeyBox = encryptSecretbox(albumKey, account.masterKey);
|
||||||
|
const name = encryptSecretbox(
|
||||||
|
new TextEncoder().encode("Other album"),
|
||||||
|
albumKey,
|
||||||
|
);
|
||||||
|
const { row, original } = photoRow(
|
||||||
|
{
|
||||||
|
id: 200,
|
||||||
|
collectionID: 1,
|
||||||
|
title: "other-200.jpg",
|
||||||
|
updationTime: 1,
|
||||||
|
},
|
||||||
|
albumKey,
|
||||||
|
);
|
||||||
|
const other = buildMetaFetch({
|
||||||
|
verifier: account.verifier,
|
||||||
|
srpAttributes: account.srpAttributes,
|
||||||
|
keyAttributes: account.keyAttributes,
|
||||||
|
encryptedToken: account.encryptedToken,
|
||||||
|
collections: [
|
||||||
|
{
|
||||||
|
id: 1,
|
||||||
|
owner: { id: USER_ID },
|
||||||
|
encryptedKey: toBase64(albumKeyBox.ciphertext),
|
||||||
|
keyDecryptionNonce: toBase64(albumKeyBox.nonce),
|
||||||
|
encryptedName: toBase64(name.ciphertext),
|
||||||
|
nameDecryptionNonce: toBase64(name.nonce),
|
||||||
|
type: "album",
|
||||||
|
updationTime: 1,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
filesByCollection: { 1: [row] },
|
||||||
|
encryptedMLData: {},
|
||||||
|
fileCiphertexts: { 200: original },
|
||||||
|
});
|
||||||
|
const fetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||||
|
const { host } = new URL(
|
||||||
|
input instanceof Request ? input.url : input,
|
||||||
|
);
|
||||||
|
return (host === "photos.example.org" ? other : server)(
|
||||||
|
input,
|
||||||
|
init,
|
||||||
|
);
|
||||||
|
}) as typeof globalThis.fetch;
|
||||||
|
return {
|
||||||
|
...context(),
|
||||||
|
login: (opts) =>
|
||||||
|
Client.login({
|
||||||
|
...opts,
|
||||||
|
apiOptions: { ...opts.apiOptions, fetch },
|
||||||
|
}),
|
||||||
|
loadSession: (path) => loadSession(path, { fetch }),
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
it("collections lists every album from one request, fetching no album's files", async () => {
|
it("collections lists every album from one request, fetching no album's files", async () => {
|
||||||
const ctx = accountContext();
|
const ctx = accountContext();
|
||||||
expect(await collectionsCommand(ctx, {})).toBe(0);
|
expect(await collectionsCommand(ctx, {})).toBe(0);
|
||||||
@@ -1906,7 +2003,7 @@ describe("an account with many albums", () => {
|
|||||||
"/collections/v2/diff?collectionID=2&sinceTime=0",
|
"/collections/v2/diff?collectionID=2&sinceTime=0",
|
||||||
"/?fileID=200",
|
"/?fileID=200",
|
||||||
]);
|
]);
|
||||||
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
|
expect(cachedAlbums()).toEqual([]);
|
||||||
});
|
});
|
||||||
|
|
||||||
// A self-hosted server serves downloads under its own address, at
|
// A self-hosted server serves downloads under its own address, at
|
||||||
@@ -1950,6 +2047,58 @@ describe("an account with many albums", () => {
|
|||||||
expect([...hosts]).toEqual(["ente.example.com"]);
|
expect([...hosts]).toEqual(["ente.example.com"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// The two servers' accounts share the cache directory `--cache-dir` names.
|
||||||
|
it("after a login to another server whose account has the same user ID, files and get see none of the first server's albums and originals", async () => {
|
||||||
|
const ctx = await twoServerContext();
|
||||||
|
const first = join(root, "first.jpg");
|
||||||
|
const second = join(root, "second.jpg");
|
||||||
|
vi.stubEnv("QUAK_EMAIL", "many@example.com");
|
||||||
|
vi.stubEnv("QUAK_PASSWORD", PASSWORD);
|
||||||
|
vi.stubEnv("QUAK_SERVER", undefined);
|
||||||
|
try {
|
||||||
|
const opts = { server: "https://ente.example.com" };
|
||||||
|
expect(await loginCommand(ctx, opts)).toBe(0);
|
||||||
|
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||||
|
expect(await getCommand(ctx, "200", { out: first })).toBe(0);
|
||||||
|
|
||||||
|
const otherOpts = { server: "https://photos.example.org" };
|
||||||
|
expect(await loginCommand(ctx, otherOpts)).toBe(0);
|
||||||
|
stdout.text = "";
|
||||||
|
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||||
|
expect(await getCommand(ctx, "200", { out: second })).toBe(0);
|
||||||
|
} finally {
|
||||||
|
vi.unstubAllEnvs();
|
||||||
|
}
|
||||||
|
|
||||||
|
expect(stdout.text).toBe("200\timage\tother-200.jpg\n");
|
||||||
|
expect(readFileSync(first, "utf-8")).toBe("photo-2.jpg");
|
||||||
|
expect(readFileSync(second, "utf-8")).toBe("other-200.jpg");
|
||||||
|
});
|
||||||
|
|
||||||
|
// `get` on an empty cache stores the original but no album.
|
||||||
|
it("get on an empty cache, after the same on another server whose account has the same user ID, saves this server's photo", async () => {
|
||||||
|
const ctx = await twoServerContext();
|
||||||
|
const first = join(root, "first.jpg");
|
||||||
|
const second = join(root, "second.jpg");
|
||||||
|
vi.stubEnv("QUAK_EMAIL", "many@example.com");
|
||||||
|
vi.stubEnv("QUAK_PASSWORD", PASSWORD);
|
||||||
|
vi.stubEnv("QUAK_SERVER", undefined);
|
||||||
|
try {
|
||||||
|
const opts = { server: "https://ente.example.com" };
|
||||||
|
expect(await loginCommand(ctx, opts)).toBe(0);
|
||||||
|
expect(await getCommand(ctx, "200", { out: first })).toBe(0);
|
||||||
|
|
||||||
|
const otherOpts = { server: "https://photos.example.org" };
|
||||||
|
expect(await loginCommand(ctx, otherOpts)).toBe(0);
|
||||||
|
expect(await getCommand(ctx, "200", { out: second })).toBe(0);
|
||||||
|
} finally {
|
||||||
|
vi.unstubAllEnvs();
|
||||||
|
}
|
||||||
|
|
||||||
|
expect(readFileSync(first, "utf-8")).toBe("photo-2.jpg");
|
||||||
|
expect(readFileSync(second, "utf-8")).toBe("other-200.jpg");
|
||||||
|
});
|
||||||
|
|
||||||
// `files` first fills the cache, so `get` waits for a refresh. No album has
|
// `files` first fills the cache, so `get` waits for a refresh. No album has
|
||||||
// changed since, so the refresh asks for no album's files.
|
// changed since, so the refresh asks for no album's files.
|
||||||
it("get on a cache that holds the account asks for no unchanged album's files, also for a photo in no album", async () => {
|
it("get on a cache that holds the account asks for no unchanged album's files, also for a photo in no album", async () => {
|
||||||
@@ -2005,7 +2154,7 @@ describe("an account with many albums", () => {
|
|||||||
"/collections/v2/diff?collectionID=39&sinceTime=0",
|
"/collections/v2/diff?collectionID=39&sinceTime=0",
|
||||||
"/?fileID=3900",
|
"/?fileID=3900",
|
||||||
]);
|
]);
|
||||||
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
|
expect(cachedAlbums()).toEqual([]);
|
||||||
});
|
});
|
||||||
|
|
||||||
// `files` first fills the cache, so the helper waits for a refresh. No
|
// `files` first fills the cache, so the helper waits for a refresh. No
|
||||||
|
|||||||
@@ -55,10 +55,25 @@ describe("Client.toJSON", () => {
|
|||||||
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
|
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("names the server apiOptions gave fromJSON", () => {
|
it("names the server apiOptions gave fromJSON", async () => {
|
||||||
const apiOrigin = "https://ente.example.com";
|
const apiOrigin = "https://ente.example.com";
|
||||||
const client = Client.fromJSON(validSnapshot(), { apiOrigin });
|
const urls: string[] = [];
|
||||||
|
const client = Client.fromJSON(validSnapshot(), {
|
||||||
|
apiOrigin,
|
||||||
|
fetch: async (input) => {
|
||||||
|
urls.push(String(input));
|
||||||
|
return new Response(JSON.stringify({ collections: [] }), {
|
||||||
|
status: 200,
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
});
|
||||||
|
},
|
||||||
|
});
|
||||||
expect(client.toJSON().apiOrigin).toBe(apiOrigin);
|
expect(client.toJSON().apiOrigin).toBe(apiOrigin);
|
||||||
|
|
||||||
|
await client.listCollections();
|
||||||
|
expect(urls).toEqual([
|
||||||
|
"https://ente.example.com/collections/v2?sinceTime=0",
|
||||||
|
]);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("throws instead of emitting a snapshot without a token", () => {
|
it("throws instead of emitting a snapshot without a token", () => {
|
||||||
@@ -93,6 +108,8 @@ describe("Client.fromJSON", () => {
|
|||||||
["publicKey", shortKey],
|
["publicKey", shortKey],
|
||||||
["apiOrigin", 7],
|
["apiOrigin", 7],
|
||||||
["apiOrigin", ""],
|
["apiOrigin", ""],
|
||||||
|
["apiOrigin", "ente.example.com"],
|
||||||
|
["apiOrigin", "ftp://ente.example.com"],
|
||||||
])("rejects %s = %j, naming the field", (field, value) => {
|
])("rejects %s = %j, naming the field", (field, value) => {
|
||||||
const snapshot: Record<string, unknown> = { ...validSnapshot() };
|
const snapshot: Record<string, unknown> = { ...validSnapshot() };
|
||||||
snapshot[field] = value;
|
snapshot[field] = value;
|
||||||
|
|||||||
@@ -19,16 +19,22 @@
|
|||||||
* refresh fails (offline start from cache).
|
* refresh fails (offline start from cache).
|
||||||
* 5. `close()` stops the timer and is idempotent, and its promise resolves
|
* 5. `close()` stops the timer and is idempotent, and its promise resolves
|
||||||
* only once an in-flight refresh has written the cache file.
|
* only once an in-flight refresh has written the cache file.
|
||||||
* 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id.
|
* 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id,
|
||||||
* 7. `open()` branches on the cache: an empty cache awaits the first refresh
|
* with a self-hosted server's address before the user id.
|
||||||
* (it has nothing to serve yet); an existing cache serves its copy at once
|
* 7. `open()` branches on the cache: a cache that holds no collection awaits
|
||||||
* and refreshes in the background, so a slow or dead server never stalls
|
* the first refresh (it has nothing to serve yet); an existing cache serves
|
||||||
* opening.
|
* its copy at once and refreshes in the background, so a slow or dead
|
||||||
|
* server never stalls opening.
|
||||||
* 8. A save failure that leaves RAM ahead of disk keeps `status().lastError`
|
* 8. A save failure that leaves RAM ahead of disk keeps `status().lastError`
|
||||||
* set and keeps retrying the write; a later empty refresh does not clear it.
|
* set and keeps retrying the write; a later empty refresh does not clear it.
|
||||||
* 9. With `refreshOnOpen: false`, `open()` starts no refresh at all, and a
|
* 9. With `refreshOnOpen: false`, `open()` starts no refresh at all, and a
|
||||||
* refresh reports through `onProgress` each collection whose files it
|
* refresh reports through `onProgress` each collection whose files it
|
||||||
* fetched.
|
* fetched.
|
||||||
|
* 10. A cache another account filled starts empty. One filled from another
|
||||||
|
* server also loses its originals and thumbnails, since two servers can
|
||||||
|
* give two files the same ID. `thumbnailOf` on an empty cache saves the
|
||||||
|
* account's user ID and server. A refresh saves the account's server, so a
|
||||||
|
* self-hosted account keeps its own cache from one `open()` to the next.
|
||||||
*
|
*
|
||||||
* The client is a mock: no crypto, no network. It serves scripted pages and
|
* The client is a mock: no crypto, no network. It serves scripted pages and
|
||||||
* records the `sinceTime` each call carried so cursor threading is provable.
|
* records the `sinceTime` each call carried so cursor threading is provable.
|
||||||
@@ -49,7 +55,13 @@
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||||
import { mkdtempSync, rmSync } from "node:fs";
|
import {
|
||||||
|
existsSync,
|
||||||
|
mkdirSync,
|
||||||
|
mkdtempSync,
|
||||||
|
rmSync,
|
||||||
|
writeFileSync,
|
||||||
|
} from "node:fs";
|
||||||
import { tmpdir } from "node:os";
|
import { tmpdir } from "node:os";
|
||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
import envPaths from "env-paths";
|
import envPaths from "env-paths";
|
||||||
@@ -107,6 +119,7 @@ const file = (
|
|||||||
*/
|
*/
|
||||||
class MockClient {
|
class MockClient {
|
||||||
userID = USER_ID;
|
userID = USER_ID;
|
||||||
|
apiOrigin?: string;
|
||||||
failCollections = false;
|
failCollections = false;
|
||||||
collectionsQueue: CollectionsPage[] = [];
|
collectionsQueue: CollectionsPage[] = [];
|
||||||
filesByCollection = new Map<number, FilesPage[]>();
|
filesByCollection = new Map<number, FilesPage[]>();
|
||||||
@@ -584,6 +597,44 @@ describe("Library.open and background refresh", () => {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// `originalOf` on an empty cache leaves such a cache behind: it names the
|
||||||
|
// account before it stores the original.
|
||||||
|
it("awaits the first refresh on a cache that names the account but holds no collection", async () => {
|
||||||
|
const seed = await MetadataStore.load(
|
||||||
|
join(cacheDirectory, "metadata.json"),
|
||||||
|
);
|
||||||
|
seed.userID = USER_ID;
|
||||||
|
await seed.save();
|
||||||
|
let releaseFirstFetch: (page: CollectionsPage) => void = () => {};
|
||||||
|
const gate = new Promise<CollectionsPage>((resolve) => {
|
||||||
|
releaseFirstFetch = resolve;
|
||||||
|
});
|
||||||
|
const client = new MockClient();
|
||||||
|
client.collectionsSince = async () => gate;
|
||||||
|
|
||||||
|
let opened = false;
|
||||||
|
const openPromise = Library.open({ client, cacheDirectory }).then(
|
||||||
|
(l) => {
|
||||||
|
opened = true;
|
||||||
|
return l;
|
||||||
|
},
|
||||||
|
);
|
||||||
|
await new Promise((r) => setTimeout(r, 20));
|
||||||
|
expect(opened).toBe(false);
|
||||||
|
|
||||||
|
releaseFirstFetch({
|
||||||
|
collections: [collection(1, 100)],
|
||||||
|
deleted: [],
|
||||||
|
cursor: 100,
|
||||||
|
});
|
||||||
|
const lib = await openPromise;
|
||||||
|
try {
|
||||||
|
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
|
||||||
|
} finally {
|
||||||
|
await lib.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
// A caller that asks for each refresh it needs, as the CLI's read commands
|
// A caller that asks for each refresh it needs, as the CLI's read commands
|
||||||
// do, opens without one: on an empty cache the refresh `open()` would
|
// do, opens without one: on an empty cache the refresh `open()` would
|
||||||
// await fetches the files of every collection, which `quak collections`
|
// await fetches the files of every collection, which `quak collections`
|
||||||
@@ -842,4 +893,165 @@ describe("Library.open and background refresh", () => {
|
|||||||
else process.env.XDG_CACHE_HOME = prev;
|
else process.env.XDG_CACHE_HOME = prev;
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("puts a self-hosted server's address, URL-encoded, before the user id in the default cacheDirectory", async () => {
|
||||||
|
const xdg = join(dir, "xdg-cache");
|
||||||
|
const prev = process.env.XDG_CACHE_HOME;
|
||||||
|
process.env.XDG_CACHE_HOME = xdg;
|
||||||
|
try {
|
||||||
|
const client = new MockClient();
|
||||||
|
client.apiOrigin = "https://ente.example.com:8443/api";
|
||||||
|
const lib = await Library.open({ client });
|
||||||
|
try {
|
||||||
|
expect(lib.cacheDirectory).toBe(
|
||||||
|
join(
|
||||||
|
envPaths("quak", { suffix: "" }).cache,
|
||||||
|
"https%3A%2F%2Fente.example.com%3A8443%2Fapi",
|
||||||
|
String(USER_ID),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
expect(lib.cacheDirectory.startsWith(xdg)).toBe(true);
|
||||||
|
} finally {
|
||||||
|
await lib.close();
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
if (prev === undefined) delete process.env.XDG_CACHE_HOME;
|
||||||
|
else process.env.XDG_CACHE_HOME = prev;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// A cache that account USER_ID on `apiOrigin` filled: its records, and an
|
||||||
|
// original and a thumbnail of file 1001.
|
||||||
|
const fillCache = async (apiOrigin?: string): Promise<void> => {
|
||||||
|
const seed = await MetadataStore.load(
|
||||||
|
join(cacheDirectory, "metadata.json"),
|
||||||
|
);
|
||||||
|
seed.userID = USER_ID;
|
||||||
|
seed.apiOrigin = apiOrigin;
|
||||||
|
seed.collectionsSinceTime = 500;
|
||||||
|
seed.putCollection(collection(1, 400));
|
||||||
|
seed.putFile(file(1001, 1, 400));
|
||||||
|
await seed.save();
|
||||||
|
for (const name of ["originals", "thumbnails"]) {
|
||||||
|
mkdirSync(join(cacheDirectory, name));
|
||||||
|
writeFileSync(join(cacheDirectory, name, "1001.jpg"), "1001");
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
it("deletes the whole cache an account on another server filled", async () => {
|
||||||
|
await fillCache("https://ente.example.com");
|
||||||
|
const client = new MockClient();
|
||||||
|
client.apiOrigin = "https://photos.example.org";
|
||||||
|
const lib = await Library.open({
|
||||||
|
client,
|
||||||
|
cacheDirectory,
|
||||||
|
refreshIntervalSeconds: 3600,
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
expect(client.collectionsSinceTimes).toEqual([0]);
|
||||||
|
expect(lib.listCollections()).toEqual([]);
|
||||||
|
for (const name of ["originals", "thumbnails"]) {
|
||||||
|
expect(existsSync(join(cacheDirectory, name, "1001.jpg"))).toBe(
|
||||||
|
false,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
await lib.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// A server gives each file its own ID, so another account's cached file
|
||||||
|
// is never served for this account's.
|
||||||
|
it("keeps the originals and thumbnails another account on the same server cached", async () => {
|
||||||
|
await fillCache();
|
||||||
|
const client = new MockClient();
|
||||||
|
client.userID = USER_ID + 1;
|
||||||
|
const lib = await Library.open({
|
||||||
|
client,
|
||||||
|
cacheDirectory,
|
||||||
|
refreshIntervalSeconds: 3600,
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
expect(client.collectionsSinceTimes).toEqual([0]);
|
||||||
|
expect(lib.listCollections()).toEqual([]);
|
||||||
|
for (const name of ["originals", "thumbnails"]) {
|
||||||
|
expect(existsSync(join(cacheDirectory, name, "1001.jpg"))).toBe(
|
||||||
|
true,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
await lib.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// `quak get-thumb` on an empty cache stores a thumbnail before any refresh
|
||||||
|
// runs. `open()` for an account on another server deletes that thumbnail
|
||||||
|
// only if the cache names the account that stored it.
|
||||||
|
it("thumbnailOf on an empty cache saves the account's user ID and server in metadata.json", async () => {
|
||||||
|
const client = new MockClient();
|
||||||
|
client.apiOrigin = "https://ente.example.com";
|
||||||
|
const lib = await Library.open({
|
||||||
|
client,
|
||||||
|
cacheDirectory,
|
||||||
|
refreshOnOpen: false,
|
||||||
|
refreshIntervalSeconds: Infinity,
|
||||||
|
contentSource: {
|
||||||
|
original: async () => {
|
||||||
|
throw new Error("original not expected");
|
||||||
|
},
|
||||||
|
thumbnail: async ({ destination }) => {
|
||||||
|
writeFileSync(destination, "thumb");
|
||||||
|
return { bytesWritten: 5 };
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
await lib.thumbnailOf(file(1001, 1, 400));
|
||||||
|
} finally {
|
||||||
|
await lib.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
const saved = await MetadataStore.load(
|
||||||
|
join(cacheDirectory, "metadata.json"),
|
||||||
|
);
|
||||||
|
expect(saved.userID).toBe(USER_ID);
|
||||||
|
expect(saved.apiOrigin).toBe("https://ente.example.com");
|
||||||
|
});
|
||||||
|
|
||||||
|
// A refresh saves the account's server in `metadata.json`. Without it,
|
||||||
|
// `open()` would take a self-hosted account's own cache for another
|
||||||
|
// server's and delete it.
|
||||||
|
it("keeps a self-hosted account's own cache from one open to the next", async () => {
|
||||||
|
const client = new MockClient();
|
||||||
|
client.apiOrigin = "https://ente.example.com";
|
||||||
|
client.collectionsQueue.push({
|
||||||
|
collections: [collection(1, 100)],
|
||||||
|
deleted: [],
|
||||||
|
cursor: 100,
|
||||||
|
});
|
||||||
|
const first = await Library.open({
|
||||||
|
client,
|
||||||
|
cacheDirectory,
|
||||||
|
refreshIntervalSeconds: Infinity,
|
||||||
|
});
|
||||||
|
await first.close();
|
||||||
|
mkdirSync(join(cacheDirectory, "originals"));
|
||||||
|
writeFileSync(join(cacheDirectory, "originals", "1001.jpg"), "1001");
|
||||||
|
|
||||||
|
const lib = await Library.open({
|
||||||
|
client,
|
||||||
|
cacheDirectory,
|
||||||
|
refreshIntervalSeconds: Infinity,
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
|
||||||
|
expect(
|
||||||
|
existsSync(join(cacheDirectory, "originals", "1001.jpg")),
|
||||||
|
).toBe(true);
|
||||||
|
} finally {
|
||||||
|
await lib.close();
|
||||||
|
}
|
||||||
|
// The second open's refresh started from the saved cursor.
|
||||||
|
expect(client.collectionsSinceTimes).toEqual([0, 100]);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
Reference in new issue
Block a user