sneak/quak
1
0
Fork 0
Code Issues 8 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

Port the quak CLI to the library API (closes #52)
check / check (push) Successful in 39s
Details

Browse Source 
Route every command through Library.open instead of scanning the client.
collections/files read lib.albums; get/get-thumb resolve lib.photos.byID
and copy the cached original/thumbnail to --out, so --collection is now
accepted but ignored. backup-metadata and the thumbnail helpers enumerate
through the library and read originals via photo.original(); ML fetch,
EXIF, and thumbnail upload are unchanged. A new global --cache-dir sets the
cache location; point commands open with the background precache off so a
one-shot command never starts downloading the whole account.

Also addresses #17: fix-missing-thumbnails
now reports a non-JPEG image or a video as skipped (unsupported), distinct
from failed, and only a genuine failure exits non-zero.

Model: opus-4-8
This commit is contained in:
sneak
2026-09-22 20:19:18 +00:00
parent 2e00139d3c
commit 005612ef76
6 changed files with 586 additions and 371 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/cli/metadata-backup.test.ts
+64 -64
View File
@@ -2,13 +2,17 @@
* Tests for `quak backup-metadata <dir>`.
*
* This command dumps all decrypted account metadata into a directory
* tree of plain JSON files, without downloading any file content. It
* is fast (no multi-megabyte downloads) and produces a complete
* plaintext record of every collection name, file title, creation
* date, GPS coordinate, camera model, caption, face label, and any
* other metadata the Ente clients have attached.
* tree of plain JSON files, without downloading any file content (unless
* `--exif` is given). It is fast and produces a complete plaintext record of
* every collection name, file title, creation date, GPS coordinate, camera
* model, caption, face label, and any other metadata the Ente clients have
* attached.
*
* Layout:
* As of issue #52 it runs on the library API: `runMetadataBackup(lib, client,
* dir)` enumerates collections and files from the library's cache rather than
* scanning the client directly, and `--exif` reads each original through the
* library's content cache (`photo.original()`). The ML fetch is unchanged. The
* output tree is identical:
*
* <dir>/
* account.json { email, userID }
@@ -44,7 +48,11 @@ import {
} from "../../src/crypto/index.js";
import * as jpegJs from "jpeg-js";
import { Client } from "../../src/client.js";
import { runMetadataBackup } from "../../src/metadata-backup.js";
import { Library } from "../../src/library/index.js";
import {
runMetadataBackup,
type MetadataBackupOptions,
} from "../../src/metadata-backup.js";
import type { KeyAttributes } from "../../src/auth/types.js";
const TEST_EMAIL = "metabackup@example.com";
@@ -431,16 +439,50 @@ afterAll(() => {
rmSync(testDir, { recursive: true, force: true });
});
// Log in against the mock and open a library over its cache. The point commands
// open the library with the background precache off and a long refresh interval;
// the same here keeps the test deterministic (no thumbnail/original prefetch it
// did not ask for, no second refresh mid-test).
const openLib = async (client: Client): Promise<Library> =>
Library.open({
// The library client omits `fetchMLData`, matching how the CLI opens
// point commands: `runMetadataBackup` fetches ML data itself through
// the client, so the library's background backfill would only be a
// redundant second pass over the same endpoint.
client: {
whoami: () => client.whoami(),
collectionsSince: (args) => client.collectionsSince(args),
filesSince: (args) => client.filesSince(args),
contentSource: () => client.contentSource(),
},
cacheDirectory: mkdtempSync(join(testDir, "cache-")),
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
});
// Run one metadata backup end to end: fresh client, fresh library, then close.
const runBackup = async (
outDir: string,
opts?: MetadataBackupOptions,
): Promise<void> => {
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
const lib = await openLib(client);
try {
await runMetadataBackup(lib, client, outDir, opts);
} finally {
lib.close();
}
};
describe("quak backup-metadata", () => {
it("writes account.json with email and userID", async () => {
const outDir = join(testDir, "full");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runBackup(outDir);
const account = JSON.parse(
readFileSync(join(outDir, "account.json"), "utf-8"),
@@ -451,13 +493,7 @@ describe("quak backup-metadata", () => {
it("creates per-collection directories with _collection.json", async () => {
const outDir = join(testDir, "collections");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runBackup(outDir);
const collDirs = readdirSync(join(outDir, "collections"));
expect(collDirs.length).toBe(2);
@@ -478,13 +514,7 @@ describe("quak backup-metadata", () => {
it("decrypts collection-level pubMagicMetadata", async () => {
const outDir = join(testDir, "coll-magic");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runBackup(outDir);
const collDirs = readdirSync(join(outDir, "collections"));
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
@@ -501,13 +531,7 @@ describe("quak backup-metadata", () => {
it("writes per-file JSON with all three metadata layers", async () => {
const outDir = join(testDir, "file-meta");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runBackup(outDir);
const collDirs = readdirSync(join(outDir, "collections"));
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
@@ -526,13 +550,7 @@ describe("quak backup-metadata", () => {
it("handles files with no magic metadata gracefully", async () => {
const outDir = join(testDir, "no-magic");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runBackup(outDir);
const collDirs = readdirSync(join(outDir, "collections"));
const workDir = collDirs.find((d) => d.includes("Work"))!;
@@ -550,14 +568,8 @@ describe("quak backup-metadata", () => {
it("is incremental: second run does not fail", async () => {
const outDir = join(testDir, "incremental");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runMetadataBackup(client, outDir);
await runBackup(outDir);
await runBackup(outDir);
const account = JSON.parse(
readFileSync(join(outDir, "account.json"), "utf-8"),
@@ -567,13 +579,7 @@ describe("quak backup-metadata", () => {
it("fetches and decrypts ML data by default", async () => {
const outDir = join(testDir, "ml-data");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir);
await runBackup(outDir);
const collDirs = readdirSync(join(outDir, "collections"));
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
@@ -597,13 +603,7 @@ describe("quak backup-metadata", () => {
it("extracts EXIF from downloaded files when --exif is set", async () => {
const outDir = join(testDir, "exif-data");
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildMetaFetch(mock) },
});
await runMetadataBackup(client, outDir, { exif: true });
await runBackup(outDir, { exif: true });
const collDirs = readdirSync(join(outDir, "collections"));
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
test/thumbnails/thumbnails.test.ts
+175 -83
View File
@@ -6,17 +6,30 @@
* working thumbnails, others return 404 or empty bodies. The tests
* verify that the detection and repair logic handles each case correctly.
*
* As of issue #52 both helpers take an open `Library` for enumeration and the
* `Client` for the API operations that stay unchanged (the thumbnail existence
* check, and the encrypt-and-upload path). `fixMissingThumbnails` reads each
* original through the library's content cache (`photo.original()`).
*
* `fixMissingThumbnails` is the most complex function in quak: it
* downloads the original file, generates a JPEG thumbnail with jpeg-js,
* encrypts it with secretstream push, gets a presigned upload URL,
* uploads to S3, and registers the new thumbnail with the API. The
* test verifies each step actually happened and the uploaded data is
* a valid encrypted blob that decrypts to a JPEG.
*
* It regenerates thumbnails for baseline JPEGs only, because `jpeg-js` decodes
* only JPEG. A non-JPEG image (PNG, HEIC) or a video is reported as "skipped
* (unsupported)" rather than crashing the decoder into an opaque failure
* (issue #17); the mixed test below locks that distinction down.
*/
import { existsSync, mkdtempSync, rmSync } from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import sodium from "libsodium-wrappers-sumo";
import * as jpegJs from "jpeg-js";
import { beforeAll, describe, expect, it } from "vitest";
import { afterAll, beforeAll, describe, expect, it } from "vitest";
import {
init,
toBase64,
@@ -27,6 +40,7 @@ import {
} from "../../src/crypto/index.js";
import { SRP, SrpServer } from "fast-srp-hap";
import { Client } from "../../src/client.js";
import { Library } from "../../src/library/index.js";
import {
listMissingThumbnails,
fixMissingThumbnails,
@@ -42,6 +56,7 @@ const TEST_EMAIL = "thumb@example.com";
const TEST_PASSWORD = "thumbpass";
const TEST_OPS = 2;
const TEST_MEM = 64 * 1024 * 1024;
const TEST_TIME = 1700000000000000;
interface ThumbMockState {
verifier: Buffer;
@@ -63,8 +78,17 @@ interface ThumbMockState {
}
let mock: ThumbMockState;
let tmpRoot: string;
const buildThumbMock = async (): Promise<ThumbMockState> => {
// PNG signature bytes — enough for `fixMissingThumbnails` to recognise a
// non-JPEG image and skip it. It need not be a decodable PNG.
const PNG_BYTES = new Uint8Array([
0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0x00, 0x00, 0x00, 0x0d,
]);
const buildThumbMock = async (opts?: {
extraFormats?: boolean;
}): Promise<ThumbMockState> => {
const kekSalt = sodium.randombytes_buf(sodium.crypto_pwhash_SALTBYTES);
const kek = await deriveKEK(TEST_PASSWORD, kekSalt, TEST_OPS, TEST_MEM);
const loginSubKeyBytes = deriveLoginSubkey(kek);
@@ -102,7 +126,6 @@ const buildThumbMock = async (): Promise<ThumbMockState> => {
opsLimit: TEST_OPS,
};
// One collection with 3 files: ok thumbnail, empty thumbnail, 404 thumbnail
const collKey = sodium.crypto_secretbox_keygen();
const ckN = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
const encCK = sodium.crypto_secretbox_easy(collKey, ckN, masterKey);
@@ -118,10 +141,11 @@ const buildThumbMock = async (): Promise<ThumbMockState> => {
encryptedName: toBase64(encCN),
nameDecryptionNonce: toBase64(cnN),
type: "album",
updationTime: 1700000000000000,
updationTime: TEST_TIME,
};
// Generate a real tiny JPEG via jpeg-js
// Generate a real tiny JPEG via jpeg-js, used as the encrypted body of the
// JPEG files so a repair actually decodes and re-encodes real pixels.
const w = 100;
const h = 80;
const pixels = new Uint8Array(w * h * 4);
@@ -131,26 +155,32 @@ const buildThumbMock = async (): Promise<ThumbMockState> => {
pixels[i + 2] = 0; // B
pixels[i + 3] = 255; // A
}
const tinyJpeg = jpegJs.encode(
{ data: pixels, width: w, height: h },
80,
).data;
const tinyJpeg = new Uint8Array(
jpegJs.encode({ data: pixels, width: w, height: h }, 80).data,
);
const fileKeys: Record<number, Uint8Array> = {};
const fileCiphertexts: Record<number, Uint8Array> = {};
const rawFiles: Record<string, unknown>[] = [];
for (const fileID of [100, 101, 102]) {
// Build one raw file record: encrypt its metadata and its body under a
// fresh per-file key, and record the key and ciphertext for the mock to
// serve and for the test to verify against.
const makeRawFile = (
fileID: number,
fileType: number,
title: string,
body: Uint8Array,
): Record<string, unknown> => {
const fk = sodium.crypto_secretstream_xchacha20poly1305_keygen();
fileKeys[fileID] = fk;
const fkN = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
const encFK = sodium.crypto_secretbox_easy(fk, fkN, collKey);
const meta = JSON.stringify({
title: `file-${fileID}.jpg`,
fileType: 0,
creationTime: 1700000000000000,
modificationTime: 1700000000000000,
title,
fileType,
creationTime: TEST_TIME,
modificationTime: TEST_TIME,
});
const metaPush =
sodium.crypto_secretstream_xchacha20poly1305_init_push(fk);
@@ -161,18 +191,17 @@ const buildThumbMock = async (): Promise<ThumbMockState> => {
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
);
// Encrypt the tiny JPEG as the file body
const filePush =
sodium.crypto_secretstream_xchacha20poly1305_init_push(fk);
const encFile = sodium.crypto_secretstream_xchacha20poly1305_push(
filePush.state,
new Uint8Array(tinyJpeg),
body,
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
);
fileCiphertexts[fileID] = encFile;
rawFiles.push({
return {
id: fileID,
collectionID: 1,
ownerID: 42,
@@ -186,8 +215,30 @@ const buildThumbMock = async (): Promise<ThumbMockState> => {
thumbnail: {
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
},
updationTime: 1700000000000000,
});
updationTime: TEST_TIME,
};
};
// Three JPEG files: ok thumbnail, empty thumbnail, 404 thumbnail.
const rawFiles: Record<string, unknown>[] = [];
for (const fileID of [100, 101, 102]) {
rawFiles.push(makeRawFile(fileID, 0, `file-${fileID}.jpg`, tinyJpeg));
}
const thumbnailBehavior: Record<number, "ok" | "empty" | "404" | "500"> = {
100: "ok",
101: "empty",
102: "404",
};
// For the issue #17 mixed test: a non-JPEG image and a video, both with a
// missing (404) thumbnail so they surface in the missing list too.
if (opts?.extraFormats) {
rawFiles.push(makeRawFile(103, 0, "file-103.png", PNG_BYTES));
rawFiles.push(
makeRawFile(104, 1, "file-104.mp4", new Uint8Array([0, 0, 0, 1])),
);
thumbnailBehavior[103] = "404";
thumbnailBehavior[104] = "404";
}
return {
@@ -206,11 +257,7 @@ const buildThumbMock = async (): Promise<ThumbMockState> => {
filesByCollection: { 1: rawFiles },
fileCiphertexts,
fileKeys,
thumbnailBehavior: {
100: "ok",
101: "empty",
102: "404",
},
thumbnailBehavior,
uploadedThumbnails: [],
};
};
@@ -381,6 +428,31 @@ const countingFetch = (
return { fetch: fake as typeof globalThis.fetch, matched: () => matched };
};
// Open a library over a mock-backed client. As the CLI does for point commands,
// the background precache is off and the refresh interval is long, and the
// library client omits `fetchMLData` so no background ML fetch runs. The real
// `Client` is still used for the API operations the helpers perform directly.
const openLib = (client: Client): Promise<Library> =>
Library.open({
client: {
whoami: () => client.whoami(),
collectionsSince: (args) => client.collectionsSince(args),
filesSince: (args) => client.filesSince(args),
contentSource: () => client.contentSource(),
},
cacheDirectory: mkdtempSync(join(tmpRoot, "cache-")),
refreshIntervalSeconds: 3600,
precacheThumbnails: false,
precacheOriginals: false,
});
const login = (fetch: typeof globalThis.fetch, retry?: RetryOptions) =>
Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: retry ? { fetch, retry } : { fetch },
});
// ---------------------------------------------------------------------------
// Tests
// ---------------------------------------------------------------------------
@@ -389,17 +461,21 @@ beforeAll(async () => {
await init();
await sodium.ready;
mock = await buildThumbMock();
tmpRoot = mkdtempSync(join(tmpdir(), "quak-thumb-test-"));
});
afterAll(() => {
if (tmpRoot && existsSync(tmpRoot))
rmSync(tmpRoot, { recursive: true, force: true });
});
describe("listMissingThumbnails", () => {
it("identifies files with empty and 404 thumbnails, ignores working ones", async () => {
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildThumbFetch(mock) },
});
const client = await login(buildThumbFetch(mock));
const lib = await openLib(client);
const missing = await listMissingThumbnails(client);
const missing = await listMissingThumbnails(lib, client);
lib.close();
// File 100 has a working thumbnail → not reported
// File 101 has an empty thumbnail → reported
@@ -436,13 +512,11 @@ describe("listMissingThumbnails", () => {
buildThumbFetch(failingMock),
(url) => url.includes("thumbnails.ente.io") && url.includes("102"),
);
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: counted.fetch, retry: { ...noWait } },
});
const client = await login(counted.fetch, { ...noWait });
const lib = await openLib(client);
const missing = await listMissingThumbnails(client);
const missing = await listMissingThumbnails(lib, client);
lib.close();
// Only the genuinely empty thumbnail is reported.
expect(missing.map((m) => m.fileID)).toEqual([101]);
@@ -475,13 +549,11 @@ describe("listMissingThumbnails", () => {
return inner(input, init);
}) as typeof globalThis.fetch;
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch, retry: { ...noWait } },
});
const client = await login(fetch, { ...noWait });
const lib = await openLib(client);
const missing = await listMissingThumbnails(client);
const missing = await listMissingThumbnails(lib, client);
lib.close();
expect(missing.map((m) => m.fileID)).toEqual([101]);
expect(thumbRequests).toBe(4);
@@ -500,13 +572,11 @@ describe("listMissingThumbnails", () => {
mockWithDupes.filesByCollection[2] =
mockWithDupes.filesByCollection[1]!;
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildThumbFetch(mockWithDupes) },
});
const client = await login(buildThumbFetch(mockWithDupes));
const lib = await openLib(client);
const missing = await listMissingThumbnails(client);
const missing = await listMissingThumbnails(lib, client);
lib.close();
// Should still be 2, not 4 (each file checked only once)
expect(missing.length).toBe(2);
@@ -516,16 +586,14 @@ describe("listMissingThumbnails", () => {
describe("fixMissingThumbnails", () => {
it("downloads original, generates thumbnail, encrypts, uploads, and registers", async () => {
const fixMock = await buildThumbMock();
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildThumbFetch(fixMock) },
});
const client = await login(buildThumbFetch(fixMock));
const lib = await openLib(client);
const results = await fixMissingThumbnails(client, [101]);
const results = await fixMissingThumbnails(lib, client, [101]);
lib.close();
expect(results.length).toBe(1);
expect(results[0]!.success).toBe(true);
expect(results[0]!.status).toBe("fixed");
expect(results[0]!.fileID).toBe(101);
expect(results[0]!.title).toBe("file-101.jpg");
expect(results[0]!.collection).toBe("Photos");
@@ -555,48 +623,76 @@ describe("fixMissingThumbnails", () => {
it("reports failure for nonexistent file IDs without crashing", async () => {
const fixMock = await buildThumbMock();
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildThumbFetch(fixMock) },
});
const client = await login(buildThumbFetch(fixMock));
const lib = await openLib(client);
const results = await fixMissingThumbnails(client, [999]);
const results = await fixMissingThumbnails(lib, client, [999]);
lib.close();
expect(results.length).toBe(1);
expect(results[0]!.success).toBe(false);
expect(results[0]!.status).toBe("failed");
expect(results[0]!.fileID).toBe(999);
expect(results[0]!.error).toContain("not found");
expect(results[0]!.reason).toContain("not found");
});
it("continues after one file fails and reports mixed results", async () => {
const fixMock = await buildThumbMock();
// Make file 102 fail by removing its ciphertext so download fails
// Make file 102 fail by removing its ciphertext so the download 404s.
delete fixMock.fileCiphertexts[102];
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildThumbFetch(fixMock) },
});
const client = await login(buildThumbFetch(fixMock));
const lib = await openLib(client);
const results = await fixMissingThumbnails(client, [101, 102]);
const results = await fixMissingThumbnails(lib, client, [101, 102]);
lib.close();
expect(results.length).toBe(2);
const success = results.find((r) => r.fileID === 101)!;
const failure = results.find((r) => r.fileID === 102)!;
expect(success.success).toBe(true);
expect(failure.success).toBe(false);
expect(success.status).toBe("fixed");
expect(failure.status).toBe("failed");
});
it("skips a non-JPEG image and a video as unsupported, not failed (issue #17)", async () => {
// A PNG and a video both throw inside the JPEG decoder. The helper must
// recognise them up front and report "skipped", distinct from a genuine
// "failed", and must not upload anything for them. The JPEG in the same
// batch is still repaired.
const fixMock = await buildThumbMock({ extraFormats: true });
const client = await login(buildThumbFetch(fixMock));
const lib = await openLib(client);
const results = await fixMissingThumbnails(
lib,
client,
[101, 103, 104],
);
lib.close();
const jpeg = results.find((r) => r.fileID === 101)!;
const png = results.find((r) => r.fileID === 103)!;
const video = results.find((r) => r.fileID === 104)!;
expect(jpeg.status).toBe("fixed");
// The PNG is a still image but not a JPEG: skipped only after its bytes
// are inspected.
expect(png.status).toBe("skipped");
expect(png.reason).toContain("JPEG");
// The video is skipped from its type alone, before any download.
expect(video.status).toBe("skipped");
expect(video.reason).toContain("video");
// Only the JPEG was uploaded; the two skipped files touched no upload.
expect(fixMock.uploadedThumbnails.length).toBe(1);
expect(fixMock.uploadedThumbnails[0]!.fileID).toBe(101);
});
});
describe("Client.getApiClient", () => {
it("returns the ApiClient when logged in", async () => {
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildThumbFetch(mock) },
});
const client = await login(buildThumbFetch(mock));
const api = client.getApiClient();
expect(api).toBeDefined();
@@ -604,11 +700,7 @@ describe("Client.getApiClient", () => {
});
it("throws after logout", async () => {
const client = await Client.login({
email: TEST_EMAIL,
password: TEST_PASSWORD,
apiOptions: { fetch: buildThumbFetch(mock) },
});
const client = await login(buildThumbFetch(mock));
client.logout();
expect(() => client.getApiClient()).toThrow(/logged out/);