Compare commits
1
Commits
next
..
2fa1e7c3aa
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
2fa1e7c3aa |
@@ -38,47 +38,34 @@ yarn quak get 67890 --out ./photo.jpg
|
||||
yarn quak backup ./my-backup
|
||||
```
|
||||
|
||||
For library use, the primary surface is the cache-backed `Library`:
|
||||
For library use:
|
||||
|
||||
```ts
|
||||
import { Client, Library } from "quak";
|
||||
import { Client } from "quak";
|
||||
|
||||
// Log in once; the client satisfies the library's client interface.
|
||||
const client = await Client.login({
|
||||
email: "you@example.com",
|
||||
password: "your-password",
|
||||
});
|
||||
|
||||
// Open a cache-backed library. On an empty cache this awaits one server
|
||||
// refresh; on an existing cache it returns immediately and refreshes in the
|
||||
// background every `refreshIntervalSeconds` (default 3).
|
||||
const lib = await Library.open({ client });
|
||||
|
||||
// Default reads answer synchronously from the local cache — no network.
|
||||
for (const album of lib.albums.list()) {
|
||||
console.log(album.collectionID, album.name);
|
||||
for (const photo of album.photos.list()) {
|
||||
console.log(` ${photo.title} [${photo.fileType}]`);
|
||||
for (const c of await client.listCollections()) {
|
||||
console.log(c.id, c.name);
|
||||
const files = await client.listFiles(c.id, c.key);
|
||||
for (const f of files) {
|
||||
console.log(` ${f.metadata.title} [${f.metadata.fileType}]`);
|
||||
}
|
||||
}
|
||||
|
||||
// Fresh reads await a server round-trip and answer with current state.
|
||||
const { albums } = await lib.fresh();
|
||||
console.log(`${albums.list().length} albums as of now`);
|
||||
// Download a file
|
||||
const files = await client.listFiles(collectionID, collectionKey);
|
||||
await client.downloadFile(files[0], "./photo.jpg");
|
||||
|
||||
// Fetch (and cache) one photo's full-resolution bytes.
|
||||
const photo = lib.photos.byID({ fileID: 12345 });
|
||||
if (photo) {
|
||||
const { path } = await photo.original();
|
||||
console.log(`original at ${path}`);
|
||||
}
|
||||
|
||||
lib.close();
|
||||
// Serialize session for later (consumer handles persistence)
|
||||
const snapshot = client.toJSON();
|
||||
// ... later:
|
||||
const restored = Client.fromJSON(snapshot);
|
||||
```
|
||||
|
||||
The lower-level `Client` (login, session serialization, and the raw
|
||||
enumeration/download calls) is exported too and documented under Design below.
|
||||
|
||||
## Entrypoints
|
||||
|
||||
This repository adheres to the
|
||||
@@ -433,7 +420,6 @@ you would treat the password itself.
|
||||
### CLI surface
|
||||
|
||||
```
|
||||
quak [--cache-dir <path>] <command> global: local metadata/content cache location
|
||||
quak login interactive or QUAK_EMAIL/QUAK_PASSWORD
|
||||
quak whoami print logged-in account as JSON
|
||||
quak logout delete saved session
|
||||
@@ -442,28 +428,13 @@ quak files --collection <id> [--json] list files in a collection
|
||||
quak get <fileID> [--out path] [--collection] download and decrypt a file
|
||||
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
|
||||
quak backup <dir> [--json] full incremental backup
|
||||
quak backup-metadata <dir> [--exif] dump all decrypted metadata as JSON
|
||||
quak helper list-missing-thumbnails [--json] find files with missing thumbnails
|
||||
quak helper fix-missing-thumbnails [--file ids] generate + upload missing thumbnails
|
||||
```
|
||||
|
||||
Every command runs on the same cache-backed library. The read commands —
|
||||
`collections`, `files`, `get`, and `get-thumb` — force a fresh server round-trip
|
||||
before they answer, so they report current account state rather than whatever
|
||||
the cache last held. `--cache-dir` overrides where the cache lives; without it
|
||||
each account gets its own directory under the per-user cache path.
|
||||
|
||||
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
|
||||
accepted for backward compatibility but ignored. `backup-metadata --exif` (alias
|
||||
`--all`) additionally downloads each file to extract full EXIF/IPTC/XMP
|
||||
metadata. The listing and backup commands support `--json` for machine-readable
|
||||
output.
|
||||
|
||||
`helper fix-missing-thumbnails` regenerates thumbnails for baseline JPEG images
|
||||
only, because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG
|
||||
image (PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept
|
||||
distinct from a `failed` repair, and does not affect the exit code; a genuine
|
||||
failure still exits non-zero.
|
||||
`get` and `get-thumb` search all collections for the file ID when `--collection`
|
||||
is not specified. All listing and backup commands support `--json` for
|
||||
machine-readable output.
|
||||
|
||||
### Backup layout
|
||||
|
||||
@@ -489,7 +460,7 @@ code is non-zero if any files failed.
|
||||
|
||||
- [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
|
||||
errors
|
||||
- [x] Update the API reference section below to match the current implementation
|
||||
- [ ] Update the API reference section below to match the current implementation
|
||||
- [x] `make docker` green
|
||||
- [ ] Tag `v1.0.0`
|
||||
|
||||
@@ -503,179 +474,25 @@ Future (desktop client, separate repo):
|
||||
|
||||
## API reference
|
||||
|
||||
The library's primary surface is the cache-backed `Library`; the lower-level
|
||||
`Client` sits underneath it and is covered by the Design sections above. The
|
||||
test suite is the canonical, executable documentation — `test/library/` and
|
||||
`test/client/usage.test.ts` walk every operation, and `yarn test` verifies them.
|
||||
The API reference section below is from an earlier draft and does not fully
|
||||
reflect the current implementation. The authoritative API documentation is in
|
||||
the test files, particularly `test/client/usage.test.ts` which is a literate
|
||||
tutorial walking through every operation. Run `yarn test` to verify the examples
|
||||
are correct.
|
||||
|
||||
### Opening a library
|
||||
The key types and their actual signatures can be found in:
|
||||
|
||||
`Library.open(options)` loads the on-disk cache, starts the background refresh
|
||||
loop, and resolves to a `Library`. On an empty cache it awaits the first refresh
|
||||
so it never opens onto empty data; on an existing cache it returns immediately
|
||||
and refreshes in the background, so an unreachable server does not block
|
||||
opening.
|
||||
|
||||
`LibraryOptions`:
|
||||
|
||||
| Option | Default | Meaning |
|
||||
| ------------------------ | --------------------------- | --------------------------------------------------------------------- |
|
||||
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
|
||||
| `cacheDirectory` | `<XDG cache>/quak/<userID>` | where `metadata.json` and the content cache live |
|
||||
| `downloadDirectory` | none | backup destination; an original already stored there counts as cached |
|
||||
| `refreshIntervalSeconds` | `3` | background refresh cadence |
|
||||
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
|
||||
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
|
||||
| `precacheOriginalsDays` | `7` | length in days of that latest window |
|
||||
| `cacheOriginalsMaxBytes` | 100 GiB | hard ceiling on the originals cache |
|
||||
| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down |
|
||||
| `isOriginalPinned` | none | extra predicate for originals that must never be evicted |
|
||||
| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) |
|
||||
| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) |
|
||||
| `contentSource` | the client's own | override the byte source (mainly for tests) |
|
||||
|
||||
Concurrency is set through `pools`: construct
|
||||
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
|
||||
and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).
|
||||
|
||||
`lib.status()` returns a `LibraryStatus` (collection/file counts, last
|
||||
refresh/ML times and errors, originals usage and effective limit, precache
|
||||
progress, and `closed`). `lib.close()` stops the background timer; it is
|
||||
idempotent, and an in-flight refresh is left to finish.
|
||||
|
||||
### Default reads vs. fresh reads
|
||||
|
||||
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
|
||||
synchronously from the last refreshed copy held in RAM and never touch the
|
||||
network. The background timer refreshes that copy every
|
||||
`refreshIntervalSeconds`, so a default read is immediate but may be up to one
|
||||
interval stale.
|
||||
|
||||
`await lib.fresh()` forces a refresh, waits for it to complete and persist, and
|
||||
returns the same `{ albums, photos, timeline }` namespaces — now guaranteed to
|
||||
reflect a completed server round-trip. Concurrent `fresh()` calls coalesce onto
|
||||
one refresh, and a refresh that fails rejects the caller (default reads stay
|
||||
silent and keep serving the last good copy). The CLI's read commands use fresh
|
||||
reads (issue https://git.eeqj.de/sneak/quak/issues/75).
|
||||
|
||||
### Read surface
|
||||
|
||||
- `lib.albums.list()` → `Album[]`, newest-updated first.
|
||||
`lib.albums.byID({ collectionID })` and `byName({ albumName })` →
|
||||
`Album | undefined`.
|
||||
- `lib.photos.byID({ fileID })` → `Photo | undefined`.
|
||||
`lib.photos.records({ fileIDs })` → `PhotoRecord[]` in the requested order,
|
||||
each id once, unknown ids dropped.
|
||||
- `lib.timeline.groups({ groupBy, filter? })` → `TimelineGroup[]`, grouped by
|
||||
`"day" | "week" | "month"` (keys `YYYY-MM-DD`, ISO `YYYY-Www`, `YYYY-MM`),
|
||||
newest group first. A `PhotoFilter` combines `albumID`, `text`
|
||||
(title/caption/album-name substring), `fileTypes`, `hasLocation`, and
|
||||
`includeArchived`; hidden photos are always excluded.
|
||||
|
||||
An `Album` exposes its record fields and `album.photos.list()` → `Photo[]`
|
||||
(newest first). A `Photo` exposes its record fields, `photo.record()` →
|
||||
`PhotoRecord`, and two content methods:
|
||||
|
||||
- `await photo.original(opts?)` → `{ path, bytes }` — the full-resolution file.
|
||||
- `await photo.thumbnail(opts?)` → `{ path, bytes }`.
|
||||
|
||||
Both serve from the on-disk content cache when the bytes are present and
|
||||
otherwise fetch through the pools; `opts.onProgress` reports per-file progress.
|
||||
They throw when the library was opened without a content source.
|
||||
|
||||
Lower-level accessors that return decrypted model objects (which hold key
|
||||
material) are also available: `listCollections()`, `getCollection(id)`,
|
||||
`listFiles(collectionID)`, `getFile(collectionID, fileID)`, and
|
||||
`getFileByID(fileID)`.
|
||||
|
||||
### Records and change notifications
|
||||
|
||||
The GUI-facing records hold no key material and no binary, so they survive
|
||||
`structuredClone`/JSON across the Electron IPC boundary:
|
||||
|
||||
- `PhotoRecord`: `fileID`, `albumIDs`, `title`, `takenAt` (milliseconds),
|
||||
`fileType`, optional `caption` / `width` / `height` / `latitude` /
|
||||
`longitude`, `isArchived`, `isHidden`, and `thumbnailPath` / `originalPath`
|
||||
once the bytes are cached.
|
||||
- `AlbumRecord`: `collectionID`, `name`, `type`, `isShared`, `updationTime`, and
|
||||
`fileIDs` (newest first).
|
||||
- `LibrarySnapshot`: `{ albums, photos, takenAt }`.
|
||||
|
||||
`lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first,
|
||||
photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
(`albumsChanged`, `photosChanged`, `fileIDsRemoved`, `albumIDsRemoved`,
|
||||
`refreshedAt`) whenever a refresh alters the projection, and returns
|
||||
`{ unsubscribe }`; a refresh that changes nothing delivers nothing.
|
||||
|
||||
### Thumbnails, ML search, and backup
|
||||
|
||||
- `lib.thumbnails.ensure({ fileIDs, priority, signal?, onProgress? })`
|
||||
prefetches thumbnails through the thumbnail pool, deduped by fileID, returning
|
||||
one `EnsureResult` (`{ fileID, path?, error? }`) per file. `priority` is
|
||||
`"visible" | "ahead" | "background"`; only `"visible"` preempts background
|
||||
work.
|
||||
- `lib.mldata` searches the CLIP index built from Ente's per-file ML data:
|
||||
`forFile({ fileID })` → `Promise<MLData | undefined>` (the whole stored
|
||||
payload — face boxes, landmarks, embedding — read from disk on demand);
|
||||
`similar({ fileID, limit? })` and `searchByEmbedding({ embedding, limit? })` →
|
||||
`SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
|
||||
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
|
||||
a query vector the caller produced elsewhere.
|
||||
- `await lib.backup(opts?)` → `BackupResult`. It refreshes, fetches every
|
||||
in-scope original (and, with `includeThumbnails`, thumbnails) through the
|
||||
content cache, and rebuilds the on-disk backup tree with a durable failure
|
||||
ledger. `BackupOptions`: `downloadDirectory` (falls back to the one `open()`
|
||||
was given), `includeOriginals` (default `true`), `includeThumbnails` (default
|
||||
`false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above for the
|
||||
tree it writes.
|
||||
|
||||
### Request pools
|
||||
|
||||
`RequestPools` holds three independent bounded pools — metadata (10), content
|
||||
(5), thumbnails (25) — because Ente meters these traffic classes differently.
|
||||
Each pool orders on-demand work ahead of background/precache work and dedups
|
||||
in-flight fetches by key, and an idle pool never lends its slots to a busy one.
|
||||
|
||||
### On-disk cache layout
|
||||
|
||||
Under `cacheDirectory`:
|
||||
|
||||
```
|
||||
<cacheDirectory>/
|
||||
metadata.json decrypted account state + refresh cursor
|
||||
originals/<fileID>.<ext> cached full-resolution files
|
||||
thumbnails/<fileID>.jpg cached thumbnails
|
||||
mldata/
|
||||
<fileID>.json one decrypted ML payload per file
|
||||
clip.f32, clip.json the packed CLIP index and its id list
|
||||
fetched.json per-file fetch bookkeeping
|
||||
```
|
||||
|
||||
A stored file appears only via an atomic temp-then-rename, so its presence means
|
||||
it is complete. The design also calls for a content-hash comparison against
|
||||
`FileMetadata.hash` on each fetched original; that check is deferred (issue
|
||||
https://git.eeqj.de/sneak/quak/issues/68) because the exact hash construction
|
||||
cannot yet be confirmed against the repo's fixtures.
|
||||
|
||||
### Key types by source file
|
||||
|
||||
- `src/library/index.ts`: `Library`, `LibraryOptions`, `LibraryStatus`,
|
||||
`LibraryClient`, `RefreshEvent`
|
||||
- `src/library/read.ts`: `Album`, `Photo`, `AlbumsAPI`, `PhotosAPI`,
|
||||
`TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`
|
||||
- `src/library/content.ts`: `ContentResult`, `ContentOptions`, `ThumbnailsAPI`,
|
||||
`EnsureOptions`, `EnsureResult`, `ContentSource`
|
||||
- `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`,
|
||||
`LibraryChange`
|
||||
- `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult`
|
||||
- `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool`
|
||||
- `src/backup.ts`: `BackupOptions`, `BackupResult`, `BackupError`
|
||||
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
|
||||
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `StreamOptions`
|
||||
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `ApiError`,
|
||||
`StreamOptions`
|
||||
- `src/errors.ts`: `ApiError`, `TruncatedStreamError`
|
||||
- `src/retry.ts`: `withRetry`, `isRetryable`, `isSafeToReplay`, `RetryOptions`
|
||||
- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileType`,
|
||||
`CollectionType`, `RawCollection`, `RawEnteFile`
|
||||
- `src/auth/types.ts`: `KeyAttributes`, `SRPAttributes`,
|
||||
`AuthorizationResponse`, `LoginChallenge`
|
||||
- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileBlob`,
|
||||
`RawCollection`, `RawEnteFile`, `RawMagicMetadata`
|
||||
- `src/download/index.ts`: `DownloadResult`
|
||||
- `src/backup.ts`: `BackupResult`, `BackupError`
|
||||
- `src/thumbnails.ts`: `MissingThumbnailInfo`, `ThumbnailFixResult`
|
||||
|
||||
## Source attribution
|
||||
|
||||
@@ -14,19 +14,10 @@ pre-1.0
|
||||
|
||||
# Next Step
|
||||
|
||||
Tag v1.0.0.
|
||||
Update the README API reference section to match the current implementation.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
|
||||
snippets) to match the shipped cache/API library on `next` (issue 53, issue
|
||||
13). Documented `Library.open` and its options, the default-read vs `fresh()`
|
||||
distinction (and that the CLI's read commands are fresh), the record types and
|
||||
`snapshot()`/`subscribe()`, the `albums`/`photos`/`timeline` read surface,
|
||||
`Photo` content methods, `thumbnails.ensure`, the `mldata` search surface,
|
||||
`backup()`, the three request pools (10/5/25), and the on-disk cache layout;
|
||||
noted the deferred content-hash integrity check (issue 68). Docs-only; no code
|
||||
changed.
|
||||
- 2026-09-22: Added resumable, deletion-aware enumeration to `Client` (issue 38,
|
||||
closes issue 7). `collectionsSince`/`filesSince` take a starting cursor,
|
||||
decrypt live records, surface tombstoned ids in a separate `deleted` list (a
|
||||
@@ -131,6 +122,7 @@ Tag v1.0.0.
|
||||
|
||||
# Future Steps
|
||||
|
||||
- Tag v1.0.0.
|
||||
- Future desktop client, separate repo:
|
||||
- Electron app skeleton consuming this library.
|
||||
- Local SQLite cache keyed on (collectionID, fileID, updationTime).
|
||||
|
||||
+124
-185
@@ -2,26 +2,13 @@
|
||||
|
||||
import { input, password as passwordPrompt } from "@inquirer/prompts";
|
||||
import { stdout, stderr } from "node:process";
|
||||
import {
|
||||
copyFileSync,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
readFileSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { Command } from "commander";
|
||||
import envPaths from "env-paths";
|
||||
import { Client, type ClientSnapshot } from "../src/client.js";
|
||||
import { init } from "../src/crypto/index.js";
|
||||
import { Library, type LibraryClient } from "../src/library/index.js";
|
||||
import {
|
||||
fileListRow,
|
||||
fileListLine,
|
||||
originalName,
|
||||
thumbnailName,
|
||||
} from "../src/cli-output.js";
|
||||
import { freshCollections, freshFiles, freshFile } from "../src/cli-read.js";
|
||||
import { Library } from "../src/library/index.js";
|
||||
import { runMetadataBackup } from "../src/metadata-backup.js";
|
||||
import {
|
||||
listMissingThumbnails,
|
||||
@@ -68,61 +55,7 @@ const program = new Command();
|
||||
program
|
||||
.name("quak")
|
||||
.description("CLI for the Ente end-to-end encrypted photo service")
|
||||
.version("0.0.0")
|
||||
.option(
|
||||
"--cache-dir <path>",
|
||||
"Directory for the local metadata/content cache " +
|
||||
"(default: the per-user cache directory)",
|
||||
);
|
||||
|
||||
// The `--cache-dir` global, or undefined to let the library pick its per-user
|
||||
// default keyed by the account id.
|
||||
const cacheDirOption = (): string | undefined =>
|
||||
program.opts<{ cacheDir?: string }>().cacheDir;
|
||||
|
||||
// A library client that omits `fetchMLData`, so the point commands below do not
|
||||
// kick the library's background ML backfill: they read metadata, or fetch one
|
||||
// file's content, and exit. `backup` and `backup-metadata` handle ML on their
|
||||
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
|
||||
// originals through the on-disk cache.
|
||||
const readLibraryClient = (client: Client): LibraryClient => ({
|
||||
whoami: () => client.whoami(),
|
||||
collectionsSince: (args) => client.collectionsSince(args),
|
||||
filesSince: (args) => client.filesSince(args),
|
||||
contentSource: () => client.contentSource(),
|
||||
});
|
||||
|
||||
// Open a library for a single point command: the aggressive background precache
|
||||
// (issue #48) is off — a one-shot `collections` or `get` must not start
|
||||
// downloading the whole account — and the refresh interval is long so no second
|
||||
// refresh fires mid-command.
|
||||
const openReadLibrary = (client: Client): Promise<Library> =>
|
||||
Library.open({
|
||||
client: readLibraryClient(client),
|
||||
cacheDirectory: cacheDirOption(),
|
||||
refreshIntervalSeconds: 3600,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
|
||||
// Close the library and exit once stdout/stderr have drained. `process.exit`
|
||||
// alone can truncate buffered piped output, and the library keeps the event
|
||||
// loop alive with a background refresh, so a plain return could hang; this does
|
||||
// neither.
|
||||
const finish = (lib: Library | undefined, code: number): void => {
|
||||
lib?.close();
|
||||
const pending = [stdout, stderr].filter((s) => s.writableLength > 0);
|
||||
if (pending.length === 0) {
|
||||
process.exit(code);
|
||||
return;
|
||||
}
|
||||
let remaining = pending.length;
|
||||
for (const s of pending) {
|
||||
s.once("drain", () => {
|
||||
if (--remaining === 0) process.exit(code);
|
||||
});
|
||||
}
|
||||
};
|
||||
.version("0.0.0");
|
||||
|
||||
program
|
||||
.command("login")
|
||||
@@ -183,11 +116,7 @@ program
|
||||
.action(async (opts: { json?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const lib = await openReadLibrary(client);
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52): the pre-library CLI printed current state in
|
||||
// this order, not the albums projection's newest-first order.
|
||||
const collections = await freshCollections(lib);
|
||||
const collections = await client.listCollections();
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(
|
||||
@@ -211,7 +140,6 @@ program
|
||||
);
|
||||
}
|
||||
}
|
||||
finish(lib, 0);
|
||||
});
|
||||
|
||||
program
|
||||
@@ -231,29 +159,36 @@ program
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(client);
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52). Each file prints from its own decrypted
|
||||
// metadata (raw title, microsecond creationTime) via cli-output, and in
|
||||
// the pre-library CLI's enumeration order, not the projection's
|
||||
// newest-first order.
|
||||
const files = await freshFiles(lib, collectionID);
|
||||
if (!files) {
|
||||
const collections = await client.listCollections();
|
||||
const col = collections.find((c) => c.id === collectionID);
|
||||
if (!col) {
|
||||
stderr.write(`Collection ${collectionID} not found\n`);
|
||||
finish(lib, 1);
|
||||
return;
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const files = await client.listFiles(col.id, col.key);
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(
|
||||
JSON.stringify(files.map(fileListRow), null, 2) + "\n",
|
||||
JSON.stringify(
|
||||
files.map((f) => ({
|
||||
id: f.id,
|
||||
title: f.metadata.title,
|
||||
fileType: f.metadata.fileType,
|
||||
creationTime: f.metadata.creationTime,
|
||||
collectionID: f.collectionID,
|
||||
})),
|
||||
null,
|
||||
2,
|
||||
) + "\n",
|
||||
);
|
||||
} else {
|
||||
for (const file of files) {
|
||||
stdout.write(fileListLine(file) + "\n");
|
||||
for (const f of files) {
|
||||
stdout.write(
|
||||
`${f.id}\t${f.metadata.fileType}\t${f.metadata.title}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
finish(lib, 0);
|
||||
});
|
||||
|
||||
program
|
||||
@@ -261,70 +196,98 @@ program
|
||||
.description("Download and decrypt a single file")
|
||||
.argument("<fileID>", "File ID (from `quak files`)")
|
||||
.option("--out <path>", "Output file path")
|
||||
.option("--collection <id>", "Accepted for compatibility; ignored")
|
||||
.action(async (fileIDStr: string, opts: { out?: string }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
stderr.write("Invalid file ID\n");
|
||||
process.exit(1);
|
||||
}
|
||||
.option(
|
||||
"--collection <id>",
|
||||
"Collection ID (required to look up the file key)",
|
||||
)
|
||||
.action(
|
||||
async (
|
||||
fileIDStr: string,
|
||||
opts: { out?: string; collection?: string },
|
||||
) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
stderr.write("Invalid file ID\n");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(client);
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
const collections = await client.listCollections();
|
||||
let targetCol;
|
||||
if (opts.collection) {
|
||||
targetCol = collections.find(
|
||||
(c) => c.id === Number(opts.collection),
|
||||
);
|
||||
}
|
||||
|
||||
// Search all collections (or the specified one) for the file
|
||||
const searchCols = targetCol ? [targetCol] : collections;
|
||||
for (const col of searchCols) {
|
||||
const files = await client.listFiles(col.id, col.key);
|
||||
const file = files.find((f) => f.id === fileID);
|
||||
if (file) {
|
||||
const result = await client.downloadFile(file, opts.out);
|
||||
stderr.write(
|
||||
`${result.bytesWritten} bytes -> ${result.path}\n`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
}
|
||||
stderr.write(`File ${fileID} not found\n`);
|
||||
finish(lib, 1);
|
||||
return;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await photo.original();
|
||||
// Default name is the file's own title, as the pre-library CLI used
|
||||
// (not the editedName-preferring projection title) (issue #52).
|
||||
const outPath = opts.out ?? originalName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
finish(lib, 0);
|
||||
});
|
||||
process.exit(1);
|
||||
},
|
||||
);
|
||||
|
||||
program
|
||||
.command("get-thumb")
|
||||
.description("Download and decrypt a thumbnail")
|
||||
.argument("<fileID>", "File ID (from `quak files`)")
|
||||
.option("--out <path>", "Output file path")
|
||||
.option("--collection <id>", "Accepted for compatibility; ignored")
|
||||
.action(async (fileIDStr: string, opts: { out?: string }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
stderr.write("Invalid file ID\n");
|
||||
process.exit(1);
|
||||
}
|
||||
.option(
|
||||
"--collection <id>",
|
||||
"Collection ID (required to look up the file key)",
|
||||
)
|
||||
.action(
|
||||
async (
|
||||
fileIDStr: string,
|
||||
opts: { out?: string; collection?: string },
|
||||
) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const fileID = Number(fileIDStr);
|
||||
if (!Number.isFinite(fileID)) {
|
||||
stderr.write("Invalid file ID\n");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(client);
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
const collections = await client.listCollections();
|
||||
let targetCol;
|
||||
if (opts.collection) {
|
||||
targetCol = collections.find(
|
||||
(c) => c.id === Number(opts.collection),
|
||||
);
|
||||
}
|
||||
|
||||
const searchCols = targetCol ? [targetCol] : collections;
|
||||
for (const col of searchCols) {
|
||||
const files = await client.listFiles(col.id, col.key);
|
||||
const file = files.find((f) => f.id === fileID);
|
||||
if (file) {
|
||||
const result = await client.downloadThumbnail(
|
||||
file,
|
||||
opts.out,
|
||||
);
|
||||
stderr.write(
|
||||
`${result.bytesWritten} bytes -> ${result.path}\n`,
|
||||
);
|
||||
return;
|
||||
}
|
||||
}
|
||||
stderr.write(`File ${fileID} not found\n`);
|
||||
finish(lib, 1);
|
||||
return;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await photo.thumbnail();
|
||||
// Default name is thumb_<file's own title>, as the pre-library CLI
|
||||
// used (not the projection title) (issue #52).
|
||||
const outPath = opts.out ?? thumbnailName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
finish(lib, 0);
|
||||
});
|
||||
process.exit(1);
|
||||
},
|
||||
);
|
||||
|
||||
program
|
||||
.command("backup-metadata")
|
||||
@@ -340,12 +303,10 @@ program
|
||||
.action(async (dir: string, opts: { exif?: boolean; all?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const lib = await openReadLibrary(client);
|
||||
await runMetadataBackup(lib, client, dir, {
|
||||
await runMetadataBackup(client, dir, {
|
||||
exif: opts.exif || opts.all,
|
||||
onProgress: (msg) => stderr.write(msg + "\n"),
|
||||
});
|
||||
finish(lib, 0);
|
||||
});
|
||||
|
||||
program
|
||||
@@ -360,17 +321,14 @@ program
|
||||
const client = requireSession();
|
||||
|
||||
stderr.write("Starting backup...\n");
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
downloadDirectory: dir,
|
||||
cacheDirectory: cacheDirOption(),
|
||||
});
|
||||
const lib = await Library.open({ client, downloadDirectory: dir });
|
||||
const result = await lib.backup({
|
||||
downloadDirectory: dir,
|
||||
onProgress: (msg) => {
|
||||
if (!opts.json) stderr.write(msg + "\n");
|
||||
},
|
||||
});
|
||||
lib.close();
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(JSON.stringify(result, null, 2) + "\n");
|
||||
@@ -390,7 +348,7 @@ program
|
||||
}
|
||||
}
|
||||
|
||||
finish(lib, result.failed > 0 ? 1 : 0);
|
||||
process.exit(result.failed > 0 ? 1 : 0);
|
||||
});
|
||||
|
||||
const helper = program
|
||||
@@ -404,8 +362,7 @@ helper
|
||||
.action(async (opts: { json?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const lib = await openReadLibrary(client);
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
const missing = await listMissingThumbnails(client, (msg) => {
|
||||
if (!opts.json) stderr.write(msg + "\n");
|
||||
});
|
||||
|
||||
@@ -425,7 +382,6 @@ helper
|
||||
}
|
||||
}
|
||||
}
|
||||
finish(lib, 0);
|
||||
});
|
||||
|
||||
helper
|
||||
@@ -441,61 +397,44 @@ helper
|
||||
.action(async (opts: { file?: string[]; json?: boolean }) => {
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const lib = await openReadLibrary(client);
|
||||
|
||||
let fileIDs: number[];
|
||||
if (opts.file && opts.file.length > 0) {
|
||||
fileIDs = opts.file.map(Number).filter(Number.isFinite);
|
||||
} else {
|
||||
stderr.write("Scanning for missing thumbnails...\n");
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) => {
|
||||
const missing = await listMissingThumbnails(client, (msg) => {
|
||||
if (!opts.json) stderr.write(msg + "\n");
|
||||
});
|
||||
fileIDs = missing.map((m) => m.fileID);
|
||||
if (fileIDs.length === 0) {
|
||||
stderr.write("No missing thumbnails found.\n");
|
||||
finish(lib, 0);
|
||||
return;
|
||||
}
|
||||
stderr.write(`Found ${fileIDs.length} file(s) to fix.\n`);
|
||||
}
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
fileIDs,
|
||||
(msg) => {
|
||||
if (!opts.json) stderr.write(msg + "\n");
|
||||
},
|
||||
);
|
||||
const results = await fixMissingThumbnails(client, fileIDs, (msg) => {
|
||||
if (!opts.json) stderr.write(msg + "\n");
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(JSON.stringify(results, null, 2) + "\n");
|
||||
} else {
|
||||
const fixed = results.filter((r) => r.status === "fixed").length;
|
||||
const skipped = results.filter(
|
||||
(r) => r.status === "skipped",
|
||||
).length;
|
||||
const failed = results.filter((r) => r.status === "failed").length;
|
||||
const ok = results.filter((r) => r.success).length;
|
||||
const fail = results.filter((r) => !r.success).length;
|
||||
stderr.write(`\n--- Done ---\n`);
|
||||
stderr.write(` Fixed: ${fixed}\n`);
|
||||
stderr.write(` Skipped: ${skipped}\n`);
|
||||
stderr.write(` Failed: ${failed}\n`);
|
||||
if (skipped > 0) {
|
||||
stderr.write("\nSkipped (unsupported format):\n");
|
||||
for (const r of results.filter((r) => r.status === "skipped")) {
|
||||
stderr.write(` ${r.fileID}\t${r.title}\t${r.reason}\n`);
|
||||
}
|
||||
}
|
||||
if (failed > 0) {
|
||||
stderr.write(` Fixed: ${ok}\n`);
|
||||
stderr.write(` Failed: ${fail}\n`);
|
||||
if (fail > 0) {
|
||||
stderr.write("\nFailed files:\n");
|
||||
for (const r of results.filter((r) => r.status === "failed")) {
|
||||
stderr.write(` ${r.fileID}\t${r.title}\t${r.reason}\n`);
|
||||
for (const r of results.filter((r) => !r.success)) {
|
||||
stderr.write(` ${r.fileID}\t${r.title}\t${r.error}\n`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
finish(lib, results.some((r) => r.status === "failed") ? 1 : 0);
|
||||
process.exit(results.some((r) => !r.success) ? 1 : 0);
|
||||
});
|
||||
|
||||
await init();
|
||||
|
||||
@@ -1,40 +0,0 @@
|
||||
// How the CLI presents a file's identity in `files`, `get`, and `get-thumb`.
|
||||
//
|
||||
// These read the file's own decrypted metadata — the raw title and the
|
||||
// creationTime in microseconds — rather than the `PhotoRecord` projection the
|
||||
// rest of the library exposes. The projection prefers `editedName`/`editedTime`
|
||||
// and reports time in milliseconds, which is right for a photo browser but
|
||||
// would change the CLI's externally-visible output. The pre-library CLI printed
|
||||
// `metadata.title` and `metadata.creationTime` and named downloads after
|
||||
// `metadata.title`, and issue #52 requires that output stay byte-identical, so
|
||||
// the commands shape their output from the raw `EnteFile` through here.
|
||||
|
||||
import type { EnteFile, FileType, Microseconds } from "./model/types.js";
|
||||
|
||||
// One row of `quak files --json`.
|
||||
export interface FileListRow {
|
||||
id: number;
|
||||
title: string;
|
||||
fileType: FileType;
|
||||
creationTime: Microseconds;
|
||||
collectionID: number;
|
||||
}
|
||||
|
||||
export const fileListRow = (file: EnteFile): FileListRow => ({
|
||||
id: file.id,
|
||||
title: file.metadata.title,
|
||||
fileType: file.metadata.fileType,
|
||||
creationTime: file.metadata.creationTime,
|
||||
collectionID: file.collectionID,
|
||||
});
|
||||
|
||||
// One line of `quak files` in its human, tab-separated form.
|
||||
export const fileListLine = (file: EnteFile): string =>
|
||||
`${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
|
||||
|
||||
// Default output path for `quak get` when `--out` is not given.
|
||||
export const originalName = (file: EnteFile): string => file.metadata.title;
|
||||
|
||||
// Default output path for `quak get-thumb` when `--out` is not given.
|
||||
export const thumbnailName = (file: EnteFile): string =>
|
||||
`thumb_${file.metadata.title}`;
|
||||
@@ -1,62 +0,0 @@
|
||||
// How the CLI's read commands obtain current data.
|
||||
//
|
||||
// `collections`, `files --collection`, `get`, and `get-thumb` must answer for
|
||||
// the account's state at the moment the command runs, not for whatever the
|
||||
// local cache last happened to hold (owner amendment, issue #36). Each helper
|
||||
// therefore forces a server round-trip through `Library.fresh()` and only then
|
||||
// reads — so a collection, file, or metadata change made elsewhere is visible.
|
||||
//
|
||||
// `collections` and `files` also list in the library's own enumeration order —
|
||||
// `listCollections()`/`listFiles()`, the order the pre-library CLI printed —
|
||||
// rather than the `albums`/`photos` projection's newest-first order, which
|
||||
// re-sorts the rows. The field values still come from each record's raw
|
||||
// metadata via `cli-output.ts`.
|
||||
|
||||
import type { Collection, EnteFile } from "./model/types.js";
|
||||
import type { Photo, PhotosAPI } from "./library/index.js";
|
||||
|
||||
// The slice of `Library` these helpers read. `Library` satisfies it
|
||||
// structurally; a test can drive them with a stand-in that records the
|
||||
// `fresh()` call and serves records in a known enumeration order.
|
||||
export interface FreshReadLibrary {
|
||||
fresh(): Promise<unknown>;
|
||||
listCollections(): Collection[];
|
||||
getCollection(id: number): Collection | undefined;
|
||||
listFiles(collectionID: number): EnteFile[];
|
||||
getFileByID(fileID: number): EnteFile | undefined;
|
||||
photos: Pick<PhotosAPI, "byID">;
|
||||
}
|
||||
|
||||
// Every live collection, current as of a forced refresh, in enumeration order.
|
||||
export const freshCollections = async (
|
||||
lib: FreshReadLibrary,
|
||||
): Promise<Collection[]> => {
|
||||
await lib.fresh();
|
||||
return lib.listCollections();
|
||||
};
|
||||
|
||||
// The files of one collection, current as of a forced refresh, in enumeration
|
||||
// order. `undefined` (not an empty list) when the collection does not exist, so
|
||||
// the caller can tell "no such collection" from "an empty collection".
|
||||
export const freshFiles = async (
|
||||
lib: FreshReadLibrary,
|
||||
collectionID: number,
|
||||
): Promise<EnteFile[] | undefined> => {
|
||||
await lib.fresh();
|
||||
if (!lib.getCollection(collectionID)) return undefined;
|
||||
return lib.listFiles(collectionID);
|
||||
};
|
||||
|
||||
// One file, current as of a forced refresh, resolved to both its content
|
||||
// handle (`Photo`, for fetching bytes) and its raw record (`EnteFile`, for the
|
||||
// default output name and field values). `undefined` when the file is unknown.
|
||||
export const freshFile = async (
|
||||
lib: FreshReadLibrary,
|
||||
fileID: number,
|
||||
): Promise<{ photo: Photo; file: EnteFile } | undefined> => {
|
||||
await lib.fresh();
|
||||
const photo = lib.photos.byID({ fileID });
|
||||
const file = lib.getFileByID(fileID);
|
||||
if (!photo || !file) return undefined;
|
||||
return { photo, file };
|
||||
};
|
||||
+16
-96
@@ -6,17 +6,10 @@
|
||||
// existing copy loaded, that first refresh runs in the background and `open()`
|
||||
// returns as soon as the cached data is ready to serve — a slow or unreachable
|
||||
// server no longer stalls opening. A background timer then refreshes every
|
||||
// `refreshIntervalSeconds`. Every default read is answered from RAM — no
|
||||
// default read touches the network. There is deliberately no `sync()`, no
|
||||
// `refresh()`, no `serverReachable` flag, and no "before each read" mode
|
||||
// (design #36).
|
||||
//
|
||||
// `fresh()` is the one exception (issue #75, an owner amendment to #36): it
|
||||
// forces a refresh, awaits it, and only then hands back the read namespaces, so
|
||||
// a caller that needs server-current data can ask for it. Concurrent `fresh()`
|
||||
// calls coalesce onto one in-flight refresh, and a refresh that fails rejects
|
||||
// the caller (the default reads stay silent and serve the last good copy). The
|
||||
// default methods and the background loop are unchanged.
|
||||
// `refreshIntervalSeconds`. Every read is answered from RAM — no read touches
|
||||
// the network. There is deliberately no `sync()`, no `refresh()`, no
|
||||
// `serverReachable` flag, and no "before each read" mode (design #36): the
|
||||
// only ways state changes are the refreshes above.
|
||||
//
|
||||
// A refresh stages all of its network work first and only mutates the store
|
||||
// once every fetch has succeeded. A refresh that fails partway therefore never
|
||||
@@ -47,7 +40,6 @@ import {
|
||||
type AlbumsAPI,
|
||||
type PhotosAPI,
|
||||
type TimelineAPI,
|
||||
type FreshReads,
|
||||
} from "./read.js";
|
||||
import {
|
||||
ContentCache,
|
||||
@@ -56,7 +48,6 @@ import {
|
||||
type EnsureOptions,
|
||||
type EnsureResult,
|
||||
} from "./content.js";
|
||||
import { makeMLDataAPI, type MLDataAPI } from "./mlsearch.js";
|
||||
import { Precache } from "./precache.js";
|
||||
|
||||
export {
|
||||
@@ -65,7 +56,6 @@ export {
|
||||
type AlbumsAPI,
|
||||
type PhotosAPI,
|
||||
type TimelineAPI,
|
||||
type FreshReads,
|
||||
type PhotoFilter,
|
||||
type TimelineGroup,
|
||||
type GroupBy,
|
||||
@@ -82,7 +72,6 @@ export {
|
||||
type EnsureResult,
|
||||
type EnsureEvent,
|
||||
} from "./content.js";
|
||||
export { type MLDataAPI, type SimilarResult } from "./mlsearch.js";
|
||||
import type { CollectionsPage, FilesPage } from "../client.js";
|
||||
import { MLDATA_BATCH_SIZE, type MLData } from "../mldata-fetch.js";
|
||||
import type { Collection, EnteFile } from "../model/types.js";
|
||||
@@ -239,10 +228,6 @@ export class Library {
|
||||
// The thumbnail-prefetch surface (issue #46): drives the thumbnail pool
|
||||
// with priority, dedup, and abort.
|
||||
readonly thumbnails: ThumbnailsAPI;
|
||||
// The content-similarity search surface over the CLIP index (issue #50).
|
||||
// Present whether or not ML fetching is enabled; with no ML store it
|
||||
// returns empty results.
|
||||
readonly mldata: MLDataAPI;
|
||||
|
||||
private readonly client: LibraryClient;
|
||||
private readonly store: MetadataStore;
|
||||
@@ -254,17 +239,12 @@ export class Library {
|
||||
private readonly onProgress?: RefreshProgressCallback;
|
||||
private readonly pools: RequestPools;
|
||||
// The ML-data cache, present only when the client can fetch ML data.
|
||||
private readonly mlStore?: MLDataStore;
|
||||
private readonly mldata?: MLDataStore;
|
||||
// The local precache (#48), present only when the content cache is.
|
||||
private readonly precache?: Precache;
|
||||
|
||||
private timer?: ReturnType<typeof setTimeout>;
|
||||
// The in-flight refresh cycle, or undefined when none runs. One slot serves
|
||||
// both paths: the background loop skips when it is set, and a fresh read
|
||||
// (issue #75) coalesces onto it or starts one. The promise carries the
|
||||
// cycle's real outcome (it rejects on failure); the background loop ignores
|
||||
// that, a fresh read propagates it.
|
||||
private cycle?: Promise<void>;
|
||||
private refreshing = false;
|
||||
// Guards the ML fetch pass so a slow backfill never runs twice at once; a
|
||||
// refresh whose pass is still running kicks nothing new.
|
||||
private mlFetching = false;
|
||||
@@ -305,7 +285,7 @@ export class Library {
|
||||
this.intervalMs = args.intervalMs;
|
||||
this.onProgress = args.onProgress;
|
||||
this.pools = args.pools;
|
||||
this.mlStore = args.mldata;
|
||||
this.mldata = args.mldata;
|
||||
this.cache = args.cache;
|
||||
this.precache = args.precache;
|
||||
this.lastRecords = this.deriveNow();
|
||||
@@ -328,8 +308,6 @@ export class Library {
|
||||
return this.cache.ensureThumbnails(opts);
|
||||
},
|
||||
};
|
||||
// Reads the ML store live so results grow as ML data is fetched.
|
||||
this.mldata = makeMLDataAPI(() => this.mlStore);
|
||||
}
|
||||
|
||||
// Load the cache and start the refresh loop. With an empty cache the first
|
||||
@@ -447,13 +425,6 @@ export class Library {
|
||||
return this.store.getFile(collectionID, fileID);
|
||||
}
|
||||
|
||||
// Any membership of a file, addressed by file id alone. A file's own
|
||||
// metadata (title, creationTime) is identical across the collections it
|
||||
// belongs to, so this serves the point commands that hold only a fileID.
|
||||
getFileByID(fileID: number): EnteFile | undefined {
|
||||
return this.store.getFileByID(fileID);
|
||||
}
|
||||
|
||||
// A synchronous, RAM-only projection of the whole library into plain
|
||||
// records (no keys), the surface the GUI reads across IPC. Photos are
|
||||
// deduplicated to one record per file and ordered newest first.
|
||||
@@ -482,7 +453,7 @@ export class Library {
|
||||
for (const c of collections) {
|
||||
files += this.store.listFiles(c.id).length;
|
||||
}
|
||||
const ml = this.mlStore?.stats();
|
||||
const ml = this.mldata?.stats();
|
||||
const originals = this.cache?.originalsStatus();
|
||||
const pre = this.precache?.status();
|
||||
return {
|
||||
@@ -505,21 +476,6 @@ export class Library {
|
||||
};
|
||||
}
|
||||
|
||||
// Fresh reads (issue #75, owner amendment to design #36). Force a refresh,
|
||||
// wait for it to complete and persist, then hand back the same
|
||||
// `albums`/`photos`/`timeline` namespaces — now guaranteed to reflect a
|
||||
// completed server round-trip. Concurrent calls coalesce onto one refresh;
|
||||
// a refresh that fails rejects here, where the default namespaces would
|
||||
// instead stay silent and serve the last good copy.
|
||||
async fresh(): Promise<FreshReads> {
|
||||
await this.refreshNow();
|
||||
return {
|
||||
albums: this.albums,
|
||||
photos: this.photos,
|
||||
timeline: this.timeline,
|
||||
};
|
||||
}
|
||||
|
||||
// Back up every in-scope file to `downloadDirectory` in the historical
|
||||
// on-disk layout, with a durable failure ledger (issue #51). Refreshes
|
||||
// first, fetches pending originals (and optional thumbnails) through the
|
||||
@@ -579,48 +535,11 @@ export class Library {
|
||||
this.timer.unref?.();
|
||||
}
|
||||
|
||||
// The background loop's refresh: run a cycle unless one is already in flight
|
||||
// (or the library is closed), and never let a failure escape — the
|
||||
// background path reports errors through `status()`/`onProgress`, it does
|
||||
// not throw. Resolves once the cycle it started (or skipped past) settles.
|
||||
private runRefresh(): Promise<void> {
|
||||
if (this.closed || this.cycle) return Promise.resolve();
|
||||
return this.startCycle().catch(() => {});
|
||||
}
|
||||
|
||||
// A fresh read's refresh (issue #75): force a cycle and await it, rejecting
|
||||
// if it fails. Concurrent fresh reads coalesce onto the one in-flight cycle
|
||||
// — the background loop's included — so they never fan out into redundant
|
||||
// server round-trips.
|
||||
private refreshNow(): Promise<void> {
|
||||
if (this.closed) {
|
||||
return Promise.reject(new Error("the library is closed"));
|
||||
}
|
||||
return this.cycle ?? this.startCycle();
|
||||
}
|
||||
|
||||
// Start one refresh cycle and record it as the in-flight cycle so every
|
||||
// caller coalesces onto it. The returned promise carries the cycle's real
|
||||
// outcome; each caller attaches the handling its own path needs, and the
|
||||
// slot is cleared once the cycle settles.
|
||||
private startCycle(): Promise<void> {
|
||||
const cycle = this.refreshCycle();
|
||||
this.cycle = cycle;
|
||||
void cycle.then(
|
||||
() => {
|
||||
if (this.cycle === cycle) this.cycle = undefined;
|
||||
},
|
||||
() => {
|
||||
if (this.cycle === cycle) this.cycle = undefined;
|
||||
},
|
||||
);
|
||||
return cycle;
|
||||
}
|
||||
|
||||
// One refresh cycle: the network fetch and commit, wrapped in the progress
|
||||
// events and status bookkeeping. Throws when the refresh fails so a fresh
|
||||
// read can reject; `runRefresh` swallows that throw for the background loop.
|
||||
private async refreshCycle(): Promise<void> {
|
||||
// One refresh cycle, guarded so a failure never escapes and overlapping
|
||||
// cycles never run. Errors are reported, not thrown.
|
||||
private async runRefresh(): Promise<void> {
|
||||
if (this.closed || this.refreshing) return;
|
||||
this.refreshing = true;
|
||||
this.emit({ operation: "refresh", status: "started" });
|
||||
try {
|
||||
await this.refreshOnce();
|
||||
@@ -636,7 +555,8 @@ export class Library {
|
||||
const error = err instanceof Error ? err.message : String(err);
|
||||
this.lastError = error;
|
||||
this.emit({ operation: "refresh", status: "failed", error });
|
||||
throw err;
|
||||
} finally {
|
||||
this.refreshing = false;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -741,7 +661,7 @@ export class Library {
|
||||
// advanced), through the metadata pool, and update the CLIP index. Guarded
|
||||
// so passes never overlap; a failure is reported, not thrown.
|
||||
private async runMLFetch(): Promise<void> {
|
||||
const mldata = this.mlStore;
|
||||
const mldata = this.mldata;
|
||||
// Bind so the call keeps the client as its receiver when invoked
|
||||
// through the pool below.
|
||||
const fetchMLData = this.client.fetchMLData?.bind(this.client);
|
||||
|
||||
@@ -1,129 +0,0 @@
|
||||
// The content-similarity search surface over the CLIP index (issue #50).
|
||||
//
|
||||
// This is `lib.mldata`. It answers three questions against the ML-data cache
|
||||
// (#49) without touching the network:
|
||||
//
|
||||
// - `forFile` returns the whole stored payload (face boxes, landmarks,
|
||||
// embedding) for a file, read from disk on demand — the only method here
|
||||
// that touches the disk, and the only one that is async.
|
||||
// - `similar` and `searchByEmbedding` rank fileIDs by cosine similarity over
|
||||
// the packed `Float32Array` index alone. That index (~50k×512) already
|
||||
// lives in RAM, so each query is a plain loop over it and nothing else.
|
||||
//
|
||||
// quak bundles no text encoder (owner-deferred), so `searchByEmbedding` takes
|
||||
// the query vector the caller has produced elsewhere; `similar` uses the
|
||||
// query file's own indexed embedding.
|
||||
|
||||
import type { MLData } from "../mldata-fetch.js";
|
||||
import type { MLDataStore, MLIndex } from "./mldata.js";
|
||||
|
||||
// How many nearest files a query returns when the caller names no limit.
|
||||
const DEFAULT_LIMIT = 20;
|
||||
|
||||
// One ranked result: a fileID and its cosine similarity to the query, in
|
||||
// [-1, 1]. Callers wanting only the ids read `.fileID`.
|
||||
export interface SimilarResult {
|
||||
fileID: number;
|
||||
score: number;
|
||||
}
|
||||
|
||||
export interface MLDataAPI {
|
||||
// The whole stored ML payload for a file, or undefined when it is not
|
||||
// cached. Reads the payload from disk, so it is async.
|
||||
forFile(args: { fileID: number }): Promise<MLData | undefined>;
|
||||
// The files nearest the given file by cosine over their CLIP embeddings,
|
||||
// most similar first, excluding the file itself. Empty when the file has
|
||||
// no indexed embedding.
|
||||
similar(args: { fileID: number; limit?: number }): SimilarResult[];
|
||||
// The files nearest a caller-supplied query embedding by cosine, most
|
||||
// similar first. Empty when the query is the wrong length for the index,
|
||||
// has zero magnitude, or the index is empty.
|
||||
searchByEmbedding(args: {
|
||||
embedding: ArrayLike<number>;
|
||||
limit?: number;
|
||||
}): SimilarResult[];
|
||||
}
|
||||
|
||||
// Rank the packed index by cosine similarity to `query`, most similar first,
|
||||
// and return the top `limit`. `skip` (a query file's own id) is left out. Both
|
||||
// each row's magnitude and the query's are computed here rather than cached:
|
||||
// the index mutates as ML data is fetched, and one plain pass over ~50k×512
|
||||
// floats is fast enough that a norm cache would only add a staleness bug. A
|
||||
// zero-magnitude vector has no direction, so it is dropped rather than divided
|
||||
// by zero.
|
||||
const topByCosine = (
|
||||
index: MLIndex,
|
||||
query: ArrayLike<number>,
|
||||
limit: number,
|
||||
skip?: number,
|
||||
): SimilarResult[] => {
|
||||
const { fileIDs, embeddingLength, embeddings } = index;
|
||||
if (embeddingLength === 0 || query.length !== embeddingLength) return [];
|
||||
|
||||
// Every indexed read below is in range: the inner loops run to
|
||||
// `embeddingLength`, the query is exactly that long (checked above), and
|
||||
// the packed buffer holds `fileIDs.length * embeddingLength` floats.
|
||||
// `noUncheckedIndexedAccess` still widens each read to `number | undefined`,
|
||||
// so they are asserted non-null rather than paying a per-element guard in
|
||||
// this hot ~50k×512 loop.
|
||||
let queryNorm = 0;
|
||||
for (let k = 0; k < embeddingLength; k++) {
|
||||
const q = query[k]!;
|
||||
queryNorm += q * q;
|
||||
}
|
||||
queryNorm = Math.sqrt(queryNorm);
|
||||
if (queryNorm === 0) return [];
|
||||
|
||||
const results: SimilarResult[] = [];
|
||||
for (let i = 0; i < fileIDs.length; i++) {
|
||||
const id = fileIDs[i]!;
|
||||
if (id === skip) continue;
|
||||
const base = i * embeddingLength;
|
||||
let dot = 0;
|
||||
let norm = 0;
|
||||
for (let k = 0; k < embeddingLength; k++) {
|
||||
const v = embeddings[base + k]!;
|
||||
dot += query[k]! * v;
|
||||
norm += v * v;
|
||||
}
|
||||
if (norm === 0) continue;
|
||||
results.push({
|
||||
fileID: id,
|
||||
score: dot / (queryNorm * Math.sqrt(norm)),
|
||||
});
|
||||
}
|
||||
|
||||
// Descending score, ties broken by ascending fileID for a stable order.
|
||||
results.sort((a, b) => b.score - a.score || a.fileID - b.fileID);
|
||||
return results.slice(0, Math.max(0, Math.trunc(limit)));
|
||||
};
|
||||
|
||||
// Build the search surface over a store the library supplies lazily (the store
|
||||
// is absent when the client cannot fetch ML data). Reading it per call keeps
|
||||
// the surface current as the index grows.
|
||||
export const makeMLDataAPI = (
|
||||
store: () => MLDataStore | undefined,
|
||||
): MLDataAPI => ({
|
||||
forFile: ({ fileID }): Promise<MLData | undefined> => {
|
||||
const s = store();
|
||||
return s ? s.readPayload(fileID) : Promise.resolve(undefined);
|
||||
},
|
||||
similar: ({ fileID, limit }): SimilarResult[] => {
|
||||
const s = store();
|
||||
if (!s) return [];
|
||||
const index = s.getIndex();
|
||||
const pos = index.fileIDs.indexOf(fileID);
|
||||
if (pos < 0) return [];
|
||||
const base = pos * index.embeddingLength;
|
||||
const query = index.embeddings.subarray(
|
||||
base,
|
||||
base + index.embeddingLength,
|
||||
);
|
||||
return topByCosine(index, query, limit ?? DEFAULT_LIMIT, fileID);
|
||||
},
|
||||
searchByEmbedding: ({ embedding, limit }): SimilarResult[] => {
|
||||
const s = store();
|
||||
if (!s) return [];
|
||||
return topByCosine(s.getIndex(), embedding, limit ?? DEFAULT_LIMIT);
|
||||
},
|
||||
});
|
||||
@@ -193,15 +193,6 @@ export interface TimelineAPI {
|
||||
groups(args: { groupBy: GroupBy; filter?: PhotoFilter }): TimelineGroup[];
|
||||
}
|
||||
|
||||
// The surface `Library.fresh()` resolves to (issue #75). It is the same three
|
||||
// read namespaces as the default `albums`/`photos`/`timeline`, handed back only
|
||||
// after a forced refresh has brought the local copy current.
|
||||
export interface FreshReads {
|
||||
albums: AlbumsAPI;
|
||||
photos: PhotosAPI;
|
||||
timeline: TimelineAPI;
|
||||
}
|
||||
|
||||
export const makeAlbumsAPI = (
|
||||
derive: () => DerivedRecords,
|
||||
content?: PhotoContent,
|
||||
|
||||
+25
-32
@@ -1,9 +1,15 @@
|
||||
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import {
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
readFileSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import * as jpeg from "jpeg-js";
|
||||
import exifReader from "exif-reader";
|
||||
import type { Client } from "./client.js";
|
||||
import type { Library, Photo } from "./library/index.js";
|
||||
import { fetchMLData } from "./mldata-fetch.js";
|
||||
import type { EnteFile } from "./model/types.js";
|
||||
|
||||
@@ -98,29 +104,24 @@ const extractImageMetadata = (
|
||||
}
|
||||
};
|
||||
|
||||
// Read a file's original bytes through the library's content cache and extract
|
||||
// its embedded image metadata. The bytes come from `photo.original()` — the
|
||||
// same on-disk cache the rest of the library fills — rather than a fresh
|
||||
// per-call download to a throwaway temp file.
|
||||
const extractExif = async (
|
||||
photo: Photo,
|
||||
client: Client,
|
||||
file: EnteFile,
|
||||
): Promise<Record<string, unknown> | undefined> => {
|
||||
const tmpDir = mkdtempSync(join(tmpdir(), "quak-exif-"));
|
||||
try {
|
||||
const { path } = await photo.original();
|
||||
const fileBytes = new Uint8Array(readFileSync(path));
|
||||
const origPath = join(tmpDir, "original");
|
||||
await client.downloadFile(file, origPath);
|
||||
const fileBytes = new Uint8Array(readFileSync(origPath));
|
||||
return extractImageMetadata(fileBytes);
|
||||
} catch {
|
||||
return undefined;
|
||||
} finally {
|
||||
rmSync(tmpDir, { recursive: true, force: true });
|
||||
}
|
||||
};
|
||||
|
||||
// Dump every decrypted metadata layer the account holds into a directory tree
|
||||
// of plain JSON: account, per-collection, and per-file records including the
|
||||
// private and public magic metadata and (by default) the ML data. Collections
|
||||
// and files are enumerated from the library's cache rather than a fresh server
|
||||
// scan; the ML fetch and EXIF extraction are unchanged.
|
||||
export const runMetadataBackup = async (
|
||||
lib: Library,
|
||||
client: Client,
|
||||
outDir: string,
|
||||
opts?: MetadataBackupOptions,
|
||||
@@ -138,19 +139,13 @@ export const runMetadataBackup = async (
|
||||
);
|
||||
|
||||
log("Fetching collections...");
|
||||
const collections = await client.listCollections();
|
||||
|
||||
// Enumerate through the library's read surface. Each album carries its
|
||||
// photos, but the full decrypted `Collection`/`EnteFile` records (with the
|
||||
// magic-metadata layers this dump exists to preserve) come from the
|
||||
// library's by-id accessors.
|
||||
const allFiles: { file: EnteFile; photo: Photo; colDirName: string }[] = [];
|
||||
const allFiles: { file: EnteFile; colDirName: string }[] = [];
|
||||
const fileKeys = new Map<number, Uint8Array>();
|
||||
const seenFileIDs = new Set<number>();
|
||||
|
||||
for (const album of lib.albums.list()) {
|
||||
const col = lib.getCollection(album.collectionID);
|
||||
if (!col) continue;
|
||||
|
||||
for (const col of collections) {
|
||||
const dirName = `${col.id}-${sanitizePath(col.name || "unnamed")}`;
|
||||
const colDir = join(outDir, "collections", dirName);
|
||||
mkdirSync(colDir, { recursive: true });
|
||||
@@ -175,13 +170,11 @@ export const runMetadataBackup = async (
|
||||
);
|
||||
|
||||
log(`[${col.name}] Fetching files...`);
|
||||
const photos = album.photos.list();
|
||||
log(`[${col.name}] ${photos.length} file(s)`);
|
||||
const files = await client.listFiles(col.id, col.key);
|
||||
log(`[${col.name}] ${files.length} file(s)`);
|
||||
|
||||
for (const photo of photos) {
|
||||
const file = lib.getFile(col.id, photo.fileID);
|
||||
if (!file) continue;
|
||||
allFiles.push({ file, photo, colDirName: dirName });
|
||||
for (const file of files) {
|
||||
allFiles.push({ file, colDirName: dirName });
|
||||
if (!seenFileIDs.has(file.id)) {
|
||||
fileKeys.set(file.id, file.key);
|
||||
seenFileIDs.add(file.id);
|
||||
@@ -198,7 +191,7 @@ export const runMetadataBackup = async (
|
||||
log(`Got ML data for ${mlDataMap.size} file(s)`);
|
||||
|
||||
const writtenFileIDs = new Set<number>();
|
||||
for (const { file, photo, colDirName } of allFiles) {
|
||||
for (const { file, colDirName } of allFiles) {
|
||||
const colDir = join(outDir, "collections", colDirName);
|
||||
|
||||
const fileMeta: Record<string, unknown> = {
|
||||
@@ -217,7 +210,7 @@ export const runMetadataBackup = async (
|
||||
|
||||
if (wantExif && !writtenFileIDs.has(file.id)) {
|
||||
log(`[${file.metadata.title}] Extracting EXIF...`);
|
||||
const exifData = await extractExif(photo);
|
||||
const exifData = await extractExif(client, file);
|
||||
if (exifData) fileMeta.imageMetadata = exifData;
|
||||
}
|
||||
writtenFileIDs.add(file.id);
|
||||
|
||||
+71
-128
@@ -2,10 +2,13 @@ import { createHash } from "node:crypto";
|
||||
import { readFileSync } from "node:fs";
|
||||
import * as jpeg from "jpeg-js";
|
||||
import type { Client } from "./client.js";
|
||||
import type { Library } from "./library/index.js";
|
||||
import { ApiError } from "./api/client.js";
|
||||
import { encryptBlob, toBase64 } from "./crypto/index.js";
|
||||
import { downloadFile } from "./download/index.js";
|
||||
import type { EnteFile } from "./model/types.js";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
|
||||
const THUMB_MAX_DIMENSION = 720;
|
||||
const THUMB_JPEG_QUALITY = 50;
|
||||
@@ -17,51 +20,34 @@ export interface MissingThumbnailInfo {
|
||||
reason: string;
|
||||
}
|
||||
|
||||
// Three outcomes, not two. "fixed": a thumbnail was generated and uploaded.
|
||||
// "failed": something went wrong (download, encode, upload) and the file still
|
||||
// has no thumbnail. "skipped": the file is a format this helper cannot
|
||||
// regenerate — a video, or an image that is not a baseline JPEG. Skipped is a
|
||||
// deliberate, expected outcome, not an error (issue #17): the repair path is
|
||||
// JPEG-only because `jpeg-js` is, and a PNG or HEIC is left for a format-aware
|
||||
// tool rather than reported as a failure.
|
||||
export type ThumbnailFixStatus = "fixed" | "skipped" | "failed";
|
||||
|
||||
export interface ThumbnailFixResult {
|
||||
fileID: number;
|
||||
title: string;
|
||||
collection: string;
|
||||
status: ThumbnailFixStatus;
|
||||
// Why the file was skipped or failed; unset when it was fixed.
|
||||
reason?: string;
|
||||
success: boolean;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
export type ProgressCallback = (message: string) => void;
|
||||
|
||||
// Enumerate every file the library knows about, newest album first, each file
|
||||
// once, and report those whose server-side thumbnail is missing. "Missing" is
|
||||
// only two answers: an empty body, or a 404. Any other error reaching this
|
||||
// point has already exhausted its retries — a failing server, a dropped
|
||||
// connection, a deadline — and says nothing about whether the thumbnail
|
||||
// exists, so it is logged and the file is left unreported. That distinction is
|
||||
// what stops `fix-missing-thumbnails` from regenerating and uploading over
|
||||
// thumbnails that were fine all along while the CDN was briefly returning 500s.
|
||||
export const listMissingThumbnails = async (
|
||||
lib: Library,
|
||||
client: Client,
|
||||
onProgress?: ProgressCallback,
|
||||
): Promise<MissingThumbnailInfo[]> => {
|
||||
const log = onProgress ?? (() => {});
|
||||
const api = client.getApiClient();
|
||||
const missing: MissingThumbnailInfo[] = [];
|
||||
const seen = new Set<number>();
|
||||
|
||||
for (const album of lib.albums.list()) {
|
||||
log(`[${album.name}] Checking thumbnails...`);
|
||||
for (const photo of album.photos.list()) {
|
||||
if (seen.has(photo.fileID)) continue;
|
||||
seen.add(photo.fileID);
|
||||
const collections = await client.listCollections();
|
||||
for (const col of collections) {
|
||||
log(`[${col.name}] Checking thumbnails...`);
|
||||
const files = await client.listFiles(col.id, col.key);
|
||||
for (const file of files) {
|
||||
if (seen.has(file.id)) continue;
|
||||
seen.add(file.id);
|
||||
try {
|
||||
const stream = await api.getThumbnailStream(photo.fileID);
|
||||
const api = client.getApiClient();
|
||||
const stream = await api.getThumbnailStream(file.id);
|
||||
const reader = stream.getReader();
|
||||
let totalBytes = 0;
|
||||
for (;;) {
|
||||
@@ -71,23 +57,35 @@ export const listMissingThumbnails = async (
|
||||
}
|
||||
if (totalBytes === 0) {
|
||||
missing.push({
|
||||
fileID: photo.fileID,
|
||||
title: photo.title,
|
||||
collection: album.name,
|
||||
fileID: file.id,
|
||||
title: file.metadata.title,
|
||||
collection: col.name,
|
||||
reason: "empty thumbnail (0 bytes)",
|
||||
});
|
||||
}
|
||||
} catch (err) {
|
||||
// A 404 is the server stating the thumbnail is not there:
|
||||
// that, and an empty body, are the only two answers that mean
|
||||
// "missing". Anything else reaching this point is a failure
|
||||
// that already exhausted its retries — a failing server, a
|
||||
// dropped connection, a deadline — and says nothing about
|
||||
// whether the thumbnail exists.
|
||||
//
|
||||
// The distinction is what stops `helper
|
||||
// fix-missing-thumbnails` from downloading originals,
|
||||
// regenerating thumbnails and uploading them over thumbnails
|
||||
// that were fine all along, because the CDN was briefly
|
||||
// returning 500s while this ran.
|
||||
if (err instanceof ApiError && err.status === 404) {
|
||||
missing.push({
|
||||
fileID: photo.fileID,
|
||||
title: photo.title,
|
||||
collection: album.name,
|
||||
fileID: file.id,
|
||||
title: file.metadata.title,
|
||||
collection: col.name,
|
||||
reason: "thumbnail not found (HTTP 404)",
|
||||
});
|
||||
} else {
|
||||
log(
|
||||
`[${album.name}] Could not check ${photo.title}: ${err instanceof Error ? err.message : String(err)} (not reported as missing)`,
|
||||
`[${col.name}] Could not check ${file.metadata.title}: ${err instanceof Error ? err.message : String(err)} (not reported as missing)`,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -96,7 +94,7 @@ export const listMissingThumbnails = async (
|
||||
return missing;
|
||||
};
|
||||
|
||||
// Bilinear resize of an RGBA pixel buffer.
|
||||
// Bilinear resize of RGBA pixel buffer
|
||||
const resizeRGBA = (
|
||||
src: Uint8Array,
|
||||
srcW: number,
|
||||
@@ -163,33 +161,7 @@ const generateThumbnail = (fileBytes: Uint8Array): Uint8Array => {
|
||||
return new Uint8Array(encoded.data);
|
||||
};
|
||||
|
||||
// A baseline/JFIF JPEG starts with the SOI marker 0xFFD8. `jpeg-js` decodes
|
||||
// only JPEG, so this signature check is what separates a file the helper can
|
||||
// regenerate from one it must skip: a PNG, HEIC, or the odd non-image byte
|
||||
// stream all fail this and are reported as skipped rather than crashing the
|
||||
// decoder into an opaque failure (issue #17).
|
||||
const isJpeg = (bytes: Uint8Array): boolean =>
|
||||
bytes.length >= 2 && bytes[0] === 0xff && bytes[1] === 0xd8;
|
||||
|
||||
// The reason a file cannot have a JPEG thumbnail regenerated for it from its
|
||||
// metadata alone, before any bytes are fetched, or undefined when it might. A
|
||||
// non-image (video, live photo) is unsupported outright; a still image still
|
||||
// has to be checked against its actual bytes once downloaded.
|
||||
const unsupportedByType = (file: EnteFile): string | undefined => {
|
||||
if (file.metadata.fileType !== "image") {
|
||||
return `unsupported file type: ${file.metadata.fileType} (only JPEG images can be regenerated)`;
|
||||
}
|
||||
return undefined;
|
||||
};
|
||||
|
||||
// Regenerate and upload a thumbnail for each requested file. Originals are read
|
||||
// through the library's content cache (`photo.original()`); the generated
|
||||
// thumbnail is JPEG-encoded, encrypted under the file's own key, and registered
|
||||
// with the server — the encrypt-and-upload path is unchanged. Each file is
|
||||
// resolved to one outcome (fixed / skipped / failed) and a failure on one file
|
||||
// never stops the others.
|
||||
export const fixMissingThumbnails = async (
|
||||
lib: Library,
|
||||
client: Client,
|
||||
fileIDs: number[],
|
||||
onProgress?: ProgressCallback,
|
||||
@@ -198,24 +170,19 @@ export const fixMissingThumbnails = async (
|
||||
const results: ThumbnailFixResult[] = [];
|
||||
const api = client.getApiClient();
|
||||
|
||||
// Resolve each requested fileID to its file record and owning album by
|
||||
// enumerating the library, each file taken from the first album that holds
|
||||
// it. The raw `EnteFile` carries the per-file key the thumbnail is
|
||||
// encrypted under, which the projected records deliberately do not.
|
||||
const wanted = new Set(fileIDs);
|
||||
const collections = await client.listCollections();
|
||||
const fileMap = new Map<
|
||||
number,
|
||||
{ file: EnteFile; collectionName: string }
|
||||
>();
|
||||
for (const album of lib.albums.list()) {
|
||||
for (const photo of album.photos.list()) {
|
||||
if (!wanted.has(photo.fileID) || fileMap.has(photo.fileID))
|
||||
continue;
|
||||
const file = lib.getFile(album.collectionID, photo.fileID);
|
||||
if (file) {
|
||||
fileMap.set(photo.fileID, {
|
||||
|
||||
for (const col of collections) {
|
||||
const files = await client.listFiles(col.id, col.key);
|
||||
for (const file of files) {
|
||||
if (fileIDs.includes(file.id) && !fileMap.has(file.id)) {
|
||||
fileMap.set(file.id, {
|
||||
file,
|
||||
collectionName: album.name,
|
||||
collectionName: col.name,
|
||||
});
|
||||
}
|
||||
}
|
||||
@@ -228,61 +195,33 @@ export const fixMissingThumbnails = async (
|
||||
fileID,
|
||||
title: "unknown",
|
||||
collection: "unknown",
|
||||
status: "failed",
|
||||
reason: "file not found in any collection",
|
||||
success: false,
|
||||
error: "file not found in any collection",
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
const { file, collectionName } = entry;
|
||||
const title = file.metadata.title;
|
||||
|
||||
const typeReason = unsupportedByType(file);
|
||||
if (typeReason) {
|
||||
log(`[${collectionName}] Skipping ${title}: ${typeReason}`);
|
||||
results.push({
|
||||
fileID,
|
||||
title,
|
||||
collection: collectionName,
|
||||
status: "skipped",
|
||||
reason: typeReason,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const tmpDir = mkdtempSync(join(tmpdir(), "quak-thumb-"));
|
||||
|
||||
try {
|
||||
const photo = lib.photos.byID({ fileID });
|
||||
if (!photo) {
|
||||
throw new Error("file not present in the library cache");
|
||||
}
|
||||
log(
|
||||
`[${collectionName}] Downloading ${file.metadata.title} for thumbnail generation...`,
|
||||
);
|
||||
const origPath = join(tmpDir, "original");
|
||||
await downloadFile(api, file, origPath);
|
||||
|
||||
log(
|
||||
`[${collectionName}] Downloading ${title} for thumbnail generation...`,
|
||||
`[${collectionName}] Generating thumbnail for ${file.metadata.title}...`,
|
||||
);
|
||||
const { path } = await photo.original();
|
||||
const fileBytes = new Uint8Array(readFileSync(path));
|
||||
|
||||
if (!isJpeg(fileBytes)) {
|
||||
const reason =
|
||||
"unsupported image format (only baseline JPEG can be regenerated)";
|
||||
log(`[${collectionName}] Skipping ${title}: ${reason}`);
|
||||
results.push({
|
||||
fileID,
|
||||
title,
|
||||
collection: collectionName,
|
||||
status: "skipped",
|
||||
reason,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
log(`[${collectionName}] Generating thumbnail for ${title}...`);
|
||||
const thumbJpeg = generateThumbnail(fileBytes);
|
||||
const fileBytes = readFileSync(origPath);
|
||||
const thumbJpeg = generateThumbnail(new Uint8Array(fileBytes));
|
||||
|
||||
log(
|
||||
`[${collectionName}] Encrypting and uploading thumbnail (${thumbJpeg.length} bytes)...`,
|
||||
);
|
||||
const { header, ciphertext } = encryptBlob(thumbJpeg, file.key);
|
||||
|
||||
const md5 = createHash("md5").update(ciphertext).digest("base64");
|
||||
const { objectKey, url } = await api.getUploadURL(
|
||||
ciphertext.length,
|
||||
@@ -291,24 +230,28 @@ export const fixMissingThumbnails = async (
|
||||
await api.putFile(url, ciphertext);
|
||||
await api.updateThumbnail(file.id, objectKey, toBase64(header));
|
||||
|
||||
log(`[${collectionName}] Thumbnail uploaded for ${title}`);
|
||||
results.push({
|
||||
fileID,
|
||||
title,
|
||||
collection: collectionName,
|
||||
status: "fixed",
|
||||
});
|
||||
} catch (err) {
|
||||
log(
|
||||
`[${collectionName}] FAILED ${title}: ${err instanceof Error ? err.message : err}`,
|
||||
`[${collectionName}] Thumbnail uploaded for ${file.metadata.title}`,
|
||||
);
|
||||
results.push({
|
||||
fileID,
|
||||
title,
|
||||
title: file.metadata.title,
|
||||
collection: collectionName,
|
||||
status: "failed",
|
||||
reason: err instanceof Error ? err.message : String(err),
|
||||
success: true,
|
||||
});
|
||||
} catch (err) {
|
||||
log(
|
||||
`[${collectionName}] FAILED ${file.metadata.title}: ${err instanceof Error ? err.message : err}`,
|
||||
);
|
||||
results.push({
|
||||
fileID,
|
||||
title: file.metadata.title,
|
||||
collection: collectionName,
|
||||
success: false,
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
});
|
||||
} finally {
|
||||
rmSync(tmpDir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2,17 +2,13 @@
|
||||
* Tests for `quak backup-metadata <dir>`.
|
||||
*
|
||||
* This command dumps all decrypted account metadata into a directory
|
||||
* tree of plain JSON files, without downloading any file content (unless
|
||||
* `--exif` is given). It is fast and produces a complete plaintext record of
|
||||
* every collection name, file title, creation date, GPS coordinate, camera
|
||||
* model, caption, face label, and any other metadata the Ente clients have
|
||||
* attached.
|
||||
* tree of plain JSON files, without downloading any file content. It
|
||||
* is fast (no multi-megabyte downloads) and produces a complete
|
||||
* plaintext record of every collection name, file title, creation
|
||||
* date, GPS coordinate, camera model, caption, face label, and any
|
||||
* other metadata the Ente clients have attached.
|
||||
*
|
||||
* As of issue #52 it runs on the library API: `runMetadataBackup(lib, client,
|
||||
* dir)` enumerates collections and files from the library's cache rather than
|
||||
* scanning the client directly, and `--exif` reads each original through the
|
||||
* library's content cache (`photo.original()`). The ML fetch is unchanged. The
|
||||
* output tree is identical:
|
||||
* Layout:
|
||||
*
|
||||
* <dir>/
|
||||
* account.json { email, userID }
|
||||
@@ -48,11 +44,7 @@ import {
|
||||
} from "../../src/crypto/index.js";
|
||||
import * as jpegJs from "jpeg-js";
|
||||
import { Client } from "../../src/client.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import {
|
||||
runMetadataBackup,
|
||||
type MetadataBackupOptions,
|
||||
} from "../../src/metadata-backup.js";
|
||||
import { runMetadataBackup } from "../../src/metadata-backup.js";
|
||||
import type { KeyAttributes } from "../../src/auth/types.js";
|
||||
|
||||
const TEST_EMAIL = "metabackup@example.com";
|
||||
@@ -439,50 +431,16 @@ afterAll(() => {
|
||||
rmSync(testDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// Log in against the mock and open a library over its cache. The point commands
|
||||
// open the library with the background precache off and a long refresh interval;
|
||||
// the same here keeps the test deterministic (no thumbnail/original prefetch it
|
||||
// did not ask for, no second refresh mid-test).
|
||||
const openLib = async (client: Client): Promise<Library> =>
|
||||
Library.open({
|
||||
// The library client omits `fetchMLData`, matching how the CLI opens
|
||||
// point commands: `runMetadataBackup` fetches ML data itself through
|
||||
// the client, so the library's background backfill would only be a
|
||||
// redundant second pass over the same endpoint.
|
||||
client: {
|
||||
whoami: () => client.whoami(),
|
||||
collectionsSince: (args) => client.collectionsSince(args),
|
||||
filesSince: (args) => client.filesSince(args),
|
||||
contentSource: () => client.contentSource(),
|
||||
},
|
||||
cacheDirectory: mkdtempSync(join(testDir, "cache-")),
|
||||
refreshIntervalSeconds: 3600,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
|
||||
// Run one metadata backup end to end: fresh client, fresh library, then close.
|
||||
const runBackup = async (
|
||||
outDir: string,
|
||||
opts?: MetadataBackupOptions,
|
||||
): Promise<void> => {
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildMetaFetch(mock) },
|
||||
});
|
||||
const lib = await openLib(client);
|
||||
try {
|
||||
await runMetadataBackup(lib, client, outDir, opts);
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
};
|
||||
|
||||
describe("quak backup-metadata", () => {
|
||||
it("writes account.json with email and userID", async () => {
|
||||
const outDir = join(testDir, "full");
|
||||
await runBackup(outDir);
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildMetaFetch(mock) },
|
||||
});
|
||||
|
||||
await runMetadataBackup(client, outDir);
|
||||
|
||||
const account = JSON.parse(
|
||||
readFileSync(join(outDir, "account.json"), "utf-8"),
|
||||
@@ -493,7 +451,13 @@ describe("quak backup-metadata", () => {
|
||||
|
||||
it("creates per-collection directories with _collection.json", async () => {
|
||||
const outDir = join(testDir, "collections");
|
||||
await runBackup(outDir);
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildMetaFetch(mock) },
|
||||
});
|
||||
|
||||
await runMetadataBackup(client, outDir);
|
||||
|
||||
const collDirs = readdirSync(join(outDir, "collections"));
|
||||
expect(collDirs.length).toBe(2);
|
||||
@@ -514,7 +478,13 @@ describe("quak backup-metadata", () => {
|
||||
|
||||
it("decrypts collection-level pubMagicMetadata", async () => {
|
||||
const outDir = join(testDir, "coll-magic");
|
||||
await runBackup(outDir);
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildMetaFetch(mock) },
|
||||
});
|
||||
|
||||
await runMetadataBackup(client, outDir);
|
||||
|
||||
const collDirs = readdirSync(join(outDir, "collections"));
|
||||
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
|
||||
@@ -531,7 +501,13 @@ describe("quak backup-metadata", () => {
|
||||
|
||||
it("writes per-file JSON with all three metadata layers", async () => {
|
||||
const outDir = join(testDir, "file-meta");
|
||||
await runBackup(outDir);
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildMetaFetch(mock) },
|
||||
});
|
||||
|
||||
await runMetadataBackup(client, outDir);
|
||||
|
||||
const collDirs = readdirSync(join(outDir, "collections"));
|
||||
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
|
||||
@@ -550,7 +526,13 @@ describe("quak backup-metadata", () => {
|
||||
|
||||
it("handles files with no magic metadata gracefully", async () => {
|
||||
const outDir = join(testDir, "no-magic");
|
||||
await runBackup(outDir);
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildMetaFetch(mock) },
|
||||
});
|
||||
|
||||
await runMetadataBackup(client, outDir);
|
||||
|
||||
const collDirs = readdirSync(join(outDir, "collections"));
|
||||
const workDir = collDirs.find((d) => d.includes("Work"))!;
|
||||
@@ -568,8 +550,14 @@ describe("quak backup-metadata", () => {
|
||||
|
||||
it("is incremental: second run does not fail", async () => {
|
||||
const outDir = join(testDir, "incremental");
|
||||
await runBackup(outDir);
|
||||
await runBackup(outDir);
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildMetaFetch(mock) },
|
||||
});
|
||||
|
||||
await runMetadataBackup(client, outDir);
|
||||
await runMetadataBackup(client, outDir);
|
||||
|
||||
const account = JSON.parse(
|
||||
readFileSync(join(outDir, "account.json"), "utf-8"),
|
||||
@@ -579,7 +567,13 @@ describe("quak backup-metadata", () => {
|
||||
|
||||
it("fetches and decrypts ML data by default", async () => {
|
||||
const outDir = join(testDir, "ml-data");
|
||||
await runBackup(outDir);
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildMetaFetch(mock) },
|
||||
});
|
||||
|
||||
await runMetadataBackup(client, outDir);
|
||||
|
||||
const collDirs = readdirSync(join(outDir, "collections"));
|
||||
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
|
||||
@@ -603,7 +597,13 @@ describe("quak backup-metadata", () => {
|
||||
|
||||
it("extracts EXIF from downloaded files when --exif is set", async () => {
|
||||
const outDir = join(testDir, "exif-data");
|
||||
await runBackup(outDir, { exif: true });
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildMetaFetch(mock) },
|
||||
});
|
||||
|
||||
await runMetadataBackup(client, outDir, { exif: true });
|
||||
|
||||
const collDirs = readdirSync(join(outDir, "collections"));
|
||||
const vacDir = collDirs.find((d) => d.includes("Vacation"))!;
|
||||
|
||||
@@ -1,76 +0,0 @@
|
||||
// The CLI presents a file by its own decrypted metadata, not the PhotoRecord
|
||||
// projection (issue #52). For a renamed file the two disagree: the projection
|
||||
// prefers `editedName` and reports `editedTime` in milliseconds, while the CLI
|
||||
// must print the raw `metadata.title` and `metadata.creationTime` (microseconds)
|
||||
// and name downloads after the raw title, byte-identical to the pre-library CLI.
|
||||
//
|
||||
// This locks in that contrast: the shared output helpers emit the raw values,
|
||||
// and the projection of the same file emits the edited ones — so a regression
|
||||
// that re-sourced the CLI from the projection would fail here.
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
|
||||
import {
|
||||
fileListRow,
|
||||
fileListLine,
|
||||
originalName,
|
||||
thumbnailName,
|
||||
} from "../../src/cli-output.js";
|
||||
import { deriveRecords } from "../../src/library/records.js";
|
||||
import type { EnteFile } from "../../src/model/types.js";
|
||||
|
||||
// Microseconds, as Ente stores times.
|
||||
const RAW_CREATION = 1700000000000000;
|
||||
const EDITED_TIME = 1710000000000000;
|
||||
const RAW_TITLE = "IMG_0001.HEIC";
|
||||
const EDITED_NAME = "Sunset.heic";
|
||||
|
||||
// A file the user has renamed and re-dated: basic metadata holds the original
|
||||
// title and capture time; public magic metadata holds the edits.
|
||||
const renamedFile: EnteFile = {
|
||||
id: 100,
|
||||
collectionID: 10,
|
||||
ownerID: 42,
|
||||
key: new Uint8Array(),
|
||||
metadata: {
|
||||
title: RAW_TITLE,
|
||||
fileType: "image",
|
||||
creationTime: RAW_CREATION,
|
||||
modificationTime: RAW_CREATION,
|
||||
},
|
||||
pubMagicMetadata: { editedName: EDITED_NAME, editedTime: EDITED_TIME },
|
||||
file: { decryptionHeader: "" },
|
||||
thumbnail: { decryptionHeader: "" },
|
||||
updationTime: RAW_CREATION,
|
||||
};
|
||||
|
||||
describe("CLI file output (issue #52)", () => {
|
||||
it("emits the raw title and microsecond creationTime for --json", () => {
|
||||
expect(fileListRow(renamedFile)).toEqual({
|
||||
id: 100,
|
||||
title: RAW_TITLE,
|
||||
fileType: "image",
|
||||
creationTime: RAW_CREATION,
|
||||
collectionID: 10,
|
||||
});
|
||||
});
|
||||
|
||||
it("emits the raw title in the human column", () => {
|
||||
expect(fileListLine(renamedFile)).toBe(`100\timage\t${RAW_TITLE}`);
|
||||
});
|
||||
|
||||
it("names downloads after the raw title", () => {
|
||||
expect(originalName(renamedFile)).toBe(RAW_TITLE);
|
||||
expect(thumbnailName(renamedFile)).toBe(`thumb_${RAW_TITLE}`);
|
||||
});
|
||||
|
||||
it("does not use the editedName/editedTime projection", () => {
|
||||
const record = deriveRecords([], [renamedFile]).photos.get(100);
|
||||
// The projection prefers the edits and reports milliseconds; the CLI
|
||||
// helpers above deliberately do not.
|
||||
expect(record?.title).toBe(EDITED_NAME);
|
||||
expect(record?.takenAt).toBe(Math.floor(EDITED_TIME / 1000));
|
||||
expect(fileListRow(renamedFile).title).not.toBe(record?.title);
|
||||
expect(fileListRow(renamedFile).creationTime).not.toBe(record?.takenAt);
|
||||
});
|
||||
});
|
||||
@@ -1,195 +0,0 @@
|
||||
/**
|
||||
* Tests for the CLI read helpers (`src/cli-read.ts`, owner amendment to
|
||||
* issue #36, issue #52).
|
||||
*
|
||||
* The `collections`, `files`, `get`, and `get-thumb` commands must answer for
|
||||
* current server state, not the local cache, so each helper forces a
|
||||
* `Library.fresh()` round-trip before it reads. The stand-in library below
|
||||
* serves nothing until `fresh()` has been awaited, so a helper that read
|
||||
* without refreshing would come back empty and fail here.
|
||||
*
|
||||
* `collections` and `files` also list in the library's enumeration order
|
||||
* (`listCollections`/`listFiles`) — the order the pre-library CLI printed — not
|
||||
* the albums/photos projection's newest-first order. The fixtures are seeded in
|
||||
* an enumeration order that a newest-first sort would rearrange, so a
|
||||
* regression to the projection order would fail here too. Field values still
|
||||
* come from the raw metadata via `cli-output.ts`.
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
|
||||
import {
|
||||
freshCollections,
|
||||
freshFiles,
|
||||
freshFile,
|
||||
type FreshReadLibrary,
|
||||
} from "../../src/cli-read.js";
|
||||
import { fileListRow } from "../../src/cli-output.js";
|
||||
import type { Photo } from "../../src/library/index.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
|
||||
const collection = (id: number, updationTime: number): Collection => ({
|
||||
id,
|
||||
ownerID: 42,
|
||||
key: new Uint8Array(),
|
||||
name: `album-${id}`,
|
||||
type: "album",
|
||||
updationTime,
|
||||
isShared: false,
|
||||
});
|
||||
|
||||
// Microseconds, as Ente stores times.
|
||||
const file = (
|
||||
id: number,
|
||||
collectionID: number,
|
||||
creationTime: number,
|
||||
): EnteFile => ({
|
||||
id,
|
||||
collectionID,
|
||||
ownerID: 42,
|
||||
key: new Uint8Array(),
|
||||
metadata: {
|
||||
title: `file-${id}.jpg`,
|
||||
fileType: "image",
|
||||
creationTime,
|
||||
modificationTime: creationTime,
|
||||
},
|
||||
file: { decryptionHeader: "" },
|
||||
thumbnail: { decryptionHeader: "" },
|
||||
updationTime: creationTime,
|
||||
});
|
||||
|
||||
// A library that reveals its records only after `fresh()` has been awaited, and
|
||||
// serves them in the enumeration order it was given. `photos.byID` returns a
|
||||
// stand-in `Photo` carrying just the fileID the helper passes through.
|
||||
class FakeLibrary implements FreshReadLibrary {
|
||||
freshCalls = 0;
|
||||
private refreshed = false;
|
||||
|
||||
constructor(
|
||||
private readonly collections: Collection[],
|
||||
private readonly files: EnteFile[],
|
||||
) {}
|
||||
|
||||
async fresh(): Promise<unknown> {
|
||||
this.freshCalls++;
|
||||
this.refreshed = true;
|
||||
return {};
|
||||
}
|
||||
|
||||
listCollections(): Collection[] {
|
||||
return this.refreshed ? this.collections : [];
|
||||
}
|
||||
|
||||
getCollection(id: number): Collection | undefined {
|
||||
return this.listCollections().find((c) => c.id === id);
|
||||
}
|
||||
|
||||
listFiles(collectionID: number): EnteFile[] {
|
||||
return this.refreshed
|
||||
? this.files.filter((f) => f.collectionID === collectionID)
|
||||
: [];
|
||||
}
|
||||
|
||||
getFileByID(fileID: number): EnteFile | undefined {
|
||||
if (!this.refreshed) return undefined;
|
||||
return this.files.find((f) => f.id === fileID);
|
||||
}
|
||||
|
||||
photos = {
|
||||
byID: ({ fileID }: { fileID: number }): Photo | undefined => {
|
||||
if (!this.refreshed) return undefined;
|
||||
if (!this.files.some((f) => f.id === fileID)) return undefined;
|
||||
return { fileID } as unknown as Photo;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
||||
it("freshCollections refreshes first, then lists in enumeration order", async () => {
|
||||
// Enumeration order 2, 1, 3; a newest-first sort would be 3, 2, 1.
|
||||
const lib = new FakeLibrary(
|
||||
[collection(2, 200), collection(1, 300), collection(3, 100)],
|
||||
[],
|
||||
);
|
||||
const rows = await freshCollections(lib);
|
||||
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(rows.map((c) => c.id)).toEqual([2, 1, 3]);
|
||||
// The projection's newest-first order is a different sequence, so this
|
||||
// is not accidentally that order.
|
||||
const newestFirst = [...rows]
|
||||
.sort((a, b) => b.updationTime - a.updationTime)
|
||||
.map((c) => c.id);
|
||||
expect(newestFirst).toEqual([1, 2, 3]);
|
||||
expect(rows.map((c) => c.id)).not.toEqual(newestFirst);
|
||||
});
|
||||
|
||||
it("freshFiles refreshes first, lists in enumeration order, keeps raw fields", async () => {
|
||||
// Enumeration order by id 10, 11, 12; creationTimes ascending, so a
|
||||
// newest-first sort would reverse them.
|
||||
const files = [
|
||||
file(10, 1, 1_700_000_000_000_000),
|
||||
file(11, 1, 1_700_000_000_000_001),
|
||||
file(12, 1, 1_700_000_000_000_002),
|
||||
];
|
||||
const lib = new FakeLibrary([collection(1, 100)], files);
|
||||
|
||||
const rows = await freshFiles(lib, 1);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(rows?.map((f) => f.id)).toEqual([10, 11, 12]);
|
||||
|
||||
// Field values come from raw metadata: microsecond creationTime and the
|
||||
// raw title, unchanged.
|
||||
expect(rows?.map(fileListRow)).toEqual([
|
||||
{
|
||||
id: 10,
|
||||
title: "file-10.jpg",
|
||||
fileType: "image",
|
||||
creationTime: 1_700_000_000_000_000,
|
||||
collectionID: 1,
|
||||
},
|
||||
{
|
||||
id: 11,
|
||||
title: "file-11.jpg",
|
||||
fileType: "image",
|
||||
creationTime: 1_700_000_000_000_001,
|
||||
collectionID: 1,
|
||||
},
|
||||
{
|
||||
id: 12,
|
||||
title: "file-12.jpg",
|
||||
fileType: "image",
|
||||
creationTime: 1_700_000_000_000_002,
|
||||
collectionID: 1,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("freshFiles returns undefined for an unknown collection", async () => {
|
||||
const lib = new FakeLibrary([collection(1, 100)], []);
|
||||
const rows = await freshFiles(lib, 999);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(rows).toBeUndefined();
|
||||
});
|
||||
|
||||
it("freshFile refreshes first, then resolves the photo and its raw record", async () => {
|
||||
const f = file(10, 1, 1_700_000_000_000_000);
|
||||
const lib = new FakeLibrary([collection(1, 100)], [f]);
|
||||
|
||||
const resolved = await freshFile(lib, 10);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(resolved?.photo.fileID).toBe(10);
|
||||
expect(resolved?.file.metadata.title).toBe("file-10.jpg");
|
||||
expect(resolved?.file.metadata.creationTime).toBe(
|
||||
1_700_000_000_000_000,
|
||||
);
|
||||
});
|
||||
|
||||
it("freshFile returns undefined for an unknown file", async () => {
|
||||
const lib = new FakeLibrary([collection(1, 100)], []);
|
||||
const resolved = await freshFile(lib, 404);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(resolved).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -1,272 +0,0 @@
|
||||
/**
|
||||
* Tests for fresh reads (issue #75, an owner amendment to design #36).
|
||||
*
|
||||
* The default read namespaces answer from RAM and never touch the network; a
|
||||
* background loop keeps the local copy current. `fresh()` adds an awaited path:
|
||||
* it forces a refresh, waits for it to complete and persist, and only then
|
||||
* hands back the `albums`/`photos`/`timeline` namespaces, guaranteeing the
|
||||
* local copy reflects a completed server round-trip. The contracts here:
|
||||
*
|
||||
* 1. A fresh read observes a server change that a same-instant default read
|
||||
* would miss (the default path has not refreshed yet).
|
||||
* 2. Concurrent fresh reads coalesce onto one in-flight refresh — three
|
||||
* concurrent `fresh()` calls make exactly one collections round-trip, not
|
||||
* three.
|
||||
* 3. A refresh that fails rejects the fresh read (currency was unavailable),
|
||||
* while the default reads stay silent and keep serving the last good copy.
|
||||
*
|
||||
* The client is the same metadata-only mock the background-refresh tests use:
|
||||
* no crypto, no network, scripted pages, and a record of each call. A long
|
||||
* refresh interval keeps the background timer out of the way so each test's
|
||||
* refreshes are exactly the ones it triggers.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
|
||||
const USER_ID = 42;
|
||||
|
||||
// Long enough that no background tick fires during a test; each test's
|
||||
// refreshes are only the ones its own `fresh()` calls force.
|
||||
const SLOW_INTERVAL = 3600;
|
||||
|
||||
const collection = (id: number, updationTime: number): Collection => ({
|
||||
id,
|
||||
ownerID: USER_ID,
|
||||
key: new Uint8Array([id & 0xff]),
|
||||
name: `album-${id}`,
|
||||
type: "album",
|
||||
updationTime,
|
||||
isShared: false,
|
||||
});
|
||||
|
||||
const file = (
|
||||
id: number,
|
||||
collectionID: number,
|
||||
updationTime: number,
|
||||
): EnteFile => ({
|
||||
id,
|
||||
collectionID,
|
||||
ownerID: USER_ID,
|
||||
key: new Uint8Array([id & 0xff]),
|
||||
metadata: {
|
||||
title: `file-${id}.jpg`,
|
||||
fileType: "image",
|
||||
creationTime: updationTime,
|
||||
modificationTime: updationTime,
|
||||
},
|
||||
file: { decryptionHeader: "aGVhZGVy" },
|
||||
thumbnail: { decryptionHeader: "dGh1bWI=" },
|
||||
updationTime,
|
||||
});
|
||||
|
||||
class MockClient {
|
||||
userID = USER_ID;
|
||||
failCollections = false;
|
||||
collectionsQueue: CollectionsPage[] = [];
|
||||
filesByCollection = new Map<number, FilesPage[]>();
|
||||
|
||||
collectionsSinceTimes: number[] = [];
|
||||
filesCalls: { collectionID: number; sinceTime: number }[] = [];
|
||||
|
||||
whoami(): { email: string; userID: number } {
|
||||
return { email: "user@example.com", userID: this.userID };
|
||||
}
|
||||
|
||||
async collectionsSince(args: {
|
||||
sinceTime: number;
|
||||
}): Promise<CollectionsPage> {
|
||||
this.collectionsSinceTimes.push(args.sinceTime);
|
||||
if (this.failCollections) throw new Error("network down");
|
||||
return (
|
||||
this.collectionsQueue.shift() ?? {
|
||||
collections: [],
|
||||
deleted: [],
|
||||
cursor: args.sinceTime,
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
async filesSince(args: {
|
||||
collectionID: number;
|
||||
collectionKey: Uint8Array;
|
||||
sinceTime: number;
|
||||
}): Promise<FilesPage> {
|
||||
this.filesCalls.push({
|
||||
collectionID: args.collectionID,
|
||||
sinceTime: args.sinceTime,
|
||||
});
|
||||
const queue = this.filesByCollection.get(args.collectionID);
|
||||
return (
|
||||
queue?.shift() ?? {
|
||||
files: [],
|
||||
deleted: [],
|
||||
cursor: args.sinceTime,
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
filesFor(collectionID: number, ...pages: FilesPage[]): void {
|
||||
this.filesByCollection.set(collectionID, pages);
|
||||
}
|
||||
}
|
||||
|
||||
// A client seeded with one collection and one file, opened with an empty cache
|
||||
// so the initial refresh is awaited and post-`open()` state is deterministic.
|
||||
const openSeeded = async (
|
||||
cacheDirectory: string,
|
||||
): Promise<{ client: MockClient; lib: Library }> => {
|
||||
const client = new MockClient();
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 100)],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
client.filesFor(1, {
|
||||
files: [file(1001, 1, 90)],
|
||||
deleted: [],
|
||||
cursor: 90,
|
||||
});
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: SLOW_INTERVAL,
|
||||
});
|
||||
return { client, lib };
|
||||
};
|
||||
|
||||
describe("Library.fresh", () => {
|
||||
let dir: string;
|
||||
let cacheDirectory: string;
|
||||
|
||||
beforeEach(() => {
|
||||
dir = mkdtempSync(join(tmpdir(), "quak-fresh-"));
|
||||
cacheDirectory = join(dir, "cache");
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("observes a server change a same-instant default read would miss", async () => {
|
||||
const { client, lib } = await openSeeded(cacheDirectory);
|
||||
try {
|
||||
// A new file appears on the server after open, advancing its
|
||||
// collection so the next refresh re-enumerates it.
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 200)],
|
||||
deleted: [],
|
||||
cursor: 200,
|
||||
});
|
||||
client.filesFor(1, {
|
||||
files: [file(1002, 1, 190)],
|
||||
deleted: [],
|
||||
cursor: 190,
|
||||
});
|
||||
|
||||
// A default read at this instant has not refreshed: it misses 1002.
|
||||
expect(lib.photos.byID({ fileID: 1002 })).toBeUndefined();
|
||||
|
||||
// A fresh read forces the round-trip and sees it.
|
||||
const reads = await lib.fresh();
|
||||
expect(reads.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
|
||||
// And the change is now live for the default namespaces too.
|
||||
expect(lib.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("coalesces concurrent fresh reads onto one in-flight refresh", async () => {
|
||||
const { client, lib } = await openSeeded(cacheDirectory);
|
||||
try {
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 200)],
|
||||
deleted: [],
|
||||
cursor: 200,
|
||||
});
|
||||
client.filesFor(1, {
|
||||
files: [file(1002, 1, 190)],
|
||||
deleted: [],
|
||||
cursor: 190,
|
||||
});
|
||||
|
||||
const collectionsBefore = client.collectionsSinceTimes.length;
|
||||
const filesBefore = client.filesCalls.length;
|
||||
|
||||
// Gate the next collections fetch so all three fresh reads are in
|
||||
// flight together before any of them completes.
|
||||
let release: () => void = () => {};
|
||||
const gate = new Promise<void>((r) => {
|
||||
release = r;
|
||||
});
|
||||
const inner = client.collectionsSince.bind(client);
|
||||
client.collectionsSince = async (args: { sinceTime: number }) => {
|
||||
await gate;
|
||||
return inner(args);
|
||||
};
|
||||
|
||||
const all = Promise.all([lib.fresh(), lib.fresh(), lib.fresh()]);
|
||||
release();
|
||||
const [a, b, c] = await all;
|
||||
|
||||
// Exactly one collections round-trip and one file round-trip served
|
||||
// all three fresh reads.
|
||||
expect(client.collectionsSinceTimes.length).toBe(
|
||||
collectionsBefore + 1,
|
||||
);
|
||||
expect(client.filesCalls.length).toBe(filesBefore + 1);
|
||||
|
||||
// All three observed the change.
|
||||
for (const reads of [a, b, c]) {
|
||||
expect(reads.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
|
||||
}
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects the fresh read when the refresh fails, leaving defaults intact", async () => {
|
||||
const { client, lib } = await openSeeded(cacheDirectory);
|
||||
try {
|
||||
client.failCollections = true;
|
||||
|
||||
// Two concurrent fresh reads both reject, and share one failed
|
||||
// round-trip rather than each making its own.
|
||||
const collectionsBefore = client.collectionsSinceTimes.length;
|
||||
const first = lib.fresh();
|
||||
const second = lib.fresh();
|
||||
await expect(first).rejects.toThrow(/network down/);
|
||||
await expect(second).rejects.toThrow(/network down/);
|
||||
expect(client.collectionsSinceTimes.length).toBe(
|
||||
collectionsBefore + 1,
|
||||
);
|
||||
|
||||
// The default reads never rejected: they still serve the last good
|
||||
// copy, and the failure surfaced through status().
|
||||
expect(lib.photos.byID({ fileID: 1001 })?.fileID).toBe(1001);
|
||||
expect(lib.status().lastError).toMatch(/network down/);
|
||||
|
||||
// Recovery: once the server answers, a fresh read resolves current.
|
||||
client.failCollections = false;
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(2, 300)],
|
||||
deleted: [],
|
||||
cursor: 300,
|
||||
});
|
||||
const reads = await lib.fresh();
|
||||
expect(reads.albums.byID({ collectionID: 2 })?.collectionID).toBe(
|
||||
2,
|
||||
);
|
||||
expect(lib.status().lastError).toBeUndefined();
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -1,108 +0,0 @@
|
||||
/**
|
||||
* Tests for the content-similarity search surface over the CLIP index
|
||||
* (issue #50).
|
||||
*
|
||||
* The surface is `lib.mldata`: `forFile` reads the full stored payload from
|
||||
* disk, while `similar` and `searchByEmbedding` rank fileIDs by cosine
|
||||
* similarity over the in-RAM `Float32Array` index alone (no disk, no network).
|
||||
* The fixture uses axis-aligned vectors so the correct cosine ranking is
|
||||
* obvious by inspection; cosine ignores magnitude, so `[2, 0, 0]` ranks above
|
||||
* `[0.8, 0.6, 0]` for a `[1, 0, 0]` query.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { MLDataStore } from "../../src/library/mldata.js";
|
||||
import { makeMLDataAPI, type MLDataAPI } from "../../src/library/mlsearch.js";
|
||||
import type { MLData } from "../../src/mldata-fetch.js";
|
||||
|
||||
// A payload shaped like Ente's: a CLIP embedding plus face data that only the
|
||||
// on-disk payload carries (never the RAM index).
|
||||
const payload = (embedding: number[]): MLData => ({
|
||||
face: { faces: [{ faceID: "f", detection: { box: { x: 0.5 } } }] },
|
||||
clip: { embedding },
|
||||
});
|
||||
|
||||
// A small fixture index. Directions are chosen so every cosine ranking below
|
||||
// is unambiguous.
|
||||
const fixture = (): Map<number, MLData> =>
|
||||
new Map([
|
||||
[10, payload([1, 0, 0])],
|
||||
[20, payload([0.8, 0.6, 0])],
|
||||
[30, payload([0, 1, 0])],
|
||||
[40, payload([-1, 0, 0])],
|
||||
[50, payload([2, 0, 0])],
|
||||
]);
|
||||
|
||||
describe("lib.mldata content-similarity search", () => {
|
||||
let dir: string;
|
||||
let store: MLDataStore;
|
||||
let api: MLDataAPI;
|
||||
|
||||
beforeEach(async () => {
|
||||
dir = mkdtempSync(join(tmpdir(), "quak-mlsearch-"));
|
||||
store = await MLDataStore.open(dir);
|
||||
const updation = new Map([...fixture().keys()].map((id) => [id, 1]));
|
||||
await store.storeFetched(fixture(), updation);
|
||||
api = makeMLDataAPI(() => store);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("forFile returns the whole stored payload, or undefined when uncached", async () => {
|
||||
const full = await api.forFile({ fileID: 20 });
|
||||
expect(full).toBeDefined();
|
||||
// Face data lives only in the payload, never in the RAM index.
|
||||
expect(full?.face).toBeDefined();
|
||||
expect(full?.clip).toEqual({ embedding: [0.8, 0.6, 0] });
|
||||
expect(await api.forFile({ fileID: 999 })).toBeUndefined();
|
||||
});
|
||||
|
||||
it("similar ranks other files by cosine and excludes the query itself", () => {
|
||||
// Query is file 10 = [1, 0, 0]. By cosine: 50 (1.0) > 20 (0.8) >
|
||||
// 30 (0) > 40 (-1); 10 itself is left out.
|
||||
const ranked = api.similar({ fileID: 10 });
|
||||
expect(ranked.map((r) => r.fileID)).toEqual([50, 20, 30, 40]);
|
||||
// Cosine ignores magnitude: [2,0,0] is a perfect match for [1,0,0].
|
||||
expect(ranked[0]).toMatchObject({ fileID: 50 });
|
||||
expect(ranked[0].score).toBeCloseTo(1, 5);
|
||||
});
|
||||
|
||||
it("similar honours limit and returns [] for an unindexed file", () => {
|
||||
expect(
|
||||
api.similar({ fileID: 10, limit: 2 }).map((r) => r.fileID),
|
||||
).toEqual([50, 20]);
|
||||
expect(api.similar({ fileID: 999 })).toEqual([]);
|
||||
});
|
||||
|
||||
it("searchByEmbedding ranks the index by cosine to the query vector", () => {
|
||||
// Query [0, 1, 0]: 30 (1.0) > 20 (0.6) > {10, 40, 50} all 0, broken by
|
||||
// ascending fileID.
|
||||
const ranked = api.searchByEmbedding({ embedding: [0, 1, 0] });
|
||||
expect(ranked.map((r) => r.fileID)).toEqual([30, 20, 10, 40, 50]);
|
||||
expect(ranked[0].score).toBeCloseTo(1, 5);
|
||||
|
||||
expect(
|
||||
api
|
||||
.searchByEmbedding({ embedding: [0, 1, 0], limit: 2 })
|
||||
.map((r) => r.fileID),
|
||||
).toEqual([30, 20]);
|
||||
});
|
||||
|
||||
it("searchByEmbedding returns [] for a wrong-length or zero query", () => {
|
||||
expect(api.searchByEmbedding({ embedding: [1, 0] })).toEqual([]);
|
||||
expect(api.searchByEmbedding({ embedding: [0, 0, 0] })).toEqual([]);
|
||||
});
|
||||
|
||||
it("degrades to empty results when no ML store is present", async () => {
|
||||
const none = makeMLDataAPI(() => undefined);
|
||||
expect(await none.forFile({ fileID: 10 })).toBeUndefined();
|
||||
expect(none.similar({ fileID: 10 })).toEqual([]);
|
||||
expect(none.searchByEmbedding({ embedding: [1, 0, 0] })).toEqual([]);
|
||||
});
|
||||
});
|
||||
@@ -6,30 +6,17 @@
|
||||
* working thumbnails, others return 404 or empty bodies. The tests
|
||||
* verify that the detection and repair logic handles each case correctly.
|
||||
*
|
||||
* As of issue #52 both helpers take an open `Library` for enumeration and the
|
||||
* `Client` for the API operations that stay unchanged (the thumbnail existence
|
||||
* check, and the encrypt-and-upload path). `fixMissingThumbnails` reads each
|
||||
* original through the library's content cache (`photo.original()`).
|
||||
*
|
||||
* `fixMissingThumbnails` is the most complex function in quak: it
|
||||
* downloads the original file, generates a JPEG thumbnail with jpeg-js,
|
||||
* encrypts it with secretstream push, gets a presigned upload URL,
|
||||
* uploads to S3, and registers the new thumbnail with the API. The
|
||||
* test verifies each step actually happened and the uploaded data is
|
||||
* a valid encrypted blob that decrypts to a JPEG.
|
||||
*
|
||||
* It regenerates thumbnails for baseline JPEGs only, because `jpeg-js` decodes
|
||||
* only JPEG. A non-JPEG image (PNG, HEIC) or a video is reported as "skipped
|
||||
* (unsupported)" rather than crashing the decoder into an opaque failure
|
||||
* (issue #17); the mixed test below locks that distinction down.
|
||||
*/
|
||||
|
||||
import { existsSync, mkdtempSync, rmSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import sodium from "libsodium-wrappers-sumo";
|
||||
import * as jpegJs from "jpeg-js";
|
||||
import { afterAll, beforeAll, describe, expect, it } from "vitest";
|
||||
import { beforeAll, describe, expect, it } from "vitest";
|
||||
import {
|
||||
init,
|
||||
toBase64,
|
||||
@@ -40,7 +27,6 @@ import {
|
||||
} from "../../src/crypto/index.js";
|
||||
import { SRP, SrpServer } from "fast-srp-hap";
|
||||
import { Client } from "../../src/client.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import {
|
||||
listMissingThumbnails,
|
||||
fixMissingThumbnails,
|
||||
@@ -56,7 +42,6 @@ const TEST_EMAIL = "thumb@example.com";
|
||||
const TEST_PASSWORD = "thumbpass";
|
||||
const TEST_OPS = 2;
|
||||
const TEST_MEM = 64 * 1024 * 1024;
|
||||
const TEST_TIME = 1700000000000000;
|
||||
|
||||
interface ThumbMockState {
|
||||
verifier: Buffer;
|
||||
@@ -78,17 +63,8 @@ interface ThumbMockState {
|
||||
}
|
||||
|
||||
let mock: ThumbMockState;
|
||||
let tmpRoot: string;
|
||||
|
||||
// PNG signature bytes — enough for `fixMissingThumbnails` to recognise a
|
||||
// non-JPEG image and skip it. It need not be a decodable PNG.
|
||||
const PNG_BYTES = new Uint8Array([
|
||||
0x89, 0x50, 0x4e, 0x47, 0x0d, 0x0a, 0x1a, 0x0a, 0x00, 0x00, 0x00, 0x0d,
|
||||
]);
|
||||
|
||||
const buildThumbMock = async (opts?: {
|
||||
extraFormats?: boolean;
|
||||
}): Promise<ThumbMockState> => {
|
||||
const buildThumbMock = async (): Promise<ThumbMockState> => {
|
||||
const kekSalt = sodium.randombytes_buf(sodium.crypto_pwhash_SALTBYTES);
|
||||
const kek = await deriveKEK(TEST_PASSWORD, kekSalt, TEST_OPS, TEST_MEM);
|
||||
const loginSubKeyBytes = deriveLoginSubkey(kek);
|
||||
@@ -126,6 +102,7 @@ const buildThumbMock = async (opts?: {
|
||||
opsLimit: TEST_OPS,
|
||||
};
|
||||
|
||||
// One collection with 3 files: ok thumbnail, empty thumbnail, 404 thumbnail
|
||||
const collKey = sodium.crypto_secretbox_keygen();
|
||||
const ckN = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
|
||||
const encCK = sodium.crypto_secretbox_easy(collKey, ckN, masterKey);
|
||||
@@ -141,11 +118,10 @@ const buildThumbMock = async (opts?: {
|
||||
encryptedName: toBase64(encCN),
|
||||
nameDecryptionNonce: toBase64(cnN),
|
||||
type: "album",
|
||||
updationTime: TEST_TIME,
|
||||
updationTime: 1700000000000000,
|
||||
};
|
||||
|
||||
// Generate a real tiny JPEG via jpeg-js, used as the encrypted body of the
|
||||
// JPEG files so a repair actually decodes and re-encodes real pixels.
|
||||
// Generate a real tiny JPEG via jpeg-js
|
||||
const w = 100;
|
||||
const h = 80;
|
||||
const pixels = new Uint8Array(w * h * 4);
|
||||
@@ -155,32 +131,26 @@ const buildThumbMock = async (opts?: {
|
||||
pixels[i + 2] = 0; // B
|
||||
pixels[i + 3] = 255; // A
|
||||
}
|
||||
const tinyJpeg = new Uint8Array(
|
||||
jpegJs.encode({ data: pixels, width: w, height: h }, 80).data,
|
||||
);
|
||||
const tinyJpeg = jpegJs.encode(
|
||||
{ data: pixels, width: w, height: h },
|
||||
80,
|
||||
).data;
|
||||
|
||||
const fileKeys: Record<number, Uint8Array> = {};
|
||||
const fileCiphertexts: Record<number, Uint8Array> = {};
|
||||
const rawFiles: Record<string, unknown>[] = [];
|
||||
|
||||
// Build one raw file record: encrypt its metadata and its body under a
|
||||
// fresh per-file key, and record the key and ciphertext for the mock to
|
||||
// serve and for the test to verify against.
|
||||
const makeRawFile = (
|
||||
fileID: number,
|
||||
fileType: number,
|
||||
title: string,
|
||||
body: Uint8Array,
|
||||
): Record<string, unknown> => {
|
||||
for (const fileID of [100, 101, 102]) {
|
||||
const fk = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
fileKeys[fileID] = fk;
|
||||
const fkN = sodium.randombytes_buf(sodium.crypto_secretbox_NONCEBYTES);
|
||||
const encFK = sodium.crypto_secretbox_easy(fk, fkN, collKey);
|
||||
|
||||
const meta = JSON.stringify({
|
||||
title,
|
||||
fileType,
|
||||
creationTime: TEST_TIME,
|
||||
modificationTime: TEST_TIME,
|
||||
title: `file-${fileID}.jpg`,
|
||||
fileType: 0,
|
||||
creationTime: 1700000000000000,
|
||||
modificationTime: 1700000000000000,
|
||||
});
|
||||
const metaPush =
|
||||
sodium.crypto_secretstream_xchacha20poly1305_init_push(fk);
|
||||
@@ -191,17 +161,18 @@ const buildThumbMock = async (opts?: {
|
||||
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
|
||||
);
|
||||
|
||||
// Encrypt the tiny JPEG as the file body
|
||||
const filePush =
|
||||
sodium.crypto_secretstream_xchacha20poly1305_init_push(fk);
|
||||
const encFile = sodium.crypto_secretstream_xchacha20poly1305_push(
|
||||
filePush.state,
|
||||
body,
|
||||
new Uint8Array(tinyJpeg),
|
||||
null,
|
||||
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
|
||||
);
|
||||
fileCiphertexts[fileID] = encFile;
|
||||
|
||||
return {
|
||||
rawFiles.push({
|
||||
id: fileID,
|
||||
collectionID: 1,
|
||||
ownerID: 42,
|
||||
@@ -215,30 +186,8 @@ const buildThumbMock = async (opts?: {
|
||||
thumbnail: {
|
||||
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
|
||||
},
|
||||
updationTime: TEST_TIME,
|
||||
};
|
||||
};
|
||||
|
||||
// Three JPEG files: ok thumbnail, empty thumbnail, 404 thumbnail.
|
||||
const rawFiles: Record<string, unknown>[] = [];
|
||||
for (const fileID of [100, 101, 102]) {
|
||||
rawFiles.push(makeRawFile(fileID, 0, `file-${fileID}.jpg`, tinyJpeg));
|
||||
}
|
||||
const thumbnailBehavior: Record<number, "ok" | "empty" | "404" | "500"> = {
|
||||
100: "ok",
|
||||
101: "empty",
|
||||
102: "404",
|
||||
};
|
||||
|
||||
// For the issue #17 mixed test: a non-JPEG image and a video, both with a
|
||||
// missing (404) thumbnail so they surface in the missing list too.
|
||||
if (opts?.extraFormats) {
|
||||
rawFiles.push(makeRawFile(103, 0, "file-103.png", PNG_BYTES));
|
||||
rawFiles.push(
|
||||
makeRawFile(104, 1, "file-104.mp4", new Uint8Array([0, 0, 0, 1])),
|
||||
);
|
||||
thumbnailBehavior[103] = "404";
|
||||
thumbnailBehavior[104] = "404";
|
||||
updationTime: 1700000000000000,
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
@@ -257,7 +206,11 @@ const buildThumbMock = async (opts?: {
|
||||
filesByCollection: { 1: rawFiles },
|
||||
fileCiphertexts,
|
||||
fileKeys,
|
||||
thumbnailBehavior,
|
||||
thumbnailBehavior: {
|
||||
100: "ok",
|
||||
101: "empty",
|
||||
102: "404",
|
||||
},
|
||||
uploadedThumbnails: [],
|
||||
};
|
||||
};
|
||||
@@ -428,31 +381,6 @@ const countingFetch = (
|
||||
return { fetch: fake as typeof globalThis.fetch, matched: () => matched };
|
||||
};
|
||||
|
||||
// Open a library over a mock-backed client. As the CLI does for point commands,
|
||||
// the background precache is off and the refresh interval is long, and the
|
||||
// library client omits `fetchMLData` so no background ML fetch runs. The real
|
||||
// `Client` is still used for the API operations the helpers perform directly.
|
||||
const openLib = (client: Client): Promise<Library> =>
|
||||
Library.open({
|
||||
client: {
|
||||
whoami: () => client.whoami(),
|
||||
collectionsSince: (args) => client.collectionsSince(args),
|
||||
filesSince: (args) => client.filesSince(args),
|
||||
contentSource: () => client.contentSource(),
|
||||
},
|
||||
cacheDirectory: mkdtempSync(join(tmpRoot, "cache-")),
|
||||
refreshIntervalSeconds: 3600,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
|
||||
const login = (fetch: typeof globalThis.fetch, retry?: RetryOptions) =>
|
||||
Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: retry ? { fetch, retry } : { fetch },
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Tests
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -461,21 +389,17 @@ beforeAll(async () => {
|
||||
await init();
|
||||
await sodium.ready;
|
||||
mock = await buildThumbMock();
|
||||
tmpRoot = mkdtempSync(join(tmpdir(), "quak-thumb-test-"));
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
if (tmpRoot && existsSync(tmpRoot))
|
||||
rmSync(tmpRoot, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
describe("listMissingThumbnails", () => {
|
||||
it("identifies files with empty and 404 thumbnails, ignores working ones", async () => {
|
||||
const client = await login(buildThumbFetch(mock));
|
||||
const lib = await openLib(client);
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildThumbFetch(mock) },
|
||||
});
|
||||
|
||||
const missing = await listMissingThumbnails(lib, client);
|
||||
lib.close();
|
||||
const missing = await listMissingThumbnails(client);
|
||||
|
||||
// File 100 has a working thumbnail → not reported
|
||||
// File 101 has an empty thumbnail → reported
|
||||
@@ -512,11 +436,13 @@ describe("listMissingThumbnails", () => {
|
||||
buildThumbFetch(failingMock),
|
||||
(url) => url.includes("thumbnails.ente.io") && url.includes("102"),
|
||||
);
|
||||
const client = await login(counted.fetch, { ...noWait });
|
||||
const lib = await openLib(client);
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: counted.fetch, retry: { ...noWait } },
|
||||
});
|
||||
|
||||
const missing = await listMissingThumbnails(lib, client);
|
||||
lib.close();
|
||||
const missing = await listMissingThumbnails(client);
|
||||
|
||||
// Only the genuinely empty thumbnail is reported.
|
||||
expect(missing.map((m) => m.fileID)).toEqual([101]);
|
||||
@@ -549,11 +475,13 @@ describe("listMissingThumbnails", () => {
|
||||
return inner(input, init);
|
||||
}) as typeof globalThis.fetch;
|
||||
|
||||
const client = await login(fetch, { ...noWait });
|
||||
const lib = await openLib(client);
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch, retry: { ...noWait } },
|
||||
});
|
||||
|
||||
const missing = await listMissingThumbnails(lib, client);
|
||||
lib.close();
|
||||
const missing = await listMissingThumbnails(client);
|
||||
|
||||
expect(missing.map((m) => m.fileID)).toEqual([101]);
|
||||
expect(thumbRequests).toBe(4);
|
||||
@@ -572,11 +500,13 @@ describe("listMissingThumbnails", () => {
|
||||
mockWithDupes.filesByCollection[2] =
|
||||
mockWithDupes.filesByCollection[1]!;
|
||||
|
||||
const client = await login(buildThumbFetch(mockWithDupes));
|
||||
const lib = await openLib(client);
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildThumbFetch(mockWithDupes) },
|
||||
});
|
||||
|
||||
const missing = await listMissingThumbnails(lib, client);
|
||||
lib.close();
|
||||
const missing = await listMissingThumbnails(client);
|
||||
|
||||
// Should still be 2, not 4 (each file checked only once)
|
||||
expect(missing.length).toBe(2);
|
||||
@@ -586,14 +516,16 @@ describe("listMissingThumbnails", () => {
|
||||
describe("fixMissingThumbnails", () => {
|
||||
it("downloads original, generates thumbnail, encrypts, uploads, and registers", async () => {
|
||||
const fixMock = await buildThumbMock();
|
||||
const client = await login(buildThumbFetch(fixMock));
|
||||
const lib = await openLib(client);
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildThumbFetch(fixMock) },
|
||||
});
|
||||
|
||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
||||
lib.close();
|
||||
const results = await fixMissingThumbnails(client, [101]);
|
||||
|
||||
expect(results.length).toBe(1);
|
||||
expect(results[0]!.status).toBe("fixed");
|
||||
expect(results[0]!.success).toBe(true);
|
||||
expect(results[0]!.fileID).toBe(101);
|
||||
expect(results[0]!.title).toBe("file-101.jpg");
|
||||
expect(results[0]!.collection).toBe("Photos");
|
||||
@@ -623,76 +555,48 @@ describe("fixMissingThumbnails", () => {
|
||||
|
||||
it("reports failure for nonexistent file IDs without crashing", async () => {
|
||||
const fixMock = await buildThumbMock();
|
||||
const client = await login(buildThumbFetch(fixMock));
|
||||
const lib = await openLib(client);
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildThumbFetch(fixMock) },
|
||||
});
|
||||
|
||||
const results = await fixMissingThumbnails(lib, client, [999]);
|
||||
lib.close();
|
||||
const results = await fixMissingThumbnails(client, [999]);
|
||||
|
||||
expect(results.length).toBe(1);
|
||||
expect(results[0]!.status).toBe("failed");
|
||||
expect(results[0]!.success).toBe(false);
|
||||
expect(results[0]!.fileID).toBe(999);
|
||||
expect(results[0]!.reason).toContain("not found");
|
||||
expect(results[0]!.error).toContain("not found");
|
||||
});
|
||||
|
||||
it("continues after one file fails and reports mixed results", async () => {
|
||||
const fixMock = await buildThumbMock();
|
||||
// Make file 102 fail by removing its ciphertext so the download 404s.
|
||||
// Make file 102 fail by removing its ciphertext so download fails
|
||||
delete fixMock.fileCiphertexts[102];
|
||||
|
||||
const client = await login(buildThumbFetch(fixMock));
|
||||
const lib = await openLib(client);
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildThumbFetch(fixMock) },
|
||||
});
|
||||
|
||||
const results = await fixMissingThumbnails(lib, client, [101, 102]);
|
||||
lib.close();
|
||||
const results = await fixMissingThumbnails(client, [101, 102]);
|
||||
|
||||
expect(results.length).toBe(2);
|
||||
const success = results.find((r) => r.fileID === 101)!;
|
||||
const failure = results.find((r) => r.fileID === 102)!;
|
||||
expect(success.status).toBe("fixed");
|
||||
expect(failure.status).toBe("failed");
|
||||
});
|
||||
|
||||
it("skips a non-JPEG image and a video as unsupported, not failed (issue #17)", async () => {
|
||||
// A PNG and a video both throw inside the JPEG decoder. The helper must
|
||||
// recognise them up front and report "skipped", distinct from a genuine
|
||||
// "failed", and must not upload anything for them. The JPEG in the same
|
||||
// batch is still repaired.
|
||||
const fixMock = await buildThumbMock({ extraFormats: true });
|
||||
const client = await login(buildThumbFetch(fixMock));
|
||||
const lib = await openLib(client);
|
||||
|
||||
const results = await fixMissingThumbnails(
|
||||
lib,
|
||||
client,
|
||||
[101, 103, 104],
|
||||
);
|
||||
lib.close();
|
||||
|
||||
const jpeg = results.find((r) => r.fileID === 101)!;
|
||||
const png = results.find((r) => r.fileID === 103)!;
|
||||
const video = results.find((r) => r.fileID === 104)!;
|
||||
|
||||
expect(jpeg.status).toBe("fixed");
|
||||
|
||||
// The PNG is a still image but not a JPEG: skipped only after its bytes
|
||||
// are inspected.
|
||||
expect(png.status).toBe("skipped");
|
||||
expect(png.reason).toContain("JPEG");
|
||||
|
||||
// The video is skipped from its type alone, before any download.
|
||||
expect(video.status).toBe("skipped");
|
||||
expect(video.reason).toContain("video");
|
||||
|
||||
// Only the JPEG was uploaded; the two skipped files touched no upload.
|
||||
expect(fixMock.uploadedThumbnails.length).toBe(1);
|
||||
expect(fixMock.uploadedThumbnails[0]!.fileID).toBe(101);
|
||||
expect(success.success).toBe(true);
|
||||
expect(failure.success).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Client.getApiClient", () => {
|
||||
it("returns the ApiClient when logged in", async () => {
|
||||
const client = await login(buildThumbFetch(mock));
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildThumbFetch(mock) },
|
||||
});
|
||||
|
||||
const api = client.getApiClient();
|
||||
expect(api).toBeDefined();
|
||||
@@ -700,7 +604,11 @@ describe("Client.getApiClient", () => {
|
||||
});
|
||||
|
||||
it("throws after logout", async () => {
|
||||
const client = await login(buildThumbFetch(mock));
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: { fetch: buildThumbFetch(mock) },
|
||||
});
|
||||
client.logout();
|
||||
|
||||
expect(() => client.getApiClient()).toThrow(/logged out/);
|
||||
|
||||
Reference in New Issue
Block a user