Compare commits
3
Commits
005612ef76
..
next
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
fe952d3e62 | ||
|
|
d23d3f8f47 | ||
|
|
aeccb489b5 |
@@ -38,34 +38,47 @@ yarn quak get 67890 --out ./photo.jpg
|
||||
yarn quak backup ./my-backup
|
||||
```
|
||||
|
||||
For library use:
|
||||
For library use, the primary surface is the cache-backed `Library`:
|
||||
|
||||
```ts
|
||||
import { Client } from "quak";
|
||||
import { Client, Library } from "quak";
|
||||
|
||||
// Log in once; the client satisfies the library's client interface.
|
||||
const client = await Client.login({
|
||||
email: "you@example.com",
|
||||
password: "your-password",
|
||||
});
|
||||
|
||||
for (const c of await client.listCollections()) {
|
||||
console.log(c.id, c.name);
|
||||
const files = await client.listFiles(c.id, c.key);
|
||||
for (const f of files) {
|
||||
console.log(` ${f.metadata.title} [${f.metadata.fileType}]`);
|
||||
// Open a cache-backed library. On an empty cache this awaits one server
|
||||
// refresh; on an existing cache it returns immediately and refreshes in the
|
||||
// background every `refreshIntervalSeconds` (default 3).
|
||||
const lib = await Library.open({ client });
|
||||
|
||||
// Default reads answer synchronously from the local cache — no network.
|
||||
for (const album of lib.albums.list()) {
|
||||
console.log(album.collectionID, album.name);
|
||||
for (const photo of album.photos.list()) {
|
||||
console.log(` ${photo.title} [${photo.fileType}]`);
|
||||
}
|
||||
}
|
||||
|
||||
// Download a file
|
||||
const files = await client.listFiles(collectionID, collectionKey);
|
||||
await client.downloadFile(files[0], "./photo.jpg");
|
||||
// Fresh reads await a server round-trip and answer with current state.
|
||||
const { albums } = await lib.fresh();
|
||||
console.log(`${albums.list().length} albums as of now`);
|
||||
|
||||
// Serialize session for later (consumer handles persistence)
|
||||
const snapshot = client.toJSON();
|
||||
// ... later:
|
||||
const restored = Client.fromJSON(snapshot);
|
||||
// Fetch (and cache) one photo's full-resolution bytes.
|
||||
const photo = lib.photos.byID({ fileID: 12345 });
|
||||
if (photo) {
|
||||
const { path } = await photo.original();
|
||||
console.log(`original at ${path}`);
|
||||
}
|
||||
|
||||
lib.close();
|
||||
```
|
||||
|
||||
The lower-level `Client` (login, session serialization, and the raw
|
||||
enumeration/download calls) is exported too and documented under Design below.
|
||||
|
||||
## Entrypoints
|
||||
|
||||
This repository adheres to the
|
||||
@@ -429,18 +442,22 @@ quak files --collection <id> [--json] list files in a collection
|
||||
quak get <fileID> [--out path] [--collection] download and decrypt a file
|
||||
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
|
||||
quak backup <dir> [--json] full incremental backup
|
||||
quak backup-metadata <dir> [--exif] dump all decrypted metadata as JSON
|
||||
quak helper list-missing-thumbnails [--json] find files with missing thumbnails
|
||||
quak helper fix-missing-thumbnails [--file ids] generate + upload missing thumbnails
|
||||
```
|
||||
|
||||
Every command reads through the local library cache: the first run fetches the
|
||||
account's metadata from the server, and later runs serve from the cache and
|
||||
refresh in the background. `--cache-dir` overrides where that cache lives;
|
||||
without it each account gets its own directory under the per-user cache path.
|
||||
Every command runs on the same cache-backed library. The read commands —
|
||||
`collections`, `files`, `get`, and `get-thumb` — force a fresh server round-trip
|
||||
before they answer, so they report current account state rather than whatever
|
||||
the cache last held. `--cache-dir` overrides where the cache lives; without it
|
||||
each account gets its own directory under the per-user cache path.
|
||||
|
||||
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
|
||||
accepted for backward compatibility but ignored. All listing and backup commands
|
||||
support `--json` for machine-readable output.
|
||||
accepted for backward compatibility but ignored. `backup-metadata --exif` (alias
|
||||
`--all`) additionally downloads each file to extract full EXIF/IPTC/XMP
|
||||
metadata. The listing and backup commands support `--json` for machine-readable
|
||||
output.
|
||||
|
||||
`helper fix-missing-thumbnails` regenerates thumbnails for baseline JPEG images
|
||||
only, because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG
|
||||
@@ -472,7 +489,7 @@ code is non-zero if any files failed.
|
||||
|
||||
- [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
|
||||
errors
|
||||
- [ ] Update the API reference section below to match the current implementation
|
||||
- [x] Update the API reference section below to match the current implementation
|
||||
- [x] `make docker` green
|
||||
- [ ] Tag `v1.0.0`
|
||||
|
||||
@@ -486,25 +503,179 @@ Future (desktop client, separate repo):
|
||||
|
||||
## API reference
|
||||
|
||||
The API reference section below is from an earlier draft and does not fully
|
||||
reflect the current implementation. The authoritative API documentation is in
|
||||
the test files, particularly `test/client/usage.test.ts` which is a literate
|
||||
tutorial walking through every operation. Run `yarn test` to verify the examples
|
||||
are correct.
|
||||
The library's primary surface is the cache-backed `Library`; the lower-level
|
||||
`Client` sits underneath it and is covered by the Design sections above. The
|
||||
test suite is the canonical, executable documentation — `test/library/` and
|
||||
`test/client/usage.test.ts` walk every operation, and `yarn test` verifies them.
|
||||
|
||||
The key types and their actual signatures can be found in:
|
||||
### Opening a library
|
||||
|
||||
`Library.open(options)` loads the on-disk cache, starts the background refresh
|
||||
loop, and resolves to a `Library`. On an empty cache it awaits the first refresh
|
||||
so it never opens onto empty data; on an existing cache it returns immediately
|
||||
and refreshes in the background, so an unreachable server does not block
|
||||
opening.
|
||||
|
||||
`LibraryOptions`:
|
||||
|
||||
| Option | Default | Meaning |
|
||||
| ------------------------ | --------------------------- | --------------------------------------------------------------------- |
|
||||
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
|
||||
| `cacheDirectory` | `<XDG cache>/quak/<userID>` | where `metadata.json` and the content cache live |
|
||||
| `downloadDirectory` | none | backup destination; an original already stored there counts as cached |
|
||||
| `refreshIntervalSeconds` | `3` | background refresh cadence |
|
||||
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
|
||||
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
|
||||
| `precacheOriginalsDays` | `7` | length in days of that latest window |
|
||||
| `cacheOriginalsMaxBytes` | 100 GiB | hard ceiling on the originals cache |
|
||||
| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down |
|
||||
| `isOriginalPinned` | none | extra predicate for originals that must never be evicted |
|
||||
| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) |
|
||||
| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) |
|
||||
| `contentSource` | the client's own | override the byte source (mainly for tests) |
|
||||
|
||||
Concurrency is set through `pools`: construct
|
||||
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
|
||||
and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).
|
||||
|
||||
`lib.status()` returns a `LibraryStatus` (collection/file counts, last
|
||||
refresh/ML times and errors, originals usage and effective limit, precache
|
||||
progress, and `closed`). `lib.close()` stops the background timer; it is
|
||||
idempotent, and an in-flight refresh is left to finish.
|
||||
|
||||
### Default reads vs. fresh reads
|
||||
|
||||
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
|
||||
synchronously from the last refreshed copy held in RAM and never touch the
|
||||
network. The background timer refreshes that copy every
|
||||
`refreshIntervalSeconds`, so a default read is immediate but may be up to one
|
||||
interval stale.
|
||||
|
||||
`await lib.fresh()` forces a refresh, waits for it to complete and persist, and
|
||||
returns the same `{ albums, photos, timeline }` namespaces — now guaranteed to
|
||||
reflect a completed server round-trip. Concurrent `fresh()` calls coalesce onto
|
||||
one refresh, and a refresh that fails rejects the caller (default reads stay
|
||||
silent and keep serving the last good copy). The CLI's read commands use fresh
|
||||
reads (issue https://git.eeqj.de/sneak/quak/issues/75).
|
||||
|
||||
### Read surface
|
||||
|
||||
- `lib.albums.list()` → `Album[]`, newest-updated first.
|
||||
`lib.albums.byID({ collectionID })` and `byName({ albumName })` →
|
||||
`Album | undefined`.
|
||||
- `lib.photos.byID({ fileID })` → `Photo | undefined`.
|
||||
`lib.photos.records({ fileIDs })` → `PhotoRecord[]` in the requested order,
|
||||
each id once, unknown ids dropped.
|
||||
- `lib.timeline.groups({ groupBy, filter? })` → `TimelineGroup[]`, grouped by
|
||||
`"day" | "week" | "month"` (keys `YYYY-MM-DD`, ISO `YYYY-Www`, `YYYY-MM`),
|
||||
newest group first. A `PhotoFilter` combines `albumID`, `text`
|
||||
(title/caption/album-name substring), `fileTypes`, `hasLocation`, and
|
||||
`includeArchived`; hidden photos are always excluded.
|
||||
|
||||
An `Album` exposes its record fields and `album.photos.list()` → `Photo[]`
|
||||
(newest first). A `Photo` exposes its record fields, `photo.record()` →
|
||||
`PhotoRecord`, and two content methods:
|
||||
|
||||
- `await photo.original(opts?)` → `{ path, bytes }` — the full-resolution file.
|
||||
- `await photo.thumbnail(opts?)` → `{ path, bytes }`.
|
||||
|
||||
Both serve from the on-disk content cache when the bytes are present and
|
||||
otherwise fetch through the pools; `opts.onProgress` reports per-file progress.
|
||||
They throw when the library was opened without a content source.
|
||||
|
||||
Lower-level accessors that return decrypted model objects (which hold key
|
||||
material) are also available: `listCollections()`, `getCollection(id)`,
|
||||
`listFiles(collectionID)`, `getFile(collectionID, fileID)`, and
|
||||
`getFileByID(fileID)`.
|
||||
|
||||
### Records and change notifications
|
||||
|
||||
The GUI-facing records hold no key material and no binary, so they survive
|
||||
`structuredClone`/JSON across the Electron IPC boundary:
|
||||
|
||||
- `PhotoRecord`: `fileID`, `albumIDs`, `title`, `takenAt` (milliseconds),
|
||||
`fileType`, optional `caption` / `width` / `height` / `latitude` /
|
||||
`longitude`, `isArchived`, `isHidden`, and `thumbnailPath` / `originalPath`
|
||||
once the bytes are cached.
|
||||
- `AlbumRecord`: `collectionID`, `name`, `type`, `isShared`, `updationTime`, and
|
||||
`fileIDs` (newest first).
|
||||
- `LibrarySnapshot`: `{ albums, photos, takenAt }`.
|
||||
|
||||
`lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first,
|
||||
photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
(`albumsChanged`, `photosChanged`, `fileIDsRemoved`, `albumIDsRemoved`,
|
||||
`refreshedAt`) whenever a refresh alters the projection, and returns
|
||||
`{ unsubscribe }`; a refresh that changes nothing delivers nothing.
|
||||
|
||||
### Thumbnails, ML search, and backup
|
||||
|
||||
- `lib.thumbnails.ensure({ fileIDs, priority, signal?, onProgress? })`
|
||||
prefetches thumbnails through the thumbnail pool, deduped by fileID, returning
|
||||
one `EnsureResult` (`{ fileID, path?, error? }`) per file. `priority` is
|
||||
`"visible" | "ahead" | "background"`; only `"visible"` preempts background
|
||||
work.
|
||||
- `lib.mldata` searches the CLIP index built from Ente's per-file ML data:
|
||||
`forFile({ fileID })` → `Promise<MLData | undefined>` (the whole stored
|
||||
payload — face boxes, landmarks, embedding — read from disk on demand);
|
||||
`similar({ fileID, limit? })` and `searchByEmbedding({ embedding, limit? })` →
|
||||
`SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
|
||||
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
|
||||
a query vector the caller produced elsewhere.
|
||||
- `await lib.backup(opts?)` → `BackupResult`. It refreshes, fetches every
|
||||
in-scope original (and, with `includeThumbnails`, thumbnails) through the
|
||||
content cache, and rebuilds the on-disk backup tree with a durable failure
|
||||
ledger. `BackupOptions`: `downloadDirectory` (falls back to the one `open()`
|
||||
was given), `includeOriginals` (default `true`), `includeThumbnails` (default
|
||||
`false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above for the
|
||||
tree it writes.
|
||||
|
||||
### Request pools
|
||||
|
||||
`RequestPools` holds three independent bounded pools — metadata (10), content
|
||||
(5), thumbnails (25) — because Ente meters these traffic classes differently.
|
||||
Each pool orders on-demand work ahead of background/precache work and dedups
|
||||
in-flight fetches by key, and an idle pool never lends its slots to a busy one.
|
||||
|
||||
### On-disk cache layout
|
||||
|
||||
Under `cacheDirectory`:
|
||||
|
||||
```
|
||||
<cacheDirectory>/
|
||||
metadata.json decrypted account state + refresh cursor
|
||||
originals/<fileID>.<ext> cached full-resolution files
|
||||
thumbnails/<fileID>.jpg cached thumbnails
|
||||
mldata/
|
||||
<fileID>.json one decrypted ML payload per file
|
||||
clip.f32, clip.json the packed CLIP index and its id list
|
||||
fetched.json per-file fetch bookkeeping
|
||||
```
|
||||
|
||||
A stored file appears only via an atomic temp-then-rename, so its presence means
|
||||
it is complete. The design also calls for a content-hash comparison against
|
||||
`FileMetadata.hash` on each fetched original; that check is deferred (issue
|
||||
https://git.eeqj.de/sneak/quak/issues/68) because the exact hash construction
|
||||
cannot yet be confirmed against the repo's fixtures.
|
||||
|
||||
### Key types by source file
|
||||
|
||||
- `src/library/index.ts`: `Library`, `LibraryOptions`, `LibraryStatus`,
|
||||
`LibraryClient`, `RefreshEvent`
|
||||
- `src/library/read.ts`: `Album`, `Photo`, `AlbumsAPI`, `PhotosAPI`,
|
||||
`TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`
|
||||
- `src/library/content.ts`: `ContentResult`, `ContentOptions`, `ThumbnailsAPI`,
|
||||
`EnsureOptions`, `EnsureResult`, `ContentSource`
|
||||
- `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`,
|
||||
`LibraryChange`
|
||||
- `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult`
|
||||
- `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool`
|
||||
- `src/backup.ts`: `BackupOptions`, `BackupResult`, `BackupError`
|
||||
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
|
||||
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `ApiError`,
|
||||
`StreamOptions`
|
||||
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `StreamOptions`
|
||||
- `src/errors.ts`: `ApiError`, `TruncatedStreamError`
|
||||
- `src/retry.ts`: `withRetry`, `isRetryable`, `isSafeToReplay`, `RetryOptions`
|
||||
- `src/auth/types.ts`: `KeyAttributes`, `SRPAttributes`,
|
||||
`AuthorizationResponse`, `LoginChallenge`
|
||||
- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileBlob`,
|
||||
`RawCollection`, `RawEnteFile`, `RawMagicMetadata`
|
||||
- `src/download/index.ts`: `DownloadResult`
|
||||
- `src/backup.ts`: `BackupResult`, `BackupError`
|
||||
- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileType`,
|
||||
`CollectionType`, `RawCollection`, `RawEnteFile`
|
||||
- `src/thumbnails.ts`: `MissingThumbnailInfo`, `ThumbnailFixResult`
|
||||
|
||||
## Source attribution
|
||||
|
||||
@@ -14,10 +14,19 @@ pre-1.0
|
||||
|
||||
# Next Step
|
||||
|
||||
Update the README API reference section to match the current implementation.
|
||||
Tag v1.0.0.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
|
||||
snippets) to match the shipped cache/API library on `next` (issue 53, issue
|
||||
13). Documented `Library.open` and its options, the default-read vs `fresh()`
|
||||
distinction (and that the CLI's read commands are fresh), the record types and
|
||||
`snapshot()`/`subscribe()`, the `albums`/`photos`/`timeline` read surface,
|
||||
`Photo` content methods, `thumbnails.ensure`, the `mldata` search surface,
|
||||
`backup()`, the three request pools (10/5/25), and the on-disk cache layout;
|
||||
noted the deferred content-hash integrity check (issue 68). Docs-only; no code
|
||||
changed.
|
||||
- 2026-09-22: Added resumable, deletion-aware enumeration to `Client` (issue 38,
|
||||
closes issue 7). `collectionsSince`/`filesSince` take a starting cursor,
|
||||
decrypt live records, surface tombstoned ids in a separate `deleted` list (a
|
||||
@@ -122,7 +131,6 @@ Update the README API reference section to match the current implementation.
|
||||
|
||||
# Future Steps
|
||||
|
||||
- Tag v1.0.0.
|
||||
- Future desktop client, separate repo:
|
||||
- Electron app skeleton consuming this library.
|
||||
- Local SQLite cache keyed on (collectionID, fileID, updationTime).
|
||||
|
||||
+46
-32
@@ -15,6 +15,13 @@ import envPaths from "env-paths";
|
||||
import { Client, type ClientSnapshot } from "../src/client.js";
|
||||
import { init } from "../src/crypto/index.js";
|
||||
import { Library, type LibraryClient } from "../src/library/index.js";
|
||||
import {
|
||||
fileListRow,
|
||||
fileListLine,
|
||||
originalName,
|
||||
thumbnailName,
|
||||
} from "../src/cli-output.js";
|
||||
import { freshCollections, freshFiles, freshFile } from "../src/cli-read.js";
|
||||
import { runMetadataBackup } from "../src/metadata-backup.js";
|
||||
import {
|
||||
listMissingThumbnails,
|
||||
@@ -177,27 +184,30 @@ program
|
||||
await init();
|
||||
const client = requireSession();
|
||||
const lib = await openReadLibrary(client);
|
||||
const albums = lib.albums.list();
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52): the pre-library CLI printed current state in
|
||||
// this order, not the albums projection's newest-first order.
|
||||
const collections = await freshCollections(lib);
|
||||
|
||||
if (opts.json) {
|
||||
stdout.write(
|
||||
JSON.stringify(
|
||||
albums.map((a) => ({
|
||||
id: a.collectionID,
|
||||
name: a.name,
|
||||
type: a.type,
|
||||
ownerID: lib.getCollection(a.collectionID)?.ownerID,
|
||||
isShared: a.isShared,
|
||||
updationTime: a.updationTime,
|
||||
collections.map((c) => ({
|
||||
id: c.id,
|
||||
name: c.name,
|
||||
type: c.type,
|
||||
ownerID: c.ownerID,
|
||||
isShared: c.isShared,
|
||||
updationTime: c.updationTime,
|
||||
})),
|
||||
null,
|
||||
2,
|
||||
) + "\n",
|
||||
);
|
||||
} else {
|
||||
for (const a of albums) {
|
||||
for (const c of collections) {
|
||||
stdout.write(
|
||||
`${a.collectionID}\t${a.type}\t${a.name}${a.isShared ? " (shared)" : ""}\n`,
|
||||
`${c.id}\t${c.type}\t${c.name}${c.isShared ? " (shared)" : ""}\n`,
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -222,31 +232,25 @@ program
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(client);
|
||||
const album = lib.albums.byID({ collectionID });
|
||||
if (!album) {
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52). Each file prints from its own decrypted
|
||||
// metadata (raw title, microsecond creationTime) via cli-output, and in
|
||||
// the pre-library CLI's enumeration order, not the projection's
|
||||
// newest-first order.
|
||||
const files = await freshFiles(lib, collectionID);
|
||||
if (!files) {
|
||||
stderr.write(`Collection ${collectionID} not found\n`);
|
||||
finish(lib, 1);
|
||||
return;
|
||||
}
|
||||
|
||||
const photos = album.photos.list();
|
||||
if (opts.json) {
|
||||
stdout.write(
|
||||
JSON.stringify(
|
||||
photos.map((p) => ({
|
||||
id: p.fileID,
|
||||
title: p.title,
|
||||
fileType: p.fileType,
|
||||
creationTime: p.takenAt,
|
||||
collectionID,
|
||||
})),
|
||||
null,
|
||||
2,
|
||||
) + "\n",
|
||||
JSON.stringify(files.map(fileListRow), null, 2) + "\n",
|
||||
);
|
||||
} else {
|
||||
for (const p of photos) {
|
||||
stdout.write(`${p.fileID}\t${p.fileType}\t${p.title}\n`);
|
||||
for (const file of files) {
|
||||
stdout.write(fileListLine(file) + "\n");
|
||||
}
|
||||
}
|
||||
finish(lib, 0);
|
||||
@@ -268,15 +272,20 @@ program
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(client);
|
||||
const photo = lib.photos.byID({ fileID });
|
||||
if (!photo) {
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
stderr.write(`File ${fileID} not found\n`);
|
||||
finish(lib, 1);
|
||||
return;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await photo.original();
|
||||
const outPath = opts.out ?? photo.title;
|
||||
// Default name is the file's own title, as the pre-library CLI used
|
||||
// (not the editedName-preferring projection title) (issue #52).
|
||||
const outPath = opts.out ?? originalName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
finish(lib, 0);
|
||||
@@ -298,15 +307,20 @@ program
|
||||
}
|
||||
|
||||
const lib = await openReadLibrary(client);
|
||||
const photo = lib.photos.byID({ fileID });
|
||||
if (!photo) {
|
||||
// Force a server round-trip so the file resolves against current state
|
||||
// (issue #36 amendment, issue #52).
|
||||
const resolved = await freshFile(lib, fileID);
|
||||
if (!resolved) {
|
||||
stderr.write(`File ${fileID} not found\n`);
|
||||
finish(lib, 1);
|
||||
return;
|
||||
}
|
||||
const { photo, file } = resolved;
|
||||
|
||||
const result = await photo.thumbnail();
|
||||
const outPath = opts.out ?? `thumb_${photo.title}`;
|
||||
// Default name is thumb_<file's own title>, as the pre-library CLI
|
||||
// used (not the projection title) (issue #52).
|
||||
const outPath = opts.out ?? thumbnailName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
finish(lib, 0);
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
// How the CLI presents a file's identity in `files`, `get`, and `get-thumb`.
|
||||
//
|
||||
// These read the file's own decrypted metadata — the raw title and the
|
||||
// creationTime in microseconds — rather than the `PhotoRecord` projection the
|
||||
// rest of the library exposes. The projection prefers `editedName`/`editedTime`
|
||||
// and reports time in milliseconds, which is right for a photo browser but
|
||||
// would change the CLI's externally-visible output. The pre-library CLI printed
|
||||
// `metadata.title` and `metadata.creationTime` and named downloads after
|
||||
// `metadata.title`, and issue #52 requires that output stay byte-identical, so
|
||||
// the commands shape their output from the raw `EnteFile` through here.
|
||||
|
||||
import type { EnteFile, FileType, Microseconds } from "./model/types.js";
|
||||
|
||||
// One row of `quak files --json`.
|
||||
export interface FileListRow {
|
||||
id: number;
|
||||
title: string;
|
||||
fileType: FileType;
|
||||
creationTime: Microseconds;
|
||||
collectionID: number;
|
||||
}
|
||||
|
||||
export const fileListRow = (file: EnteFile): FileListRow => ({
|
||||
id: file.id,
|
||||
title: file.metadata.title,
|
||||
fileType: file.metadata.fileType,
|
||||
creationTime: file.metadata.creationTime,
|
||||
collectionID: file.collectionID,
|
||||
});
|
||||
|
||||
// One line of `quak files` in its human, tab-separated form.
|
||||
export const fileListLine = (file: EnteFile): string =>
|
||||
`${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
|
||||
|
||||
// Default output path for `quak get` when `--out` is not given.
|
||||
export const originalName = (file: EnteFile): string => file.metadata.title;
|
||||
|
||||
// Default output path for `quak get-thumb` when `--out` is not given.
|
||||
export const thumbnailName = (file: EnteFile): string =>
|
||||
`thumb_${file.metadata.title}`;
|
||||
@@ -0,0 +1,62 @@
|
||||
// How the CLI's read commands obtain current data.
|
||||
//
|
||||
// `collections`, `files --collection`, `get`, and `get-thumb` must answer for
|
||||
// the account's state at the moment the command runs, not for whatever the
|
||||
// local cache last happened to hold (owner amendment, issue #36). Each helper
|
||||
// therefore forces a server round-trip through `Library.fresh()` and only then
|
||||
// reads — so a collection, file, or metadata change made elsewhere is visible.
|
||||
//
|
||||
// `collections` and `files` also list in the library's own enumeration order —
|
||||
// `listCollections()`/`listFiles()`, the order the pre-library CLI printed —
|
||||
// rather than the `albums`/`photos` projection's newest-first order, which
|
||||
// re-sorts the rows. The field values still come from each record's raw
|
||||
// metadata via `cli-output.ts`.
|
||||
|
||||
import type { Collection, EnteFile } from "./model/types.js";
|
||||
import type { Photo, PhotosAPI } from "./library/index.js";
|
||||
|
||||
// The slice of `Library` these helpers read. `Library` satisfies it
|
||||
// structurally; a test can drive them with a stand-in that records the
|
||||
// `fresh()` call and serves records in a known enumeration order.
|
||||
export interface FreshReadLibrary {
|
||||
fresh(): Promise<unknown>;
|
||||
listCollections(): Collection[];
|
||||
getCollection(id: number): Collection | undefined;
|
||||
listFiles(collectionID: number): EnteFile[];
|
||||
getFileByID(fileID: number): EnteFile | undefined;
|
||||
photos: Pick<PhotosAPI, "byID">;
|
||||
}
|
||||
|
||||
// Every live collection, current as of a forced refresh, in enumeration order.
|
||||
export const freshCollections = async (
|
||||
lib: FreshReadLibrary,
|
||||
): Promise<Collection[]> => {
|
||||
await lib.fresh();
|
||||
return lib.listCollections();
|
||||
};
|
||||
|
||||
// The files of one collection, current as of a forced refresh, in enumeration
|
||||
// order. `undefined` (not an empty list) when the collection does not exist, so
|
||||
// the caller can tell "no such collection" from "an empty collection".
|
||||
export const freshFiles = async (
|
||||
lib: FreshReadLibrary,
|
||||
collectionID: number,
|
||||
): Promise<EnteFile[] | undefined> => {
|
||||
await lib.fresh();
|
||||
if (!lib.getCollection(collectionID)) return undefined;
|
||||
return lib.listFiles(collectionID);
|
||||
};
|
||||
|
||||
// One file, current as of a forced refresh, resolved to both its content
|
||||
// handle (`Photo`, for fetching bytes) and its raw record (`EnteFile`, for the
|
||||
// default output name and field values). `undefined` when the file is unknown.
|
||||
export const freshFile = async (
|
||||
lib: FreshReadLibrary,
|
||||
fileID: number,
|
||||
): Promise<{ photo: Photo; file: EnteFile } | undefined> => {
|
||||
await lib.fresh();
|
||||
const photo = lib.photos.byID({ fileID });
|
||||
const file = lib.getFileByID(fileID);
|
||||
if (!photo || !file) return undefined;
|
||||
return { photo, file };
|
||||
};
|
||||
+84
-12
@@ -6,10 +6,17 @@
|
||||
// existing copy loaded, that first refresh runs in the background and `open()`
|
||||
// returns as soon as the cached data is ready to serve — a slow or unreachable
|
||||
// server no longer stalls opening. A background timer then refreshes every
|
||||
// `refreshIntervalSeconds`. Every read is answered from RAM — no read touches
|
||||
// the network. There is deliberately no `sync()`, no `refresh()`, no
|
||||
// `serverReachable` flag, and no "before each read" mode (design #36): the
|
||||
// only ways state changes are the refreshes above.
|
||||
// `refreshIntervalSeconds`. Every default read is answered from RAM — no
|
||||
// default read touches the network. There is deliberately no `sync()`, no
|
||||
// `refresh()`, no `serverReachable` flag, and no "before each read" mode
|
||||
// (design #36).
|
||||
//
|
||||
// `fresh()` is the one exception (issue #75, an owner amendment to #36): it
|
||||
// forces a refresh, awaits it, and only then hands back the read namespaces, so
|
||||
// a caller that needs server-current data can ask for it. Concurrent `fresh()`
|
||||
// calls coalesce onto one in-flight refresh, and a refresh that fails rejects
|
||||
// the caller (the default reads stay silent and serve the last good copy). The
|
||||
// default methods and the background loop are unchanged.
|
||||
//
|
||||
// A refresh stages all of its network work first and only mutates the store
|
||||
// once every fetch has succeeded. A refresh that fails partway therefore never
|
||||
@@ -40,6 +47,7 @@ import {
|
||||
type AlbumsAPI,
|
||||
type PhotosAPI,
|
||||
type TimelineAPI,
|
||||
type FreshReads,
|
||||
} from "./read.js";
|
||||
import {
|
||||
ContentCache,
|
||||
@@ -57,6 +65,7 @@ export {
|
||||
type AlbumsAPI,
|
||||
type PhotosAPI,
|
||||
type TimelineAPI,
|
||||
type FreshReads,
|
||||
type PhotoFilter,
|
||||
type TimelineGroup,
|
||||
type GroupBy,
|
||||
@@ -250,7 +259,12 @@ export class Library {
|
||||
private readonly precache?: Precache;
|
||||
|
||||
private timer?: ReturnType<typeof setTimeout>;
|
||||
private refreshing = false;
|
||||
// The in-flight refresh cycle, or undefined when none runs. One slot serves
|
||||
// both paths: the background loop skips when it is set, and a fresh read
|
||||
// (issue #75) coalesces onto it or starts one. The promise carries the
|
||||
// cycle's real outcome (it rejects on failure); the background loop ignores
|
||||
// that, a fresh read propagates it.
|
||||
private cycle?: Promise<void>;
|
||||
// Guards the ML fetch pass so a slow backfill never runs twice at once; a
|
||||
// refresh whose pass is still running kicks nothing new.
|
||||
private mlFetching = false;
|
||||
@@ -433,6 +447,13 @@ export class Library {
|
||||
return this.store.getFile(collectionID, fileID);
|
||||
}
|
||||
|
||||
// Any membership of a file, addressed by file id alone. A file's own
|
||||
// metadata (title, creationTime) is identical across the collections it
|
||||
// belongs to, so this serves the point commands that hold only a fileID.
|
||||
getFileByID(fileID: number): EnteFile | undefined {
|
||||
return this.store.getFileByID(fileID);
|
||||
}
|
||||
|
||||
// A synchronous, RAM-only projection of the whole library into plain
|
||||
// records (no keys), the surface the GUI reads across IPC. Photos are
|
||||
// deduplicated to one record per file and ordered newest first.
|
||||
@@ -484,6 +505,21 @@ export class Library {
|
||||
};
|
||||
}
|
||||
|
||||
// Fresh reads (issue #75, owner amendment to design #36). Force a refresh,
|
||||
// wait for it to complete and persist, then hand back the same
|
||||
// `albums`/`photos`/`timeline` namespaces — now guaranteed to reflect a
|
||||
// completed server round-trip. Concurrent calls coalesce onto one refresh;
|
||||
// a refresh that fails rejects here, where the default namespaces would
|
||||
// instead stay silent and serve the last good copy.
|
||||
async fresh(): Promise<FreshReads> {
|
||||
await this.refreshNow();
|
||||
return {
|
||||
albums: this.albums,
|
||||
photos: this.photos,
|
||||
timeline: this.timeline,
|
||||
};
|
||||
}
|
||||
|
||||
// Back up every in-scope file to `downloadDirectory` in the historical
|
||||
// on-disk layout, with a durable failure ledger (issue #51). Refreshes
|
||||
// first, fetches pending originals (and optional thumbnails) through the
|
||||
@@ -543,11 +579,48 @@ export class Library {
|
||||
this.timer.unref?.();
|
||||
}
|
||||
|
||||
// One refresh cycle, guarded so a failure never escapes and overlapping
|
||||
// cycles never run. Errors are reported, not thrown.
|
||||
private async runRefresh(): Promise<void> {
|
||||
if (this.closed || this.refreshing) return;
|
||||
this.refreshing = true;
|
||||
// The background loop's refresh: run a cycle unless one is already in flight
|
||||
// (or the library is closed), and never let a failure escape — the
|
||||
// background path reports errors through `status()`/`onProgress`, it does
|
||||
// not throw. Resolves once the cycle it started (or skipped past) settles.
|
||||
private runRefresh(): Promise<void> {
|
||||
if (this.closed || this.cycle) return Promise.resolve();
|
||||
return this.startCycle().catch(() => {});
|
||||
}
|
||||
|
||||
// A fresh read's refresh (issue #75): force a cycle and await it, rejecting
|
||||
// if it fails. Concurrent fresh reads coalesce onto the one in-flight cycle
|
||||
// — the background loop's included — so they never fan out into redundant
|
||||
// server round-trips.
|
||||
private refreshNow(): Promise<void> {
|
||||
if (this.closed) {
|
||||
return Promise.reject(new Error("the library is closed"));
|
||||
}
|
||||
return this.cycle ?? this.startCycle();
|
||||
}
|
||||
|
||||
// Start one refresh cycle and record it as the in-flight cycle so every
|
||||
// caller coalesces onto it. The returned promise carries the cycle's real
|
||||
// outcome; each caller attaches the handling its own path needs, and the
|
||||
// slot is cleared once the cycle settles.
|
||||
private startCycle(): Promise<void> {
|
||||
const cycle = this.refreshCycle();
|
||||
this.cycle = cycle;
|
||||
void cycle.then(
|
||||
() => {
|
||||
if (this.cycle === cycle) this.cycle = undefined;
|
||||
},
|
||||
() => {
|
||||
if (this.cycle === cycle) this.cycle = undefined;
|
||||
},
|
||||
);
|
||||
return cycle;
|
||||
}
|
||||
|
||||
// One refresh cycle: the network fetch and commit, wrapped in the progress
|
||||
// events and status bookkeeping. Throws when the refresh fails so a fresh
|
||||
// read can reject; `runRefresh` swallows that throw for the background loop.
|
||||
private async refreshCycle(): Promise<void> {
|
||||
this.emit({ operation: "refresh", status: "started" });
|
||||
try {
|
||||
await this.refreshOnce();
|
||||
@@ -563,8 +636,7 @@ export class Library {
|
||||
const error = err instanceof Error ? err.message : String(err);
|
||||
this.lastError = error;
|
||||
this.emit({ operation: "refresh", status: "failed", error });
|
||||
} finally {
|
||||
this.refreshing = false;
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -193,6 +193,15 @@ export interface TimelineAPI {
|
||||
groups(args: { groupBy: GroupBy; filter?: PhotoFilter }): TimelineGroup[];
|
||||
}
|
||||
|
||||
// The surface `Library.fresh()` resolves to (issue #75). It is the same three
|
||||
// read namespaces as the default `albums`/`photos`/`timeline`, handed back only
|
||||
// after a forced refresh has brought the local copy current.
|
||||
export interface FreshReads {
|
||||
albums: AlbumsAPI;
|
||||
photos: PhotosAPI;
|
||||
timeline: TimelineAPI;
|
||||
}
|
||||
|
||||
export const makeAlbumsAPI = (
|
||||
derive: () => DerivedRecords,
|
||||
content?: PhotoContent,
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
// The CLI presents a file by its own decrypted metadata, not the PhotoRecord
|
||||
// projection (issue #52). For a renamed file the two disagree: the projection
|
||||
// prefers `editedName` and reports `editedTime` in milliseconds, while the CLI
|
||||
// must print the raw `metadata.title` and `metadata.creationTime` (microseconds)
|
||||
// and name downloads after the raw title, byte-identical to the pre-library CLI.
|
||||
//
|
||||
// This locks in that contrast: the shared output helpers emit the raw values,
|
||||
// and the projection of the same file emits the edited ones — so a regression
|
||||
// that re-sourced the CLI from the projection would fail here.
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
|
||||
import {
|
||||
fileListRow,
|
||||
fileListLine,
|
||||
originalName,
|
||||
thumbnailName,
|
||||
} from "../../src/cli-output.js";
|
||||
import { deriveRecords } from "../../src/library/records.js";
|
||||
import type { EnteFile } from "../../src/model/types.js";
|
||||
|
||||
// Microseconds, as Ente stores times.
|
||||
const RAW_CREATION = 1700000000000000;
|
||||
const EDITED_TIME = 1710000000000000;
|
||||
const RAW_TITLE = "IMG_0001.HEIC";
|
||||
const EDITED_NAME = "Sunset.heic";
|
||||
|
||||
// A file the user has renamed and re-dated: basic metadata holds the original
|
||||
// title and capture time; public magic metadata holds the edits.
|
||||
const renamedFile: EnteFile = {
|
||||
id: 100,
|
||||
collectionID: 10,
|
||||
ownerID: 42,
|
||||
key: new Uint8Array(),
|
||||
metadata: {
|
||||
title: RAW_TITLE,
|
||||
fileType: "image",
|
||||
creationTime: RAW_CREATION,
|
||||
modificationTime: RAW_CREATION,
|
||||
},
|
||||
pubMagicMetadata: { editedName: EDITED_NAME, editedTime: EDITED_TIME },
|
||||
file: { decryptionHeader: "" },
|
||||
thumbnail: { decryptionHeader: "" },
|
||||
updationTime: RAW_CREATION,
|
||||
};
|
||||
|
||||
describe("CLI file output (issue #52)", () => {
|
||||
it("emits the raw title and microsecond creationTime for --json", () => {
|
||||
expect(fileListRow(renamedFile)).toEqual({
|
||||
id: 100,
|
||||
title: RAW_TITLE,
|
||||
fileType: "image",
|
||||
creationTime: RAW_CREATION,
|
||||
collectionID: 10,
|
||||
});
|
||||
});
|
||||
|
||||
it("emits the raw title in the human column", () => {
|
||||
expect(fileListLine(renamedFile)).toBe(`100\timage\t${RAW_TITLE}`);
|
||||
});
|
||||
|
||||
it("names downloads after the raw title", () => {
|
||||
expect(originalName(renamedFile)).toBe(RAW_TITLE);
|
||||
expect(thumbnailName(renamedFile)).toBe(`thumb_${RAW_TITLE}`);
|
||||
});
|
||||
|
||||
it("does not use the editedName/editedTime projection", () => {
|
||||
const record = deriveRecords([], [renamedFile]).photos.get(100);
|
||||
// The projection prefers the edits and reports milliseconds; the CLI
|
||||
// helpers above deliberately do not.
|
||||
expect(record?.title).toBe(EDITED_NAME);
|
||||
expect(record?.takenAt).toBe(Math.floor(EDITED_TIME / 1000));
|
||||
expect(fileListRow(renamedFile).title).not.toBe(record?.title);
|
||||
expect(fileListRow(renamedFile).creationTime).not.toBe(record?.takenAt);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,195 @@
|
||||
/**
|
||||
* Tests for the CLI read helpers (`src/cli-read.ts`, owner amendment to
|
||||
* issue #36, issue #52).
|
||||
*
|
||||
* The `collections`, `files`, `get`, and `get-thumb` commands must answer for
|
||||
* current server state, not the local cache, so each helper forces a
|
||||
* `Library.fresh()` round-trip before it reads. The stand-in library below
|
||||
* serves nothing until `fresh()` has been awaited, so a helper that read
|
||||
* without refreshing would come back empty and fail here.
|
||||
*
|
||||
* `collections` and `files` also list in the library's enumeration order
|
||||
* (`listCollections`/`listFiles`) — the order the pre-library CLI printed — not
|
||||
* the albums/photos projection's newest-first order. The fixtures are seeded in
|
||||
* an enumeration order that a newest-first sort would rearrange, so a
|
||||
* regression to the projection order would fail here too. Field values still
|
||||
* come from the raw metadata via `cli-output.ts`.
|
||||
*/
|
||||
|
||||
import { describe, it, expect } from "vitest";
|
||||
|
||||
import {
|
||||
freshCollections,
|
||||
freshFiles,
|
||||
freshFile,
|
||||
type FreshReadLibrary,
|
||||
} from "../../src/cli-read.js";
|
||||
import { fileListRow } from "../../src/cli-output.js";
|
||||
import type { Photo } from "../../src/library/index.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
|
||||
const collection = (id: number, updationTime: number): Collection => ({
|
||||
id,
|
||||
ownerID: 42,
|
||||
key: new Uint8Array(),
|
||||
name: `album-${id}`,
|
||||
type: "album",
|
||||
updationTime,
|
||||
isShared: false,
|
||||
});
|
||||
|
||||
// Microseconds, as Ente stores times.
|
||||
const file = (
|
||||
id: number,
|
||||
collectionID: number,
|
||||
creationTime: number,
|
||||
): EnteFile => ({
|
||||
id,
|
||||
collectionID,
|
||||
ownerID: 42,
|
||||
key: new Uint8Array(),
|
||||
metadata: {
|
||||
title: `file-${id}.jpg`,
|
||||
fileType: "image",
|
||||
creationTime,
|
||||
modificationTime: creationTime,
|
||||
},
|
||||
file: { decryptionHeader: "" },
|
||||
thumbnail: { decryptionHeader: "" },
|
||||
updationTime: creationTime,
|
||||
});
|
||||
|
||||
// A library that reveals its records only after `fresh()` has been awaited, and
|
||||
// serves them in the enumeration order it was given. `photos.byID` returns a
|
||||
// stand-in `Photo` carrying just the fileID the helper passes through.
|
||||
class FakeLibrary implements FreshReadLibrary {
|
||||
freshCalls = 0;
|
||||
private refreshed = false;
|
||||
|
||||
constructor(
|
||||
private readonly collections: Collection[],
|
||||
private readonly files: EnteFile[],
|
||||
) {}
|
||||
|
||||
async fresh(): Promise<unknown> {
|
||||
this.freshCalls++;
|
||||
this.refreshed = true;
|
||||
return {};
|
||||
}
|
||||
|
||||
listCollections(): Collection[] {
|
||||
return this.refreshed ? this.collections : [];
|
||||
}
|
||||
|
||||
getCollection(id: number): Collection | undefined {
|
||||
return this.listCollections().find((c) => c.id === id);
|
||||
}
|
||||
|
||||
listFiles(collectionID: number): EnteFile[] {
|
||||
return this.refreshed
|
||||
? this.files.filter((f) => f.collectionID === collectionID)
|
||||
: [];
|
||||
}
|
||||
|
||||
getFileByID(fileID: number): EnteFile | undefined {
|
||||
if (!this.refreshed) return undefined;
|
||||
return this.files.find((f) => f.id === fileID);
|
||||
}
|
||||
|
||||
photos = {
|
||||
byID: ({ fileID }: { fileID: number }): Photo | undefined => {
|
||||
if (!this.refreshed) return undefined;
|
||||
if (!this.files.some((f) => f.id === fileID)) return undefined;
|
||||
return { fileID } as unknown as Photo;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
||||
it("freshCollections refreshes first, then lists in enumeration order", async () => {
|
||||
// Enumeration order 2, 1, 3; a newest-first sort would be 3, 2, 1.
|
||||
const lib = new FakeLibrary(
|
||||
[collection(2, 200), collection(1, 300), collection(3, 100)],
|
||||
[],
|
||||
);
|
||||
const rows = await freshCollections(lib);
|
||||
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(rows.map((c) => c.id)).toEqual([2, 1, 3]);
|
||||
// The projection's newest-first order is a different sequence, so this
|
||||
// is not accidentally that order.
|
||||
const newestFirst = [...rows]
|
||||
.sort((a, b) => b.updationTime - a.updationTime)
|
||||
.map((c) => c.id);
|
||||
expect(newestFirst).toEqual([1, 2, 3]);
|
||||
expect(rows.map((c) => c.id)).not.toEqual(newestFirst);
|
||||
});
|
||||
|
||||
it("freshFiles refreshes first, lists in enumeration order, keeps raw fields", async () => {
|
||||
// Enumeration order by id 10, 11, 12; creationTimes ascending, so a
|
||||
// newest-first sort would reverse them.
|
||||
const files = [
|
||||
file(10, 1, 1_700_000_000_000_000),
|
||||
file(11, 1, 1_700_000_000_000_001),
|
||||
file(12, 1, 1_700_000_000_000_002),
|
||||
];
|
||||
const lib = new FakeLibrary([collection(1, 100)], files);
|
||||
|
||||
const rows = await freshFiles(lib, 1);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(rows?.map((f) => f.id)).toEqual([10, 11, 12]);
|
||||
|
||||
// Field values come from raw metadata: microsecond creationTime and the
|
||||
// raw title, unchanged.
|
||||
expect(rows?.map(fileListRow)).toEqual([
|
||||
{
|
||||
id: 10,
|
||||
title: "file-10.jpg",
|
||||
fileType: "image",
|
||||
creationTime: 1_700_000_000_000_000,
|
||||
collectionID: 1,
|
||||
},
|
||||
{
|
||||
id: 11,
|
||||
title: "file-11.jpg",
|
||||
fileType: "image",
|
||||
creationTime: 1_700_000_000_000_001,
|
||||
collectionID: 1,
|
||||
},
|
||||
{
|
||||
id: 12,
|
||||
title: "file-12.jpg",
|
||||
fileType: "image",
|
||||
creationTime: 1_700_000_000_000_002,
|
||||
collectionID: 1,
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("freshFiles returns undefined for an unknown collection", async () => {
|
||||
const lib = new FakeLibrary([collection(1, 100)], []);
|
||||
const rows = await freshFiles(lib, 999);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(rows).toBeUndefined();
|
||||
});
|
||||
|
||||
it("freshFile refreshes first, then resolves the photo and its raw record", async () => {
|
||||
const f = file(10, 1, 1_700_000_000_000_000);
|
||||
const lib = new FakeLibrary([collection(1, 100)], [f]);
|
||||
|
||||
const resolved = await freshFile(lib, 10);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(resolved?.photo.fileID).toBe(10);
|
||||
expect(resolved?.file.metadata.title).toBe("file-10.jpg");
|
||||
expect(resolved?.file.metadata.creationTime).toBe(
|
||||
1_700_000_000_000_000,
|
||||
);
|
||||
});
|
||||
|
||||
it("freshFile returns undefined for an unknown file", async () => {
|
||||
const lib = new FakeLibrary([collection(1, 100)], []);
|
||||
const resolved = await freshFile(lib, 404);
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(resolved).toBeUndefined();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,272 @@
|
||||
/**
|
||||
* Tests for fresh reads (issue #75, an owner amendment to design #36).
|
||||
*
|
||||
* The default read namespaces answer from RAM and never touch the network; a
|
||||
* background loop keeps the local copy current. `fresh()` adds an awaited path:
|
||||
* it forces a refresh, waits for it to complete and persist, and only then
|
||||
* hands back the `albums`/`photos`/`timeline` namespaces, guaranteeing the
|
||||
* local copy reflects a completed server round-trip. The contracts here:
|
||||
*
|
||||
* 1. A fresh read observes a server change that a same-instant default read
|
||||
* would miss (the default path has not refreshed yet).
|
||||
* 2. Concurrent fresh reads coalesce onto one in-flight refresh — three
|
||||
* concurrent `fresh()` calls make exactly one collections round-trip, not
|
||||
* three.
|
||||
* 3. A refresh that fails rejects the fresh read (currency was unavailable),
|
||||
* while the default reads stay silent and keep serving the last good copy.
|
||||
*
|
||||
* The client is the same metadata-only mock the background-refresh tests use:
|
||||
* no crypto, no network, scripted pages, and a record of each call. A long
|
||||
* refresh interval keeps the background timer out of the way so each test's
|
||||
* refreshes are exactly the ones it triggers.
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from "vitest";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
|
||||
const USER_ID = 42;
|
||||
|
||||
// Long enough that no background tick fires during a test; each test's
|
||||
// refreshes are only the ones its own `fresh()` calls force.
|
||||
const SLOW_INTERVAL = 3600;
|
||||
|
||||
const collection = (id: number, updationTime: number): Collection => ({
|
||||
id,
|
||||
ownerID: USER_ID,
|
||||
key: new Uint8Array([id & 0xff]),
|
||||
name: `album-${id}`,
|
||||
type: "album",
|
||||
updationTime,
|
||||
isShared: false,
|
||||
});
|
||||
|
||||
const file = (
|
||||
id: number,
|
||||
collectionID: number,
|
||||
updationTime: number,
|
||||
): EnteFile => ({
|
||||
id,
|
||||
collectionID,
|
||||
ownerID: USER_ID,
|
||||
key: new Uint8Array([id & 0xff]),
|
||||
metadata: {
|
||||
title: `file-${id}.jpg`,
|
||||
fileType: "image",
|
||||
creationTime: updationTime,
|
||||
modificationTime: updationTime,
|
||||
},
|
||||
file: { decryptionHeader: "aGVhZGVy" },
|
||||
thumbnail: { decryptionHeader: "dGh1bWI=" },
|
||||
updationTime,
|
||||
});
|
||||
|
||||
class MockClient {
|
||||
userID = USER_ID;
|
||||
failCollections = false;
|
||||
collectionsQueue: CollectionsPage[] = [];
|
||||
filesByCollection = new Map<number, FilesPage[]>();
|
||||
|
||||
collectionsSinceTimes: number[] = [];
|
||||
filesCalls: { collectionID: number; sinceTime: number }[] = [];
|
||||
|
||||
whoami(): { email: string; userID: number } {
|
||||
return { email: "user@example.com", userID: this.userID };
|
||||
}
|
||||
|
||||
async collectionsSince(args: {
|
||||
sinceTime: number;
|
||||
}): Promise<CollectionsPage> {
|
||||
this.collectionsSinceTimes.push(args.sinceTime);
|
||||
if (this.failCollections) throw new Error("network down");
|
||||
return (
|
||||
this.collectionsQueue.shift() ?? {
|
||||
collections: [],
|
||||
deleted: [],
|
||||
cursor: args.sinceTime,
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
async filesSince(args: {
|
||||
collectionID: number;
|
||||
collectionKey: Uint8Array;
|
||||
sinceTime: number;
|
||||
}): Promise<FilesPage> {
|
||||
this.filesCalls.push({
|
||||
collectionID: args.collectionID,
|
||||
sinceTime: args.sinceTime,
|
||||
});
|
||||
const queue = this.filesByCollection.get(args.collectionID);
|
||||
return (
|
||||
queue?.shift() ?? {
|
||||
files: [],
|
||||
deleted: [],
|
||||
cursor: args.sinceTime,
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
filesFor(collectionID: number, ...pages: FilesPage[]): void {
|
||||
this.filesByCollection.set(collectionID, pages);
|
||||
}
|
||||
}
|
||||
|
||||
// A client seeded with one collection and one file, opened with an empty cache
|
||||
// so the initial refresh is awaited and post-`open()` state is deterministic.
|
||||
const openSeeded = async (
|
||||
cacheDirectory: string,
|
||||
): Promise<{ client: MockClient; lib: Library }> => {
|
||||
const client = new MockClient();
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 100)],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
client.filesFor(1, {
|
||||
files: [file(1001, 1, 90)],
|
||||
deleted: [],
|
||||
cursor: 90,
|
||||
});
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: SLOW_INTERVAL,
|
||||
});
|
||||
return { client, lib };
|
||||
};
|
||||
|
||||
describe("Library.fresh", () => {
|
||||
let dir: string;
|
||||
let cacheDirectory: string;
|
||||
|
||||
beforeEach(() => {
|
||||
dir = mkdtempSync(join(tmpdir(), "quak-fresh-"));
|
||||
cacheDirectory = join(dir, "cache");
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it("observes a server change a same-instant default read would miss", async () => {
|
||||
const { client, lib } = await openSeeded(cacheDirectory);
|
||||
try {
|
||||
// A new file appears on the server after open, advancing its
|
||||
// collection so the next refresh re-enumerates it.
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 200)],
|
||||
deleted: [],
|
||||
cursor: 200,
|
||||
});
|
||||
client.filesFor(1, {
|
||||
files: [file(1002, 1, 190)],
|
||||
deleted: [],
|
||||
cursor: 190,
|
||||
});
|
||||
|
||||
// A default read at this instant has not refreshed: it misses 1002.
|
||||
expect(lib.photos.byID({ fileID: 1002 })).toBeUndefined();
|
||||
|
||||
// A fresh read forces the round-trip and sees it.
|
||||
const reads = await lib.fresh();
|
||||
expect(reads.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
|
||||
// And the change is now live for the default namespaces too.
|
||||
expect(lib.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("coalesces concurrent fresh reads onto one in-flight refresh", async () => {
|
||||
const { client, lib } = await openSeeded(cacheDirectory);
|
||||
try {
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 200)],
|
||||
deleted: [],
|
||||
cursor: 200,
|
||||
});
|
||||
client.filesFor(1, {
|
||||
files: [file(1002, 1, 190)],
|
||||
deleted: [],
|
||||
cursor: 190,
|
||||
});
|
||||
|
||||
const collectionsBefore = client.collectionsSinceTimes.length;
|
||||
const filesBefore = client.filesCalls.length;
|
||||
|
||||
// Gate the next collections fetch so all three fresh reads are in
|
||||
// flight together before any of them completes.
|
||||
let release: () => void = () => {};
|
||||
const gate = new Promise<void>((r) => {
|
||||
release = r;
|
||||
});
|
||||
const inner = client.collectionsSince.bind(client);
|
||||
client.collectionsSince = async (args: { sinceTime: number }) => {
|
||||
await gate;
|
||||
return inner(args);
|
||||
};
|
||||
|
||||
const all = Promise.all([lib.fresh(), lib.fresh(), lib.fresh()]);
|
||||
release();
|
||||
const [a, b, c] = await all;
|
||||
|
||||
// Exactly one collections round-trip and one file round-trip served
|
||||
// all three fresh reads.
|
||||
expect(client.collectionsSinceTimes.length).toBe(
|
||||
collectionsBefore + 1,
|
||||
);
|
||||
expect(client.filesCalls.length).toBe(filesBefore + 1);
|
||||
|
||||
// All three observed the change.
|
||||
for (const reads of [a, b, c]) {
|
||||
expect(reads.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
|
||||
}
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("rejects the fresh read when the refresh fails, leaving defaults intact", async () => {
|
||||
const { client, lib } = await openSeeded(cacheDirectory);
|
||||
try {
|
||||
client.failCollections = true;
|
||||
|
||||
// Two concurrent fresh reads both reject, and share one failed
|
||||
// round-trip rather than each making its own.
|
||||
const collectionsBefore = client.collectionsSinceTimes.length;
|
||||
const first = lib.fresh();
|
||||
const second = lib.fresh();
|
||||
await expect(first).rejects.toThrow(/network down/);
|
||||
await expect(second).rejects.toThrow(/network down/);
|
||||
expect(client.collectionsSinceTimes.length).toBe(
|
||||
collectionsBefore + 1,
|
||||
);
|
||||
|
||||
// The default reads never rejected: they still serve the last good
|
||||
// copy, and the failure surfaced through status().
|
||||
expect(lib.photos.byID({ fileID: 1001 })?.fileID).toBe(1001);
|
||||
expect(lib.status().lastError).toMatch(/network down/);
|
||||
|
||||
// Recovery: once the server answers, a fresh read resolves current.
|
||||
client.failCollections = false;
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(2, 300)],
|
||||
deleted: [],
|
||||
cursor: 300,
|
||||
});
|
||||
const reads = await lib.fresh();
|
||||
expect(reads.albums.byID({ collectionID: 2 })?.collectionID).toBe(
|
||||
2,
|
||||
);
|
||||
expect(lib.status().lastError).toBeUndefined();
|
||||
} finally {
|
||||
lib.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user