3 Commits
Author SHA1 Message Date
clawbot fe952d3e62 Rewrite the README API reference for the new library surface (closes #53)
check / check (push) Successful in 28s
Rewrites the README API reference to match the shipped library surface: Library.open options (XDG cache dir + userID, downloadDirectory, refreshIntervalSeconds, precache and cache-size options), default vs fresh reads (Library.fresh()), snapshot/subscribe, albums/photos/timeline, Photo.original/thumbnail, thumbnails.ensure, mldata search, backup, the request pools and on-disk layout, and the CLI (--cache-dir, fresh reads). Every documented signature verified against the code. Notes the deferred content-hash integrity check (#68). Also issue #13.

Model: opus-4-8
2026-09-23 00:27:30 +02:00
clawbot d23d3f8f47 Port the quak CLI to the library API (closes #52)
check / check (push) Successful in 17s
Ports the CLI to the library API. collections/files/get/get-thumb use the fresh read variants (Library.fresh(), current server state); backup runs lib.backup; backup-metadata and the missing-thumbnail helpers enumerate via the library. files/get output is byte-identical to the pre-port CLI — raw metadata.title, microsecond creationTime, pre-port row order — exit codes unchanged. Adds --cache-dir; fixes the helper JPEG-only assumption (closes #17).

Model: opus-4-8
2026-09-23 00:00:55 +02:00
clawbot aeccb489b5 Fresh read variants that await a server round-trip (closes #75)
check / check (push) Successful in 28s
Adds Library.fresh(): forces a refresh, awaits its completion and persist, then returns the albums/photos/timeline read namespaces now reflecting a completed server round trip — the caller awaits and is guaranteed current at resolve. Default reads and the background loop are unchanged (immediate-from-cache). Concurrent fresh reads coalesce to one in-flight refresh; a fresh read whose refresh fails rejects rather than answering stale. The CLI adopts fresh reads separately (#52).

Model: opus-4-8
2026-09-22 23:29:15 +02:00
10 changed files with 1000 additions and 81 deletions
+206 -35
View File
@@ -38,34 +38,47 @@ yarn quak get 67890 --out ./photo.jpg
yarn quak backup ./my-backup
```
For library use:
For library use, the primary surface is the cache-backed `Library`:
```ts
import { Client } from "quak";
import { Client, Library } from "quak";
// Log in once; the client satisfies the library's client interface.
const client = await Client.login({
email: "you@example.com",
password: "your-password",
});
for (const c of await client.listCollections()) {
console.log(c.id, c.name);
const files = await client.listFiles(c.id, c.key);
for (const f of files) {
console.log(` ${f.metadata.title} [${f.metadata.fileType}]`);
// Open a cache-backed library. On an empty cache this awaits one server
// refresh; on an existing cache it returns immediately and refreshes in the
// background every `refreshIntervalSeconds` (default 3).
const lib = await Library.open({ client });
// Default reads answer synchronously from the local cache — no network.
for (const album of lib.albums.list()) {
console.log(album.collectionID, album.name);
for (const photo of album.photos.list()) {
console.log(` ${photo.title} [${photo.fileType}]`);
}
}
// Download a file
const files = await client.listFiles(collectionID, collectionKey);
await client.downloadFile(files[0], "./photo.jpg");
// Fresh reads await a server round-trip and answer with current state.
const { albums } = await lib.fresh();
console.log(`${albums.list().length} albums as of now`);
// Serialize session for later (consumer handles persistence)
const snapshot = client.toJSON();
// ... later:
const restored = Client.fromJSON(snapshot);
// Fetch (and cache) one photo's full-resolution bytes.
const photo = lib.photos.byID({ fileID: 12345 });
if (photo) {
const { path } = await photo.original();
console.log(`original at ${path}`);
}
lib.close();
```
The lower-level `Client` (login, session serialization, and the raw
enumeration/download calls) is exported too and documented under Design below.
## Entrypoints
This repository adheres to the
@@ -429,18 +442,22 @@ quak files --collection <id> [--json] list files in a collection
quak get <fileID> [--out path] [--collection] download and decrypt a file
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
quak backup <dir> [--json] full incremental backup
quak backup-metadata <dir> [--exif] dump all decrypted metadata as JSON
quak helper list-missing-thumbnails [--json] find files with missing thumbnails
quak helper fix-missing-thumbnails [--file ids] generate + upload missing thumbnails
```
Every command reads through the local library cache: the first run fetches the
account's metadata from the server, and later runs serve from the cache and
refresh in the background. `--cache-dir` overrides where that cache lives;
without it each account gets its own directory under the per-user cache path.
Every command runs on the same cache-backed library. The read commands —
`collections`, `files`, `get`, and `get-thumb` — force a fresh server round-trip
before they answer, so they report current account state rather than whatever
the cache last held. `--cache-dir` overrides where the cache lives; without it
each account gets its own directory under the per-user cache path.
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
accepted for backward compatibility but ignored. All listing and backup commands
support `--json` for machine-readable output.
accepted for backward compatibility but ignored. `backup-metadata --exif` (alias
`--all`) additionally downloads each file to extract full EXIF/IPTC/XMP
metadata. The listing and backup commands support `--json` for machine-readable
output.
`helper fix-missing-thumbnails` regenerates thumbnails for baseline JPEG images
only, because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG
@@ -472,7 +489,7 @@ code is non-zero if any files failed.
- [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
errors
- [ ] Update the API reference section below to match the current implementation
- [x] Update the API reference section below to match the current implementation
- [x] `make docker` green
- [ ] Tag `v1.0.0`
@@ -486,25 +503,179 @@ Future (desktop client, separate repo):
## API reference
The API reference section below is from an earlier draft and does not fully
reflect the current implementation. The authoritative API documentation is in
the test files, particularly `test/client/usage.test.ts` which is a literate
tutorial walking through every operation. Run `yarn test` to verify the examples
are correct.
The library's primary surface is the cache-backed `Library`; the lower-level
`Client` sits underneath it and is covered by the Design sections above. The
test suite is the canonical, executable documentation — `test/library/` and
`test/client/usage.test.ts` walk every operation, and `yarn test` verifies them.
The key types and their actual signatures can be found in:
### Opening a library
`Library.open(options)` loads the on-disk cache, starts the background refresh
loop, and resolves to a `Library`. On an empty cache it awaits the first refresh
so it never opens onto empty data; on an existing cache it returns immediately
and refreshes in the background, so an unreachable server does not block
opening.
`LibraryOptions`:
| Option | Default | Meaning |
| ------------------------ | --------------------------- | --------------------------------------------------------------------- |
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
| `cacheDirectory` | `<XDG cache>/quak/<userID>` | where `metadata.json` and the content cache live |
| `downloadDirectory` | none | backup destination; an original already stored there counts as cached |
| `refreshIntervalSeconds` | `3` | background refresh cadence |
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
| `precacheOriginalsDays` | `7` | length in days of that latest window |
| `cacheOriginalsMaxBytes` | 100 GiB | hard ceiling on the originals cache |
| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down |
| `isOriginalPinned` | none | extra predicate for originals that must never be evicted |
| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) |
| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) |
| `contentSource` | the client's own | override the byte source (mainly for tests) |
Concurrency is set through `pools`: construct
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).
`lib.status()` returns a `LibraryStatus` (collection/file counts, last
refresh/ML times and errors, originals usage and effective limit, precache
progress, and `closed`). `lib.close()` stops the background timer; it is
idempotent, and an in-flight refresh is left to finish.
### Default reads vs. fresh reads
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
synchronously from the last refreshed copy held in RAM and never touch the
network. The background timer refreshes that copy every
`refreshIntervalSeconds`, so a default read is immediate but may be up to one
interval stale.
`await lib.fresh()` forces a refresh, waits for it to complete and persist, and
returns the same `{ albums, photos, timeline }` namespaces — now guaranteed to
reflect a completed server round-trip. Concurrent `fresh()` calls coalesce onto
one refresh, and a refresh that fails rejects the caller (default reads stay
silent and keep serving the last good copy). The CLI's read commands use fresh
reads (issue https://git.eeqj.de/sneak/quak/issues/75).
### Read surface
- `lib.albums.list()``Album[]`, newest-updated first.
`lib.albums.byID({ collectionID })` and `byName({ albumName })`
`Album | undefined`.
- `lib.photos.byID({ fileID })``Photo | undefined`.
`lib.photos.records({ fileIDs })``PhotoRecord[]` in the requested order,
each id once, unknown ids dropped.
- `lib.timeline.groups({ groupBy, filter? })``TimelineGroup[]`, grouped by
`"day" | "week" | "month"` (keys `YYYY-MM-DD`, ISO `YYYY-Www`, `YYYY-MM`),
newest group first. A `PhotoFilter` combines `albumID`, `text`
(title/caption/album-name substring), `fileTypes`, `hasLocation`, and
`includeArchived`; hidden photos are always excluded.
An `Album` exposes its record fields and `album.photos.list()``Photo[]`
(newest first). A `Photo` exposes its record fields, `photo.record()`
`PhotoRecord`, and two content methods:
- `await photo.original(opts?)``{ path, bytes }` — the full-resolution file.
- `await photo.thumbnail(opts?)``{ path, bytes }`.
Both serve from the on-disk content cache when the bytes are present and
otherwise fetch through the pools; `opts.onProgress` reports per-file progress.
They throw when the library was opened without a content source.
Lower-level accessors that return decrypted model objects (which hold key
material) are also available: `listCollections()`, `getCollection(id)`,
`listFiles(collectionID)`, `getFile(collectionID, fileID)`, and
`getFileByID(fileID)`.
### Records and change notifications
The GUI-facing records hold no key material and no binary, so they survive
`structuredClone`/JSON across the Electron IPC boundary:
- `PhotoRecord`: `fileID`, `albumIDs`, `title`, `takenAt` (milliseconds),
`fileType`, optional `caption` / `width` / `height` / `latitude` /
`longitude`, `isArchived`, `isHidden`, and `thumbnailPath` / `originalPath`
once the bytes are cached.
- `AlbumRecord`: `collectionID`, `name`, `type`, `isShared`, `updationTime`, and
`fileIDs` (newest first).
- `LibrarySnapshot`: `{ albums, photos, takenAt }`.
`lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first,
photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
(`albumsChanged`, `photosChanged`, `fileIDsRemoved`, `albumIDsRemoved`,
`refreshedAt`) whenever a refresh alters the projection, and returns
`{ unsubscribe }`; a refresh that changes nothing delivers nothing.
### Thumbnails, ML search, and backup
- `lib.thumbnails.ensure({ fileIDs, priority, signal?, onProgress? })`
prefetches thumbnails through the thumbnail pool, deduped by fileID, returning
one `EnsureResult` (`{ fileID, path?, error? }`) per file. `priority` is
`"visible" | "ahead" | "background"`; only `"visible"` preempts background
work.
- `lib.mldata` searches the CLIP index built from Ente's per-file ML data:
`forFile({ fileID })``Promise<MLData | undefined>` (the whole stored
payload — face boxes, landmarks, embedding — read from disk on demand);
`similar({ fileID, limit? })` and `searchByEmbedding({ embedding, limit? })`
`SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
a query vector the caller produced elsewhere.
- `await lib.backup(opts?)``BackupResult`. It refreshes, fetches every
in-scope original (and, with `includeThumbnails`, thumbnails) through the
content cache, and rebuilds the on-disk backup tree with a durable failure
ledger. `BackupOptions`: `downloadDirectory` (falls back to the one `open()`
was given), `includeOriginals` (default `true`), `includeThumbnails` (default
`false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above for the
tree it writes.
### Request pools
`RequestPools` holds three independent bounded pools — metadata (10), content
(5), thumbnails (25) — because Ente meters these traffic classes differently.
Each pool orders on-demand work ahead of background/precache work and dedups
in-flight fetches by key, and an idle pool never lends its slots to a busy one.
### On-disk cache layout
Under `cacheDirectory`:
```
<cacheDirectory>/
metadata.json decrypted account state + refresh cursor
originals/<fileID>.<ext> cached full-resolution files
thumbnails/<fileID>.jpg cached thumbnails
mldata/
<fileID>.json one decrypted ML payload per file
clip.f32, clip.json the packed CLIP index and its id list
fetched.json per-file fetch bookkeeping
```
A stored file appears only via an atomic temp-then-rename, so its presence means
it is complete. The design also calls for a content-hash comparison against
`FileMetadata.hash` on each fetched original; that check is deferred (issue
https://git.eeqj.de/sneak/quak/issues/68) because the exact hash construction
cannot yet be confirmed against the repo's fixtures.
### Key types by source file
- `src/library/index.ts`: `Library`, `LibraryOptions`, `LibraryStatus`,
`LibraryClient`, `RefreshEvent`
- `src/library/read.ts`: `Album`, `Photo`, `AlbumsAPI`, `PhotosAPI`,
`TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`
- `src/library/content.ts`: `ContentResult`, `ContentOptions`, `ThumbnailsAPI`,
`EnsureOptions`, `EnsureResult`, `ContentSource`
- `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`,
`LibraryChange`
- `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult`
- `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool`
- `src/backup.ts`: `BackupOptions`, `BackupResult`, `BackupError`
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `ApiError`,
`StreamOptions`
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `StreamOptions`
- `src/errors.ts`: `ApiError`, `TruncatedStreamError`
- `src/retry.ts`: `withRetry`, `isRetryable`, `isSafeToReplay`, `RetryOptions`
- `src/auth/types.ts`: `KeyAttributes`, `SRPAttributes`,
`AuthorizationResponse`, `LoginChallenge`
- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileBlob`,
`RawCollection`, `RawEnteFile`, `RawMagicMetadata`
- `src/download/index.ts`: `DownloadResult`
- `src/backup.ts`: `BackupResult`, `BackupError`
- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileType`,
`CollectionType`, `RawCollection`, `RawEnteFile`
- `src/thumbnails.ts`: `MissingThumbnailInfo`, `ThumbnailFixResult`
## Source attribution
+10 -2
View File
@@ -14,10 +14,19 @@ pre-1.0
# Next Step
Update the README API reference section to match the current implementation.
Tag v1.0.0.
# Completed Steps
- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
snippets) to match the shipped cache/API library on `next` (issue 53, issue
13). Documented `Library.open` and its options, the default-read vs `fresh()`
distinction (and that the CLI's read commands are fresh), the record types and
`snapshot()`/`subscribe()`, the `albums`/`photos`/`timeline` read surface,
`Photo` content methods, `thumbnails.ensure`, the `mldata` search surface,
`backup()`, the three request pools (10/5/25), and the on-disk cache layout;
noted the deferred content-hash integrity check (issue 68). Docs-only; no code
changed.
- 2026-09-22: Added resumable, deletion-aware enumeration to `Client` (issue 38,
closes issue 7). `collectionsSince`/`filesSince` take a starting cursor,
decrypt live records, surface tombstoned ids in a separate `deleted` list (a
@@ -122,7 +131,6 @@ Update the README API reference section to match the current implementation.
# Future Steps
- Tag v1.0.0.
- Future desktop client, separate repo:
- Electron app skeleton consuming this library.
- Local SQLite cache keyed on (collectionID, fileID, updationTime).
+46 -32
View File
@@ -15,6 +15,13 @@ import envPaths from "env-paths";
import { Client, type ClientSnapshot } from "../src/client.js";
import { init } from "../src/crypto/index.js";
import { Library, type LibraryClient } from "../src/library/index.js";
import {
fileListRow,
fileListLine,
originalName,
thumbnailName,
} from "../src/cli-output.js";
import { freshCollections, freshFiles, freshFile } from "../src/cli-read.js";
import { runMetadataBackup } from "../src/metadata-backup.js";
import {
listMissingThumbnails,
@@ -177,27 +184,30 @@ program
await init();
const client = requireSession();
const lib = await openReadLibrary(client);
const albums = lib.albums.list();
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52): the pre-library CLI printed current state in
// this order, not the albums projection's newest-first order.
const collections = await freshCollections(lib);
if (opts.json) {
stdout.write(
JSON.stringify(
albums.map((a) => ({
id: a.collectionID,
name: a.name,
type: a.type,
ownerID: lib.getCollection(a.collectionID)?.ownerID,
isShared: a.isShared,
updationTime: a.updationTime,
collections.map((c) => ({
id: c.id,
name: c.name,
type: c.type,
ownerID: c.ownerID,
isShared: c.isShared,
updationTime: c.updationTime,
})),
null,
2,
) + "\n",
);
} else {
for (const a of albums) {
for (const c of collections) {
stdout.write(
`${a.collectionID}\t${a.type}\t${a.name}${a.isShared ? " (shared)" : ""}\n`,
`${c.id}\t${c.type}\t${c.name}${c.isShared ? " (shared)" : ""}\n`,
);
}
}
@@ -222,31 +232,25 @@ program
}
const lib = await openReadLibrary(client);
const album = lib.albums.byID({ collectionID });
if (!album) {
// Force a server round-trip and list in enumeration order (issue #36
// amendment, issue #52). Each file prints from its own decrypted
// metadata (raw title, microsecond creationTime) via cli-output, and in
// the pre-library CLI's enumeration order, not the projection's
// newest-first order.
const files = await freshFiles(lib, collectionID);
if (!files) {
stderr.write(`Collection ${collectionID} not found\n`);
finish(lib, 1);
return;
}
const photos = album.photos.list();
if (opts.json) {
stdout.write(
JSON.stringify(
photos.map((p) => ({
id: p.fileID,
title: p.title,
fileType: p.fileType,
creationTime: p.takenAt,
collectionID,
})),
null,
2,
) + "\n",
JSON.stringify(files.map(fileListRow), null, 2) + "\n",
);
} else {
for (const p of photos) {
stdout.write(`${p.fileID}\t${p.fileType}\t${p.title}\n`);
for (const file of files) {
stdout.write(fileListLine(file) + "\n");
}
}
finish(lib, 0);
@@ -268,15 +272,20 @@ program
}
const lib = await openReadLibrary(client);
const photo = lib.photos.byID({ fileID });
if (!photo) {
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
stderr.write(`File ${fileID} not found\n`);
finish(lib, 1);
return;
}
const { photo, file } = resolved;
const result = await photo.original();
const outPath = opts.out ?? photo.title;
// Default name is the file's own title, as the pre-library CLI used
// (not the editedName-preferring projection title) (issue #52).
const outPath = opts.out ?? originalName(file);
copyFileSync(result.path, outPath);
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
finish(lib, 0);
@@ -298,15 +307,20 @@ program
}
const lib = await openReadLibrary(client);
const photo = lib.photos.byID({ fileID });
if (!photo) {
// Force a server round-trip so the file resolves against current state
// (issue #36 amendment, issue #52).
const resolved = await freshFile(lib, fileID);
if (!resolved) {
stderr.write(`File ${fileID} not found\n`);
finish(lib, 1);
return;
}
const { photo, file } = resolved;
const result = await photo.thumbnail();
const outPath = opts.out ?? `thumb_${photo.title}`;
// Default name is thumb_<file's own title>, as the pre-library CLI
// used (not the projection title) (issue #52).
const outPath = opts.out ?? thumbnailName(file);
copyFileSync(result.path, outPath);
stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
finish(lib, 0);
+40
View File
@@ -0,0 +1,40 @@
// How the CLI presents a file's identity in `files`, `get`, and `get-thumb`.
//
// These read the file's own decrypted metadata — the raw title and the
// creationTime in microseconds — rather than the `PhotoRecord` projection the
// rest of the library exposes. The projection prefers `editedName`/`editedTime`
// and reports time in milliseconds, which is right for a photo browser but
// would change the CLI's externally-visible output. The pre-library CLI printed
// `metadata.title` and `metadata.creationTime` and named downloads after
// `metadata.title`, and issue #52 requires that output stay byte-identical, so
// the commands shape their output from the raw `EnteFile` through here.
import type { EnteFile, FileType, Microseconds } from "./model/types.js";
// One row of `quak files --json`.
export interface FileListRow {
id: number;
title: string;
fileType: FileType;
creationTime: Microseconds;
collectionID: number;
}
export const fileListRow = (file: EnteFile): FileListRow => ({
id: file.id,
title: file.metadata.title,
fileType: file.metadata.fileType,
creationTime: file.metadata.creationTime,
collectionID: file.collectionID,
});
// One line of `quak files` in its human, tab-separated form.
export const fileListLine = (file: EnteFile): string =>
`${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
// Default output path for `quak get` when `--out` is not given.
export const originalName = (file: EnteFile): string => file.metadata.title;
// Default output path for `quak get-thumb` when `--out` is not given.
export const thumbnailName = (file: EnteFile): string =>
`thumb_${file.metadata.title}`;
+62
View File
@@ -0,0 +1,62 @@
// How the CLI's read commands obtain current data.
//
// `collections`, `files --collection`, `get`, and `get-thumb` must answer for
// the account's state at the moment the command runs, not for whatever the
// local cache last happened to hold (owner amendment, issue #36). Each helper
// therefore forces a server round-trip through `Library.fresh()` and only then
// reads — so a collection, file, or metadata change made elsewhere is visible.
//
// `collections` and `files` also list in the library's own enumeration order —
// `listCollections()`/`listFiles()`, the order the pre-library CLI printed —
// rather than the `albums`/`photos` projection's newest-first order, which
// re-sorts the rows. The field values still come from each record's raw
// metadata via `cli-output.ts`.
import type { Collection, EnteFile } from "./model/types.js";
import type { Photo, PhotosAPI } from "./library/index.js";
// The slice of `Library` these helpers read. `Library` satisfies it
// structurally; a test can drive them with a stand-in that records the
// `fresh()` call and serves records in a known enumeration order.
export interface FreshReadLibrary {
fresh(): Promise<unknown>;
listCollections(): Collection[];
getCollection(id: number): Collection | undefined;
listFiles(collectionID: number): EnteFile[];
getFileByID(fileID: number): EnteFile | undefined;
photos: Pick<PhotosAPI, "byID">;
}
// Every live collection, current as of a forced refresh, in enumeration order.
export const freshCollections = async (
lib: FreshReadLibrary,
): Promise<Collection[]> => {
await lib.fresh();
return lib.listCollections();
};
// The files of one collection, current as of a forced refresh, in enumeration
// order. `undefined` (not an empty list) when the collection does not exist, so
// the caller can tell "no such collection" from "an empty collection".
export const freshFiles = async (
lib: FreshReadLibrary,
collectionID: number,
): Promise<EnteFile[] | undefined> => {
await lib.fresh();
if (!lib.getCollection(collectionID)) return undefined;
return lib.listFiles(collectionID);
};
// One file, current as of a forced refresh, resolved to both its content
// handle (`Photo`, for fetching bytes) and its raw record (`EnteFile`, for the
// default output name and field values). `undefined` when the file is unknown.
export const freshFile = async (
lib: FreshReadLibrary,
fileID: number,
): Promise<{ photo: Photo; file: EnteFile } | undefined> => {
await lib.fresh();
const photo = lib.photos.byID({ fileID });
const file = lib.getFileByID(fileID);
if (!photo || !file) return undefined;
return { photo, file };
};
+84 -12
View File
@@ -6,10 +6,17 @@
// existing copy loaded, that first refresh runs in the background and `open()`
// returns as soon as the cached data is ready to serve — a slow or unreachable
// server no longer stalls opening. A background timer then refreshes every
// `refreshIntervalSeconds`. Every read is answered from RAM — no read touches
// the network. There is deliberately no `sync()`, no `refresh()`, no
// `serverReachable` flag, and no "before each read" mode (design #36): the
// only ways state changes are the refreshes above.
// `refreshIntervalSeconds`. Every default read is answered from RAM — no
// default read touches the network. There is deliberately no `sync()`, no
// `refresh()`, no `serverReachable` flag, and no "before each read" mode
// (design #36).
//
// `fresh()` is the one exception (issue #75, an owner amendment to #36): it
// forces a refresh, awaits it, and only then hands back the read namespaces, so
// a caller that needs server-current data can ask for it. Concurrent `fresh()`
// calls coalesce onto one in-flight refresh, and a refresh that fails rejects
// the caller (the default reads stay silent and serve the last good copy). The
// default methods and the background loop are unchanged.
//
// A refresh stages all of its network work first and only mutates the store
// once every fetch has succeeded. A refresh that fails partway therefore never
@@ -40,6 +47,7 @@ import {
type AlbumsAPI,
type PhotosAPI,
type TimelineAPI,
type FreshReads,
} from "./read.js";
import {
ContentCache,
@@ -57,6 +65,7 @@ export {
type AlbumsAPI,
type PhotosAPI,
type TimelineAPI,
type FreshReads,
type PhotoFilter,
type TimelineGroup,
type GroupBy,
@@ -250,7 +259,12 @@ export class Library {
private readonly precache?: Precache;
private timer?: ReturnType<typeof setTimeout>;
private refreshing = false;
// The in-flight refresh cycle, or undefined when none runs. One slot serves
// both paths: the background loop skips when it is set, and a fresh read
// (issue #75) coalesces onto it or starts one. The promise carries the
// cycle's real outcome (it rejects on failure); the background loop ignores
// that, a fresh read propagates it.
private cycle?: Promise<void>;
// Guards the ML fetch pass so a slow backfill never runs twice at once; a
// refresh whose pass is still running kicks nothing new.
private mlFetching = false;
@@ -433,6 +447,13 @@ export class Library {
return this.store.getFile(collectionID, fileID);
}
// Any membership of a file, addressed by file id alone. A file's own
// metadata (title, creationTime) is identical across the collections it
// belongs to, so this serves the point commands that hold only a fileID.
getFileByID(fileID: number): EnteFile | undefined {
return this.store.getFileByID(fileID);
}
// A synchronous, RAM-only projection of the whole library into plain
// records (no keys), the surface the GUI reads across IPC. Photos are
// deduplicated to one record per file and ordered newest first.
@@ -484,6 +505,21 @@ export class Library {
};
}
// Fresh reads (issue #75, owner amendment to design #36). Force a refresh,
// wait for it to complete and persist, then hand back the same
// `albums`/`photos`/`timeline` namespaces — now guaranteed to reflect a
// completed server round-trip. Concurrent calls coalesce onto one refresh;
// a refresh that fails rejects here, where the default namespaces would
// instead stay silent and serve the last good copy.
async fresh(): Promise<FreshReads> {
await this.refreshNow();
return {
albums: this.albums,
photos: this.photos,
timeline: this.timeline,
};
}
// Back up every in-scope file to `downloadDirectory` in the historical
// on-disk layout, with a durable failure ledger (issue #51). Refreshes
// first, fetches pending originals (and optional thumbnails) through the
@@ -543,11 +579,48 @@ export class Library {
this.timer.unref?.();
}
// One refresh cycle, guarded so a failure never escapes and overlapping
// cycles never run. Errors are reported, not thrown.
private async runRefresh(): Promise<void> {
if (this.closed || this.refreshing) return;
this.refreshing = true;
// The background loop's refresh: run a cycle unless one is already in flight
// (or the library is closed), and never let a failure escape — the
// background path reports errors through `status()`/`onProgress`, it does
// not throw. Resolves once the cycle it started (or skipped past) settles.
private runRefresh(): Promise<void> {
if (this.closed || this.cycle) return Promise.resolve();
return this.startCycle().catch(() => {});
}
// A fresh read's refresh (issue #75): force a cycle and await it, rejecting
// if it fails. Concurrent fresh reads coalesce onto the one in-flight cycle
// — the background loop's included — so they never fan out into redundant
// server round-trips.
private refreshNow(): Promise<void> {
if (this.closed) {
return Promise.reject(new Error("the library is closed"));
}
return this.cycle ?? this.startCycle();
}
// Start one refresh cycle and record it as the in-flight cycle so every
// caller coalesces onto it. The returned promise carries the cycle's real
// outcome; each caller attaches the handling its own path needs, and the
// slot is cleared once the cycle settles.
private startCycle(): Promise<void> {
const cycle = this.refreshCycle();
this.cycle = cycle;
void cycle.then(
() => {
if (this.cycle === cycle) this.cycle = undefined;
},
() => {
if (this.cycle === cycle) this.cycle = undefined;
},
);
return cycle;
}
// One refresh cycle: the network fetch and commit, wrapped in the progress
// events and status bookkeeping. Throws when the refresh fails so a fresh
// read can reject; `runRefresh` swallows that throw for the background loop.
private async refreshCycle(): Promise<void> {
this.emit({ operation: "refresh", status: "started" });
try {
await this.refreshOnce();
@@ -563,8 +636,7 @@ export class Library {
const error = err instanceof Error ? err.message : String(err);
this.lastError = error;
this.emit({ operation: "refresh", status: "failed", error });
} finally {
this.refreshing = false;
throw err;
}
}
+9
View File
@@ -193,6 +193,15 @@ export interface TimelineAPI {
groups(args: { groupBy: GroupBy; filter?: PhotoFilter }): TimelineGroup[];
}
// The surface `Library.fresh()` resolves to (issue #75). It is the same three
// read namespaces as the default `albums`/`photos`/`timeline`, handed back only
// after a forced refresh has brought the local copy current.
export interface FreshReads {
albums: AlbumsAPI;
photos: PhotosAPI;
timeline: TimelineAPI;
}
export const makeAlbumsAPI = (
derive: () => DerivedRecords,
content?: PhotoContent,
+76
View File
@@ -0,0 +1,76 @@
// The CLI presents a file by its own decrypted metadata, not the PhotoRecord
// projection (issue #52). For a renamed file the two disagree: the projection
// prefers `editedName` and reports `editedTime` in milliseconds, while the CLI
// must print the raw `metadata.title` and `metadata.creationTime` (microseconds)
// and name downloads after the raw title, byte-identical to the pre-library CLI.
//
// This locks in that contrast: the shared output helpers emit the raw values,
// and the projection of the same file emits the edited ones — so a regression
// that re-sourced the CLI from the projection would fail here.
import { describe, it, expect } from "vitest";
import {
fileListRow,
fileListLine,
originalName,
thumbnailName,
} from "../../src/cli-output.js";
import { deriveRecords } from "../../src/library/records.js";
import type { EnteFile } from "../../src/model/types.js";
// Microseconds, as Ente stores times.
const RAW_CREATION = 1700000000000000;
const EDITED_TIME = 1710000000000000;
const RAW_TITLE = "IMG_0001.HEIC";
const EDITED_NAME = "Sunset.heic";
// A file the user has renamed and re-dated: basic metadata holds the original
// title and capture time; public magic metadata holds the edits.
const renamedFile: EnteFile = {
id: 100,
collectionID: 10,
ownerID: 42,
key: new Uint8Array(),
metadata: {
title: RAW_TITLE,
fileType: "image",
creationTime: RAW_CREATION,
modificationTime: RAW_CREATION,
},
pubMagicMetadata: { editedName: EDITED_NAME, editedTime: EDITED_TIME },
file: { decryptionHeader: "" },
thumbnail: { decryptionHeader: "" },
updationTime: RAW_CREATION,
};
describe("CLI file output (issue #52)", () => {
it("emits the raw title and microsecond creationTime for --json", () => {
expect(fileListRow(renamedFile)).toEqual({
id: 100,
title: RAW_TITLE,
fileType: "image",
creationTime: RAW_CREATION,
collectionID: 10,
});
});
it("emits the raw title in the human column", () => {
expect(fileListLine(renamedFile)).toBe(`100\timage\t${RAW_TITLE}`);
});
it("names downloads after the raw title", () => {
expect(originalName(renamedFile)).toBe(RAW_TITLE);
expect(thumbnailName(renamedFile)).toBe(`thumb_${RAW_TITLE}`);
});
it("does not use the editedName/editedTime projection", () => {
const record = deriveRecords([], [renamedFile]).photos.get(100);
// The projection prefers the edits and reports milliseconds; the CLI
// helpers above deliberately do not.
expect(record?.title).toBe(EDITED_NAME);
expect(record?.takenAt).toBe(Math.floor(EDITED_TIME / 1000));
expect(fileListRow(renamedFile).title).not.toBe(record?.title);
expect(fileListRow(renamedFile).creationTime).not.toBe(record?.takenAt);
});
});
+195
View File
@@ -0,0 +1,195 @@
/**
* Tests for the CLI read helpers (`src/cli-read.ts`, owner amendment to
* issue #36, issue #52).
*
* The `collections`, `files`, `get`, and `get-thumb` commands must answer for
* current server state, not the local cache, so each helper forces a
* `Library.fresh()` round-trip before it reads. The stand-in library below
* serves nothing until `fresh()` has been awaited, so a helper that read
* without refreshing would come back empty and fail here.
*
* `collections` and `files` also list in the library's enumeration order
* (`listCollections`/`listFiles`) — the order the pre-library CLI printed — not
* the albums/photos projection's newest-first order. The fixtures are seeded in
* an enumeration order that a newest-first sort would rearrange, so a
* regression to the projection order would fail here too. Field values still
* come from the raw metadata via `cli-output.ts`.
*/
import { describe, it, expect } from "vitest";
import {
freshCollections,
freshFiles,
freshFile,
type FreshReadLibrary,
} from "../../src/cli-read.js";
import { fileListRow } from "../../src/cli-output.js";
import type { Photo } from "../../src/library/index.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
const collection = (id: number, updationTime: number): Collection => ({
id,
ownerID: 42,
key: new Uint8Array(),
name: `album-${id}`,
type: "album",
updationTime,
isShared: false,
});
// Microseconds, as Ente stores times.
const file = (
id: number,
collectionID: number,
creationTime: number,
): EnteFile => ({
id,
collectionID,
ownerID: 42,
key: new Uint8Array(),
metadata: {
title: `file-${id}.jpg`,
fileType: "image",
creationTime,
modificationTime: creationTime,
},
file: { decryptionHeader: "" },
thumbnail: { decryptionHeader: "" },
updationTime: creationTime,
});
// A library that reveals its records only after `fresh()` has been awaited, and
// serves them in the enumeration order it was given. `photos.byID` returns a
// stand-in `Photo` carrying just the fileID the helper passes through.
class FakeLibrary implements FreshReadLibrary {
freshCalls = 0;
private refreshed = false;
constructor(
private readonly collections: Collection[],
private readonly files: EnteFile[],
) {}
async fresh(): Promise<unknown> {
this.freshCalls++;
this.refreshed = true;
return {};
}
listCollections(): Collection[] {
return this.refreshed ? this.collections : [];
}
getCollection(id: number): Collection | undefined {
return this.listCollections().find((c) => c.id === id);
}
listFiles(collectionID: number): EnteFile[] {
return this.refreshed
? this.files.filter((f) => f.collectionID === collectionID)
: [];
}
getFileByID(fileID: number): EnteFile | undefined {
if (!this.refreshed) return undefined;
return this.files.find((f) => f.id === fileID);
}
photos = {
byID: ({ fileID }: { fileID: number }): Photo | undefined => {
if (!this.refreshed) return undefined;
if (!this.files.some((f) => f.id === fileID)) return undefined;
return { fileID } as unknown as Photo;
},
};
}
describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
it("freshCollections refreshes first, then lists in enumeration order", async () => {
// Enumeration order 2, 1, 3; a newest-first sort would be 3, 2, 1.
const lib = new FakeLibrary(
[collection(2, 200), collection(1, 300), collection(3, 100)],
[],
);
const rows = await freshCollections(lib);
expect(lib.freshCalls).toBe(1);
expect(rows.map((c) => c.id)).toEqual([2, 1, 3]);
// The projection's newest-first order is a different sequence, so this
// is not accidentally that order.
const newestFirst = [...rows]
.sort((a, b) => b.updationTime - a.updationTime)
.map((c) => c.id);
expect(newestFirst).toEqual([1, 2, 3]);
expect(rows.map((c) => c.id)).not.toEqual(newestFirst);
});
it("freshFiles refreshes first, lists in enumeration order, keeps raw fields", async () => {
// Enumeration order by id 10, 11, 12; creationTimes ascending, so a
// newest-first sort would reverse them.
const files = [
file(10, 1, 1_700_000_000_000_000),
file(11, 1, 1_700_000_000_000_001),
file(12, 1, 1_700_000_000_000_002),
];
const lib = new FakeLibrary([collection(1, 100)], files);
const rows = await freshFiles(lib, 1);
expect(lib.freshCalls).toBe(1);
expect(rows?.map((f) => f.id)).toEqual([10, 11, 12]);
// Field values come from raw metadata: microsecond creationTime and the
// raw title, unchanged.
expect(rows?.map(fileListRow)).toEqual([
{
id: 10,
title: "file-10.jpg",
fileType: "image",
creationTime: 1_700_000_000_000_000,
collectionID: 1,
},
{
id: 11,
title: "file-11.jpg",
fileType: "image",
creationTime: 1_700_000_000_000_001,
collectionID: 1,
},
{
id: 12,
title: "file-12.jpg",
fileType: "image",
creationTime: 1_700_000_000_000_002,
collectionID: 1,
},
]);
});
it("freshFiles returns undefined for an unknown collection", async () => {
const lib = new FakeLibrary([collection(1, 100)], []);
const rows = await freshFiles(lib, 999);
expect(lib.freshCalls).toBe(1);
expect(rows).toBeUndefined();
});
it("freshFile refreshes first, then resolves the photo and its raw record", async () => {
const f = file(10, 1, 1_700_000_000_000_000);
const lib = new FakeLibrary([collection(1, 100)], [f]);
const resolved = await freshFile(lib, 10);
expect(lib.freshCalls).toBe(1);
expect(resolved?.photo.fileID).toBe(10);
expect(resolved?.file.metadata.title).toBe("file-10.jpg");
expect(resolved?.file.metadata.creationTime).toBe(
1_700_000_000_000_000,
);
});
it("freshFile returns undefined for an unknown file", async () => {
const lib = new FakeLibrary([collection(1, 100)], []);
const resolved = await freshFile(lib, 404);
expect(lib.freshCalls).toBe(1);
expect(resolved).toBeUndefined();
});
});
+272
View File
@@ -0,0 +1,272 @@
/**
* Tests for fresh reads (issue #75, an owner amendment to design #36).
*
* The default read namespaces answer from RAM and never touch the network; a
* background loop keeps the local copy current. `fresh()` adds an awaited path:
* it forces a refresh, waits for it to complete and persist, and only then
* hands back the `albums`/`photos`/`timeline` namespaces, guaranteeing the
* local copy reflects a completed server round-trip. The contracts here:
*
* 1. A fresh read observes a server change that a same-instant default read
* would miss (the default path has not refreshed yet).
* 2. Concurrent fresh reads coalesce onto one in-flight refresh — three
* concurrent `fresh()` calls make exactly one collections round-trip, not
* three.
* 3. A refresh that fails rejects the fresh read (currency was unavailable),
* while the default reads stay silent and keep serving the last good copy.
*
* The client is the same metadata-only mock the background-refresh tests use:
* no crypto, no network, scripted pages, and a record of each call. A long
* refresh interval keeps the background timer out of the way so each test's
* refreshes are exactly the ones it triggers.
*/
import { describe, it, expect, beforeEach, afterEach } from "vitest";
import { mkdtempSync, rmSync } from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { Library } from "../../src/library/index.js";
import type { CollectionsPage, FilesPage } from "../../src/client.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
const USER_ID = 42;
// Long enough that no background tick fires during a test; each test's
// refreshes are only the ones its own `fresh()` calls force.
const SLOW_INTERVAL = 3600;
const collection = (id: number, updationTime: number): Collection => ({
id,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
name: `album-${id}`,
type: "album",
updationTime,
isShared: false,
});
const file = (
id: number,
collectionID: number,
updationTime: number,
): EnteFile => ({
id,
collectionID,
ownerID: USER_ID,
key: new Uint8Array([id & 0xff]),
metadata: {
title: `file-${id}.jpg`,
fileType: "image",
creationTime: updationTime,
modificationTime: updationTime,
},
file: { decryptionHeader: "aGVhZGVy" },
thumbnail: { decryptionHeader: "dGh1bWI=" },
updationTime,
});
class MockClient {
userID = USER_ID;
failCollections = false;
collectionsQueue: CollectionsPage[] = [];
filesByCollection = new Map<number, FilesPage[]>();
collectionsSinceTimes: number[] = [];
filesCalls: { collectionID: number; sinceTime: number }[] = [];
whoami(): { email: string; userID: number } {
return { email: "user@example.com", userID: this.userID };
}
async collectionsSince(args: {
sinceTime: number;
}): Promise<CollectionsPage> {
this.collectionsSinceTimes.push(args.sinceTime);
if (this.failCollections) throw new Error("network down");
return (
this.collectionsQueue.shift() ?? {
collections: [],
deleted: [],
cursor: args.sinceTime,
}
);
}
async filesSince(args: {
collectionID: number;
collectionKey: Uint8Array;
sinceTime: number;
}): Promise<FilesPage> {
this.filesCalls.push({
collectionID: args.collectionID,
sinceTime: args.sinceTime,
});
const queue = this.filesByCollection.get(args.collectionID);
return (
queue?.shift() ?? {
files: [],
deleted: [],
cursor: args.sinceTime,
}
);
}
filesFor(collectionID: number, ...pages: FilesPage[]): void {
this.filesByCollection.set(collectionID, pages);
}
}
// A client seeded with one collection and one file, opened with an empty cache
// so the initial refresh is awaited and post-`open()` state is deterministic.
const openSeeded = async (
cacheDirectory: string,
): Promise<{ client: MockClient; lib: Library }> => {
const client = new MockClient();
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
client.filesFor(1, {
files: [file(1001, 1, 90)],
deleted: [],
cursor: 90,
});
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: SLOW_INTERVAL,
});
return { client, lib };
};
describe("Library.fresh", () => {
let dir: string;
let cacheDirectory: string;
beforeEach(() => {
dir = mkdtempSync(join(tmpdir(), "quak-fresh-"));
cacheDirectory = join(dir, "cache");
});
afterEach(() => {
rmSync(dir, { recursive: true, force: true });
});
it("observes a server change a same-instant default read would miss", async () => {
const { client, lib } = await openSeeded(cacheDirectory);
try {
// A new file appears on the server after open, advancing its
// collection so the next refresh re-enumerates it.
client.collectionsQueue.push({
collections: [collection(1, 200)],
deleted: [],
cursor: 200,
});
client.filesFor(1, {
files: [file(1002, 1, 190)],
deleted: [],
cursor: 190,
});
// A default read at this instant has not refreshed: it misses 1002.
expect(lib.photos.byID({ fileID: 1002 })).toBeUndefined();
// A fresh read forces the round-trip and sees it.
const reads = await lib.fresh();
expect(reads.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
// And the change is now live for the default namespaces too.
expect(lib.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
} finally {
lib.close();
}
});
it("coalesces concurrent fresh reads onto one in-flight refresh", async () => {
const { client, lib } = await openSeeded(cacheDirectory);
try {
client.collectionsQueue.push({
collections: [collection(1, 200)],
deleted: [],
cursor: 200,
});
client.filesFor(1, {
files: [file(1002, 1, 190)],
deleted: [],
cursor: 190,
});
const collectionsBefore = client.collectionsSinceTimes.length;
const filesBefore = client.filesCalls.length;
// Gate the next collections fetch so all three fresh reads are in
// flight together before any of them completes.
let release: () => void = () => {};
const gate = new Promise<void>((r) => {
release = r;
});
const inner = client.collectionsSince.bind(client);
client.collectionsSince = async (args: { sinceTime: number }) => {
await gate;
return inner(args);
};
const all = Promise.all([lib.fresh(), lib.fresh(), lib.fresh()]);
release();
const [a, b, c] = await all;
// Exactly one collections round-trip and one file round-trip served
// all three fresh reads.
expect(client.collectionsSinceTimes.length).toBe(
collectionsBefore + 1,
);
expect(client.filesCalls.length).toBe(filesBefore + 1);
// All three observed the change.
for (const reads of [a, b, c]) {
expect(reads.photos.byID({ fileID: 1002 })?.fileID).toBe(1002);
}
} finally {
lib.close();
}
});
it("rejects the fresh read when the refresh fails, leaving defaults intact", async () => {
const { client, lib } = await openSeeded(cacheDirectory);
try {
client.failCollections = true;
// Two concurrent fresh reads both reject, and share one failed
// round-trip rather than each making its own.
const collectionsBefore = client.collectionsSinceTimes.length;
const first = lib.fresh();
const second = lib.fresh();
await expect(first).rejects.toThrow(/network down/);
await expect(second).rejects.toThrow(/network down/);
expect(client.collectionsSinceTimes.length).toBe(
collectionsBefore + 1,
);
// The default reads never rejected: they still serve the last good
// copy, and the failure surfaced through status().
expect(lib.photos.byID({ fileID: 1001 })?.fileID).toBe(1001);
expect(lib.status().lastError).toMatch(/network down/);
// Recovery: once the server answers, a fresh read resolves current.
client.failCollections = false;
client.collectionsQueue.push({
collections: [collection(2, 300)],
deleted: [],
cursor: 300,
});
const reads = await lib.fresh();
expect(reads.albums.byID({ collectionID: 2 })?.collectionID).toBe(
2,
);
expect(lib.status().lastError).toBeUndefined();
} finally {
lib.close();
}
});
});