From fe952d3e627117851a136c639cf53bedef43d06e Mon Sep 17 00:00:00 2001 From: clawbot <35+clawbot@noreply.example.org> Date: Wed, 23 Sep 2026 00:27:30 +0200 Subject: [PATCH] Rewrite the README API reference for the new library surface (closes #53) Rewrites the README API reference to match the shipped library surface: Library.open options (XDG cache dir + userID, downloadDirectory, refreshIntervalSeconds, precache and cache-size options), default vs fresh reads (Library.fresh()), snapshot/subscribe, albums/photos/timeline, Photo.original/thumbnail, thumbnails.ensure, mldata search, backup, the request pools and on-disk layout, and the CLI (--cache-dir, fresh reads). Every documented signature verified against the code. Notes the deferred content-hash integrity check (#68). Also issue #13. Model: opus-4-8 --- README.md | 241 ++++++++++++++++++++++++++++++++++++++++++++++-------- TODO.md | 12 ++- 2 files changed, 216 insertions(+), 37 deletions(-) diff --git a/README.md b/README.md index 8453581..38b8039 100644 --- a/README.md +++ b/README.md @@ -38,34 +38,47 @@ yarn quak get 67890 --out ./photo.jpg yarn quak backup ./my-backup ``` -For library use: +For library use, the primary surface is the cache-backed `Library`: ```ts -import { Client } from "quak"; +import { Client, Library } from "quak"; +// Log in once; the client satisfies the library's client interface. const client = await Client.login({ email: "you@example.com", password: "your-password", }); -for (const c of await client.listCollections()) { - console.log(c.id, c.name); - const files = await client.listFiles(c.id, c.key); - for (const f of files) { - console.log(` ${f.metadata.title} [${f.metadata.fileType}]`); +// Open a cache-backed library. On an empty cache this awaits one server +// refresh; on an existing cache it returns immediately and refreshes in the +// background every `refreshIntervalSeconds` (default 3). +const lib = await Library.open({ client }); + +// Default reads answer synchronously from the local cache — no network. +for (const album of lib.albums.list()) { + console.log(album.collectionID, album.name); + for (const photo of album.photos.list()) { + console.log(` ${photo.title} [${photo.fileType}]`); } } -// Download a file -const files = await client.listFiles(collectionID, collectionKey); -await client.downloadFile(files[0], "./photo.jpg"); +// Fresh reads await a server round-trip and answer with current state. +const { albums } = await lib.fresh(); +console.log(`${albums.list().length} albums as of now`); -// Serialize session for later (consumer handles persistence) -const snapshot = client.toJSON(); -// ... later: -const restored = Client.fromJSON(snapshot); +// Fetch (and cache) one photo's full-resolution bytes. +const photo = lib.photos.byID({ fileID: 12345 }); +if (photo) { + const { path } = await photo.original(); + console.log(`original at ${path}`); +} + +lib.close(); ``` +The lower-level `Client` (login, session serialization, and the raw +enumeration/download calls) is exported too and documented under Design below. + ## Entrypoints This repository adheres to the @@ -429,18 +442,22 @@ quak files --collection [--json] list files in a collection quak get [--out path] [--collection] download and decrypt a file quak get-thumb [--out] [--collection] download and decrypt a thumbnail quak backup [--json] full incremental backup +quak backup-metadata [--exif] dump all decrypted metadata as JSON quak helper list-missing-thumbnails [--json] find files with missing thumbnails quak helper fix-missing-thumbnails [--file ids] generate + upload missing thumbnails ``` -Every command reads through the local library cache: the first run fetches the -account's metadata from the server, and later runs serve from the cache and -refresh in the background. `--cache-dir` overrides where that cache lives; -without it each account gets its own directory under the per-user cache path. +Every command runs on the same cache-backed library. The read commands — +`collections`, `files`, `get`, and `get-thumb` — force a fresh server round-trip +before they answer, so they report current account state rather than whatever +the cache last held. `--cache-dir` overrides where the cache lives; without it +each account gets its own directory under the per-user cache path. `get` and `get-thumb` resolve the file by ID directly, so `--collection` is -accepted for backward compatibility but ignored. All listing and backup commands -support `--json` for machine-readable output. +accepted for backward compatibility but ignored. `backup-metadata --exif` (alias +`--all`) additionally downloads each file to extract full EXIF/IPTC/XMP +metadata. The listing and backup commands support `--json` for machine-readable +output. `helper fix-missing-thumbnails` regenerates thumbnails for baseline JPEG images only, because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG @@ -472,7 +489,7 @@ code is non-zero if any files failed. - [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network errors -- [ ] Update the API reference section below to match the current implementation +- [x] Update the API reference section below to match the current implementation - [x] `make docker` green - [ ] Tag `v1.0.0` @@ -486,25 +503,179 @@ Future (desktop client, separate repo): ## API reference -The API reference section below is from an earlier draft and does not fully -reflect the current implementation. The authoritative API documentation is in -the test files, particularly `test/client/usage.test.ts` which is a literate -tutorial walking through every operation. Run `yarn test` to verify the examples -are correct. +The library's primary surface is the cache-backed `Library`; the lower-level +`Client` sits underneath it and is covered by the Design sections above. The +test suite is the canonical, executable documentation — `test/library/` and +`test/client/usage.test.ts` walk every operation, and `yarn test` verifies them. -The key types and their actual signatures can be found in: +### Opening a library +`Library.open(options)` loads the on-disk cache, starts the background refresh +loop, and resolves to a `Library`. On an empty cache it awaits the first refresh +so it never opens onto empty data; on an existing cache it returns immediately +and refreshes in the background, so an unreachable server does not block +opening. + +`LibraryOptions`: + +| Option | Default | Meaning | +| ------------------------ | --------------------------- | --------------------------------------------------------------------- | +| `client` | required | the account client (a `Client`, or any `LibraryClient`) | +| `cacheDirectory` | `/quak/` | where `metadata.json` and the content cache live | +| `downloadDirectory` | none | backup destination; an original already stored there counts as cached | +| `refreshIntervalSeconds` | `3` | background refresh cadence | +| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first | +| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals | +| `precacheOriginalsDays` | `7` | length in days of that latest window | +| `cacheOriginalsMaxBytes` | 100 GiB | hard ceiling on the originals cache | +| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down | +| `isOriginalPinned` | none | extra predicate for originals that must never be evicted | +| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) | +| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) | +| `contentSource` | the client's own | override the byte source (mainly for tests) | + +Concurrency is set through `pools`: construct +`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })` +and pass it. The three pools default to 10 / 5 / 25 (see Request pools below). + +`lib.status()` returns a `LibraryStatus` (collection/file counts, last +refresh/ML times and errors, originals usage and effective limit, precache +progress, and `closed`). `lib.close()` stops the background timer; it is +idempotent, and an in-flight refresh is left to finish. + +### Default reads vs. fresh reads + +Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer +synchronously from the last refreshed copy held in RAM and never touch the +network. The background timer refreshes that copy every +`refreshIntervalSeconds`, so a default read is immediate but may be up to one +interval stale. + +`await lib.fresh()` forces a refresh, waits for it to complete and persist, and +returns the same `{ albums, photos, timeline }` namespaces — now guaranteed to +reflect a completed server round-trip. Concurrent `fresh()` calls coalesce onto +one refresh, and a refresh that fails rejects the caller (default reads stay +silent and keep serving the last good copy). The CLI's read commands use fresh +reads (issue https://git.eeqj.de/sneak/quak/issues/75). + +### Read surface + +- `lib.albums.list()` → `Album[]`, newest-updated first. + `lib.albums.byID({ collectionID })` and `byName({ albumName })` → + `Album | undefined`. +- `lib.photos.byID({ fileID })` → `Photo | undefined`. + `lib.photos.records({ fileIDs })` → `PhotoRecord[]` in the requested order, + each id once, unknown ids dropped. +- `lib.timeline.groups({ groupBy, filter? })` → `TimelineGroup[]`, grouped by + `"day" | "week" | "month"` (keys `YYYY-MM-DD`, ISO `YYYY-Www`, `YYYY-MM`), + newest group first. A `PhotoFilter` combines `albumID`, `text` + (title/caption/album-name substring), `fileTypes`, `hasLocation`, and + `includeArchived`; hidden photos are always excluded. + +An `Album` exposes its record fields and `album.photos.list()` → `Photo[]` +(newest first). A `Photo` exposes its record fields, `photo.record()` → +`PhotoRecord`, and two content methods: + +- `await photo.original(opts?)` → `{ path, bytes }` — the full-resolution file. +- `await photo.thumbnail(opts?)` → `{ path, bytes }`. + +Both serve from the on-disk content cache when the bytes are present and +otherwise fetch through the pools; `opts.onProgress` reports per-file progress. +They throw when the library was opened without a content source. + +Lower-level accessors that return decrypted model objects (which hold key +material) are also available: `listCollections()`, `getCollection(id)`, +`listFiles(collectionID)`, `getFile(collectionID, fileID)`, and +`getFileByID(fileID)`. + +### Records and change notifications + +The GUI-facing records hold no key material and no binary, so they survive +`structuredClone`/JSON across the Electron IPC boundary: + +- `PhotoRecord`: `fileID`, `albumIDs`, `title`, `takenAt` (milliseconds), + `fileType`, optional `caption` / `width` / `height` / `latitude` / + `longitude`, `isArchived`, `isHidden`, and `thumbnailPath` / `originalPath` + once the bytes are cached. +- `AlbumRecord`: `collectionID`, `name`, `type`, `isShared`, `updationTime`, and + `fileIDs` (newest first). +- `LibrarySnapshot`: `{ albums, photos, takenAt }`. + +`lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first, +photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange` +(`albumsChanged`, `photosChanged`, `fileIDsRemoved`, `albumIDsRemoved`, +`refreshedAt`) whenever a refresh alters the projection, and returns +`{ unsubscribe }`; a refresh that changes nothing delivers nothing. + +### Thumbnails, ML search, and backup + +- `lib.thumbnails.ensure({ fileIDs, priority, signal?, onProgress? })` + prefetches thumbnails through the thumbnail pool, deduped by fileID, returning + one `EnsureResult` (`{ fileID, path?, error? }`) per file. `priority` is + `"visible" | "ahead" | "background"`; only `"visible"` preempts background + work. +- `lib.mldata` searches the CLIP index built from Ente's per-file ML data: + `forFile({ fileID })` → `Promise` (the whole stored + payload — face boxes, landmarks, embedding — read from disk on demand); + `similar({ fileID, limit? })` and `searchByEmbedding({ embedding, limit? })` → + `SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first, + default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes + a query vector the caller produced elsewhere. +- `await lib.backup(opts?)` → `BackupResult`. It refreshes, fetches every + in-scope original (and, with `includeThumbnails`, thumbnails) through the + content cache, and rebuilds the on-disk backup tree with a durable failure + ledger. `BackupOptions`: `downloadDirectory` (falls back to the one `open()` + was given), `includeOriginals` (default `true`), `includeThumbnails` (default + `false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above for the + tree it writes. + +### Request pools + +`RequestPools` holds three independent bounded pools — metadata (10), content +(5), thumbnails (25) — because Ente meters these traffic classes differently. +Each pool orders on-demand work ahead of background/precache work and dedups +in-flight fetches by key, and an idle pool never lends its slots to a busy one. + +### On-disk cache layout + +Under `cacheDirectory`: + +``` +/ + metadata.json decrypted account state + refresh cursor + originals/. cached full-resolution files + thumbnails/.jpg cached thumbnails + mldata/ + .json one decrypted ML payload per file + clip.f32, clip.json the packed CLIP index and its id list + fetched.json per-file fetch bookkeeping +``` + +A stored file appears only via an atomic temp-then-rename, so its presence means +it is complete. The design also calls for a content-hash comparison against +`FileMetadata.hash` on each fetched original; that check is deferred (issue +https://git.eeqj.de/sneak/quak/issues/68) because the exact hash construction +cannot yet be confirmed against the repo's fixtures. + +### Key types by source file + +- `src/library/index.ts`: `Library`, `LibraryOptions`, `LibraryStatus`, + `LibraryClient`, `RefreshEvent` +- `src/library/read.ts`: `Album`, `Photo`, `AlbumsAPI`, `PhotosAPI`, + `TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy` +- `src/library/content.ts`: `ContentResult`, `ContentOptions`, `ThumbnailsAPI`, + `EnsureOptions`, `EnsureResult`, `ContentSource` +- `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`, + `LibraryChange` +- `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult` +- `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool` +- `src/backup.ts`: `BackupOptions`, `BackupResult`, `BackupError` - `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot` -- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `ApiError`, - `StreamOptions` +- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `StreamOptions` - `src/errors.ts`: `ApiError`, `TruncatedStreamError` - `src/retry.ts`: `withRetry`, `isRetryable`, `isSafeToReplay`, `RetryOptions` -- `src/auth/types.ts`: `KeyAttributes`, `SRPAttributes`, - `AuthorizationResponse`, `LoginChallenge` -- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileBlob`, - `RawCollection`, `RawEnteFile`, `RawMagicMetadata` -- `src/download/index.ts`: `DownloadResult` -- `src/backup.ts`: `BackupResult`, `BackupError` +- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileType`, + `CollectionType`, `RawCollection`, `RawEnteFile` - `src/thumbnails.ts`: `MissingThumbnailInfo`, `ThumbnailFixResult` ## Source attribution diff --git a/TODO.md b/TODO.md index faa3564..2300972 100644 --- a/TODO.md +++ b/TODO.md @@ -14,10 +14,19 @@ pre-1.0 # Next Step -Update the README API reference section to match the current implementation. +Tag v1.0.0. # Completed Steps +- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage + snippets) to match the shipped cache/API library on `next` (issue 53, issue + 13). Documented `Library.open` and its options, the default-read vs `fresh()` + distinction (and that the CLI's read commands are fresh), the record types and + `snapshot()`/`subscribe()`, the `albums`/`photos`/`timeline` read surface, + `Photo` content methods, `thumbnails.ensure`, the `mldata` search surface, + `backup()`, the three request pools (10/5/25), and the on-disk cache layout; + noted the deferred content-hash integrity check (issue 68). Docs-only; no code + changed. - 2026-09-22: Added resumable, deletion-aware enumeration to `Client` (issue 38, closes issue 7). `collectionsSince`/`filesSince` take a starting cursor, decrypt live records, surface tombstoned ids in a separate `deleted` list (a @@ -122,7 +131,6 @@ Update the README API reference section to match the current implementation. # Future Steps -- Tag v1.0.0. - Future desktop client, separate repo: - Electron app skeleton consuming this library. - Local SQLite cache keyed on (collectionID, fileID, updationTime).