Compare commits
1
Commits
next
..
dbc7338f50
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
dbc7338f50 |
@@ -38,47 +38,34 @@ yarn quak get 67890 --out ./photo.jpg
|
||||
yarn quak backup ./my-backup
|
||||
```
|
||||
|
||||
For library use, the primary surface is the cache-backed `Library`:
|
||||
For library use:
|
||||
|
||||
```ts
|
||||
import { Client, Library } from "quak";
|
||||
import { Client } from "quak";
|
||||
|
||||
// Log in once; the client satisfies the library's client interface.
|
||||
const client = await Client.login({
|
||||
email: "you@example.com",
|
||||
password: "your-password",
|
||||
});
|
||||
|
||||
// Open a cache-backed library. On an empty cache this awaits one server
|
||||
// refresh; on an existing cache it returns immediately and refreshes in the
|
||||
// background every `refreshIntervalSeconds` (default 3).
|
||||
const lib = await Library.open({ client });
|
||||
|
||||
// Default reads answer synchronously from the local cache — no network.
|
||||
for (const album of lib.albums.list()) {
|
||||
console.log(album.collectionID, album.name);
|
||||
for (const photo of album.photos.list()) {
|
||||
console.log(` ${photo.title} [${photo.fileType}]`);
|
||||
for (const c of await client.listCollections()) {
|
||||
console.log(c.id, c.name);
|
||||
const files = await client.listFiles(c.id, c.key);
|
||||
for (const f of files) {
|
||||
console.log(` ${f.metadata.title} [${f.metadata.fileType}]`);
|
||||
}
|
||||
}
|
||||
|
||||
// Fresh reads await a server round-trip and answer with current state.
|
||||
const { albums } = await lib.fresh();
|
||||
console.log(`${albums.list().length} albums as of now`);
|
||||
// Download a file
|
||||
const files = await client.listFiles(collectionID, collectionKey);
|
||||
await client.downloadFile(files[0], "./photo.jpg");
|
||||
|
||||
// Fetch (and cache) one photo's full-resolution bytes.
|
||||
const photo = lib.photos.byID({ fileID: 12345 });
|
||||
if (photo) {
|
||||
const { path } = await photo.original();
|
||||
console.log(`original at ${path}`);
|
||||
}
|
||||
|
||||
lib.close();
|
||||
// Serialize session for later (consumer handles persistence)
|
||||
const snapshot = client.toJSON();
|
||||
// ... later:
|
||||
const restored = Client.fromJSON(snapshot);
|
||||
```
|
||||
|
||||
The lower-level `Client` (login, session serialization, and the raw
|
||||
enumeration/download calls) is exported too and documented under Design below.
|
||||
|
||||
## Entrypoints
|
||||
|
||||
This repository adheres to the
|
||||
@@ -442,22 +429,18 @@ quak files --collection <id> [--json] list files in a collection
|
||||
quak get <fileID> [--out path] [--collection] download and decrypt a file
|
||||
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
|
||||
quak backup <dir> [--json] full incremental backup
|
||||
quak backup-metadata <dir> [--exif] dump all decrypted metadata as JSON
|
||||
quak helper list-missing-thumbnails [--json] find files with missing thumbnails
|
||||
quak helper fix-missing-thumbnails [--file ids] generate + upload missing thumbnails
|
||||
```
|
||||
|
||||
Every command runs on the same cache-backed library. The read commands —
|
||||
`collections`, `files`, `get`, and `get-thumb` — force a fresh server round-trip
|
||||
before they answer, so they report current account state rather than whatever
|
||||
the cache last held. `--cache-dir` overrides where the cache lives; without it
|
||||
each account gets its own directory under the per-user cache path.
|
||||
Every command reads through the local library cache: the first run fetches the
|
||||
account's metadata from the server, and later runs serve from the cache and
|
||||
refresh in the background. `--cache-dir` overrides where that cache lives;
|
||||
without it each account gets its own directory under the per-user cache path.
|
||||
|
||||
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
|
||||
accepted for backward compatibility but ignored. `backup-metadata --exif` (alias
|
||||
`--all`) additionally downloads each file to extract full EXIF/IPTC/XMP
|
||||
metadata. The listing and backup commands support `--json` for machine-readable
|
||||
output.
|
||||
accepted for backward compatibility but ignored. All listing and backup commands
|
||||
support `--json` for machine-readable output.
|
||||
|
||||
`helper fix-missing-thumbnails` regenerates thumbnails for baseline JPEG images
|
||||
only, because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG
|
||||
@@ -489,7 +472,7 @@ code is non-zero if any files failed.
|
||||
|
||||
- [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
|
||||
errors
|
||||
- [x] Update the API reference section below to match the current implementation
|
||||
- [ ] Update the API reference section below to match the current implementation
|
||||
- [x] `make docker` green
|
||||
- [ ] Tag `v1.0.0`
|
||||
|
||||
@@ -503,179 +486,25 @@ Future (desktop client, separate repo):
|
||||
|
||||
## API reference
|
||||
|
||||
The library's primary surface is the cache-backed `Library`; the lower-level
|
||||
`Client` sits underneath it and is covered by the Design sections above. The
|
||||
test suite is the canonical, executable documentation — `test/library/` and
|
||||
`test/client/usage.test.ts` walk every operation, and `yarn test` verifies them.
|
||||
The API reference section below is from an earlier draft and does not fully
|
||||
reflect the current implementation. The authoritative API documentation is in
|
||||
the test files, particularly `test/client/usage.test.ts` which is a literate
|
||||
tutorial walking through every operation. Run `yarn test` to verify the examples
|
||||
are correct.
|
||||
|
||||
### Opening a library
|
||||
The key types and their actual signatures can be found in:
|
||||
|
||||
`Library.open(options)` loads the on-disk cache, starts the background refresh
|
||||
loop, and resolves to a `Library`. On an empty cache it awaits the first refresh
|
||||
so it never opens onto empty data; on an existing cache it returns immediately
|
||||
and refreshes in the background, so an unreachable server does not block
|
||||
opening.
|
||||
|
||||
`LibraryOptions`:
|
||||
|
||||
| Option | Default | Meaning |
|
||||
| ------------------------ | --------------------------- | --------------------------------------------------------------------- |
|
||||
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
|
||||
| `cacheDirectory` | `<XDG cache>/quak/<userID>` | where `metadata.json` and the content cache live |
|
||||
| `downloadDirectory` | none | backup destination; an original already stored there counts as cached |
|
||||
| `refreshIntervalSeconds` | `3` | background refresh cadence |
|
||||
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
|
||||
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
|
||||
| `precacheOriginalsDays` | `7` | length in days of that latest window |
|
||||
| `cacheOriginalsMaxBytes` | 100 GiB | hard ceiling on the originals cache |
|
||||
| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down |
|
||||
| `isOriginalPinned` | none | extra predicate for originals that must never be evicted |
|
||||
| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) |
|
||||
| `onProgress` | none | refresh/ML/precache progress callback (`RefreshEvent`) |
|
||||
| `contentSource` | the client's own | override the byte source (mainly for tests) |
|
||||
|
||||
Concurrency is set through `pools`: construct
|
||||
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
|
||||
and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).
|
||||
|
||||
`lib.status()` returns a `LibraryStatus` (collection/file counts, last
|
||||
refresh/ML times and errors, originals usage and effective limit, precache
|
||||
progress, and `closed`). `lib.close()` stops the background timer; it is
|
||||
idempotent, and an in-flight refresh is left to finish.
|
||||
|
||||
### Default reads vs. fresh reads
|
||||
|
||||
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
|
||||
synchronously from the last refreshed copy held in RAM and never touch the
|
||||
network. The background timer refreshes that copy every
|
||||
`refreshIntervalSeconds`, so a default read is immediate but may be up to one
|
||||
interval stale.
|
||||
|
||||
`await lib.fresh()` forces a refresh, waits for it to complete and persist, and
|
||||
returns the same `{ albums, photos, timeline }` namespaces — now guaranteed to
|
||||
reflect a completed server round-trip. Concurrent `fresh()` calls coalesce onto
|
||||
one refresh, and a refresh that fails rejects the caller (default reads stay
|
||||
silent and keep serving the last good copy). The CLI's read commands use fresh
|
||||
reads (issue https://git.eeqj.de/sneak/quak/issues/75).
|
||||
|
||||
### Read surface
|
||||
|
||||
- `lib.albums.list()` → `Album[]`, newest-updated first.
|
||||
`lib.albums.byID({ collectionID })` and `byName({ albumName })` →
|
||||
`Album | undefined`.
|
||||
- `lib.photos.byID({ fileID })` → `Photo | undefined`.
|
||||
`lib.photos.records({ fileIDs })` → `PhotoRecord[]` in the requested order,
|
||||
each id once, unknown ids dropped.
|
||||
- `lib.timeline.groups({ groupBy, filter? })` → `TimelineGroup[]`, grouped by
|
||||
`"day" | "week" | "month"` (keys `YYYY-MM-DD`, ISO `YYYY-Www`, `YYYY-MM`),
|
||||
newest group first. A `PhotoFilter` combines `albumID`, `text`
|
||||
(title/caption/album-name substring), `fileTypes`, `hasLocation`, and
|
||||
`includeArchived`; hidden photos are always excluded.
|
||||
|
||||
An `Album` exposes its record fields and `album.photos.list()` → `Photo[]`
|
||||
(newest first). A `Photo` exposes its record fields, `photo.record()` →
|
||||
`PhotoRecord`, and two content methods:
|
||||
|
||||
- `await photo.original(opts?)` → `{ path, bytes }` — the full-resolution file.
|
||||
- `await photo.thumbnail(opts?)` → `{ path, bytes }`.
|
||||
|
||||
Both serve from the on-disk content cache when the bytes are present and
|
||||
otherwise fetch through the pools; `opts.onProgress` reports per-file progress.
|
||||
They throw when the library was opened without a content source.
|
||||
|
||||
Lower-level accessors that return decrypted model objects (which hold key
|
||||
material) are also available: `listCollections()`, `getCollection(id)`,
|
||||
`listFiles(collectionID)`, `getFile(collectionID, fileID)`, and
|
||||
`getFileByID(fileID)`.
|
||||
|
||||
### Records and change notifications
|
||||
|
||||
The GUI-facing records hold no key material and no binary, so they survive
|
||||
`structuredClone`/JSON across the Electron IPC boundary:
|
||||
|
||||
- `PhotoRecord`: `fileID`, `albumIDs`, `title`, `takenAt` (milliseconds),
|
||||
`fileType`, optional `caption` / `width` / `height` / `latitude` /
|
||||
`longitude`, `isArchived`, `isHidden`, and `thumbnailPath` / `originalPath`
|
||||
once the bytes are cached.
|
||||
- `AlbumRecord`: `collectionID`, `name`, `type`, `isShared`, `updationTime`, and
|
||||
`fileIDs` (newest first).
|
||||
- `LibrarySnapshot`: `{ albums, photos, takenAt }`.
|
||||
|
||||
`lib.snapshot()` returns a `LibrarySnapshot` (albums newest-updated first,
|
||||
photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
(`albumsChanged`, `photosChanged`, `fileIDsRemoved`, `albumIDsRemoved`,
|
||||
`refreshedAt`) whenever a refresh alters the projection, and returns
|
||||
`{ unsubscribe }`; a refresh that changes nothing delivers nothing.
|
||||
|
||||
### Thumbnails, ML search, and backup
|
||||
|
||||
- `lib.thumbnails.ensure({ fileIDs, priority, signal?, onProgress? })`
|
||||
prefetches thumbnails through the thumbnail pool, deduped by fileID, returning
|
||||
one `EnsureResult` (`{ fileID, path?, error? }`) per file. `priority` is
|
||||
`"visible" | "ahead" | "background"`; only `"visible"` preempts background
|
||||
work.
|
||||
- `lib.mldata` searches the CLIP index built from Ente's per-file ML data:
|
||||
`forFile({ fileID })` → `Promise<MLData | undefined>` (the whole stored
|
||||
payload — face boxes, landmarks, embedding — read from disk on demand);
|
||||
`similar({ fileID, limit? })` and `searchByEmbedding({ embedding, limit? })` →
|
||||
`SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
|
||||
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
|
||||
a query vector the caller produced elsewhere.
|
||||
- `await lib.backup(opts?)` → `BackupResult`. It refreshes, fetches every
|
||||
in-scope original (and, with `includeThumbnails`, thumbnails) through the
|
||||
content cache, and rebuilds the on-disk backup tree with a durable failure
|
||||
ledger. `BackupOptions`: `downloadDirectory` (falls back to the one `open()`
|
||||
was given), `includeOriginals` (default `true`), `includeThumbnails` (default
|
||||
`false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above for the
|
||||
tree it writes.
|
||||
|
||||
### Request pools
|
||||
|
||||
`RequestPools` holds three independent bounded pools — metadata (10), content
|
||||
(5), thumbnails (25) — because Ente meters these traffic classes differently.
|
||||
Each pool orders on-demand work ahead of background/precache work and dedups
|
||||
in-flight fetches by key, and an idle pool never lends its slots to a busy one.
|
||||
|
||||
### On-disk cache layout
|
||||
|
||||
Under `cacheDirectory`:
|
||||
|
||||
```
|
||||
<cacheDirectory>/
|
||||
metadata.json decrypted account state + refresh cursor
|
||||
originals/<fileID>.<ext> cached full-resolution files
|
||||
thumbnails/<fileID>.jpg cached thumbnails
|
||||
mldata/
|
||||
<fileID>.json one decrypted ML payload per file
|
||||
clip.f32, clip.json the packed CLIP index and its id list
|
||||
fetched.json per-file fetch bookkeeping
|
||||
```
|
||||
|
||||
A stored file appears only via an atomic temp-then-rename, so its presence means
|
||||
it is complete. The design also calls for a content-hash comparison against
|
||||
`FileMetadata.hash` on each fetched original; that check is deferred (issue
|
||||
https://git.eeqj.de/sneak/quak/issues/68) because the exact hash construction
|
||||
cannot yet be confirmed against the repo's fixtures.
|
||||
|
||||
### Key types by source file
|
||||
|
||||
- `src/library/index.ts`: `Library`, `LibraryOptions`, `LibraryStatus`,
|
||||
`LibraryClient`, `RefreshEvent`
|
||||
- `src/library/read.ts`: `Album`, `Photo`, `AlbumsAPI`, `PhotosAPI`,
|
||||
`TimelineAPI`, `PhotoFilter`, `TimelineGroup`, `GroupBy`
|
||||
- `src/library/content.ts`: `ContentResult`, `ContentOptions`, `ThumbnailsAPI`,
|
||||
`EnsureOptions`, `EnsureResult`, `ContentSource`
|
||||
- `src/library/records.ts`: `PhotoRecord`, `AlbumRecord`, `LibrarySnapshot`,
|
||||
`LibraryChange`
|
||||
- `src/library/mlsearch.ts`: `MLDataAPI`, `SimilarResult`
|
||||
- `src/library/pools.ts`: `RequestPools`, `RequestPoolsOptions`, `BoundedPool`
|
||||
- `src/backup.ts`: `BackupOptions`, `BackupResult`, `BackupError`
|
||||
- `src/client.ts`: `Client`, `LoginOptions`, `ClientSnapshot`
|
||||
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `StreamOptions`
|
||||
- `src/api/client.ts`: `ApiClient`, `ApiClientOptions`, `ApiError`,
|
||||
`StreamOptions`
|
||||
- `src/errors.ts`: `ApiError`, `TruncatedStreamError`
|
||||
- `src/retry.ts`: `withRetry`, `isRetryable`, `isSafeToReplay`, `RetryOptions`
|
||||
- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileType`,
|
||||
`CollectionType`, `RawCollection`, `RawEnteFile`
|
||||
- `src/auth/types.ts`: `KeyAttributes`, `SRPAttributes`,
|
||||
`AuthorizationResponse`, `LoginChallenge`
|
||||
- `src/model/types.ts`: `Collection`, `EnteFile`, `FileMetadata`, `FileBlob`,
|
||||
`RawCollection`, `RawEnteFile`, `RawMagicMetadata`
|
||||
- `src/download/index.ts`: `DownloadResult`
|
||||
- `src/backup.ts`: `BackupResult`, `BackupError`
|
||||
- `src/thumbnails.ts`: `MissingThumbnailInfo`, `ThumbnailFixResult`
|
||||
|
||||
## Source attribution
|
||||
|
||||
@@ -14,19 +14,10 @@ pre-1.0
|
||||
|
||||
# Next Step
|
||||
|
||||
Tag v1.0.0.
|
||||
Update the README API reference section to match the current implementation.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
|
||||
snippets) to match the shipped cache/API library on `next` (issue 53, issue
|
||||
13). Documented `Library.open` and its options, the default-read vs `fresh()`
|
||||
distinction (and that the CLI's read commands are fresh), the record types and
|
||||
`snapshot()`/`subscribe()`, the `albums`/`photos`/`timeline` read surface,
|
||||
`Photo` content methods, `thumbnails.ensure`, the `mldata` search surface,
|
||||
`backup()`, the three request pools (10/5/25), and the on-disk cache layout;
|
||||
noted the deferred content-hash integrity check (issue 68). Docs-only; no code
|
||||
changed.
|
||||
- 2026-09-22: Added resumable, deletion-aware enumeration to `Client` (issue 38,
|
||||
closes issue 7). `collectionsSince`/`filesSince` take a starting cursor,
|
||||
decrypt live records, surface tombstoned ids in a separate `deleted` list (a
|
||||
@@ -131,6 +122,7 @@ Tag v1.0.0.
|
||||
|
||||
# Future Steps
|
||||
|
||||
- Tag v1.0.0.
|
||||
- Future desktop client, separate repo:
|
||||
- Electron app skeleton consuming this library.
|
||||
- Local SQLite cache keyed on (collectionID, fileID, updationTime).
|
||||
|
||||
Reference in New Issue
Block a user