Compare commits

..
1 Commits
Author SHA1 Message Date
sneak 3ae49bd477 Bring README and TODO.md in line with next after the milestone merge (closes #132)
check / check (push) Successful in 1m40s
Docs only. TODO.md's Next Step says no implementation work is open and
the cache design waits on sneak's review. The README is corrected
wherever the code contradicts it: login's TOTP and email OTP steps and
the crypto done outside libsodium; which errors are retried and what
each backup does with a failed download; the session and logout
behavior; the CLI's --exif, --json, ML data and thumbnail-fixer
details; the cache's default directory and size limit; when refreshes
run and what fresh() and lib.backup() wait for; the Photo fields; test
coverage; the Makefile shims; and the 408/429 retries.

Model: opus-5-5
2026-09-29 02:40:19 +00:00
2 changed files with 84 additions and 57 deletions
+62 -47
View File
@@ -51,7 +51,8 @@ const client = await Client.login({
// Open a cache-backed library. On an empty cache this awaits one server // Open a cache-backed library. On an empty cache this awaits one server
// refresh; on an existing cache it returns immediately and refreshes in the // refresh; on an existing cache it returns immediately and refreshes in the
// background every `refreshIntervalSeconds` (default 3). // background. Later refreshes start `refreshIntervalSeconds` (default 3) after
// the previous one ends.
const lib = await Library.open({ client }); const lib = await Library.open({ client });
// Default reads answer synchronously from the local cache — no network. // Default reads answer synchronously from the local cache — no network.
@@ -62,7 +63,7 @@ for (const album of lib.albums.list()) {
} }
} }
// Fresh reads await a server round-trip and answer with current state. // Fresh reads await a server round-trip, joining one already running.
const { albums } = await lib.fresh(); const { albums } = await lib.fresh();
console.log(`${albums.list().length} albums as of now`); console.log(`${albums.list().length} albums as of now`);
@@ -84,7 +85,7 @@ enumeration/download calls) is exported too and documented under Design below.
This repository adheres to the This repository adheres to the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all) [Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard: normalized scripts in `script/` are the entrypoints for the standard: normalized scripts in `script/` are the entrypoints for the
development workflow, and the Makefile targets are thin shims that call them. development workflow, and most Makefile targets are thin shims that call them.
The scripts are POSIX sh (not bash) so they run in minimal containers such as The scripts are POSIX sh (not bash) so they run in minimal containers such as
alpine. We provide: alpine. We provide:
@@ -251,7 +252,8 @@ the repository root rather than `src/`, because `bin/` is compiled too and
All cryptography is done by `libsodium-wrappers-sumo` (the "sumo" build is All cryptography is done by `libsodium-wrappers-sumo` (the "sumo" build is
required for `crypto_pwhash` / Argon2id), except the SRP handshake, which uses required for `crypto_pwhash` / Argon2id), except the SRP handshake, which uses
`fast-srp-hap`. No hand-rolled crypto. `fast-srp-hap`, and the MD5 checksum sent with a thumbnail upload, which uses
Node's built-in `node:crypto`. No hand-rolled crypto.
The key hierarchy, derived during login, is: The key hierarchy, derived during login, is:
@@ -262,9 +264,12 @@ The key hierarchy, derived during login, is:
3. SRP login: `crypto_kdf_derive_from_key` (BLAKE2b) derives a 32-byte subkey 3. SRP login: `crypto_kdf_derive_from_key` (BLAKE2b) derives a 32-byte subkey
from the KEK with subkey id 1 and context `loginctx`. Its first 16 bytes are from the KEK with subkey id 1 and context `loginctx`. Its first 16 bytes are
the SRP password. the SRP password.
4. After SRP completes, or after the email OTP that replaces SRP when the 4. When SRP completes, the server returns a blob of "key attributes" plus an
account has email MFA on (`isEmailMFAEnabled`), the server returns a blob of encrypted auth token, or first asks for a second factor. quak answers a TOTP
"key attributes" plus an encrypted auth token. request with the code (`POST /users/two-factor/verify`), after which the
server returns them, and cannot answer a passkey request. When the account
has email MFA on (`isEmailMFAEnabled`), an email OTP replaces SRP and the
server returns them after it.
5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK 5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK
yields the 32-byte master key. yields the 32-byte master key.
6. `crypto_secretbox_open_easy` over the encrypted secret key with the master 6. `crypto_secretbox_open_easy` over the encrypted secret key with the master
@@ -331,8 +336,9 @@ request is repeated only when repeating it could produce a different answer:
- Every other 4xx: not retried. A 404 in particular is an answer, and - Every other 4xx: not retried. A 404 in particular is an answer, and
`listMissingThumbnails` depends on getting it promptly and once. `listMissingThumbnails` depends on getting it promptly and once.
- Transport failures — a `fetch` rejection, `ECONNRESET`, `ETIMEDOUT`, a DNS or - Transport failures — a `fetch` rejection, `ECONNRESET`, `ETIMEDOUT`, a DNS or
TLS failure — and deadline aborts: retried. The errno is looked for in the TLS failure — and deadline aborts: retried. Node's `fetch` rejects with a
error's `cause` chain, because that is where Node's `fetch` puts it. plain `TypeError`, so every `TypeError` is retried. The errno is looked for in
the error's `cause` chain, because that is where Node's `fetch` puts it.
- A truncated download: retried. - A truncated download: retried.
- Anything else, including a secretstream authentication failure that is not - Anything else, including a secretstream authentication failure that is not
truncation: not retried. The default answer is no. For a backup tool, retrying truncation: not retried. The default answer is no. For a backup tool, retrying
@@ -401,9 +407,10 @@ and these endpoints have no Range support, so a retry starts the file over. The
atomic write is part of the retried unit: each attempt writes its own temporary atomic write is part of the retried unit: each attempt writes its own temporary
files, one for most files and two for a live photo (its image and its video), files, one for most files and two for a live photo (its image and its video),
and removes them if it fails. Only the attempt that completes renames anything and removes them if it fails. Only the attempt that completes renames anything
into place. `runBackup` and `runMetadataBackup` are unchanged: the retry sits into place. The retry sits below `runBackup` and `runMetadataBackup`.
below them, and a file that fails after exhausting it is still logged, counted, `runBackup` logs, counts and steps over a file that still fails after its
and stepped over. retries; `runMetadataBackup`, which downloads only with `--exif`, records the
error in that file's JSON as `imageMetadataError` and goes on.
One imprecision is deliberate and worth knowing about. When a body ends part-way One imprecision is deliberate and worth knowing about. When a body ends part-way
through a secretstream chunk, Poly1305 fails and carries no framing signal, so a through a secretstream chunk, Poly1305 fails and carries no framing signal, so a
@@ -427,10 +434,11 @@ base64-encoded keys) that the consumer can write to disk, a database, or
whatever else fits their use case. `Client.fromJSON(snapshot)` restores a whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
working client from that snapshot without re-authenticating; it checks every working client from that snapshot without re-authenticating; it checks every
field and each key's length first, and throws an error naming the bad field. field and each key's length first, and throws an error naming the bad field.
`client.logout()` clears the token and zeroes the key buffers in place; every `client.logout()` clears the token and zeroes the key buffers in place; after
later call on that client throws. It does not contact the server, so the token it, every other method on that client throws. It does not contact the server, so
stays valid there and in any saved snapshot; `await client.logoutOnServer()` the token stays valid there and in any saved snapshot;
first ends the session on the server (`POST /users/logout`). `await client.logoutOnServer()` first ends the session on the server
(`POST /users/logout`).
The CLI stores the snapshot at the platform-appropriate data directory via The CLI stores the snapshot at the platform-appropriate data directory via
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS, `env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
@@ -438,7 +446,8 @@ The CLI stores the snapshot at the platform-appropriate data directory via
`0600`. The key material is stored in cleartext in the JSON; treat this file as `0600`. The key material is stored in cleartext in the JSON; treat this file as
you would treat the password itself. A missing file is reported as "not logged you would treat the password itself. A missing file is reported as "not logged
in"; a file that exists but is corrupt is reported as such, naming the bad in"; a file that exists but is corrupt is reported as such, naming the bad
field. Both exit with status 1. field. Both exit with status 1, except that `quak logout` with no file says
there is no session and exits 0.
`quak logout` ends the session on the server, so the token in `session.json` `quak logout` ends the session on the server, so the token in `session.json`
stops working even in a copy of the file, and then deletes the file. If the stops working even in a copy of the file, and then deletes the file. If the
@@ -488,13 +497,13 @@ its EXIF metadata and dimensions. `collections`, `files`, `backup`,
`--json` for machine-readable output. `--json` for machine-readable output.
`backup-metadata` fetches ML data in requests of up to 200 files. When a request `backup-metadata` fetches ML data in requests of up to 200 files. When a request
still fails after its retries, the error is logged, each of its files is written fails, the error is logged, each of its files is written with the reason in an
with the reason in an `mlDataError` field instead of `mlData`, and the dump goes `mlDataError` field instead of `mlData`, and the dump goes on. The exit code is
on. The exit code is non-zero if any ML data request failed. non-zero if any ML data request failed.
`helper fix-missing-thumbnails` regenerates thumbnails for baseline JPEG images `helper fix-missing-thumbnails` regenerates thumbnails for JPEG images only,
only, because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG image
image (PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept (PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept
distinct from a `failed` repair, and does not affect the exit code; a genuine distinct from a `failed` repair, and does not affect the exit code; a genuine
failure still exits non-zero. The server accepts a new thumbnail only from the failure still exits non-zero. The server accepts a new thumbnail only from the
file's owner and only when it is no larger than the thumbnail size it records file's owner and only when it is no larger than the thumbnail size it records
@@ -601,7 +610,9 @@ Future (desktop client, separate repo):
The library's primary surface is the cache-backed `Library`; the lower-level The library's primary surface is the cache-backed `Library`; the lower-level
`Client` sits underneath it and is covered by the Design sections above. The `Client` sits underneath it and is covered by the Design sections above. The
test suite is the canonical, executable documentation — `test/library/` and test suite is the canonical, executable documentation — `test/library/` and
`test/client/usage.test.ts` walk every operation, and `yarn test` verifies them. `test/client/usage.test.ts` walk most operations, `test/cli/backup.test.ts`
walks `lib.backup()`, and `yarn test` verifies them. No test calls
`getFileByID()`.
### Opening a library ### Opening a library
@@ -623,7 +634,7 @@ background, so an unreachable server does not block opening.
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first | | `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals | | `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
| `precacheOriginalsDays` | `7` | length in days of that latest window | | `precacheOriginalsDays` | `7` | length in days of that latest window |
| `cacheOriginalsMaxBytes` | 100 GiB | hard ceiling on the originals cache | | `cacheOriginalsMaxBytes` | 100 GiB | size limit on the originals cache; pinned originals can exceed it |
| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down | | `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down |
| `isOriginalPinned` | none | extra predicate for originals that must never be evicted | | `isOriginalPinned` | none | extra predicate for originals that must never be evicted |
| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) | | `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) |
@@ -645,17 +656,21 @@ can then be removed.
### Default reads vs. fresh reads ### Default reads vs. fresh reads
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
synchronously from the last refreshed copy held in RAM and never touch the synchronously from the copy held in RAM and never touch the network. A refresh
network. The background timer refreshes that copy every changes that copy only once all its server requests have succeeded, and the
`refreshIntervalSeconds`, so a default read is immediate but may be up to one background timer starts the next refresh `refreshIntervalSeconds` after the
interval stale. previous one ends. So a default read is immediate, but only as current as the
last refresh whose requests all succeeded; right after opening an existing
cache, it is the copy on disk.
`await lib.fresh()` forces a refresh, waits for it to complete and persist, and `await lib.fresh()` waits for a refresh to complete and persist, and returns the
returns the same `{ albums, photos, timeline }` namespaces — now guaranteed to same `{ albums, photos, timeline }` namespaces, which then reflect a completed
reflect a completed server round-trip. Concurrent `fresh()` calls coalesce onto server round-trip. When a refresh is already running, background or not,
one refresh, and a refresh that fails rejects the caller (default reads stay `fresh()` waits for that one, so its answer can come from requests made before
silent and keep serving the last good copy). The CLI's read commands use fresh the call; only when none is running does it start one. A refresh that fails
reads (issue https://git.eeqj.de/sneak/quak/issues/75). rejects the caller (default reads stay silent and keep serving the last good
copy). The CLI's read commands use fresh reads (issue
https://git.eeqj.de/sneak/quak/issues/75).
### Read surface ### Read surface
@@ -672,8 +687,8 @@ reads (issue https://git.eeqj.de/sneak/quak/issues/75).
`includeArchived`; hidden photos are always excluded. `includeArchived`; hidden photos are always excluded.
An `Album` exposes its record fields and `album.photos.list()` → `Photo[]` An `Album` exposes its record fields and `album.photos.list()` → `Photo[]`
(newest first). A `Photo` exposes its record fields, `photo.record()` → (newest first). A `Photo` exposes its record fields other than `thumbnailPath`
`PhotoRecord`, and two content methods: and `originalPath`, `photo.record()` → `PhotoRecord`, and two content methods:
- `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the - `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the
full-resolution file. For a live photo, `path` and `bytes` are its image's and full-resolution file. For a live photo, `path` and `bytes` are its image's and
@@ -722,16 +737,16 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
`SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first, `SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
a query vector the caller produced elsewhere. a query vector the caller produced elsewhere.
- `await lib.backup(opts?)` → `BackupResult`. It refreshes, fetches every - `await lib.backup(opts?)` → `BackupResult`. It waits for a refresh as
in-scope original not already in the backup (and, with `includeThumbnails`, `fresh()` does, fetches every in-scope original not already in the backup
thumbnails) through the content cache, and rebuilds the on-disk backup tree (and, with `includeThumbnails`, thumbnails) through the content cache, and
with a durable failure ledger. A fetched original is written straight into the rebuilds the on-disk backup tree with a durable failure ledger. A fetched
backup's `originals/` and not into the cache, which then counts it as present; original is written straight into the backup's `originals/` and not into the
one the cache already held is copied from there. `BackupOptions`: cache, which then counts it as present; one the cache already held is copied
`downloadDirectory` (falls back to the one `open()` was given), from there. `BackupOptions`: `downloadDirectory` (falls back to the one
`includeOriginals` (default `true`), `includeThumbnails` (default `false`), `open()` was given), `includeOriginals` (default `true`), `includeThumbnails`
`onlyAlbumNames`, and `onProgress`. See Backup layout above for the tree it (default `false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above
writes. for the tree it writes.
### Request pools ### Request pools
+22 -10
View File
@@ -25,16 +25,28 @@ declares one.
- 2026-09-29: Brought the README and this file in line with `next` after the - 2026-09-29: Brought the README and this file in line with `next` after the
milestone merge (issue 132). The Next Step says no implementation work is open milestone merge (issue 132). The Next Step says no implementation work is open
and the cache design (issue 36) waits on sneak's review. README corrections: and the cache design (issue 36) waits on sneak's review. README corrections:
`script/lint` and `script/test` have no host path, though `yarn test` does; the Getting Started comments say when the library refreshes; most, not all,
the SRP handshake uses `fast-srp-hap`, outside `crypto/`; the SRP password is Makefile targets call a script; `script/lint` and `script/test` have no host
the first 16 bytes of a 32-byte subkey; email OTP replaces SRP when the path, though `yarn test` does; the SRP handshake uses `fast-srp-hap`, outside
account has email MFA on; the auth token is sent as URL-safe base64 with `crypto/`, and a thumbnail upload's MD5 uses `node:crypto`; the SRP password
padding; each download attempt writes its own temporary files, two for a live is the first 16 bytes of a 32-byte subkey; the key attributes and token come
photo, and only the attempt that completes renames them into place; `login`, after SRP, after the TOTP code SRP may ask for, or after the email OTP that
`whoami` and `logout` open no library; `--exif` records XMP and, for a JPEG, replaces SRP when the account has email MFA on, and quak cannot answer a
EXIF, and no IPTC; which commands take `--json`; `quak backup` still fetches passkey; the auth token is sent as URL-safe base64 with padding; every
ML data; the default cache directory is the per-user one, not an XDG path on `TypeError` is retried; each download attempt writes its own temporary files,
macOS; and `408` and `429` are retried. two for a live photo, and only the attempt that completes renames them into
place; `runMetadataBackup` records a failed download in the file's JSON; a
second `client.logout()` does not throw; `quak logout` with no session exits
0, and names the cache directory only when it knows it; `login`, `whoami` and
`logout` open no library; `--exif` records XMP and, for a JPEG, EXIF, and no
IPTC; which commands take `--json`; a failed ML data request may not have been
retried; the thumbnail fixer is not limited to baseline JPEG; `quak backup`
still fetches ML data; the default cache directory is the per-user one, not an
XDG path on macOS; pinned originals can exceed `cacheOriginalsMaxBytes`; which
tests cover which operations; a default read is only as current as the last
refresh whose requests all succeeded; `fresh()` and `lib.backup()` join a
refresh already running; a `Photo` has no `thumbnailPath` or `originalPath`;
and `408` and `429` are retried.
- 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks - 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks
that the image's and the video's temp files are fsynced before either is that the image's and the video's temp files are fsynced before either is