Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
3ae49bd477 |
@@ -51,7 +51,8 @@ const client = await Client.login({
|
||||
|
||||
// Open a cache-backed library. On an empty cache this awaits one server
|
||||
// refresh; on an existing cache it returns immediately and refreshes in the
|
||||
// background every `refreshIntervalSeconds` (default 3).
|
||||
// background. Later refreshes start `refreshIntervalSeconds` (default 3) after
|
||||
// the previous one ends.
|
||||
const lib = await Library.open({ client });
|
||||
|
||||
// Default reads answer synchronously from the local cache — no network.
|
||||
@@ -62,7 +63,7 @@ for (const album of lib.albums.list()) {
|
||||
}
|
||||
}
|
||||
|
||||
// Fresh reads await a server round-trip and answer with current state.
|
||||
// Fresh reads await a server round-trip, joining one already running.
|
||||
const { albums } = await lib.fresh();
|
||||
console.log(`${albums.list().length} albums as of now`);
|
||||
|
||||
@@ -84,7 +85,7 @@ enumeration/download calls) is exported too and documented under Design below.
|
||||
This repository adheres to the
|
||||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||||
standard: normalized scripts in `script/` are the entrypoints for the
|
||||
development workflow, and the Makefile targets are thin shims that call them.
|
||||
development workflow, and most Makefile targets are thin shims that call them.
|
||||
The scripts are POSIX sh (not bash) so they run in minimal containers such as
|
||||
alpine. We provide:
|
||||
|
||||
@@ -251,7 +252,8 @@ the repository root rather than `src/`, because `bin/` is compiled too and
|
||||
|
||||
All cryptography is done by `libsodium-wrappers-sumo` (the "sumo" build is
|
||||
required for `crypto_pwhash` / Argon2id), except the SRP handshake, which uses
|
||||
`fast-srp-hap`. No hand-rolled crypto.
|
||||
`fast-srp-hap`, and the MD5 checksum sent with a thumbnail upload, which uses
|
||||
Node's built-in `node:crypto`. No hand-rolled crypto.
|
||||
|
||||
The key hierarchy, derived during login, is:
|
||||
|
||||
@@ -262,9 +264,12 @@ The key hierarchy, derived during login, is:
|
||||
3. SRP login: `crypto_kdf_derive_from_key` (BLAKE2b) derives a 32-byte subkey
|
||||
from the KEK with subkey id 1 and context `loginctx`. Its first 16 bytes are
|
||||
the SRP password.
|
||||
4. After SRP completes, or after the email OTP that replaces SRP when the
|
||||
account has email MFA on (`isEmailMFAEnabled`), the server returns a blob of
|
||||
"key attributes" plus an encrypted auth token.
|
||||
4. When SRP completes, the server returns a blob of "key attributes" plus an
|
||||
encrypted auth token, or first asks for a second factor. quak answers a TOTP
|
||||
request with the code (`POST /users/two-factor/verify`), after which the
|
||||
server returns them, and cannot answer a passkey request. When the account
|
||||
has email MFA on (`isEmailMFAEnabled`), an email OTP replaces SRP and the
|
||||
server returns them after it.
|
||||
5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK
|
||||
yields the 32-byte master key.
|
||||
6. `crypto_secretbox_open_easy` over the encrypted secret key with the master
|
||||
@@ -331,8 +336,9 @@ request is repeated only when repeating it could produce a different answer:
|
||||
- Every other 4xx: not retried. A 404 in particular is an answer, and
|
||||
`listMissingThumbnails` depends on getting it promptly and once.
|
||||
- Transport failures — a `fetch` rejection, `ECONNRESET`, `ETIMEDOUT`, a DNS or
|
||||
TLS failure — and deadline aborts: retried. The errno is looked for in the
|
||||
error's `cause` chain, because that is where Node's `fetch` puts it.
|
||||
TLS failure — and deadline aborts: retried. Node's `fetch` rejects with a
|
||||
plain `TypeError`, so every `TypeError` is retried. The errno is looked for in
|
||||
the error's `cause` chain, because that is where Node's `fetch` puts it.
|
||||
- A truncated download: retried.
|
||||
- Anything else, including a secretstream authentication failure that is not
|
||||
truncation: not retried. The default answer is no. For a backup tool, retrying
|
||||
@@ -401,9 +407,10 @@ and these endpoints have no Range support, so a retry starts the file over. The
|
||||
atomic write is part of the retried unit: each attempt writes its own temporary
|
||||
files, one for most files and two for a live photo (its image and its video),
|
||||
and removes them if it fails. Only the attempt that completes renames anything
|
||||
into place. `runBackup` and `runMetadataBackup` are unchanged: the retry sits
|
||||
below them, and a file that fails after exhausting it is still logged, counted,
|
||||
and stepped over.
|
||||
into place. The retry sits below `runBackup` and `runMetadataBackup`.
|
||||
`runBackup` logs, counts and steps over a file that still fails after its
|
||||
retries; `runMetadataBackup`, which downloads only with `--exif`, records the
|
||||
error in that file's JSON as `imageMetadataError` and goes on.
|
||||
|
||||
One imprecision is deliberate and worth knowing about. When a body ends part-way
|
||||
through a secretstream chunk, Poly1305 fails and carries no framing signal, so a
|
||||
@@ -427,10 +434,11 @@ base64-encoded keys) that the consumer can write to disk, a database, or
|
||||
whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
|
||||
working client from that snapshot without re-authenticating; it checks every
|
||||
field and each key's length first, and throws an error naming the bad field.
|
||||
`client.logout()` clears the token and zeroes the key buffers in place; every
|
||||
later call on that client throws. It does not contact the server, so the token
|
||||
stays valid there and in any saved snapshot; `await client.logoutOnServer()`
|
||||
first ends the session on the server (`POST /users/logout`).
|
||||
`client.logout()` clears the token and zeroes the key buffers in place; after
|
||||
it, every other method on that client throws. It does not contact the server, so
|
||||
the token stays valid there and in any saved snapshot;
|
||||
`await client.logoutOnServer()` first ends the session on the server
|
||||
(`POST /users/logout`).
|
||||
|
||||
The CLI stores the snapshot at the platform-appropriate data directory via
|
||||
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
|
||||
@@ -438,7 +446,8 @@ The CLI stores the snapshot at the platform-appropriate data directory via
|
||||
`0600`. The key material is stored in cleartext in the JSON; treat this file as
|
||||
you would treat the password itself. A missing file is reported as "not logged
|
||||
in"; a file that exists but is corrupt is reported as such, naming the bad
|
||||
field. Both exit with status 1.
|
||||
field. Both exit with status 1, except that `quak logout` with no file says
|
||||
there is no session and exits 0.
|
||||
|
||||
`quak logout` ends the session on the server, so the token in `session.json`
|
||||
stops working even in a copy of the file, and then deletes the file. If the
|
||||
@@ -488,13 +497,13 @@ its EXIF metadata and dimensions. `collections`, `files`, `backup`,
|
||||
`--json` for machine-readable output.
|
||||
|
||||
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
||||
still fails after its retries, the error is logged, each of its files is written
|
||||
with the reason in an `mlDataError` field instead of `mlData`, and the dump goes
|
||||
on. The exit code is non-zero if any ML data request failed.
|
||||
fails, the error is logged, each of its files is written with the reason in an
|
||||
`mlDataError` field instead of `mlData`, and the dump goes on. The exit code is
|
||||
non-zero if any ML data request failed.
|
||||
|
||||
`helper fix-missing-thumbnails` regenerates thumbnails for baseline JPEG images
|
||||
only, because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG
|
||||
image (PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept
|
||||
`helper fix-missing-thumbnails` regenerates thumbnails for JPEG images only,
|
||||
because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG image
|
||||
(PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept
|
||||
distinct from a `failed` repair, and does not affect the exit code; a genuine
|
||||
failure still exits non-zero. The server accepts a new thumbnail only from the
|
||||
file's owner and only when it is no larger than the thumbnail size it records
|
||||
@@ -601,7 +610,9 @@ Future (desktop client, separate repo):
|
||||
The library's primary surface is the cache-backed `Library`; the lower-level
|
||||
`Client` sits underneath it and is covered by the Design sections above. The
|
||||
test suite is the canonical, executable documentation — `test/library/` and
|
||||
`test/client/usage.test.ts` walk every operation, and `yarn test` verifies them.
|
||||
`test/client/usage.test.ts` walk most operations, `test/cli/backup.test.ts`
|
||||
walks `lib.backup()`, and `yarn test` verifies them. No test calls
|
||||
`getFileByID()`.
|
||||
|
||||
### Opening a library
|
||||
|
||||
@@ -623,7 +634,7 @@ background, so an unreachable server does not block opening.
|
||||
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
|
||||
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
|
||||
| `precacheOriginalsDays` | `7` | length in days of that latest window |
|
||||
| `cacheOriginalsMaxBytes` | 100 GiB | hard ceiling on the originals cache |
|
||||
| `cacheOriginalsMaxBytes` | 100 GiB | size limit on the originals cache; pinned originals can exceed it |
|
||||
| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down |
|
||||
| `isOriginalPinned` | none | extra predicate for originals that must never be evicted |
|
||||
| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) |
|
||||
@@ -645,17 +656,21 @@ can then be removed.
|
||||
### Default reads vs. fresh reads
|
||||
|
||||
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
|
||||
synchronously from the last refreshed copy held in RAM and never touch the
|
||||
network. The background timer refreshes that copy every
|
||||
`refreshIntervalSeconds`, so a default read is immediate but may be up to one
|
||||
interval stale.
|
||||
synchronously from the copy held in RAM and never touch the network. A refresh
|
||||
changes that copy only once all its server requests have succeeded, and the
|
||||
background timer starts the next refresh `refreshIntervalSeconds` after the
|
||||
previous one ends. So a default read is immediate, but only as current as the
|
||||
last refresh whose requests all succeeded; right after opening an existing
|
||||
cache, it is the copy on disk.
|
||||
|
||||
`await lib.fresh()` forces a refresh, waits for it to complete and persist, and
|
||||
returns the same `{ albums, photos, timeline }` namespaces — now guaranteed to
|
||||
reflect a completed server round-trip. Concurrent `fresh()` calls coalesce onto
|
||||
one refresh, and a refresh that fails rejects the caller (default reads stay
|
||||
silent and keep serving the last good copy). The CLI's read commands use fresh
|
||||
reads (issue https://git.eeqj.de/sneak/quak/issues/75).
|
||||
`await lib.fresh()` waits for a refresh to complete and persist, and returns the
|
||||
same `{ albums, photos, timeline }` namespaces, which then reflect a completed
|
||||
server round-trip. When a refresh is already running, background or not,
|
||||
`fresh()` waits for that one, so its answer can come from requests made before
|
||||
the call; only when none is running does it start one. A refresh that fails
|
||||
rejects the caller (default reads stay silent and keep serving the last good
|
||||
copy). The CLI's read commands use fresh reads (issue
|
||||
https://git.eeqj.de/sneak/quak/issues/75).
|
||||
|
||||
### Read surface
|
||||
|
||||
@@ -672,8 +687,8 @@ reads (issue https://git.eeqj.de/sneak/quak/issues/75).
|
||||
`includeArchived`; hidden photos are always excluded.
|
||||
|
||||
An `Album` exposes its record fields and `album.photos.list()` → `Photo[]`
|
||||
(newest first). A `Photo` exposes its record fields, `photo.record()` →
|
||||
`PhotoRecord`, and two content methods:
|
||||
(newest first). A `Photo` exposes its record fields other than `thumbnailPath`
|
||||
and `originalPath`, `photo.record()` → `PhotoRecord`, and two content methods:
|
||||
|
||||
- `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the
|
||||
full-resolution file. For a live photo, `path` and `bytes` are its image's and
|
||||
@@ -722,16 +737,16 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
`SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
|
||||
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
|
||||
a query vector the caller produced elsewhere.
|
||||
- `await lib.backup(opts?)` → `BackupResult`. It refreshes, fetches every
|
||||
in-scope original not already in the backup (and, with `includeThumbnails`,
|
||||
thumbnails) through the content cache, and rebuilds the on-disk backup tree
|
||||
with a durable failure ledger. A fetched original is written straight into the
|
||||
backup's `originals/` and not into the cache, which then counts it as present;
|
||||
one the cache already held is copied from there. `BackupOptions`:
|
||||
`downloadDirectory` (falls back to the one `open()` was given),
|
||||
`includeOriginals` (default `true`), `includeThumbnails` (default `false`),
|
||||
`onlyAlbumNames`, and `onProgress`. See Backup layout above for the tree it
|
||||
writes.
|
||||
- `await lib.backup(opts?)` → `BackupResult`. It waits for a refresh as
|
||||
`fresh()` does, fetches every in-scope original not already in the backup
|
||||
(and, with `includeThumbnails`, thumbnails) through the content cache, and
|
||||
rebuilds the on-disk backup tree with a durable failure ledger. A fetched
|
||||
original is written straight into the backup's `originals/` and not into the
|
||||
cache, which then counts it as present; one the cache already held is copied
|
||||
from there. `BackupOptions`: `downloadDirectory` (falls back to the one
|
||||
`open()` was given), `includeOriginals` (default `true`), `includeThumbnails`
|
||||
(default `false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above
|
||||
for the tree it writes.
|
||||
|
||||
### Request pools
|
||||
|
||||
|
||||
@@ -25,16 +25,28 @@ declares one.
|
||||
- 2026-09-29: Brought the README and this file in line with `next` after the
|
||||
milestone merge (issue 132). The Next Step says no implementation work is open
|
||||
and the cache design (issue 36) waits on sneak's review. README corrections:
|
||||
`script/lint` and `script/test` have no host path, though `yarn test` does;
|
||||
the SRP handshake uses `fast-srp-hap`, outside `crypto/`; the SRP password is
|
||||
the first 16 bytes of a 32-byte subkey; email OTP replaces SRP when the
|
||||
account has email MFA on; the auth token is sent as URL-safe base64 with
|
||||
padding; each download attempt writes its own temporary files, two for a live
|
||||
photo, and only the attempt that completes renames them into place; `login`,
|
||||
`whoami` and `logout` open no library; `--exif` records XMP and, for a JPEG,
|
||||
EXIF, and no IPTC; which commands take `--json`; `quak backup` still fetches
|
||||
ML data; the default cache directory is the per-user one, not an XDG path on
|
||||
macOS; and `408` and `429` are retried.
|
||||
the Getting Started comments say when the library refreshes; most, not all,
|
||||
Makefile targets call a script; `script/lint` and `script/test` have no host
|
||||
path, though `yarn test` does; the SRP handshake uses `fast-srp-hap`, outside
|
||||
`crypto/`, and a thumbnail upload's MD5 uses `node:crypto`; the SRP password
|
||||
is the first 16 bytes of a 32-byte subkey; the key attributes and token come
|
||||
after SRP, after the TOTP code SRP may ask for, or after the email OTP that
|
||||
replaces SRP when the account has email MFA on, and quak cannot answer a
|
||||
passkey; the auth token is sent as URL-safe base64 with padding; every
|
||||
`TypeError` is retried; each download attempt writes its own temporary files,
|
||||
two for a live photo, and only the attempt that completes renames them into
|
||||
place; `runMetadataBackup` records a failed download in the file's JSON; a
|
||||
second `client.logout()` does not throw; `quak logout` with no session exits
|
||||
0, and names the cache directory only when it knows it; `login`, `whoami` and
|
||||
`logout` open no library; `--exif` records XMP and, for a JPEG, EXIF, and no
|
||||
IPTC; which commands take `--json`; a failed ML data request may not have been
|
||||
retried; the thumbnail fixer is not limited to baseline JPEG; `quak backup`
|
||||
still fetches ML data; the default cache directory is the per-user one, not an
|
||||
XDG path on macOS; pinned originals can exceed `cacheOriginalsMaxBytes`; which
|
||||
tests cover which operations; a default read is only as current as the last
|
||||
refresh whose requests all succeeded; `fresh()` and `lib.backup()` join a
|
||||
refresh already running; a `Photo` has no `thumbnailPath` or `originalPath`;
|
||||
and `408` and `429` are retried.
|
||||
|
||||
- 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks
|
||||
that the image's and the video's temp files are fsynced before either is
|
||||
|
||||
Reference in New Issue
Block a user