Compare commits
1
Commits
main
..
e5a83cb190
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e5a83cb190 |
@@ -8,11 +8,12 @@ and downloads individual images while decrypting them on the way to disk.
|
|||||||
quak also includes a resilient backup command that downloads every file in the
|
quak also includes a resilient backup command that downloads every file in the
|
||||||
account into a deduplicated local directory tree, skipping files that already
|
account into a deduplicated local directory tree, skipping files that already
|
||||||
exist on disk and continuing past individual download failures instead of
|
exist on disk and continuing past individual download failures instead of
|
||||||
crashing. It decrypts and persists all three metadata layers (basic, private
|
crashing. For each file it persists the basic metadata fields quak keeps (title,
|
||||||
magic, public magic) per file, including camera info, GPS coordinates, captions,
|
file type, creation and modification time, latitude, longitude, content hash),
|
||||||
and any face/keyword labels the Ente clients have added. A helper subcommand can
|
and the private and public magic metadata in full, including camera info, GPS
|
||||||
detect and regenerate missing thumbnails, encrypting and uploading them back to
|
coordinates, captions, and any face/keyword labels the Ente clients have added.
|
||||||
the server.
|
A helper subcommand can detect and regenerate missing thumbnails, encrypting and
|
||||||
|
uploading them back to the server.
|
||||||
|
|
||||||
## Getting Started
|
## Getting Started
|
||||||
|
|
||||||
@@ -51,7 +52,8 @@ const client = await Client.login({
|
|||||||
|
|
||||||
// Open a cache-backed library. On an empty cache this awaits one server
|
// Open a cache-backed library. On an empty cache this awaits one server
|
||||||
// refresh; on an existing cache it returns immediately and refreshes in the
|
// refresh; on an existing cache it returns immediately and refreshes in the
|
||||||
// background every `refreshIntervalSeconds` (default 3).
|
// background. Later refreshes start `refreshIntervalSeconds` (default 3) after
|
||||||
|
// the previous one ends.
|
||||||
const lib = await Library.open({ client });
|
const lib = await Library.open({ client });
|
||||||
|
|
||||||
// Default reads answer synchronously from the local cache — no network.
|
// Default reads answer synchronously from the local cache — no network.
|
||||||
@@ -62,7 +64,7 @@ for (const album of lib.albums.list()) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Fresh reads await a server round-trip and answer with current state.
|
// Fresh reads await a server round-trip, joining one already running.
|
||||||
const { albums } = await lib.fresh();
|
const { albums } = await lib.fresh();
|
||||||
console.log(`${albums.list().length} albums as of now`);
|
console.log(`${albums.list().length} albums as of now`);
|
||||||
|
|
||||||
@@ -84,7 +86,7 @@ enumeration/download calls) is exported too and documented under Design below.
|
|||||||
This repository adheres to the
|
This repository adheres to the
|
||||||
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
||||||
standard: normalized scripts in `script/` are the entrypoints for the
|
standard: normalized scripts in `script/` are the entrypoints for the
|
||||||
development workflow, and the Makefile targets are thin shims that call them.
|
development workflow, and most Makefile targets are thin shims that call them.
|
||||||
The scripts are POSIX sh (not bash) so they run in minimal containers such as
|
The scripts are POSIX sh (not bash) so they run in minimal containers such as
|
||||||
alpine. We provide:
|
alpine. We provide:
|
||||||
|
|
||||||
@@ -122,9 +124,9 @@ alpine. We provide:
|
|||||||
Linting and testing are phases of the `Dockerfile`. The `lint` phase copies the
|
Linting and testing are phases of the `Dockerfile`. The `lint` phase copies the
|
||||||
repo into a digest-pinned node image and runs eslint and `prettier --check .`;
|
repo into a digest-pinned node image and runs eslint and `prettier --check .`;
|
||||||
the `test` phase does the same with the suite. `script/lint` and `script/test`
|
the `test` phase does the same with the suite. `script/lint` and `script/test`
|
||||||
each build one phase with `docker build --no-cache --target <phase>`. There is
|
each build one phase with `docker build --no-cache --target <phase>`. Neither
|
||||||
no host lint or test path: docker is required, and that also works where the
|
script runs the tools on the host: docker is required, and that also works where
|
||||||
docker daemon is remote and bind mounts are impossible.
|
the docker daemon is remote and bind mounts are impossible.
|
||||||
|
|
||||||
The last stage of the `Dockerfile` compiles the package, and it copies a file
|
The last stage of the `Dockerfile` compiles the package, and it copies a file
|
||||||
from each phase, so it cannot be built unless lint and the tests pass. That is
|
from each phase, so it cannot be built unless lint and the tests pass. That is
|
||||||
@@ -209,7 +211,7 @@ the CLI is for humans.
|
|||||||
```
|
```
|
||||||
quak/
|
quak/
|
||||||
src/
|
src/
|
||||||
crypto/ libsodium primitives (boxes, secretstreams, KDF, SRP)
|
crypto/ libsodium primitives (boxes, secretstreams, KDF, hash)
|
||||||
api/ HTTP client (ApiClient class)
|
api/ HTTP client (ApiClient class)
|
||||||
auth/ login flow (SRP + email OTP + TOTP), key unwrap
|
auth/ login flow (SRP + email OTP + TOTP), key unwrap
|
||||||
model/ decrypted Collection, File, Metadata types + decrypt fns
|
model/ decrypted Collection, File, Metadata types + decrypt fns
|
||||||
@@ -219,7 +221,7 @@ quak/
|
|||||||
and search, request pools
|
and search, request pools
|
||||||
backup.ts resilient full-account backup with dedup
|
backup.ts resilient full-account backup with dedup
|
||||||
metadata-backup.ts
|
metadata-backup.ts
|
||||||
backup-metadata: all decrypted metadata as JSON
|
backup-metadata: the metadata quak keeps, as JSON
|
||||||
mldata-fetch.ts fetch + decrypt per-file ML data
|
mldata-fetch.ts fetch + decrypt per-file ML data
|
||||||
filename.ts safe file names from server metadata
|
filename.ts safe file names from server metadata
|
||||||
errors.ts error types shared across layers
|
errors.ts error types shared across layers
|
||||||
@@ -250,7 +252,9 @@ the repository root rather than `src/`, because `bin/` is compiled too and
|
|||||||
### Cryptography
|
### Cryptography
|
||||||
|
|
||||||
All cryptography is done by `libsodium-wrappers-sumo` (the "sumo" build is
|
All cryptography is done by `libsodium-wrappers-sumo` (the "sumo" build is
|
||||||
required for `crypto_pwhash` / Argon2id). No hand-rolled crypto.
|
required for `crypto_pwhash` / Argon2id), except the SRP handshake, which uses
|
||||||
|
`fast-srp-hap`, and the MD5 checksum sent with a thumbnail upload, which uses
|
||||||
|
Node's built-in `node:crypto`. No hand-rolled crypto.
|
||||||
|
|
||||||
The key hierarchy, derived during login, is:
|
The key hierarchy, derived during login, is:
|
||||||
|
|
||||||
@@ -258,19 +262,23 @@ The key hierarchy, derived during login, is:
|
|||||||
2. Argon2id (`crypto_pwhash`) over the password and a server-issued `kekSalt`,
|
2. Argon2id (`crypto_pwhash`) over the password and a server-issued `kekSalt`,
|
||||||
with server-issued `memLimit` and `opsLimit`, produces a 32-byte Key
|
with server-issued `memLimit` and `opsLimit`, produces a 32-byte Key
|
||||||
Encryption Key (KEK).
|
Encryption Key (KEK).
|
||||||
3. SRP login: a 16-byte SRP login subkey is derived from the KEK using
|
3. SRP login: `crypto_kdf_derive_from_key` (BLAKE2b) derives a 32-byte subkey
|
||||||
`crypto_kdf_derive_from_key` (BLAKE2b) with subkey id 1 and context
|
from the KEK with subkey id 1 and context `loginctx`. Its first 16 bytes are
|
||||||
`loginctx`. That 16-byte value is the SRP password.
|
the SRP password.
|
||||||
4. After SRP completes (or after email-OTP fallback), the server returns a blob
|
4. When SRP completes, the server returns a blob of "key attributes" plus an
|
||||||
of "key attributes" plus an encrypted auth token.
|
encrypted auth token, or first asks for a second factor. quak answers a TOTP
|
||||||
|
request with the code (`POST /users/two-factor/verify`), after which the
|
||||||
|
server returns them, and cannot answer a passkey request. When the account
|
||||||
|
has email MFA on (`isEmailMFAEnabled`), an email OTP replaces SRP and the
|
||||||
|
server returns them after it.
|
||||||
5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK
|
5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK
|
||||||
yields the 32-byte master key.
|
yields the 32-byte master key.
|
||||||
6. `crypto_secretbox_open_easy` over the encrypted secret key with the master
|
6. `crypto_secretbox_open_easy` over the encrypted secret key with the master
|
||||||
key yields the user's X25519 private key. The matching public key is
|
key yields the user's X25519 private key. The matching public key is
|
||||||
delivered in cleartext.
|
delivered in cleartext.
|
||||||
7. `crypto_box_seal_open` over the encrypted token with the user's keypair
|
7. `crypto_box_seal_open` over the encrypted token with the user's keypair
|
||||||
yields the URL-safe base64 auth token used in `X-Auth-Token` for all
|
yields the auth token's bytes. Encoded as URL-safe base64 with padding, they
|
||||||
subsequent calls.
|
are the `X-Auth-Token` value for all subsequent calls.
|
||||||
|
|
||||||
Per-collection keys are decrypted with `crypto_secretbox_open_easy` using the
|
Per-collection keys are decrypted with `crypto_secretbox_open_easy` using the
|
||||||
master key (for owned collections). Per-file keys are decrypted with
|
master key (for owned collections). Per-file keys are decrypted with
|
||||||
@@ -306,7 +314,8 @@ Endpoints used:
|
|||||||
- `POST /users/srp/create-session`: begin SRP handshake.
|
- `POST /users/srp/create-session`: begin SRP handshake.
|
||||||
- `POST /users/srp/verify-session`: complete SRP, receive 2FA challenge or the
|
- `POST /users/srp/verify-session`: complete SRP, receive 2FA challenge or the
|
||||||
encrypted token plus key attributes.
|
encrypted token plus key attributes.
|
||||||
- `POST /users/ott` and `POST /users/verify-email`: email OTP fallback path.
|
- `POST /users/ott` and `POST /users/verify-email`: email OTP, used instead of
|
||||||
|
SRP when the SRP attributes have `isEmailMFAEnabled` set.
|
||||||
- `POST /users/two-factor/verify`: TOTP second factor.
|
- `POST /users/two-factor/verify`: TOTP second factor.
|
||||||
- `POST /users/logout`: end the calling token's session (`quak logout`).
|
- `POST /users/logout`: end the calling token's session (`quak logout`).
|
||||||
- `GET /collections/v2?sinceTime=<usec>`: list collections changed since
|
- `GET /collections/v2?sinceTime=<usec>`: list collections changed since
|
||||||
@@ -328,8 +337,9 @@ request is repeated only when repeating it could produce a different answer:
|
|||||||
- Every other 4xx: not retried. A 404 in particular is an answer, and
|
- Every other 4xx: not retried. A 404 in particular is an answer, and
|
||||||
`listMissingThumbnails` depends on getting it promptly and once.
|
`listMissingThumbnails` depends on getting it promptly and once.
|
||||||
- Transport failures — a `fetch` rejection, `ECONNRESET`, `ETIMEDOUT`, a DNS or
|
- Transport failures — a `fetch` rejection, `ECONNRESET`, `ETIMEDOUT`, a DNS or
|
||||||
TLS failure — and deadline aborts: retried. The errno is looked for in the
|
TLS failure — and deadline aborts: retried. Node's `fetch` rejects with a
|
||||||
error's `cause` chain, because that is where Node's `fetch` puts it.
|
plain `TypeError`, so every `TypeError` is retried. The errno is looked for in
|
||||||
|
the error's `cause` chain, because that is where Node's `fetch` puts it.
|
||||||
- A truncated download: retried.
|
- A truncated download: retried.
|
||||||
- Anything else, including a secretstream authentication failure that is not
|
- Anything else, including a secretstream authentication failure that is not
|
||||||
truncation: not retried. The default answer is no. For a backup tool, retrying
|
truncation: not retried. The default answer is no. For a backup tool, retrying
|
||||||
@@ -395,10 +405,13 @@ because a socket reset after the response headers have arrived surfaces in the
|
|||||||
download layer rather than in `ApiClient`, and that is the common failure for
|
download layer rather than in `ApiClient`, and that is the common failure for
|
||||||
multi-megabyte photos over a CDN. The secretstream pull state is not resumable
|
multi-megabyte photos over a CDN. The secretstream pull state is not resumable
|
||||||
and these endpoints have no Range support, so a retry starts the file over. The
|
and these endpoints have no Range support, so a retry starts the file over. The
|
||||||
atomic write stays outside the retry, so a download that needed three attempts
|
atomic write is part of the retried unit: each attempt writes its own temporary
|
||||||
still performs exactly one write and one rename. `runBackup` and
|
files, one for most files and two for a live photo (its image and its video),
|
||||||
`runMetadataBackup` are unchanged: the retry sits below them, and a file that
|
and removes them if it fails. Only the attempt that completes renames anything
|
||||||
fails after exhausting it is still logged, counted, and stepped over.
|
into place. The retry sits below `runBackup` and `runMetadataBackup`.
|
||||||
|
`runBackup` logs, counts and steps over a file that still fails after its
|
||||||
|
retries; `runMetadataBackup`, which downloads only with `--exif`, records the
|
||||||
|
error in that file's JSON as `imageMetadataError` and goes on.
|
||||||
|
|
||||||
One imprecision is deliberate and worth knowing about. When a body ends part-way
|
One imprecision is deliberate and worth knowing about. When a body ends part-way
|
||||||
through a secretstream chunk, Poly1305 fails and carries no framing signal, so a
|
through a secretstream chunk, Poly1305 fails and carries no framing signal, so a
|
||||||
@@ -422,10 +435,11 @@ base64-encoded keys) that the consumer can write to disk, a database, or
|
|||||||
whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
|
whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
|
||||||
working client from that snapshot without re-authenticating; it checks every
|
working client from that snapshot without re-authenticating; it checks every
|
||||||
field and each key's length first, and throws an error naming the bad field.
|
field and each key's length first, and throws an error naming the bad field.
|
||||||
`client.logout()` clears the token and zeroes the key buffers in place; every
|
`client.logout()` clears the token and zeroes the key buffers in place; after
|
||||||
later call on that client throws. It does not contact the server, so the token
|
it, every other method on that client throws. It does not contact the server, so
|
||||||
stays valid there and in any saved snapshot; `await client.logoutOnServer()`
|
the token stays valid there and in any saved snapshot;
|
||||||
first ends the session on the server (`POST /users/logout`).
|
`await client.logoutOnServer()` first ends the session on the server
|
||||||
|
(`POST /users/logout`).
|
||||||
|
|
||||||
The CLI stores the snapshot at the platform-appropriate data directory via
|
The CLI stores the snapshot at the platform-appropriate data directory via
|
||||||
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
|
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
|
||||||
@@ -433,15 +447,17 @@ The CLI stores the snapshot at the platform-appropriate data directory via
|
|||||||
`0600`. The key material is stored in cleartext in the JSON; treat this file as
|
`0600`. The key material is stored in cleartext in the JSON; treat this file as
|
||||||
you would treat the password itself. A missing file is reported as "not logged
|
you would treat the password itself. A missing file is reported as "not logged
|
||||||
in"; a file that exists but is corrupt is reported as such, naming the bad
|
in"; a file that exists but is corrupt is reported as such, naming the bad
|
||||||
field. Both exit with status 1.
|
field. Both exit with status 1, except that `quak logout` with no file says
|
||||||
|
there is no session and exits 0.
|
||||||
|
|
||||||
`quak logout` ends the session on the server, so the token in `session.json`
|
`quak logout` ends the session on the server, so the token in `session.json`
|
||||||
stops working even in a copy of the file, and then deletes the file. If the
|
stops working even in a copy of the file, and then deletes the file. If the
|
||||||
server call fails (or the file is corrupt), the file is still deleted, the
|
server call fails (or the file is corrupt), the file is still deleted, the
|
||||||
command says the server session could not be ended, and it exits with status 1.
|
command says the server session could not be ended, and it exits with status 1.
|
||||||
It does not delete the cache: it prints the account's cache directory and says
|
It does not delete the cache. When it knows the cache directory, from
|
||||||
it still holds decrypted data (file keys in `metadata.json`, cached originals
|
`--cache-dir` or from a session file it could read, it prints it and says it
|
||||||
and thumbnails), for the user to delete if they want it gone.
|
still holds decrypted data (file keys in `metadata.json`, cached originals and
|
||||||
|
thumbnails), for the user to delete if they want it gone.
|
||||||
|
|
||||||
### CLI surface
|
### CLI surface
|
||||||
|
|
||||||
@@ -455,19 +471,19 @@ quak files --collection <id> [--json] list files in a collection
|
|||||||
quak get <fileID> [--out path] [--collection] download and decrypt a file
|
quak get <fileID> [--out path] [--collection] download and decrypt a file
|
||||||
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
|
quak get-thumb <fileID> [--out] [--collection] download and decrypt a thumbnail
|
||||||
quak backup <dir> [--json] full incremental backup
|
quak backup <dir> [--json] full incremental backup
|
||||||
quak backup-metadata <dir> [--exif] dump all decrypted metadata as JSON
|
quak backup-metadata <dir> [--exif] dump the metadata quak keeps as JSON
|
||||||
quak helper list-missing-thumbnails [--json] find files with missing thumbnails
|
quak helper list-missing-thumbnails [--json] find files with missing thumbnails
|
||||||
quak helper fix-missing-thumbnails [--file ids] generate + upload missing thumbnails
|
quak helper fix-missing-thumbnails [--file ids] [--json] generate + upload missing thumbnails
|
||||||
```
|
```
|
||||||
|
|
||||||
Every command runs on the same cache-backed library. The read commands —
|
Every command except `login`, `whoami` and `logout` runs on the cache-backed
|
||||||
`collections`, `files`, `get`, `get-thumb`, `backup-metadata`,
|
library. The read commands — `collections`, `files`, `get`, `get-thumb`,
|
||||||
`helper list-missing-thumbnails` and `helper fix-missing-thumbnails` — force a
|
`backup-metadata`, `helper list-missing-thumbnails` and
|
||||||
fresh server round-trip before they answer, so they report current account state
|
`helper fix-missing-thumbnails` — force a fresh server round-trip before they
|
||||||
rather than whatever the cache last held. If that round-trip fails, the command
|
answer, so they report current account state rather than whatever the cache last
|
||||||
prints the error on one line and exits 1. `--cache-dir` overrides where the
|
held. If that round-trip fails, the command prints the error on one line and
|
||||||
cache lives; without it each account gets its own directory under the per-user
|
exits 1. `--cache-dir` overrides where the cache lives; without it each account
|
||||||
cache path.
|
gets its own directory under the per-user cache path.
|
||||||
|
|
||||||
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
|
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
|
||||||
accepted for backward compatibility but ignored. For a live photo, `get` writes
|
accepted for backward compatibility but ignored. For a live photo, `get` writes
|
||||||
@@ -475,18 +491,20 @@ its image and its video, each named after the title with its own extension, as
|
|||||||
Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`). With
|
Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`). With
|
||||||
`--out PATH`, the image is written to `PATH` and the video beside it, with
|
`--out PATH`, the image is written to `PATH` and the video beside it, with
|
||||||
`PATH`'s name and the video's extension; a `PATH` with the video's extension is
|
`PATH`'s name and the video's extension; a `PATH` with the video's extension is
|
||||||
refused. `backup-metadata --exif` (alias `--all`) additionally downloads each
|
refused. `backup-metadata --exif` (alias `--all`) additionally fetches each
|
||||||
file to extract full EXIF/IPTC/XMP metadata. The listing and backup commands
|
file's original through the cache and records its XMP metadata and, for a JPEG,
|
||||||
support `--json` for machine-readable output.
|
its EXIF metadata and dimensions. `collections`, `files`, `backup`,
|
||||||
|
`helper list-missing-thumbnails` and `helper fix-missing-thumbnails` take
|
||||||
|
`--json` for machine-readable output.
|
||||||
|
|
||||||
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
||||||
still fails after its retries, the error is logged, each of its files is written
|
fails, the error is logged, each of its files is written with the reason in an
|
||||||
with the reason in an `mlDataError` field instead of `mlData`, and the dump goes
|
`mlDataError` field instead of `mlData`, and the dump goes on. The exit code is
|
||||||
on. The exit code is non-zero if any ML data request failed.
|
non-zero if any ML data request failed.
|
||||||
|
|
||||||
`helper fix-missing-thumbnails` regenerates thumbnails for baseline JPEG images
|
`helper fix-missing-thumbnails` regenerates thumbnails for JPEG images only,
|
||||||
only, because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG
|
because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG image
|
||||||
image (PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept
|
(PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept
|
||||||
distinct from a `failed` repair, and does not affect the exit code; a genuine
|
distinct from a `failed` repair, and does not affect the exit code; a genuine
|
||||||
failure still exits non-zero. The server accepts a new thumbnail only from the
|
failure still exits non-zero. The server accepts a new thumbnail only from the
|
||||||
file's owner and only when it is no larger than the thumbnail size it records
|
file's owner and only when it is no larger than the thumbnail size it records
|
||||||
@@ -505,7 +523,9 @@ the smallest does not.
|
|||||||
originals/
|
originals/
|
||||||
<fileID>.<ext> actual file content (one per unique file,
|
<fileID>.<ext> actual file content (one per unique file,
|
||||||
two for a live photo: see below)
|
two for a live photo: see below)
|
||||||
<fileID>.json all decrypted metadata for that file
|
<fileID>.json the file's basic metadata fields quak
|
||||||
|
keeps, and its private and public magic
|
||||||
|
metadata
|
||||||
<fileID>.livephoto.json which of a live photo's two files is which
|
<fileID>.livephoto.json which of a live photo's two files is which
|
||||||
collections/
|
collections/
|
||||||
<name>/
|
<name>/
|
||||||
@@ -550,7 +570,9 @@ the cache. An original the cache already held is copied from there instead. On
|
|||||||
subsequent runs, existing originals are skipped. If a download fails, the error
|
subsequent runs, existing originals are skipped. If a download fails, the error
|
||||||
is logged and the backup continues with the next file. The exit code is non-zero
|
is logged and the backup continues with the next file. The exit code is non-zero
|
||||||
if any files failed. `quak backup` opens its library with the thumbnail and
|
if any files failed. `quak backup` opens its library with the thumbnail and
|
||||||
originals precache off, so it fetches only what the backup stores.
|
originals precache off, so the only file content it fetches is the originals the
|
||||||
|
backup stores. The library's ML data fetch still runs and fills the cache's
|
||||||
|
`mldata/`.
|
||||||
|
|
||||||
Each original is written to a temporary file in the same directory, synced to
|
Each original is written to a temporary file in the same directory, synced to
|
||||||
disk, and renamed into place, so an original is either complete or absent, even
|
disk, and renamed into place, so an original is either complete or absent, even
|
||||||
@@ -567,8 +589,8 @@ temporary file's permissions, not those of the file it replaced.
|
|||||||
|
|
||||||
## TODO
|
## TODO
|
||||||
|
|
||||||
- [x] Retry policy: no retry on 4xx, exponential backoff on 5xx and network
|
- [x] Retry policy: no retry on 4xx (except `408` and `429`), exponential
|
||||||
errors
|
backoff on 5xx and network errors
|
||||||
- [x] Update the API reference section below to match the current implementation
|
- [x] Update the API reference section below to match the current implementation
|
||||||
- [x] `make docker` green
|
- [x] `make docker` green
|
||||||
- [x] Store live photos in a form a photo viewer can open
|
- [x] Store live photos in a form a photo viewer can open
|
||||||
@@ -591,7 +613,8 @@ Future (desktop client, separate repo):
|
|||||||
The library's primary surface is the cache-backed `Library`; the lower-level
|
The library's primary surface is the cache-backed `Library`; the lower-level
|
||||||
`Client` sits underneath it and is covered by the Design sections above. The
|
`Client` sits underneath it and is covered by the Design sections above. The
|
||||||
test suite is the canonical, executable documentation — `test/library/` and
|
test suite is the canonical, executable documentation — `test/library/` and
|
||||||
`test/client/usage.test.ts` walk every operation, and `yarn test` verifies them.
|
`test/client/usage.test.ts` walk most operations, `test/cli/backup.test.ts`
|
||||||
|
walks `lib.backup()`, and `yarn test` verifies them.
|
||||||
|
|
||||||
### Opening a library
|
### Opening a library
|
||||||
|
|
||||||
@@ -605,15 +628,15 @@ background, so an unreachable server does not block opening.
|
|||||||
`LibraryOptions`:
|
`LibraryOptions`:
|
||||||
|
|
||||||
| Option | Default | Meaning |
|
| Option | Default | Meaning |
|
||||||
| ------------------------ | --------------------------- | --------------------------------------------------------------------- |
|
| ------------------------ | -------------------------------------------------- | --------------------------------------------------------------------- |
|
||||||
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
|
| `client` | required | the account client (a `Client`, or any `LibraryClient`) |
|
||||||
| `cacheDirectory` | `<XDG cache>/quak/<userID>` | where `metadata.json` and the content cache live |
|
| `cacheDirectory` | `quak/<userID>` under the per-user cache directory | where `metadata.json` and the content cache live |
|
||||||
| `downloadDirectory` | none | backup destination; an original already stored there counts as cached |
|
| `downloadDirectory` | none | backup destination; an original already stored there counts as cached |
|
||||||
| `refreshIntervalSeconds` | `3` | background refresh cadence |
|
| `refreshIntervalSeconds` | `3` | background refresh cadence |
|
||||||
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
|
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
|
||||||
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
|
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
|
||||||
| `precacheOriginalsDays` | `7` | length in days of that latest window |
|
| `precacheOriginalsDays` | `7` | length in days of that latest window |
|
||||||
| `cacheOriginalsMaxBytes` | 100 GiB | hard ceiling on the originals cache |
|
| `cacheOriginalsMaxBytes` | 100 GiB | size limit on the originals cache; pinned originals can exceed it |
|
||||||
| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down |
|
| `freeBelowBytes` | 50 GiB | free space to protect on the volume; the effective limit adapts down |
|
||||||
| `isOriginalPinned` | none | extra predicate for originals that must never be evicted |
|
| `isOriginalPinned` | none | extra predicate for originals that must never be evicted |
|
||||||
| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) |
|
| `pools` | fresh `RequestPools` | the bounded request pools (sets concurrency) |
|
||||||
@@ -635,17 +658,21 @@ can then be removed.
|
|||||||
### Default reads vs. fresh reads
|
### Default reads vs. fresh reads
|
||||||
|
|
||||||
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
|
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
|
||||||
synchronously from the last refreshed copy held in RAM and never touch the
|
synchronously from the copy held in RAM and never touch the network. A refresh
|
||||||
network. The background timer refreshes that copy every
|
changes that copy only once all its server requests have succeeded, and the
|
||||||
`refreshIntervalSeconds`, so a default read is immediate but may be up to one
|
background timer starts the next refresh `refreshIntervalSeconds` after the
|
||||||
interval stale.
|
previous one ends. So a default read is immediate, but only as current as the
|
||||||
|
last refresh whose requests all succeeded; right after opening an existing
|
||||||
|
cache, it is the copy on disk.
|
||||||
|
|
||||||
`await lib.fresh()` forces a refresh, waits for it to complete and persist, and
|
`await lib.fresh()` waits for a refresh to complete and persist, and returns the
|
||||||
returns the same `{ albums, photos, timeline }` namespaces — now guaranteed to
|
same `{ albums, photos, timeline }` namespaces, which then reflect a completed
|
||||||
reflect a completed server round-trip. Concurrent `fresh()` calls coalesce onto
|
server round-trip. When a refresh is already running, background or not,
|
||||||
one refresh, and a refresh that fails rejects the caller (default reads stay
|
`fresh()` waits for that one, so its answer can come from requests made before
|
||||||
silent and keep serving the last good copy). The CLI's read commands use fresh
|
the call; only when none is running does it start one. A refresh that fails
|
||||||
reads (issue https://git.eeqj.de/sneak/quak/issues/75).
|
rejects the caller (default reads stay silent and keep serving the last good
|
||||||
|
copy). The CLI's read commands use fresh reads (issue
|
||||||
|
https://git.eeqj.de/sneak/quak/issues/75).
|
||||||
|
|
||||||
### Read surface
|
### Read surface
|
||||||
|
|
||||||
@@ -662,8 +689,8 @@ reads (issue https://git.eeqj.de/sneak/quak/issues/75).
|
|||||||
`includeArchived`; hidden photos are always excluded.
|
`includeArchived`; hidden photos are always excluded.
|
||||||
|
|
||||||
An `Album` exposes its record fields and `album.photos.list()` → `Photo[]`
|
An `Album` exposes its record fields and `album.photos.list()` → `Photo[]`
|
||||||
(newest first). A `Photo` exposes its record fields, `photo.record()` →
|
(newest first). A `Photo` exposes its record fields other than `thumbnailPath`
|
||||||
`PhotoRecord`, and two content methods:
|
and `originalPath`, `photo.record()` → `PhotoRecord`, and two content methods:
|
||||||
|
|
||||||
- `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the
|
- `await photo.original(opts?)` → `{ path, bytes, videoPath? }` — the
|
||||||
full-resolution file. For a live photo, `path` and `bytes` are its image's and
|
full-resolution file. For a live photo, `path` and `bytes` are its image's and
|
||||||
@@ -712,16 +739,16 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
|||||||
`SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
|
`SimilarResult[]` (`{ fileID, score }`, cosine similarity, most similar first,
|
||||||
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
|
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
|
||||||
a query vector the caller produced elsewhere.
|
a query vector the caller produced elsewhere.
|
||||||
- `await lib.backup(opts?)` → `BackupResult`. It refreshes, fetches every
|
- `await lib.backup(opts?)` → `BackupResult`. It waits for a refresh as
|
||||||
in-scope original not already in the backup (and, with `includeThumbnails`,
|
`fresh()` does, fetches every in-scope original not already in the backup
|
||||||
thumbnails) through the content cache, and rebuilds the on-disk backup tree
|
(and, with `includeThumbnails`, thumbnails) through the content cache, and
|
||||||
with a durable failure ledger. A fetched original is written straight into the
|
rebuilds the on-disk backup tree with a durable failure ledger. A fetched
|
||||||
backup's `originals/` and not into the cache, which then counts it as present;
|
original is written straight into the backup's `originals/` and not into the
|
||||||
one the cache already held is copied from there. `BackupOptions`:
|
cache, which then counts it as present; one the cache already held is copied
|
||||||
`downloadDirectory` (falls back to the one `open()` was given),
|
from there. `BackupOptions`: `downloadDirectory` (falls back to the one
|
||||||
`includeOriginals` (default `true`), `includeThumbnails` (default `false`),
|
`open()` was given), `includeOriginals` (default `true`), `includeThumbnails`
|
||||||
`onlyAlbumNames`, and `onProgress`. See Backup layout above for the tree it
|
(default `false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above
|
||||||
writes.
|
for the tree it writes.
|
||||||
|
|
||||||
### Request pools
|
### Request pools
|
||||||
|
|
||||||
|
|||||||
@@ -14,13 +14,42 @@ pre-1.0
|
|||||||
|
|
||||||
# Next Step
|
# Next Step
|
||||||
|
|
||||||
None: every issue still open is done on `next` and waits for it to reach `main`.
|
None: no implementation work is open. The cache design,
|
||||||
|
https://git.eeqj.de/sneak/quak/issues/36, waits on sneak's review.
|
||||||
|
|
||||||
Tagging and releases are decided by sneak alone, and happen only when he
|
Tagging and releases are decided by sneak alone, and happen only when he
|
||||||
declares one.
|
declares one.
|
||||||
|
|
||||||
# Completed Steps
|
# Completed Steps
|
||||||
|
|
||||||
|
- 2026-09-29: Brought the README and this file in line with `next` after the
|
||||||
|
milestone merge (issue 132). The Next Step says no implementation work is open
|
||||||
|
and the cache design (issue 36) waits on sneak's review. README corrections:
|
||||||
|
the Getting Started comments say when the library refreshes; most, not all,
|
||||||
|
Makefile targets call a script; `script/lint` and `script/test` have no host
|
||||||
|
path, though `yarn test` does; the SRP handshake uses `fast-srp-hap`, outside
|
||||||
|
`crypto/`, and a thumbnail upload's MD5 uses `node:crypto`; the SRP password
|
||||||
|
is the first 16 bytes of a 32-byte subkey; the key attributes and token come
|
||||||
|
after SRP, after the TOTP code SRP may ask for, or after the email OTP that
|
||||||
|
replaces SRP when the account has email MFA on, and quak cannot answer a
|
||||||
|
passkey; the auth token is sent as URL-safe base64 with padding; every
|
||||||
|
`TypeError` is retried; each download attempt writes its own temporary files,
|
||||||
|
two for a live photo, and only the attempt that completes renames them into
|
||||||
|
place; `runMetadataBackup` records a failed download in the file's JSON; a
|
||||||
|
second `client.logout()` does not throw; `quak logout` with no session exits
|
||||||
|
0, and names the cache directory only when it knows it; `login`, `whoami` and
|
||||||
|
`logout` open no library; `--exif` records XMP and, for a JPEG, EXIF, and no
|
||||||
|
IPTC; which commands take `--json`; a failed ML data request may not have been
|
||||||
|
retried; the thumbnail fixer is not limited to baseline JPEG; `quak backup`
|
||||||
|
still fetches ML data; a backup's JSON holds the basic metadata fields quak
|
||||||
|
keeps and the private and public magic metadata, not every decrypted field;
|
||||||
|
the default cache directory is the per-user one, not an XDG path on macOS;
|
||||||
|
pinned originals can exceed `cacheOriginalsMaxBytes`; which tests cover which
|
||||||
|
operations; a default read is only as current as the last refresh whose
|
||||||
|
requests all succeeded; `fresh()` and `lib.backup()` join a refresh already
|
||||||
|
running; a `Photo` has no `thumbnailPath` or `originalPath`; and `408` and
|
||||||
|
`429` are retried.
|
||||||
|
|
||||||
- 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks
|
- 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks
|
||||||
that the image's and the video's temp files are fsynced before either is
|
that the image's and the video's temp files are fsynced before either is
|
||||||
renamed into place, and the directory after both renames, as the `writeAtomic`
|
renamed into place, and the directory after both renames, as the `writeAtomic`
|
||||||
|
|||||||
Reference in New Issue
Block a user