1 Commits
Author SHA1 Message Date
sneak 6d9852af13 README.md rewritten sentence by sentence (closes #219)
check / check (push) Canceled after 0s
The old text served as the list of what a reader needs covered. The new
text leads with what quak is, what it is for and how to use it. Every
command, option, exit code and file layout follows in reference sections,
then the library reference, then design, entrypoints and development.
Every statement was checked against the code on next; the places where
the old text disagreed with it are listed in the pull request. No sentence
runs past about 30 words, every table is now a list, and no issue links or
project history remain in text meant for users. Two code comments that
named README sections now name the new ones.

Model: fable-5-1
2026-10-26 12:11:36 +00:00
3 changed files with 158 additions and 202 deletions
+150 -191
View File
@@ -73,7 +73,8 @@ and `dist/` is not committed. Clone and build it as above, then add the clone to
your own program's `package.json` by its path, for example your own program's `package.json` by its path, for example
`yarn add file:../quak`. The package's `main`, `types` and `exports` fields `yarn add file:../quak`. The package's `main`, `types` and `exports` fields
point into `dist/`, so the clone must be built first. The package is an ES point into `dist/`, so the clone must be built first. The package is an ES
module only, so a CommonJS program cannot `require` it. module only: `package.json` sets `"type": "module"`, and its `exports` field
offers only `import`, so a CommonJS program cannot `require` it.
```ts ```ts
import { Client, Library } from "quak"; import { Client, Library } from "quak";
@@ -89,9 +90,8 @@ const client = await Client.login({
const lib = await Library.open({ const lib = await Library.open({
client, client,
// Without these two, open() also starts downloading every thumbnail in // Without these two, open() also starts downloading every thumbnail in
// the account, and the originals of the favorites album and the latest // the account and the latest week's originals into the cache. It fetches
// week, into the cache. It fetches machine-learning data after each // machine-learning data after each refresh either way.
// refresh either way.
precacheThumbnails: false, precacheThumbnails: false,
precacheOriginals: false, precacheOriginals: false,
}); });
@@ -122,12 +122,10 @@ await lib.close();
for. It refreshes every 3 seconds until `close()`. It also downloads, in the for. It refreshes every 3 seconds until `close()`. It also downloads, in the
background, every thumbnail in the account and the originals of the favorites background, every thumbnail in the account and the originals of the favorites
album and of the latest week (see "Opening a library"). The example turns those album and of the latest week (see "Opening a library"). The example turns those
two downloads off. After each refresh that succeeds, the library also fetches two downloads off. After each refresh the library also fetches each file's
each file's machine-learning data into the cache. That is the faces and the machine-learning data into the cache. That is the faces and the other data
other data Ente's apps compute for a photo (see "The JSON beside each Ente's apps compute for a photo (see "The JSON beside each original"). No option
original"). No option in `LibraryOptions` turns this off. A `client` without a turns this off.
`fetchMLData` method fetches none, which is how quak's own read commands open
their library.
The `Client` under the library handles login, saved sessions and the raw The `Client` under the library handles login, saved sessions and the raw
requests. The "Library reference" below covers both. requests. The "Library reference" below covers both.
@@ -136,9 +134,9 @@ requests. The "Library reference" below covers both.
`quak backup` never prompts, so cron can run it. `make install` builds quak as `quak backup` never prompts, so cron can run it. `make install` builds quak as
one binary and copies it to `~/bin/quak`. Besides what Getting Started needs, one binary and copies it to `~/bin/quak`. Besides what Getting Started needs,
`make install` needs `make` and [nix](https://nixos.org/download). It uses bun, `make install` needs [nix](https://nixos.org/download). It uses bun, a
a JavaScript runtime, to bundle the built `dist/` into one executable. It runs JavaScript runtime, to bundle the built `dist/` into one executable. It runs bun
bun through `nix-shell -p bun`. through `nix-shell -p bun`.
Log in once with `~/bin/quak`, as the user the cron job will run as: Log in once with `~/bin/quak`, as the user the cron job will run as:
@@ -158,10 +156,9 @@ output to `~/quak-backup.log`. Sunday's run adds `--verify`:
A run with `--verify` does all a plain run does. It also hashes each original A run with `--verify` does all a plain run does. It also hashes each original
already in the backup and compares the result with the content hash. The content already in the backup and compares the result with the content hash. The content
hash is the hash of the file's bytes that Ente recorded at upload. The run hash is the hash of the file's bytes that Ente recorded at upload. It replaces
fetches a new copy of each original that does not match, and replaces the any original that does not match. Sunday's run is the `--verify` run, rather
original once the copy matches. Sunday's run is the `--verify` run, rather than than a second job that night, because two backups of one directory cannot run at
a second job that night, because two backups of one directory cannot run at
once. The second one exits 2 without doing anything. once. The second one exits 2 without doing anything.
Cron runs the job with a short `PATH`, usually `/usr/bin:/bin`, so the crontab Cron runs the job with a short `PATH`, usually `/usr/bin:/bin`, so the crontab
@@ -230,12 +227,9 @@ set, it asks on the terminal. The password shows as `*` characters while it is
typed. typed.
When the account has a second factor, quak asks for the code. It asks When the account has a second factor, quak asks for the code. It asks
`TOTP code: ` for a time-based code from an authenticator app. An account whose `TOTP code: ` for a time-based code from an authenticator app, and
only second factor is a passkey cannot log in with quak. When the account has `Email verification code: ` for a code sent by email. An account whose only
email verification on, quak asks `Email verification code: ` for the code sent second factor is a passkey cannot log in with quak.
by email. The password is then used only to unlock the keys. After the email
code quak expects the login to complete. When the server asks for a second
factor at that point, login fails.
On success it logs `Logged in as <email> (user <id>)` and On success it logs `Logged in as <email> (user <id>)` and
`Session saved to <path>`. On failure it logs `Login failed: <reason>`, and the `Session saved to <path>`. On failure it logs `Login failed: <reason>`, and the
@@ -245,9 +239,7 @@ to stdout.
#### `quak whoami` #### `quak whoami`
Prints the saved session's account on stdout as one line of JSON: Prints the saved session's account on stdout as one line of JSON:
`{"email":"you@example.com","userID":12345}`. It reads the session file only and `{"email":"you@example.com","userID":12345}`.
sends no request, so it succeeds even when the server no longer accepts the
session.
#### `quak logout` #### `quak logout`
@@ -314,16 +306,16 @@ refused, because both files would land on it.
#### `quak get-thumb <fileID> [--out <path>]` #### `quak get-thumb <fileID> [--out <path>]`
Downloads and decrypts one file's thumbnail, the small JPEG Ente shows in a Downloads and decrypts one file's thumbnail, the small JPEG Ente shows in a
grid. It writes one file, also for a live photo. Without `--out`, the name is grid. It works as `get` does. Without `--out`, the name is `thumb_` followed by
`thumb_` followed by the title. In all else it works as `get` does. the title.
#### `quak backup <dir> [--json] [--verify]` #### `quak backup <dir> [--json] [--verify]`
Backs up every file in the account to `<dir>`. The layout is described under Backs up every file in the account to `<dir>`. The layout is described under
"The backup directory". The command downloads only the originals that are not "The backup directory". It downloads only the originals that are not yet in the
yet in the backup, and it goes on past a file that fails. A file is downloaded backup, and it goes on past a file that fails. A file is downloaded once,
once, however many collections hold it. An original the cache already holds is however many collections hold it. An original the cache already holds is copied
copied from there instead. from there instead.
Progress goes to stderr: one line per original fetched, and a `FAILED` line for Progress goes to stderr: one line per original fetched, and a `FAILED` line for
each original or album that failed. The run ends with a summary of counts and, each original or album that failed. The run ends with a summary of counts and,
@@ -362,8 +354,7 @@ Writes the metadata quak keeps about the account, its collections and its files
as JSON files under `<dir>`. The layout is described under "The metadata dump as JSON files under `<dir>`. The layout is described under "The metadata dump
directory". Photo content is not written. With `--exif`, or its alias `--all`, directory". Photo content is not written. With `--exif`, or its alias `--all`,
the command also fetches each file's original and records the image's own the command also fetches each file's original and records the image's own
metadata from it (see "The metadata dump directory"). Each original it fetches metadata from it (see "The metadata dump directory").
stays in the cache.
Ente's apps compute machine-learning data for each photo, such as the faces in Ente's apps compute machine-learning data for each photo, such as the faces in
it (see "The JSON beside each original"). The command fetches it in requests of it (see "The JSON beside each original"). The command fetches it in requests of
@@ -378,13 +369,12 @@ word, and does not change the exit status.
#### `quak helper list-missing-thumbnails [--json]` #### `quak helper list-missing-thumbnails [--json]`
Finds the files whose thumbnail is missing on the server. It downloads the Finds the files whose thumbnail is missing on the server. It downloads every
thumbnail of every file you own, one collection at a time, newest updated first, thumbnail in the account, one collection at a time, newest collection first, so
so on a large account it takes a while. A thumbnail counts as missing when the on a large account it takes a while. A thumbnail counts as missing when the
server answers with an empty body or with HTTP 404. Any other failure is logged server answers with an empty body or with HTTP 404. Any other failure is logged
as `[<collection>] Could not check <title>: <reason> (not reported as missing)`, as `[<collection>] Could not check <title>: <reason> (not reported as missing)`,
and the file is left out of the list. The command exits 0 even when some files and the file is left out of the list.
could not be checked.
The server lets only a file's owner replace its thumbnail. A file another The server lets only a file's owner replace its thumbnail. A file another
account owns, in a collection shared with you, is skipped and not checked. account owns, in a collection shared with you, is skipped and not checked.
@@ -427,13 +417,12 @@ Each file ends in one of three states:
`--file` is held by no collection. `--file` is held by no collection.
Without `--json`, the command ends with counts of fixed, skipped and failed Without `--json`, the command ends with counts of fixed, skipped and failed
files. A `Skipped:` list and a `Failed files:` list follow when they have files, a `Skipped:` list and a `Failed files:` list. With `--json`, one JSON
entries. With `--json`, one JSON array on stdout replaces them; each object array on stdout replaces them; each object holds `fileID`, `title`,
holds `fileID`, `title`, `collection`, `status` and, for a skipped or failed `collection`, `status` and, for a skipped or failed file, `reason`. When any
file, `reason`. When any file failed, the exit status is 1. A skipped file does file failed, the exit status is 1. A skipped file does not change it. When
not change it. When nothing needs fixing, the command logs nothing needs fixing, the command logs `No missing thumbnails found.` and
`No missing thumbnails found.` and exits 0. With `--json` it then writes nothing exits 0. With `--json` it then writes nothing on stdout.
on stdout.
quak refuses the whole command, before any request, when one `--file` ID is quak refuses the whole command, before any request, when one `--file` ID is
invalid (see "IDs"). invalid (see "IDs").
@@ -464,9 +453,9 @@ collection fetched, such as `Fetched the files of 3 of 120 collections`.
When a refresh or a search fails, the command logs the error on one line and When a refresh or a search fails, the command logs the error on one line and
exits 1, or 3 when the server no longer accepts the saved session. A collection exits 1, or 3 when the server no longer accepts the saved session. A collection
or file that quak cannot decrypt does not stop a refresh or a search: the or file that quak cannot decrypt does not stop a refresh: the command answers
command leaves it out and answers from what did decrypt (see "Default reads and from what did decrypt (see "Default reads and fresh reads"). `backup` counts it
fresh reads"). `backup` counts it as failed (see "What quak cannot decrypt"). as failed (see "What quak cannot decrypt").
### IDs ### IDs
@@ -478,11 +467,20 @@ before any request is sent. The command logs `Invalid file ID: <arg>` or
### What quak prints ### What quak prints
stdout holds a command's output, such as the listing or the JSON, and the stdout holds only a command's output: the listing, the JSON, or nothing. The
prompts of `login`. Every other line goes to stderr as a log line at one of six prompts of `login` go to stdout too. Every other line goes to stderr as a log
levels. From the quietest, the levels are fatal, error, warning, info, debug and line at one of six levels. From the quietest, the levels are fatal, error,
trace. A run prints the lines at its own level and at every quieter one. The warning, info, debug and trace. A run prints the lines at its own level and at
level is info unless `-v`, `-vv` or `-q` says otherwise. every quieter one. The level is info unless `-v`, `-vv` or `-q` says otherwise.
The one exception is a command line quak cannot parse, such as a missing
argument, an unknown option or `-q` with `-v`. quak refuses it with one
`error: …` line that is not a log line, and exits 1. `-q` does not hide that
line. `quak` with no command prints the usage on stderr and exits 1.
`yarn quak` prints yarn's own lines on stdout as well. To pipe a command's
output, run `yarn --silent quak files --json` or
`node dist/bin/quak.js files --json` instead.
- Fatal: why a command exits non-zero. A command that runs and exits non-zero - Fatal: why a command exits non-zero. A command that runs and exits non-zero
logs at least one fatal line. A command that exits 1 because files or albums logs at least one fatal line. A command that exits 1 because files or albums
@@ -508,22 +506,12 @@ level is info unless `-v`, `-vv` or `-q` says otherwise.
one that exits non-zero prints only its fatal lines. The prompts of `login` are one that exits non-zero prints only its fatal lines. The prompts of `login` are
not log lines, and `-q` leaves them alone. not log lines, and `-q` leaves them alone.
Two things on stderr are not log lines. A command line that quak cannot parse,
such as a missing argument, an unknown option or `-q` with `-v`, is refused with
an `error: …` message, and quak exits 1. `-q` does not hide the message. `quak`
with no command prints the usage on stderr and exits 1.
`yarn quak` prints yarn's own lines on stdout as well. To pipe a command's
output, run `yarn --silent quak files --json` or
`node dist/bin/quak.js files --json` instead.
With `--json`, a command's JSON on stdout takes the place of its summary on With `--json`, a command's JSON on stdout takes the place of its summary on
stderr. For `backup`, that is the `--- Backup complete ---` line, the counts and stderr. For `backup`, that is the lines from `--- Backup complete ---` to the
the `Failed files:` list. For `list-missing-thumbnails`, it is end of the `Failed files:` list. For `list-missing-thumbnails`, it is
`No missing thumbnails found.` or `<n> file(s) with missing thumbnails:`. For `No missing thumbnails found.` or `<n> file(s) with missing thumbnails:`. For
`fix-missing-thumbnails`, it is the `--- Done ---` line, the counts, the `fix-missing-thumbnails`, it is the lines from `--- Done ---` to the end of the
`Skipped:` list and the `Failed files:` list. Every other line is logged as `Failed files:` list. Every other line is logged as without `--json`.
without `--json`.
### Exit codes ### Exit codes
@@ -689,13 +677,13 @@ The JSON file beside an original holds:
Update times are in microseconds, as Ente records them. Update times are in microseconds, as Ente records them.
The JSON also holds the machine-learning data Ente computed for the photo, as The file also holds the machine-learning data Ente computed for the file, as
`mlData`. Ente's apps detect faces in each photo and compute a CLIP embedding `mlData`. Ente's apps detect faces in each photo and compute a CLIP embedding
for it, a list of 512 numbers that describes what the photo shows. `mlData` for it, a list of 512 numbers that describes what the photo shows. `mlData`
holds everything but the `clip` block, which holds that embedding. In its place holds everything but that embedding. In its place the file holds `keywords`,
the JSON holds `keywords`, derived from the embedding as described under derived from the embedding as described under "Keywords". A file with no
"Keywords". A file with no machine-learning data has neither field. A file whose machine-learning data has neither field. A file whose data holds no usable
data holds no usable embedding has no `keywords`. embedding has no `keywords`.
The backup waits for the library's fetch of machine-learning data to finish The backup waits for the library's fetch of machine-learning data to finish
before it writes the JSON files. When a file's data cannot be fetched, before it writes the JSON files. When a file's data cannot be fetched,
@@ -748,11 +736,11 @@ derives a file's keywords the way the app sorts it. The categories are:
Each category has a query, and a text embedding of that query from Ente's CLIP Each category has a query, and a text embedding of that query from Ente's CLIP
text model. A file's score for a category is the cosine similarity of the file's text model. A file's score for a category is the cosine similarity of the file's
embedding and the category's. That is a number from -1 to 1; the more alike the embedding and the category's: a number from -1 to 1, higher the more alike the
two embeddings are, the higher it is. The file falls in the category when its two embeddings are. The file falls in the category when its score reaches the
score reaches the category's minimum score. The keywords are the titles of the category's minimum score. The keywords are the titles of the categories the file
categories the file falls in, highest score first. A video never falls in a falls in, highest score first. A video never falls in a category marked "no
category marked "no videos", as in the app. videos", as in the app.
#### Collection directories #### Collection directories
@@ -775,21 +763,17 @@ ignoring case, each gets its ID added: two albums named `Trip` become
`Trip (10)/` and `Trip (11)/`. Two files in one collection whose titles match, `Trip (10)/` and `Trip (11)/`. Two files in one collection whose titles match,
ignoring case, get their IDs added before the extension: `IMG_0001 (12345).JPG` ignoring case, get their IDs added before the extension: `IMG_0001 (12345).JPG`
and `IMG_0001 (12346).JPG`. A collection named like another collection's JSON and `IMG_0001 (12346).JPG`. A collection named like another collection's JSON
file, such as `Trip.json` beside `Trip`, also gets its ID added. A file name file, such as `Trip.json` beside `Trip`, also gets its ID added. A name too long
longer than 255 bytes, the limit on most file systems, gets its ID added too. So for a file name, which most file systems limit to 255 bytes, gets its ID added
does a collection name longer than 250 bytes, which leaves room for `.json`. too. It is then cut short before the ID, just far enough to fit. IDs never
Either name is then cut short before the ID, just far enough to fit. IDs never
change, so a name stays the same from run to run until such a clash appears or change, so a name stays the same from run to run until such a clash appears or
goes away. goes away.
Each run removes every symlink in a collection's directory that points into the Each run removes the symlinks that no longer belong in a collection's directory.
date folders and is not one of its current links, yours included. A file or It removes the directory and JSON of a collection that was deleted or renamed,
symlink of yours at the name of one of its links is replaced by the link. A once the directory holds nothing else. Nothing else under `collections/` is
directory of yours at that name stays, and the photo whose link it blocks goes touched: a file or symlink you put there stays, and a directory that still holds
into `failures.json`. Anything else you put under `collections/` stays. A run one stays too, with its JSON.
removes the directory and JSON of a collection that was deleted or renamed, once
the directory holds nothing else. One that still holds something of yours stays,
with its JSON.
A collection whose directory or JSON cannot be written does not stop the run. A collection whose directory or JSON cannot be written does not stop the run.
For example, a file you put there may already have the directory's name. The run For example, a file you put there may already have the directory's name. The run
@@ -834,9 +818,8 @@ Each entry holds:
- `fileID` and `title`; - `fileID` and `title`;
- `classification`: the kind of failure; - `classification`: the kind of failure;
- `attempts`: how many runs in a row have failed the file; - `attempts`: how many runs have tried the file;
- `lastTriedAt`: when the run that last tried it finished its refresh, as a Unix - `lastTriedAt`: when it was last tried, as a Unix time in milliseconds;
time in milliseconds;
- `error`: the error, as quak printed it. - `error`: the error, as quak printed it.
The kinds are: The kinds are:
@@ -854,13 +837,13 @@ file outside them. `failures.json` is deleted when the list is empty.
`backup.lock` keeps two backups of the same directory from running at once, such `backup.lock` keeps two backups of the same directory from running at once, such
as a cron run that starts while the previous one is still going. A backup as a cron run that starts while the previous one is still going. A backup
creates `<dir>` if it is missing and takes the lock before its refresh, so a creates `<dir>` if it is missing and takes the lock before its refresh, so a
refused run sends no request. A run that still holds the lock removes it when it refused run sends no request. It removes the lock when it ends, whether it
ends, whether it succeeded or failed. succeeded or failed.
The lock is a directory that The lock is a directory that
[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates. [proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates.
While the backup runs, it updates the modification time of `backup.lock` every 5 While the backup runs, the backup updates the directory's modification time
seconds. A second backup of `<dir>` fails at once: it prints every 5 seconds. A second backup of the directory fails at once: it prints
`quak: another backup of <dir> is running` and exits 2. A run that is killed, or `quak: another backup of <dir> is running` and exits 2. A run that is killed, or
cut off by a crash, can leave the lock behind. Once the lock has gone 10 seconds cut off by a crash, can leave the lock behind. Once the lock has gone 10 seconds
without an update, the next run takes it over, so nobody has to remove it by without an update, the next run takes it over, so nobody has to remove it by
@@ -903,13 +886,13 @@ The summary and the `--json` output add the counts `verified`, `mismatched` and
`unchecked`. Each counts only originals that were already stored. A mismatch `unchecked`. Each counts only originals that were already stored. A mismatch
that was replaced with matching bytes does not make the exit code non-zero. that was replaced with matching bytes does not make the exit code non-zero.
A run without `--verify` drops two kinds of failure from `failures.json`: Two kinds of failure stay in `failures.json` only until the next run:
- a mismatched original whose new copy failed; - a mismatched original whose new copy failed;
- a stored original that cannot be read. - a stored original that cannot be read.
That run finds the original in place, counts it as skipped and drops the entry, A run without `--verify` finds the original in place, counts it as skipped, and
so it can exit 0. Only a `--verify` run hashes the original again. drops the entry, so that run can exit 0. Only a `--verify` run hashes it again.
Without `--verify`, no stored original is hashed, the summary is unchanged, and Without `--verify`, no stored original is hashed, the summary is unchanged, and
the three counts are 0 in the JSON. the three counts are 0 in the JSON.
@@ -935,11 +918,7 @@ scheme. Opening a library deletes such leftovers in the cache's `originals/` and
The rename replaces whatever was at the destination rather than writing through The rename replaces whatever was at the destination rather than writing through
it. A symlink there is replaced, not followed, and the new file has the it. A symlink there is replaced, not followed, and the new file has the
temporary file's permissions, not those of the file it replaced. Each original temporary file's permissions, not those of the file it replaced. Each original
is written with mode `0600`, readable by its owner only. Everything else the is written with mode `0600`, readable by its owner only.
backup writes, its folders and JSON files included, gets the mode your umask
allows, which usually lets every user on the machine read it. The JSON beside an
original holds the file's location and faces. The `backup-metadata` tree is
written the same way.
### The metadata dump directory ### The metadata dump directory
@@ -964,9 +943,9 @@ of the same name apart.
`files`. Each `<fileID>.json` holds the fields a backup's JSON beside an `files`. Each `<fileID>.json` holds the fields a backup's JSON beside an
original holds, with these differences: original holds, with these differences:
- `imageMetadata` is written only with `--exif`. The original of every file, - `imageMetadata` is written only with `--exif`. The original of every file is
videos included, is then downloaded and read for its EXIF, XMP and JPEG then downloaded, videos included, and read as a backup reads it (see "The JSON
dimensions. beside each original").
- A file whose original holds no image metadata gets no `imageMetadata` field. - A file whose original holds no image metadata gets no `imageMetadata` field.
- An original that cannot be downloaded or read gives `imageMetadataError` - An original that cannot be downloaded or read gives `imageMetadataError`
instead. instead.
@@ -1047,24 +1026,23 @@ onto what the cache holds, empty or not. A refresh then starts only when
- `contentSource`: a replacement for the client's own byte source. Mainly for - `contentSource`: a replacement for the client's own byte source. Mainly for
tests. tests.
In the content and thumbnail pools, your own requests take the next free slot In each pool, your own requests take the next free slot ahead of the two
ahead of the two background downloads. A file already in the cache is not background downloads. A file already in the cache is not fetched again. When the
fetched again. When the originals cache grows past its limit, the least recently originals cache grows past its limit, the least recently used originals are
used originals are evicted, but never a pinned one, and never one whose fetch is evicted, but never a pinned one, and never one whose fetch is still in flight.
still in flight.
`onProgress` receives a `RefreshEvent`. Its `operation` names the work: `onProgress` receives a `RefreshEvent`. Its `operation` names the work:
`"refresh"`, `"fetchMLData"` for a machine-learning fetch, and `"refresh"`, `"fetchMLData"` for a machine-learning fetch, and
`"precacheThumbnails"` or `"precacheOriginals"` for the two background `"precacheThumbnails"` or `"precacheOriginals"` for the two background
downloads. Its `status` is `"started"` as the work starts and `"done"` as it downloads. Its `status` is `"started"` as the work starts and `"done"` as it
ends. A refresh or a machine-learning fetch that fails ends with `"failed"` ends. A refresh or a machine-learning fetch that fails ends with `"failed"` and
instead, with the message in `error`. In between, a refresh sends `"progress"` the error message instead. In between, a refresh sends `"progress"` after each
after each collection whose files it fetched: `fetched` is the number of collection whose files it fetched: `fetched` is the number of collections
collections fetched so far, and `total` the number it will fetch. A refresh fetched so far, and `total` the number it will fetch. A refresh fetches the
fetches the files of each collection that is new or changed since the cache's files of each collection that is new or changed since the cache's copy, so on an
copy, so on an empty cache it fetches those of every collection. The `"done"` empty cache it fetches those of every collection. The `"done"` event of a
event of a machine-learning fetch, and of each background download, also carries machine-learning fetch, and of each background download, also carries `fetched`:
`fetched`: how many items it newly stored. how many items it newly stored.
`lib.status()` returns a `LibraryStatus`. It holds the collection and file `lib.status()` returns a `LibraryStatus`. It holds the collection and file
counts. `lastRefreshAt` is the time of the last refresh that succeeded, and counts. `lastRefreshAt` is the time of the last refresh that succeeded, and
@@ -1079,10 +1057,10 @@ cached, so the next fetch asks for it again. The reasons are kept in memory
only. only.
`lib.close()` stops the background timer. It can be called more than once. A `lib.close()` stops the background timer. It can be called more than once. A
refresh in flight is left to finish, with its write to the cache. `close()` refresh in flight is left to finish, with its write to the cache. The promise
returns a promise. It resolves once that refresh, the machine-learning fetch and `close()` returns resolves once that refresh, the machine-learning fetch and the
the background downloads already running have all finished. The cache directory background downloads already running have all finished. The cache directory can
can then be removed. After `close()`, `fresh()`, `freshCollections()`, then be removed. After `close()`, `fresh()`, `freshCollections()`,
`freshFiles()` and `lib.backup()` reject with `the library is closed`. `freshFiles()` and `lib.backup()` reject with `the library is closed`.
### Default reads and fresh reads ### Default reads and fresh reads
@@ -1187,11 +1165,11 @@ progress of that one file.
keyed by tag name, each as exifreader decodes it with its `id`, `value`, keyed by tag name, each as exifreader decodes it with its `id`, `value`,
`description` and `computed` value. For example, `Make` is `description` and `computed` value. For example, `Make` is
`{ id: 271, value: ["Example"], description: "Example", computed: "Example" }`. `{ id: 271, value: ["Example"], description: "Example", computed: "Example" }`.
A tag that exifreader has no name for is keyed `undefined-<tag number>`. The A tag exifreader has no name for is keyed `undefined-<tag number>`. The tags
tags of the thumbnail embedded in the file are under `Thumbnail`, so they of the thumbnail embedded in the file are under `Thumbnail`, so they cannot
cannot hide the main image's tags of the same name. The thumbnail image itself hide the main image's tags of the same name. The thumbnail image itself is
is left out. Any file exifreader cannot read gives `{}`, and a video gives left out. Any file exifreader cannot read gives `{}`, and a video gives `{}`
`{}` without being downloaded. without being downloaded.
- `await photo.make(opts?)` returns one common EXIF field, picked from the tags - `await photo.make(opts?)` returns one common EXIF field, picked from the tags
`exif()` returns and typed as in `PhotoExif`, or `undefined` when the file `exif()` returns and typed as in `PhotoExif`, or `undefined` when the file
lacks it. These methods do the same for their fields: lacks it. These methods do the same for their fields:
@@ -1279,9 +1257,7 @@ symlinks. Once the lock is lost, it rejects with
`BackupOptions`: `BackupOptions`:
- `downloadDirectory`: where the tree lives. Default: the library's. - `downloadDirectory`: where the tree lives. Default: the library's.
- `includeOriginals`: fetch and store originals. Default `true`. With `false`, - `includeOriginals`: fetch and store originals. Default `true`.
the backup also fetches no machine-learning data, and writes no JSON beside
the originals and no symlinks.
- `includeThumbnails`: also store thumbnails as `thumbnails/<fileID>.jpg` beside - `includeThumbnails`: also store thumbnails as `thumbnails/<fileID>.jpg` beside
`collections/`. Default `false`. `quak backup` does not set it. `collections/`. Default `false`. `quak backup` does not set it.
- `onlyAlbumNames`: back up only the albums with these names. - `onlyAlbumNames`: back up only the albums with these names.
@@ -1302,18 +1278,15 @@ releases it once the backup ends.
many requests run at once, because Ente meters these three kinds of traffic many requests run at once, because Ente meters these three kinds of traffic
differently: differently:
- metadata, with 10 slots: only the machine-learning data requests; - metadata, with 10 slots;
- content, with 5: originals; - content, with 5;
- thumbnails, with 25: thumbnails. - thumbnails, with 25.
Only the requests named above go through the pools. A refresh, Construct one with
`freshCollections()` and `freshFiles()` send theirs outside them. Construct one
with
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })` `new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
and pass it as `pools`. Each pool serves on-demand work ahead of background and pass it as `pools`. Each pool serves on-demand work ahead of background
work. Two tasks given to `BoundedPool.run()` with the same `key` run once while work. A task submitted under a key that is still pending is not run a second
the first is pending, and both get its result. An idle pool never lends its time. An idle pool never lends its slots to a busy one.
slots to a busy one.
`lib.backup()` works on as many originals at once as the content pool allows, 5 `lib.backup()` works on as many originals at once as the content pool allows, 5
by default. `quak backup` uses the default. The limit covers everything it does by default. `quak backup` uses the default. The limit covers everything it does
@@ -1383,9 +1356,9 @@ They differ because a download's length depends on the file and the link. A
whole-transfer deadline short enough to catch a hung connection would cancel a whole-transfer deadline short enough to catch a hung connection would cancel a
large video on a slow link that is still making progress. The download deadline large video on a slow link that is still making progress. The download deadline
restarts each time bytes arrive, so a slow download runs as long as it keeps restarts each time bytes arrive, so a slow download runs as long as it keeps
moving, and one that stalls is aborted after 60 seconds of silence. The download moving, and one that stalls is aborted after 60 seconds of silence. It covers
deadline covers the wait for the headers as well as the body. There is no limit the wait for the headers as well as the body. There is no limit on the total
on the total length of a download. length of a download.
A request that changes state is not blindly replayed. Every `POST` and `PUT` in A request that changes state is not blindly replayed. Every `POST` and `PUT` in
the endpoint list under "HTTP API" goes through this rule. Verifying a second the endpoint list under "HTTP API" goes through this rule. Verifying a second
@@ -1407,13 +1380,13 @@ that rule although they look like connect-time failures. On Linux they are also
delivered on an established socket, after the request may have been written. delivered on an established socket, after the request may have been written.
They stay retryable for requests that change nothing. Bun's `ConnectionRefused` They stay retryable for requests that change nothing. Bun's `ConnectionRefused`
and `FailedToOpenSocket` are treated the same way: a request that changes and `FailedToOpenSocket` are treated the same way: a request that changes
nothing is retried on them, and a `POST` or `PUT` is not replayed on them. Bun nothing is retried on them, and a `POST` or `PUT` is not replayed on them. When
reuses a connection it kept open from an earlier request. When that connection a connection bun kept open from an earlier request closes before any answer, bun
closes before any answer, bun sends the request again on a new one. If that one sends the request again on a new connection. If that one fails, bun gives one of
fails, bun gives one of these codes although the server may already have the these codes although the server may already have the request. The upload of a
request. The upload of a thumbnail to its presigned URL is exempt from the rule. thumbnail to its presigned URL is exempt from the rule. It stores one whole
It stores one whole object at one key in one request, so replaying it has no object at one key in one request, so replaying it has no partial state to
partial state to damage. damage.
A download is retried as a whole: the request, the reading of the body and the A download is retried as a whole: the request, the reading of the body and the
decryption. A socket reset after the response headers have arrived surfaces in decryption. A socket reset after the response headers have arrived surfaces in
@@ -1438,9 +1411,8 @@ return, quak never keeps a truncated file.
`Client.login({ email, password, totp?, emailOTP?, apiOptions? })` logs in and `Client.login({ email, password, totp?, emailOTP?, apiOptions? })` logs in and
resolves to a `Client`. `totp` and `emailOTP` are callbacks that return the code resolves to a `Client`. `totp` and `emailOTP` are callbacks that return the code
when the account asks for one. Without the callback the account needs, login when the account asks for one. Without the callback the account needs, login
throws. A passkey challenge always throws. After an email code, login expects throws. A passkey challenge always throws. `apiOptions` configures the HTTP
the key attributes and the token. When the server asks for a second factor client (see below).
instead, login throws. `apiOptions` configures the HTTP client (see below).
The `Client` holds the session token, the master key, the secret key and the The `Client` holds the session token, the master key, the secret key and the
public key in memory. There is no on-disk session store in the library; the public key in memory. There is no on-disk session store in the library; the
@@ -1450,16 +1422,9 @@ write wherever it likes. `Client.fromJSON(snapshot, apiOptions?)` restores a
working client from that snapshot without logging in again. It checks every working client from that snapshot without logging in again. It checks every
field and each key's length first, and throws an error naming the bad field. field and each key's length first, and throws an error naming the bad field.
`Client.fromJSON` works only once libsodium, the cryptography library quak uses,
is loaded. `Client.login` loads it, and `Client.fromJSON` does not. A program
that restores a session in a new process first runs `await sodium.ready`, with
`sodium` imported from `libsodium-wrappers-sumo`, which quak depends on. The
exported `beginLogin`, `unwrapAuth`, `decryptCollection`, `decryptFile`,
`downloadFile` and `downloadThumbnail` need the same.
`client.logout()` clears the token and zeroes the key buffers in place. After `client.logout()` clears the token and zeroes the key buffers in place. After
it, every other method on that client throws. `logout()` does not contact the it, every other method on that client throws. It does not contact the server, so
server, so the token stays valid there and in any saved snapshot. Call the token stays valid there and in any saved snapshot. Call
`await client.logoutOnServer()` before `logout()` to end the session on the `await client.logoutOnServer()` before `logout()` to end the session on the
server. server.
@@ -1479,10 +1444,8 @@ return only the decrypted records that are not deleted, from `sinceTime` `0`.
`client.fetchMLData({ fileIDs, fileKeys })` fetches and decrypts the `client.fetchMLData({ fileIDs, fileKeys })` fetches and decrypts the
machine-learning data of up to 200 files. `client.downloadFile(file, outPath?)` machine-learning data of up to 200 files. `client.downloadFile(file, outPath?)`
and `client.downloadThumbnail(file, outPath?)` download and decrypt one file to and `client.downloadThumbnail(file, outPath?)` download and decrypt one file to
`outPath`. For a live photo, `downloadFile` writes the image and the video, each `outPath`. Without `outPath`, they write to the working directory under the
named `outPath` with its own extension. `quak get --out` instead puts the image names `quak get` and `quak get-thumb` would use.
at the path itself. Without `outPath`, they write to the working directory under
the names `quak get` and `quak get-thumb` would use.
`ApiClientOptions`: `ApiClientOptions`:
@@ -1677,8 +1640,8 @@ Login derives the keys in this order:
the code (`POST /users/two-factor/verify`); the server then returns the key the code (`POST /users/two-factor/verify`); the server then returns the key
attributes and the token. quak cannot answer a passkey request. When the attributes and the token. quak cannot answer a passkey request. When the
account has email verification on (`isEmailMFAEnabled`), an email code account has email verification on (`isEmailMFAEnabled`), an email code
replaces SRP. quak then expects the key attributes and the token in the replaces SRP, and the server returns the key attributes and the token after
answer, and fails the login when the server asks for a second factor instead. it.
5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK 5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK
yields the 32-byte master key. yields the 32-byte master key.
6. `crypto_secretbox_open_easy` over the encrypted secret key with the master 6. `crypto_secretbox_open_easy` over the encrypted secret key with the master
@@ -1757,35 +1720,36 @@ This repository adheres to the
standard. Normalized scripts in `script/` are the entrypoints for the standard. Normalized scripts in `script/` are the entrypoints for the
development workflow, and most Makefile targets are thin shims that call them. development workflow, and most Makefile targets are thin shims that call them.
The scripts are POSIX `sh`, not bash, so they run in minimal containers such as The scripts are POSIX `sh`, not bash, so they run in minimal containers such as
Alpine. Of these, only `bootstrap`, `setup`, `test` and `cibuild` come from the Alpine. The scripts marked "ours" are this repository's own additions to the
standard. The others are additions to it. standard.
- `script/bootstrap`: install all dependencies. It installs `make`, `git`, Node - `script/bootstrap`: install all dependencies. It installs `make`, `git`, Node
and `yarn` when they are missing, then runs `yarn install --frozen-lockfile`. and `yarn` when they are missing, then runs `yarn install --frozen-lockfile`.
- `script/setup`: set up the repo for development after a fresh clone. It runs - `script/setup`: set up the repo for development after a fresh clone. It runs
`script/bootstrap`, then `script/install-precommit`. `script/bootstrap`, then `script/install-precommit`.
- `script/projectname`: print the project name. `script/docker` uses it for the - `script/projectname` (ours): print the project name. `script/docker` uses it
image tag. for the image tag.
- `script/build`: compile the TypeScript sources into `dist/` and stamp the - `script/build` (ours): compile the TypeScript sources into `dist/` and stamp
version into `dist/package.json`. Then check that the entrypoints the version into `dist/package.json`. Then check that the entrypoints
`package.json` declares (`main`, `types`, `bin`) were written, and make the `package.json` declares (`main`, `types`, `bin`) were written, and make the
command line executable. command line executable.
- `script/version`: print the version `script/build` stamps (see "Version"). - `script/version` (ours): print the version `script/build` stamps (see
"Version").
- `script/test`: run the test suite, by building the `test` phase of the - `script/test`: run the test suite, by building the `test` phase of the
`Dockerfile`. Requires Docker. `Dockerfile`. Requires Docker.
- `script/lint`: run eslint and a prettier check, by building the `lint` phase - `script/lint`: run eslint and a prettier check, by building the `lint` phase
of the `Dockerfile`. Requires Docker. of the `Dockerfile`. Requires Docker.
- `script/fmt`: format all files with prettier. It rewrites files in place. - `script/fmt`: format all files with prettier. It rewrites files in place.
- `script/fmt-check`: check formatting on the host. It is read-only. - `script/fmt-check`: check formatting on the host. It is read-only.
- `script/check`: run all checks: `test`, `lint`, `fmt-check`. - `script/check` (ours): run all checks: `test`, `lint`, `fmt-check`.
- `script/docker`: build the image, tagged with `script/projectname`. - `script/docker`: build the image, tagged with `script/projectname`.
- `script/cibuild`: what CI runs: `script/bootstrap`, then `script/check`, then - `script/cibuild`: what CI runs: `script/bootstrap`, then `script/check`, then
the image build. the image build.
- `script/precommit`: run by the git pre-commit hook. It runs `script/lint` and - `script/precommit` (ours): run by the git pre-commit hook. It runs
`script/fmt-check`, and not the tests, so that a commit of failing tests can `script/lint` and `script/fmt-check`, and not the tests, so that a commit of
land (see "Development workflow"). failing tests can land (see "Development workflow").
- `script/install-precommit`: install the git pre-commit hook. `make hooks` - `script/install-precommit` (ours): install the git pre-commit hook.
calls it. `make hooks` calls it.
The Makefile targets are: The Makefile targets are:
@@ -1852,8 +1816,8 @@ empty, `dev` or `unknown`, because such a build could not be traced to its
commit. commit.
`.dockerignore` therefore does not leave out `.git`, so any `docker build .` of `.dockerignore` therefore does not leave out `.git`, so any `docker build .` of
a clone stamps the commit it was built from. A clone of depth 1 stamps a tag a clone stamps the commit it was built from. A shallow clone stamps a tag only
only when its commit carries one, and otherwise the short commit. when the cloned commit itself carries one, and otherwise the short commit.
`.dockerignore` does leave out `.git/config`, which holds the clone's remote URL `.dockerignore` does leave out `.git/config`, which holds the clone's remote URL
and any credential in it; `git describe` does not need that file. and any credential in it; `git describe` does not need that file.
`script/docker` and `script/cibuild` work out the version on the host, with `script/docker` and `script/cibuild` work out the version on the host, with
@@ -1909,12 +1873,7 @@ These apply to people and to coding agents alike.
- the nvm archive `script/bootstrap` downloads, and the Python packages - the nvm archive `script/bootstrap` downloads, and the Python packages
`bin/clip-keyword-vectors` installs, by SHA-256 hash; `bin/clip-keyword-vectors` installs, by SHA-256 hash;
- Node and yarn, when `script/bootstrap` installs them, by version. - Node and yarn, when `script/bootstrap` installs them, by version.
- Not pinned: - bun is not pinned: `make build-bin` uses whatever `nix-shell -p bun` provides.
- bun, which `make build-bin` takes from `nix-shell -p bun`;
- `make` and `git`, which `script/bootstrap` installs from the system
package manager when they are missing, as every `Dockerfile` stage does;
- `bash` and `curl`, which `script/bootstrap` installs the same way when it
has to install Node.
- Follow the workflow above: branches off `next`, tests first and committed - Follow the workflow above: branches off `next`, tests first and committed
failing, and `next` merged into `main` by sneak alone. failing, and `next` merged into `main` by sneak alone.
- Before every commit, `make lint` and `make fmt-check` must pass. `make lint` - Before every commit, `make lint` and `make fmt-check` must pass. `make lint`
+7 -10
View File
@@ -1,18 +1,15 @@
// The CLI's log level, and the one function the commands write stderr lines // The CLI's log level, and the one function the commands write stderr lines
// through. // through.
// //
// Every line quak logs goes to stderr, at one of six levels, from the // Every line quak prints on stderr is a log line at one of six levels, from
// quietest: fatal, error, warning, info, debug, trace. A run prints the lines // the quietest: fatal, error, warning, info, debug, trace. A run prints the
// at its own level and at every quieter one. Its level is info unless `-v` or // lines at its own level and at every quieter one. Its level is info unless
// `-q` asks for another (see "What quak prints" in the README). A command that // `-v` or `-q` asks for another (see "What quak prints" in the README). Every
// runs and exits non-zero logs a fatal line saying why, so even `-q`, which // command that exits non-zero logs a fatal line saying why, so even `-q`, which
// prints only fatal lines, never leaves such an exit unexplained. An error // prints only fatal lines, never leaves a non-zero exit unexplained. An error
// line reports something that failed, such as a file or an album, and a // line reports something that failed, such as a file or an album, and a
// warning something that did not fail but needs the user's attention. stdout // warning something that did not fail but needs the user's attention. stdout
// carries a command's output and the prompts of `login`, and is not a log. // carries only a command's output and is not a log.
// Two things on stderr are not log lines: the `error:` message for a
// command line that quak cannot parse, and the usage printed when no command
// is given.
// //
// The library itself writes no lines: its layers report through callbacks, and // The library itself writes no lines: its layers report through callbacks, and
// the CLI logs what they report. `logRequests` gives the API client options // the CLI logs what they report. `logRequests` gives the API client options
+1 -1
View File
@@ -328,7 +328,7 @@ describe("what the library reports", () => {
]); ]);
}); });
// Each operation is named in plain words. // Each operation is named in plain words, as the README names it.
it("logs a refresh's progress at info, and its start and end at debug", () => { it("logs a refresh's progress at info, and its start and end at debug", () => {
const { log, lines } = collect(); const { log, lines } = collect();
const onProgress = logRefresh(log); const onProgress = logRefresh(log);