Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
aeb275e137 | ||
|
|
67794bf5f8 | ||
|
|
b5fcbf4400 |
@@ -73,8 +73,7 @@ and `dist/` is not committed. Clone and build it as above, then add the clone to
|
||||
your own program's `package.json` by its path, for example
|
||||
`yarn add file:../quak`. The package's `main`, `types` and `exports` fields
|
||||
point into `dist/`, so the clone must be built first. The package is an ES
|
||||
module only: `package.json` sets `"type": "module"`, and its `exports` field
|
||||
offers only `import`, so a CommonJS program cannot `require` it.
|
||||
module only, so a CommonJS program cannot `require` it.
|
||||
|
||||
```ts
|
||||
import { Client, Library } from "quak";
|
||||
@@ -90,8 +89,9 @@ const client = await Client.login({
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
// Without these two, open() also starts downloading every thumbnail in
|
||||
// the account and the latest week's originals into the cache. It fetches
|
||||
// machine-learning data after each refresh either way.
|
||||
// the account, and the originals of the favorites album and the latest
|
||||
// week, into the cache. It fetches machine-learning data after each
|
||||
// refresh either way.
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
});
|
||||
@@ -122,10 +122,12 @@ await lib.close();
|
||||
for. It refreshes every 3 seconds until `close()`. It also downloads, in the
|
||||
background, every thumbnail in the account and the originals of the favorites
|
||||
album and of the latest week (see "Opening a library"). The example turns those
|
||||
two downloads off. After each refresh the library also fetches each file's
|
||||
machine-learning data into the cache. That is the faces and the other data
|
||||
Ente's apps compute for a photo (see "The JSON beside each original"). No option
|
||||
turns this off.
|
||||
two downloads off. After each refresh that succeeds, the library also fetches
|
||||
each file's machine-learning data into the cache. That is the faces and the
|
||||
other data Ente's apps compute for a photo (see "The JSON beside each
|
||||
original"). No option in `LibraryOptions` turns this off. A `client` without a
|
||||
`fetchMLData` method fetches none, which is how quak's own read commands open
|
||||
their library.
|
||||
|
||||
The `Client` under the library handles login, saved sessions and the raw
|
||||
requests. The "Library reference" below covers both.
|
||||
@@ -134,9 +136,9 @@ requests. The "Library reference" below covers both.
|
||||
|
||||
`quak backup` never prompts, so cron can run it. `make install` builds quak as
|
||||
one binary and copies it to `~/bin/quak`. Besides what Getting Started needs,
|
||||
`make install` needs [nix](https://nixos.org/download). It uses bun, a
|
||||
JavaScript runtime, to bundle the built `dist/` into one executable. It runs bun
|
||||
through `nix-shell -p bun`.
|
||||
`make install` needs `make` and [nix](https://nixos.org/download). It uses bun,
|
||||
a JavaScript runtime, to bundle the built `dist/` into one executable. It runs
|
||||
bun through `nix-shell -p bun`.
|
||||
|
||||
Log in once with `~/bin/quak`, as the user the cron job will run as:
|
||||
|
||||
@@ -156,9 +158,10 @@ output to `~/quak-backup.log`. Sunday's run adds `--verify`:
|
||||
|
||||
A run with `--verify` does all a plain run does. It also hashes each original
|
||||
already in the backup and compares the result with the content hash. The content
|
||||
hash is the hash of the file's bytes that Ente recorded at upload. It replaces
|
||||
any original that does not match. Sunday's run is the `--verify` run, rather
|
||||
than a second job that night, because two backups of one directory cannot run at
|
||||
hash is the hash of the file's bytes that Ente recorded at upload. The run
|
||||
fetches a new copy of each original that does not match, and replaces the
|
||||
original once the copy matches. Sunday's run is the `--verify` run, rather than
|
||||
a second job that night, because two backups of one directory cannot run at
|
||||
once. The second one exits 2 without doing anything.
|
||||
|
||||
Cron runs the job with a short `PATH`, usually `/usr/bin:/bin`, so the crontab
|
||||
@@ -227,9 +230,12 @@ set, it asks on the terminal. The password shows as `*` characters while it is
|
||||
typed.
|
||||
|
||||
When the account has a second factor, quak asks for the code. It asks
|
||||
`TOTP code: ` for a time-based code from an authenticator app, and
|
||||
`Email verification code: ` for a code sent by email. An account whose only
|
||||
second factor is a passkey cannot log in with quak.
|
||||
`TOTP code: ` for a time-based code from an authenticator app. An account whose
|
||||
only second factor is a passkey cannot log in with quak. When the account has
|
||||
email verification on, quak asks `Email verification code: ` for the code sent
|
||||
by email. The password is then used only to unlock the keys. After the email
|
||||
code quak expects the login to complete. When the server asks for a second
|
||||
factor at that point, login fails.
|
||||
|
||||
On success it logs `Logged in as <email> (user <id>)` and
|
||||
`Session saved to <path>`. On failure it logs `Login failed: <reason>`, and the
|
||||
@@ -239,7 +245,9 @@ to stdout.
|
||||
#### `quak whoami`
|
||||
|
||||
Prints the saved session's account on stdout as one line of JSON:
|
||||
`{"email":"you@example.com","userID":12345}`.
|
||||
`{"email":"you@example.com","userID":12345}`. It reads the session file only and
|
||||
sends no request, so it succeeds even when the server no longer accepts the
|
||||
session.
|
||||
|
||||
#### `quak logout`
|
||||
|
||||
@@ -306,16 +314,16 @@ refused, because both files would land on it.
|
||||
#### `quak get-thumb <fileID> [--out <path>]`
|
||||
|
||||
Downloads and decrypts one file's thumbnail, the small JPEG Ente shows in a
|
||||
grid. It works as `get` does. Without `--out`, the name is `thumb_` followed by
|
||||
the title.
|
||||
grid. It writes one file, also for a live photo. Without `--out`, the name is
|
||||
`thumb_` followed by the title. In all else it works as `get` does.
|
||||
|
||||
#### `quak backup <dir> [--json] [--verify]`
|
||||
|
||||
Backs up every file in the account to `<dir>`. The layout is described under
|
||||
"The backup directory". It downloads only the originals that are not yet in the
|
||||
backup, and it goes on past a file that fails. A file is downloaded once,
|
||||
however many collections hold it. An original the cache already holds is copied
|
||||
from there instead.
|
||||
"The backup directory". The command downloads only the originals that are not
|
||||
yet in the backup, and it goes on past a file that fails. A file is downloaded
|
||||
once, however many collections hold it. An original the cache already holds is
|
||||
copied from there instead.
|
||||
|
||||
Progress goes to stderr: one line per original fetched, and a `FAILED` line for
|
||||
each original or album that failed. The run ends with a summary of counts and,
|
||||
@@ -354,7 +362,8 @@ Writes the metadata quak keeps about the account, its collections and its files
|
||||
as JSON files under `<dir>`. The layout is described under "The metadata dump
|
||||
directory". Photo content is not written. With `--exif`, or its alias `--all`,
|
||||
the command also fetches each file's original and records the image's own
|
||||
metadata from it (see "The metadata dump directory").
|
||||
metadata from it (see "The metadata dump directory"). Each original it fetches
|
||||
stays in the cache.
|
||||
|
||||
Ente's apps compute machine-learning data for each photo, such as the faces in
|
||||
it (see "The JSON beside each original"). The command fetches it in requests of
|
||||
@@ -369,12 +378,13 @@ word, and does not change the exit status.
|
||||
|
||||
#### `quak helper list-missing-thumbnails [--json]`
|
||||
|
||||
Finds the files whose thumbnail is missing on the server. It downloads every
|
||||
thumbnail in the account, one collection at a time, newest collection first, so
|
||||
on a large account it takes a while. A thumbnail counts as missing when the
|
||||
Finds the files whose thumbnail is missing on the server. It downloads the
|
||||
thumbnail of every file you own, one collection at a time, newest updated first,
|
||||
so on a large account it takes a while. A thumbnail counts as missing when the
|
||||
server answers with an empty body or with HTTP 404. Any other failure is logged
|
||||
as `[<collection>] Could not check <title>: <reason> (not reported as missing)`,
|
||||
and the file is left out of the list.
|
||||
and the file is left out of the list. The command exits 0 even when some files
|
||||
could not be checked.
|
||||
|
||||
The server lets only a file's owner replace its thumbnail. A file another
|
||||
account owns, in a collection shared with you, is skipped and not checked.
|
||||
@@ -417,12 +427,13 @@ Each file ends in one of three states:
|
||||
`--file` is held by no collection.
|
||||
|
||||
Without `--json`, the command ends with counts of fixed, skipped and failed
|
||||
files, a `Skipped:` list and a `Failed files:` list. With `--json`, one JSON
|
||||
array on stdout replaces them; each object holds `fileID`, `title`,
|
||||
`collection`, `status` and, for a skipped or failed file, `reason`. When any
|
||||
file failed, the exit status is 1. A skipped file does not change it. When
|
||||
nothing needs fixing, the command logs `No missing thumbnails found.` and
|
||||
exits 0. With `--json` it then writes nothing on stdout.
|
||||
files. A `Skipped:` list and a `Failed files:` list follow when they have
|
||||
entries. With `--json`, one JSON array on stdout replaces them; each object
|
||||
holds `fileID`, `title`, `collection`, `status` and, for a skipped or failed
|
||||
file, `reason`. When any file failed, the exit status is 1. A skipped file does
|
||||
not change it. When nothing needs fixing, the command logs
|
||||
`No missing thumbnails found.` and exits 0. With `--json` it then writes nothing
|
||||
on stdout.
|
||||
|
||||
quak refuses the whole command, before any request, when one `--file` ID is
|
||||
invalid (see "IDs").
|
||||
@@ -453,9 +464,9 @@ collection fetched, such as `Fetched the files of 3 of 120 collections`.
|
||||
|
||||
When a refresh or a search fails, the command logs the error on one line and
|
||||
exits 1, or 3 when the server no longer accepts the saved session. A collection
|
||||
or file that quak cannot decrypt does not stop a refresh: the command answers
|
||||
from what did decrypt (see "Default reads and fresh reads"). `backup` counts it
|
||||
as failed (see "What quak cannot decrypt").
|
||||
or file that quak cannot decrypt does not stop a refresh or a search: the
|
||||
command leaves it out and answers from what did decrypt (see "Default reads and
|
||||
fresh reads"). `backup` counts it as failed (see "What quak cannot decrypt").
|
||||
|
||||
### IDs
|
||||
|
||||
@@ -467,20 +478,11 @@ before any request is sent. The command logs `Invalid file ID: <arg>` or
|
||||
|
||||
### What quak prints
|
||||
|
||||
stdout holds only a command's output: the listing, the JSON, or nothing. The
|
||||
prompts of `login` go to stdout too. Every other line goes to stderr as a log
|
||||
line at one of six levels. From the quietest, the levels are fatal, error,
|
||||
warning, info, debug and trace. A run prints the lines at its own level and at
|
||||
every quieter one. The level is info unless `-v`, `-vv` or `-q` says otherwise.
|
||||
|
||||
The one exception is a command line quak cannot parse, such as a missing
|
||||
argument, an unknown option or `-q` with `-v`. quak refuses it with one
|
||||
`error: …` line that is not a log line, and exits 1. `-q` does not hide that
|
||||
line. `quak` with no command prints the usage on stderr and exits 1.
|
||||
|
||||
`yarn quak` prints yarn's own lines on stdout as well. To pipe a command's
|
||||
output, run `yarn --silent quak files --json` or
|
||||
`node dist/bin/quak.js files --json` instead.
|
||||
stdout holds a command's output, such as the listing or the JSON, and the
|
||||
prompts of `login`. Every other line goes to stderr as a log line at one of six
|
||||
levels. From the quietest, the levels are fatal, error, warning, info, debug and
|
||||
trace. A run prints the lines at its own level and at every quieter one. The
|
||||
level is info unless `-v`, `-vv` or `-q` says otherwise.
|
||||
|
||||
- Fatal: why a command exits non-zero. A command that runs and exits non-zero
|
||||
logs at least one fatal line. A command that exits 1 because files or albums
|
||||
@@ -506,12 +508,22 @@ output, run `yarn --silent quak files --json` or
|
||||
one that exits non-zero prints only its fatal lines. The prompts of `login` are
|
||||
not log lines, and `-q` leaves them alone.
|
||||
|
||||
Two things on stderr are not log lines. A command line that quak cannot parse,
|
||||
such as a missing argument, an unknown option or `-q` with `-v`, is refused with
|
||||
an `error: …` message, and quak exits 1. `-q` does not hide the message. `quak`
|
||||
with no command prints the usage on stderr and exits 1.
|
||||
|
||||
`yarn quak` prints yarn's own lines on stdout as well. To pipe a command's
|
||||
output, run `yarn --silent quak files --json` or
|
||||
`node dist/bin/quak.js files --json` instead.
|
||||
|
||||
With `--json`, a command's JSON on stdout takes the place of its summary on
|
||||
stderr. For `backup`, that is the lines from `--- Backup complete ---` to the
|
||||
end of the `Failed files:` list. For `list-missing-thumbnails`, it is
|
||||
stderr. For `backup`, that is the `--- Backup complete ---` line, the counts and
|
||||
the `Failed files:` list. For `list-missing-thumbnails`, it is
|
||||
`No missing thumbnails found.` or `<n> file(s) with missing thumbnails:`. For
|
||||
`fix-missing-thumbnails`, it is the lines from `--- Done ---` to the end of the
|
||||
`Failed files:` list. Every other line is logged as without `--json`.
|
||||
`fix-missing-thumbnails`, it is the `--- Done ---` line, the counts, the
|
||||
`Skipped:` list and the `Failed files:` list. Every other line is logged as
|
||||
without `--json`.
|
||||
|
||||
### Exit codes
|
||||
|
||||
@@ -677,13 +689,13 @@ The JSON file beside an original holds:
|
||||
|
||||
Update times are in microseconds, as Ente records them.
|
||||
|
||||
The file also holds the machine-learning data Ente computed for the file, as
|
||||
The JSON also holds the machine-learning data Ente computed for the photo, as
|
||||
`mlData`. Ente's apps detect faces in each photo and compute a CLIP embedding
|
||||
for it, a list of 512 numbers that describes what the photo shows. `mlData`
|
||||
holds everything but that embedding. In its place the file holds `keywords`,
|
||||
derived from the embedding as described under "Keywords". A file with no
|
||||
machine-learning data has neither field. A file whose data holds no usable
|
||||
embedding has no `keywords`.
|
||||
holds everything but the `clip` block, which holds that embedding. In its place
|
||||
the JSON holds `keywords`, derived from the embedding as described under
|
||||
"Keywords". A file with no machine-learning data has neither field. A file whose
|
||||
data holds no usable embedding has no `keywords`.
|
||||
|
||||
The backup waits for the library's fetch of machine-learning data to finish
|
||||
before it writes the JSON files. When a file's data cannot be fetched,
|
||||
@@ -736,11 +748,11 @@ derives a file's keywords the way the app sorts it. The categories are:
|
||||
|
||||
Each category has a query, and a text embedding of that query from Ente's CLIP
|
||||
text model. A file's score for a category is the cosine similarity of the file's
|
||||
embedding and the category's: a number from -1 to 1, higher the more alike the
|
||||
two embeddings are. The file falls in the category when its score reaches the
|
||||
category's minimum score. The keywords are the titles of the categories the file
|
||||
falls in, highest score first. A video never falls in a category marked "no
|
||||
videos", as in the app.
|
||||
embedding and the category's. That is a number from -1 to 1; the more alike the
|
||||
two embeddings are, the higher it is. The file falls in the category when its
|
||||
score reaches the category's minimum score. The keywords are the titles of the
|
||||
categories the file falls in, highest score first. A video never falls in a
|
||||
category marked "no videos", as in the app.
|
||||
|
||||
#### Collection directories
|
||||
|
||||
@@ -763,17 +775,21 @@ ignoring case, each gets its ID added: two albums named `Trip` become
|
||||
`Trip (10)/` and `Trip (11)/`. Two files in one collection whose titles match,
|
||||
ignoring case, get their IDs added before the extension: `IMG_0001 (12345).JPG`
|
||||
and `IMG_0001 (12346).JPG`. A collection named like another collection's JSON
|
||||
file, such as `Trip.json` beside `Trip`, also gets its ID added. A name too long
|
||||
for a file name, which most file systems limit to 255 bytes, gets its ID added
|
||||
too. It is then cut short before the ID, just far enough to fit. IDs never
|
||||
file, such as `Trip.json` beside `Trip`, also gets its ID added. A file name
|
||||
longer than 255 bytes, the limit on most file systems, gets its ID added too. So
|
||||
does a collection name longer than 250 bytes, which leaves room for `.json`.
|
||||
Either name is then cut short before the ID, just far enough to fit. IDs never
|
||||
change, so a name stays the same from run to run until such a clash appears or
|
||||
goes away.
|
||||
|
||||
Each run removes the symlinks that no longer belong in a collection's directory.
|
||||
It removes the directory and JSON of a collection that was deleted or renamed,
|
||||
once the directory holds nothing else. Nothing else under `collections/` is
|
||||
touched: a file or symlink you put there stays, and a directory that still holds
|
||||
one stays too, with its JSON.
|
||||
Each run removes every symlink in a collection's directory that points into the
|
||||
date folders and is not one of its current links, yours included. A file or
|
||||
symlink of yours at the name of one of its links is replaced by the link. A
|
||||
directory of yours at that name stays, and the photo whose link it blocks goes
|
||||
into `failures.json`. Anything else you put under `collections/` stays. A run
|
||||
removes the directory and JSON of a collection that was deleted or renamed, once
|
||||
the directory holds nothing else. One that still holds something of yours stays,
|
||||
with its JSON.
|
||||
|
||||
A collection whose directory or JSON cannot be written does not stop the run.
|
||||
For example, a file you put there may already have the directory's name. The run
|
||||
@@ -818,8 +834,9 @@ Each entry holds:
|
||||
|
||||
- `fileID` and `title`;
|
||||
- `classification`: the kind of failure;
|
||||
- `attempts`: how many runs have tried the file;
|
||||
- `lastTriedAt`: when it was last tried, as a Unix time in milliseconds;
|
||||
- `attempts`: how many runs in a row have failed the file;
|
||||
- `lastTriedAt`: when the run that last tried it finished its refresh, as a Unix
|
||||
time in milliseconds;
|
||||
- `error`: the error, as quak printed it.
|
||||
|
||||
The kinds are:
|
||||
@@ -837,13 +854,13 @@ file outside them. `failures.json` is deleted when the list is empty.
|
||||
`backup.lock` keeps two backups of the same directory from running at once, such
|
||||
as a cron run that starts while the previous one is still going. A backup
|
||||
creates `<dir>` if it is missing and takes the lock before its refresh, so a
|
||||
refused run sends no request. It removes the lock when it ends, whether it
|
||||
succeeded or failed.
|
||||
refused run sends no request. A run that still holds the lock removes it when it
|
||||
ends, whether it succeeded or failed.
|
||||
|
||||
The lock is a directory that
|
||||
[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates.
|
||||
While the backup runs, the backup updates the directory's modification time
|
||||
every 5 seconds. A second backup of the directory fails at once: it prints
|
||||
While the backup runs, it updates the modification time of `backup.lock` every 5
|
||||
seconds. A second backup of `<dir>` fails at once: it prints
|
||||
`quak: another backup of <dir> is running` and exits 2. A run that is killed, or
|
||||
cut off by a crash, can leave the lock behind. Once the lock has gone 10 seconds
|
||||
without an update, the next run takes it over, so nobody has to remove it by
|
||||
@@ -886,13 +903,13 @@ The summary and the `--json` output add the counts `verified`, `mismatched` and
|
||||
`unchecked`. Each counts only originals that were already stored. A mismatch
|
||||
that was replaced with matching bytes does not make the exit code non-zero.
|
||||
|
||||
Two kinds of failure stay in `failures.json` only until the next run:
|
||||
A run without `--verify` drops two kinds of failure from `failures.json`:
|
||||
|
||||
- a mismatched original whose new copy failed;
|
||||
- a stored original that cannot be read.
|
||||
|
||||
A run without `--verify` finds the original in place, counts it as skipped, and
|
||||
drops the entry, so that run can exit 0. Only a `--verify` run hashes it again.
|
||||
That run finds the original in place, counts it as skipped and drops the entry,
|
||||
so it can exit 0. Only a `--verify` run hashes the original again.
|
||||
|
||||
Without `--verify`, no stored original is hashed, the summary is unchanged, and
|
||||
the three counts are 0 in the JSON.
|
||||
@@ -918,7 +935,11 @@ scheme. Opening a library deletes such leftovers in the cache's `originals/` and
|
||||
The rename replaces whatever was at the destination rather than writing through
|
||||
it. A symlink there is replaced, not followed, and the new file has the
|
||||
temporary file's permissions, not those of the file it replaced. Each original
|
||||
is written with mode `0600`, readable by its owner only.
|
||||
is written with mode `0600`, readable by its owner only. Everything else the
|
||||
backup writes, its folders and JSON files included, gets the mode your umask
|
||||
allows, which usually lets every user on the machine read it. The JSON beside an
|
||||
original holds the file's location and faces. The `backup-metadata` tree is
|
||||
written the same way.
|
||||
|
||||
### The metadata dump directory
|
||||
|
||||
@@ -943,9 +964,9 @@ of the same name apart.
|
||||
`files`. Each `<fileID>.json` holds the fields a backup's JSON beside an
|
||||
original holds, with these differences:
|
||||
|
||||
- `imageMetadata` is written only with `--exif`. The original of every file is
|
||||
then downloaded, videos included, and read as a backup reads it (see "The JSON
|
||||
beside each original").
|
||||
- `imageMetadata` is written only with `--exif`. The original of every file,
|
||||
videos included, is then downloaded and read for its EXIF, XMP and JPEG
|
||||
dimensions.
|
||||
- A file whose original holds no image metadata gets no `imageMetadata` field.
|
||||
- An original that cannot be downloaded or read gives `imageMetadataError`
|
||||
instead.
|
||||
@@ -1026,23 +1047,24 @@ onto what the cache holds, empty or not. A refresh then starts only when
|
||||
- `contentSource`: a replacement for the client's own byte source. Mainly for
|
||||
tests.
|
||||
|
||||
In each pool, your own requests take the next free slot ahead of the two
|
||||
background downloads. A file already in the cache is not fetched again. When the
|
||||
originals cache grows past its limit, the least recently used originals are
|
||||
evicted, but never a pinned one, and never one whose fetch is still in flight.
|
||||
In the content and thumbnail pools, your own requests take the next free slot
|
||||
ahead of the two background downloads. A file already in the cache is not
|
||||
fetched again. When the originals cache grows past its limit, the least recently
|
||||
used originals are evicted, but never a pinned one, and never one whose fetch is
|
||||
still in flight.
|
||||
|
||||
`onProgress` receives a `RefreshEvent`. Its `operation` names the work:
|
||||
`"refresh"`, `"fetchMLData"` for a machine-learning fetch, and
|
||||
`"precacheThumbnails"` or `"precacheOriginals"` for the two background
|
||||
downloads. Its `status` is `"started"` as the work starts and `"done"` as it
|
||||
ends. A refresh or a machine-learning fetch that fails ends with `"failed"` and
|
||||
the error message instead. In between, a refresh sends `"progress"` after each
|
||||
collection whose files it fetched: `fetched` is the number of collections
|
||||
fetched so far, and `total` the number it will fetch. A refresh fetches the
|
||||
files of each collection that is new or changed since the cache's copy, so on an
|
||||
empty cache it fetches those of every collection. The `"done"` event of a
|
||||
machine-learning fetch, and of each background download, also carries `fetched`:
|
||||
how many items it newly stored.
|
||||
ends. A refresh or a machine-learning fetch that fails ends with `"failed"`
|
||||
instead, with the message in `error`. In between, a refresh sends `"progress"`
|
||||
after each collection whose files it fetched: `fetched` is the number of
|
||||
collections fetched so far, and `total` the number it will fetch. A refresh
|
||||
fetches the files of each collection that is new or changed since the cache's
|
||||
copy, so on an empty cache it fetches those of every collection. The `"done"`
|
||||
event of a machine-learning fetch, and of each background download, also carries
|
||||
`fetched`: how many items it newly stored.
|
||||
|
||||
`lib.status()` returns a `LibraryStatus`. It holds the collection and file
|
||||
counts. `lastRefreshAt` is the time of the last refresh that succeeded, and
|
||||
@@ -1057,10 +1079,10 @@ cached, so the next fetch asks for it again. The reasons are kept in memory
|
||||
only.
|
||||
|
||||
`lib.close()` stops the background timer. It can be called more than once. A
|
||||
refresh in flight is left to finish, with its write to the cache. The promise
|
||||
`close()` returns resolves once that refresh, the machine-learning fetch and the
|
||||
background downloads already running have all finished. The cache directory can
|
||||
then be removed. After `close()`, `fresh()`, `freshCollections()`,
|
||||
refresh in flight is left to finish, with its write to the cache. `close()`
|
||||
returns a promise. It resolves once that refresh, the machine-learning fetch and
|
||||
the background downloads already running have all finished. The cache directory
|
||||
can then be removed. After `close()`, `fresh()`, `freshCollections()`,
|
||||
`freshFiles()` and `lib.backup()` reject with `the library is closed`.
|
||||
|
||||
### Default reads and fresh reads
|
||||
@@ -1165,11 +1187,11 @@ progress of that one file.
|
||||
keyed by tag name, each as exifreader decodes it with its `id`, `value`,
|
||||
`description` and `computed` value. For example, `Make` is
|
||||
`{ id: 271, value: ["Example"], description: "Example", computed: "Example" }`.
|
||||
A tag exifreader has no name for is keyed `undefined-<tag number>`. The tags
|
||||
of the thumbnail embedded in the file are under `Thumbnail`, so they cannot
|
||||
hide the main image's tags of the same name. The thumbnail image itself is
|
||||
left out. Any file exifreader cannot read gives `{}`, and a video gives `{}`
|
||||
without being downloaded.
|
||||
A tag that exifreader has no name for is keyed `undefined-<tag number>`. The
|
||||
tags of the thumbnail embedded in the file are under `Thumbnail`, so they
|
||||
cannot hide the main image's tags of the same name. The thumbnail image itself
|
||||
is left out. Any file exifreader cannot read gives `{}`, and a video gives
|
||||
`{}` without being downloaded.
|
||||
- `await photo.make(opts?)` returns one common EXIF field, picked from the tags
|
||||
`exif()` returns and typed as in `PhotoExif`, or `undefined` when the file
|
||||
lacks it. These methods do the same for their fields:
|
||||
@@ -1257,7 +1279,9 @@ symlinks. Once the lock is lost, it rejects with
|
||||
`BackupOptions`:
|
||||
|
||||
- `downloadDirectory`: where the tree lives. Default: the library's.
|
||||
- `includeOriginals`: fetch and store originals. Default `true`.
|
||||
- `includeOriginals`: fetch and store originals. Default `true`. With `false`,
|
||||
the backup also fetches no machine-learning data, and writes no JSON beside
|
||||
the originals and no symlinks.
|
||||
- `includeThumbnails`: also store thumbnails as `thumbnails/<fileID>.jpg` beside
|
||||
`collections/`. Default `false`. `quak backup` does not set it.
|
||||
- `onlyAlbumNames`: back up only the albums with these names.
|
||||
@@ -1278,15 +1302,18 @@ releases it once the backup ends.
|
||||
many requests run at once, because Ente meters these three kinds of traffic
|
||||
differently:
|
||||
|
||||
- metadata, with 10 slots;
|
||||
- content, with 5;
|
||||
- thumbnails, with 25.
|
||||
- metadata, with 10 slots: only the machine-learning data requests;
|
||||
- content, with 5: originals;
|
||||
- thumbnails, with 25: thumbnails.
|
||||
|
||||
Construct one with
|
||||
Only the requests named above go through the pools. A refresh,
|
||||
`freshCollections()` and `freshFiles()` send theirs outside them. Construct one
|
||||
with
|
||||
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
|
||||
and pass it as `pools`. Each pool serves on-demand work ahead of background
|
||||
work. A task submitted under a key that is still pending is not run a second
|
||||
time. An idle pool never lends its slots to a busy one.
|
||||
work. Two tasks given to `BoundedPool.run()` with the same `key` run once while
|
||||
the first is pending, and both get its result. An idle pool never lends its
|
||||
slots to a busy one.
|
||||
|
||||
`lib.backup()` works on as many originals at once as the content pool allows, 5
|
||||
by default. `quak backup` uses the default. The limit covers everything it does
|
||||
@@ -1356,9 +1383,9 @@ They differ because a download's length depends on the file and the link. A
|
||||
whole-transfer deadline short enough to catch a hung connection would cancel a
|
||||
large video on a slow link that is still making progress. The download deadline
|
||||
restarts each time bytes arrive, so a slow download runs as long as it keeps
|
||||
moving, and one that stalls is aborted after 60 seconds of silence. It covers
|
||||
the wait for the headers as well as the body. There is no limit on the total
|
||||
length of a download.
|
||||
moving, and one that stalls is aborted after 60 seconds of silence. The download
|
||||
deadline covers the wait for the headers as well as the body. There is no limit
|
||||
on the total length of a download.
|
||||
|
||||
A request that changes state is not blindly replayed. Every `POST` and `PUT` in
|
||||
the endpoint list under "HTTP API" goes through this rule. Verifying a second
|
||||
@@ -1380,13 +1407,13 @@ that rule although they look like connect-time failures. On Linux they are also
|
||||
delivered on an established socket, after the request may have been written.
|
||||
They stay retryable for requests that change nothing. Bun's `ConnectionRefused`
|
||||
and `FailedToOpenSocket` are treated the same way: a request that changes
|
||||
nothing is retried on them, and a `POST` or `PUT` is not replayed on them. When
|
||||
a connection bun kept open from an earlier request closes before any answer, bun
|
||||
sends the request again on a new connection. If that one fails, bun gives one of
|
||||
these codes although the server may already have the request. The upload of a
|
||||
thumbnail to its presigned URL is exempt from the rule. It stores one whole
|
||||
object at one key in one request, so replaying it has no partial state to
|
||||
damage.
|
||||
nothing is retried on them, and a `POST` or `PUT` is not replayed on them. Bun
|
||||
reuses a connection it kept open from an earlier request. When that connection
|
||||
closes before any answer, bun sends the request again on a new one. If that one
|
||||
fails, bun gives one of these codes although the server may already have the
|
||||
request. The upload of a thumbnail to its presigned URL is exempt from the rule.
|
||||
It stores one whole object at one key in one request, so replaying it has no
|
||||
partial state to damage.
|
||||
|
||||
A download is retried as a whole: the request, the reading of the body and the
|
||||
decryption. A socket reset after the response headers have arrived surfaces in
|
||||
@@ -1411,8 +1438,9 @@ return, quak never keeps a truncated file.
|
||||
`Client.login({ email, password, totp?, emailOTP?, apiOptions? })` logs in and
|
||||
resolves to a `Client`. `totp` and `emailOTP` are callbacks that return the code
|
||||
when the account asks for one. Without the callback the account needs, login
|
||||
throws. A passkey challenge always throws. `apiOptions` configures the HTTP
|
||||
client (see below).
|
||||
throws. A passkey challenge always throws. After an email code, login expects
|
||||
the key attributes and the token. When the server asks for a second factor
|
||||
instead, login throws. `apiOptions` configures the HTTP client (see below).
|
||||
|
||||
The `Client` holds the session token, the master key, the secret key and the
|
||||
public key in memory. There is no on-disk session store in the library; the
|
||||
@@ -1422,9 +1450,16 @@ write wherever it likes. `Client.fromJSON(snapshot, apiOptions?)` restores a
|
||||
working client from that snapshot without logging in again. It checks every
|
||||
field and each key's length first, and throws an error naming the bad field.
|
||||
|
||||
`Client.fromJSON` works only once libsodium, the cryptography library quak uses,
|
||||
is loaded. `Client.login` loads it, and `Client.fromJSON` does not. A program
|
||||
that restores a session in a new process first runs `await sodium.ready`, with
|
||||
`sodium` imported from `libsodium-wrappers-sumo`, which quak depends on. The
|
||||
exported `beginLogin`, `unwrapAuth`, `decryptCollection`, `decryptFile`,
|
||||
`downloadFile` and `downloadThumbnail` need the same.
|
||||
|
||||
`client.logout()` clears the token and zeroes the key buffers in place. After
|
||||
it, every other method on that client throws. It does not contact the server, so
|
||||
the token stays valid there and in any saved snapshot. Call
|
||||
it, every other method on that client throws. `logout()` does not contact the
|
||||
server, so the token stays valid there and in any saved snapshot. Call
|
||||
`await client.logoutOnServer()` before `logout()` to end the session on the
|
||||
server.
|
||||
|
||||
@@ -1444,8 +1479,10 @@ return only the decrypted records that are not deleted, from `sinceTime` `0`.
|
||||
`client.fetchMLData({ fileIDs, fileKeys })` fetches and decrypts the
|
||||
machine-learning data of up to 200 files. `client.downloadFile(file, outPath?)`
|
||||
and `client.downloadThumbnail(file, outPath?)` download and decrypt one file to
|
||||
`outPath`. Without `outPath`, they write to the working directory under the
|
||||
names `quak get` and `quak get-thumb` would use.
|
||||
`outPath`. For a live photo, `downloadFile` writes the image and the video, each
|
||||
named `outPath` with its own extension. `quak get --out` instead puts the image
|
||||
at the path itself. Without `outPath`, they write to the working directory under
|
||||
the names `quak get` and `quak get-thumb` would use.
|
||||
|
||||
`ApiClientOptions`:
|
||||
|
||||
@@ -1640,8 +1677,8 @@ Login derives the keys in this order:
|
||||
the code (`POST /users/two-factor/verify`); the server then returns the key
|
||||
attributes and the token. quak cannot answer a passkey request. When the
|
||||
account has email verification on (`isEmailMFAEnabled`), an email code
|
||||
replaces SRP, and the server returns the key attributes and the token after
|
||||
it.
|
||||
replaces SRP. quak then expects the key attributes and the token in the
|
||||
answer, and fails the login when the server asks for a second factor instead.
|
||||
5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK
|
||||
yields the 32-byte master key.
|
||||
6. `crypto_secretbox_open_easy` over the encrypted secret key with the master
|
||||
@@ -1720,36 +1757,35 @@ This repository adheres to the
|
||||
standard. Normalized scripts in `script/` are the entrypoints for the
|
||||
development workflow, and most Makefile targets are thin shims that call them.
|
||||
The scripts are POSIX `sh`, not bash, so they run in minimal containers such as
|
||||
Alpine. The scripts marked "ours" are this repository's own additions to the
|
||||
standard.
|
||||
Alpine. Of these, only `bootstrap`, `setup`, `test` and `cibuild` come from the
|
||||
standard. The others are additions to it.
|
||||
|
||||
- `script/bootstrap`: install all dependencies. It installs `make`, `git`, Node
|
||||
and `yarn` when they are missing, then runs `yarn install --frozen-lockfile`.
|
||||
- `script/setup`: set up the repo for development after a fresh clone. It runs
|
||||
`script/bootstrap`, then `script/install-precommit`.
|
||||
- `script/projectname` (ours): print the project name. `script/docker` uses it
|
||||
for the image tag.
|
||||
- `script/build` (ours): compile the TypeScript sources into `dist/` and stamp
|
||||
the version into `dist/package.json`. Then check that the entrypoints
|
||||
- `script/projectname`: print the project name. `script/docker` uses it for the
|
||||
image tag.
|
||||
- `script/build`: compile the TypeScript sources into `dist/` and stamp the
|
||||
version into `dist/package.json`. Then check that the entrypoints
|
||||
`package.json` declares (`main`, `types`, `bin`) were written, and make the
|
||||
command line executable.
|
||||
- `script/version` (ours): print the version `script/build` stamps (see
|
||||
"Version").
|
||||
- `script/version`: print the version `script/build` stamps (see "Version").
|
||||
- `script/test`: run the test suite, by building the `test` phase of the
|
||||
`Dockerfile`. Requires Docker.
|
||||
- `script/lint`: run eslint and a prettier check, by building the `lint` phase
|
||||
of the `Dockerfile`. Requires Docker.
|
||||
- `script/fmt`: format all files with prettier. It rewrites files in place.
|
||||
- `script/fmt-check`: check formatting on the host. It is read-only.
|
||||
- `script/check` (ours): run all checks: `test`, `lint`, `fmt-check`.
|
||||
- `script/check`: run all checks: `test`, `lint`, `fmt-check`.
|
||||
- `script/docker`: build the image, tagged with `script/projectname`.
|
||||
- `script/cibuild`: what CI runs: `script/bootstrap`, then `script/check`, then
|
||||
the image build.
|
||||
- `script/precommit` (ours): run by the git pre-commit hook. It runs
|
||||
`script/lint` and `script/fmt-check`, and not the tests, so that a commit of
|
||||
failing tests can land (see "Development workflow").
|
||||
- `script/install-precommit` (ours): install the git pre-commit hook.
|
||||
`make hooks` calls it.
|
||||
- `script/precommit`: run by the git pre-commit hook. It runs `script/lint` and
|
||||
`script/fmt-check`, and not the tests, so that a commit of failing tests can
|
||||
land (see "Development workflow").
|
||||
- `script/install-precommit`: install the git pre-commit hook. `make hooks`
|
||||
calls it.
|
||||
|
||||
The Makefile targets are:
|
||||
|
||||
@@ -1816,8 +1852,8 @@ empty, `dev` or `unknown`, because such a build could not be traced to its
|
||||
commit.
|
||||
|
||||
`.dockerignore` therefore does not leave out `.git`, so any `docker build .` of
|
||||
a clone stamps the commit it was built from. A shallow clone stamps a tag only
|
||||
when the cloned commit itself carries one, and otherwise the short commit.
|
||||
a clone stamps the commit it was built from. A clone of depth 1 stamps a tag
|
||||
only when its commit carries one, and otherwise the short commit.
|
||||
`.dockerignore` does leave out `.git/config`, which holds the clone's remote URL
|
||||
and any credential in it; `git describe` does not need that file.
|
||||
`script/docker` and `script/cibuild` work out the version on the host, with
|
||||
@@ -1873,7 +1909,12 @@ These apply to people and to coding agents alike.
|
||||
- the nvm archive `script/bootstrap` downloads, and the Python packages
|
||||
`bin/clip-keyword-vectors` installs, by SHA-256 hash;
|
||||
- Node and yarn, when `script/bootstrap` installs them, by version.
|
||||
- bun is not pinned: `make build-bin` uses whatever `nix-shell -p bun` provides.
|
||||
- Not pinned:
|
||||
- bun, which `make build-bin` takes from `nix-shell -p bun`;
|
||||
- `make` and `git`, which `script/bootstrap` installs from the system
|
||||
package manager when they are missing, as every `Dockerfile` stage does;
|
||||
- `bash` and `curl`, which `script/bootstrap` installs the same way when it
|
||||
has to install Node.
|
||||
- Follow the workflow above: branches off `next`, tests first and committed
|
||||
failing, and `next` merged into `main` by sneak alone.
|
||||
- Before every commit, `make lint` and `make fmt-check` must pass. `make lint`
|
||||
|
||||
+10
-7
@@ -1,15 +1,18 @@
|
||||
// The CLI's log level, and the one function the commands write stderr lines
|
||||
// through.
|
||||
//
|
||||
// Every line quak prints on stderr is a log line at one of six levels, from
|
||||
// the quietest: fatal, error, warning, info, debug, trace. A run prints the
|
||||
// lines at its own level and at every quieter one. Its level is info unless
|
||||
// `-v` or `-q` asks for another (see "What quak prints" in the README). Every
|
||||
// command that exits non-zero logs a fatal line saying why, so even `-q`, which
|
||||
// prints only fatal lines, never leaves a non-zero exit unexplained. An error
|
||||
// Every line quak logs goes to stderr, at one of six levels, from the
|
||||
// quietest: fatal, error, warning, info, debug, trace. A run prints the lines
|
||||
// at its own level and at every quieter one. Its level is info unless `-v` or
|
||||
// `-q` asks for another (see "What quak prints" in the README). A command that
|
||||
// runs and exits non-zero logs a fatal line saying why, so even `-q`, which
|
||||
// prints only fatal lines, never leaves such an exit unexplained. An error
|
||||
// line reports something that failed, such as a file or an album, and a
|
||||
// warning something that did not fail but needs the user's attention. stdout
|
||||
// carries only a command's output and is not a log.
|
||||
// carries a command's output and the prompts of `login`, and is not a log.
|
||||
// Two things on stderr are not log lines: the `error:` message for a
|
||||
// command line that quak cannot parse, and the usage printed when no command
|
||||
// is given.
|
||||
//
|
||||
// The library itself writes no lines: its layers report through callbacks, and
|
||||
// the CLI logs what they report. `logRequests` gives the API client options
|
||||
|
||||
@@ -328,7 +328,7 @@ describe("what the library reports", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
// Each operation is named in plain words, as the README names it.
|
||||
// Each operation is named in plain words.
|
||||
it("logs a refresh's progress at info, and its start and end at debug", () => {
|
||||
const { log, lines } = collect();
|
||||
const onProgress = logRefresh(log);
|
||||
|
||||
Reference in New Issue
Block a user