1 Commits
Author SHA1 Message Date
sneak c80651b43d README.md rewritten sentence by sentence (closes #219)
check / check (push) Canceled after 0s
The old text served as the list of what a reader needs covered. The new
text leads with what quak is, what it is for and how to use it. Every
command, option, exit code and file layout follows in reference sections,
then the library reference, then design, entrypoints and development.
Every statement was checked against the code on next; the places where
the old text disagreed with it are listed in the pull request. No sentence
runs past about 30 words, every table is now a list, and no issue links or
project history remain in text meant for users. Two code comments that
named README sections now name the new ones.

Model: fable-5-1
2026-10-26 11:34:55 +00:00
+99 -129
View File
@@ -72,9 +72,7 @@ quak is not published to a package registry: `package.json` marks it `private`,
and `dist/` is not committed. Clone and build it as above, then add the clone to
your own program's `package.json` by its path, for example
`yarn add file:../quak`. The package's `main`, `types` and `exports` fields
point into `dist/`, so the clone must be built first. The package is an ES
module only: `package.json` sets `"type": "module"`, and its `exports` field
offers only `import`, so a CommonJS program cannot `require` it.
point into `dist/`, so the clone must be built first.
```ts
import { Client, Library } from "quak";
@@ -90,8 +88,7 @@ const client = await Client.login({
const lib = await Library.open({
client,
// Without these two, open() also starts downloading every thumbnail in
// the account and the latest week's originals into the cache. It fetches
// machine-learning data after each refresh either way.
// the account and the latest week's originals into the cache.
precacheThumbnails: false,
precacheOriginals: false,
});
@@ -121,11 +118,8 @@ await lib.close();
`Library.open({ client })` with no other options does more than the example asks
for. It refreshes every 3 seconds until `close()`. It also downloads, in the
background, every thumbnail in the account and the originals of the favorites
album and of the latest week (see "Opening a library"). The example turns those
two downloads off. After each refresh the library also fetches each file's
machine-learning data into the cache. That is the faces and the other data
Ente's apps compute for a photo (see "The JSON beside each original"). No option
turns this off.
album and of the latest week (see "Opening a library"). The example turns the
downloads off.
The `Client` under the library handles login, saved sessions and the raw
requests. The "Library reference" below covers both.
@@ -318,10 +312,9 @@ however many collections hold it. An original the cache already holds is copied
from there instead.
Progress goes to stderr: one line per original fetched, and a `FAILED` line for
each original or album that failed. The run ends with a summary of counts and,
when any file failed, a `Failed files:` list of every failed file. With
`--json`, the summary and the list are replaced by one JSON object on stdout.
Its fields are:
each file or album that failed. The run ends with a summary of counts and a
`Failed files:` list. With `--json`, the summary and the list are replaced by
one JSON object on stdout. Its fields are:
- `totalFiles`: how many distinct files the run covered, however many
collections hold each;
@@ -356,13 +349,12 @@ directory". Photo content is not written. With `--exif`, or its alias `--all`,
the command also fetches each file's original and records the image's own
metadata from it (see "The metadata dump directory").
Ente's apps compute machine-learning data for each photo, such as the faces in
it (see "The JSON beside each original"). The command fetches it in requests of
up to 200 files. When a request fails, each of its files is written with the
reason in `mlDataError` instead of `mlData`, and the dump goes on. A file whose
data does not decrypt, decompress or parse also gets `mlDataError`. When any
file was written with `mlDataError`, the last line says how many and the exit
status is 1.
The command fetches the machine-learning data Ente holds for each file, in
requests of up to 200 files. When a request fails, each of its files is written
with the reason in `mlDataError` instead of `mlData`, and the dump goes on. A
file whose data does not decrypt, decompress or parse also gets `mlDataError`.
When any file was written with `mlDataError`, the last line says how many and
the exit status is 1.
A collection or file that quak cannot decrypt is left out of the dump without a
word, and does not change the exit status.
@@ -407,7 +399,7 @@ Each file ends in one of three states:
- `fixed`: a thumbnail was uploaded.
- `skipped`: the helper cannot make a thumbnail for the file, or the server
would refuse it. This covers:
would refuse it. A skipped file does not affect the exit status. This covers:
- a file another account owns;
- a file that is not an image;
- an image that is not a JPEG;
@@ -420,9 +412,9 @@ Without `--json`, the command ends with counts of fixed, skipped and failed
files, a `Skipped:` list and a `Failed files:` list. With `--json`, one JSON
array on stdout replaces them; each object holds `fileID`, `title`,
`collection`, `status` and, for a skipped or failed file, `reason`. When any
file failed, the exit status is 1. A skipped file does not change it. When
nothing needs fixing, the command logs `No missing thumbnails found.` and
exits 0. With `--json` it then writes nothing on stdout.
file failed, the exit status is 1. When nothing needs fixing, the command logs
`No missing thumbnails found.` and exits 0. With `--json` it then writes nothing
on stdout.
quak refuses the whole command, before any request, when one `--file` ID is
invalid (see "IDs").
@@ -468,28 +460,23 @@ before any request is sent. The command logs `Invalid file ID: <arg>` or
### What quak prints
stdout holds only a command's output: the listing, the JSON, or nothing. The
prompts of `login` go to stdout too. Every other line goes to stderr as a log
line at one of six levels. From the quietest, the levels are fatal, error,
warning, info, debug and trace. A run prints the lines at its own level and at
every quieter one. The level is info unless `-v`, `-vv` or `-q` says otherwise.
The one exception is a command line quak cannot parse, such as a missing
argument, an unknown option or `-q` with `-v`. quak refuses it with one
`error: …` line that is not a log line, and exits 1. `-q` does not hide that
line. `quak` with no command prints the usage on stderr and exits 1.
prompts of `login` go to stdout too. Every other line goes to stderr, and every
stderr line is a log line at one of six levels. From the quietest, the levels
are fatal, error, warning, info, debug and trace. A run prints the lines at its
own level and at every quieter one. The level is info unless `-v`, `-vv` or `-q`
says otherwise.
`yarn quak` prints yarn's own lines on stdout as well. To pipe a command's
output, run `yarn --silent quak files --json` or
`node dist/bin/quak.js files --json` instead.
- Fatal: why a command exits non-zero. A command that runs and exits non-zero
logs at least one fatal line. A command that exits 1 because files or albums
failed logs it last, saying how many failed and where to look.
- Error: something that failed without ending the run. These lines are:
- the `FAILED` lines of `backup` and `fix-missing-thumbnails`;
- their `Failed files:` lists;
- the `ML data` lines of `backup-metadata`;
- `[<collection>] Could not check <title>` from the thumbnail helpers.
- Fatal: why a command exits non-zero. Every command that exits non-zero logs at
least one fatal line. A command that exits 1 because files or albums failed
logs it last, saying how many failed and where to look.
- Error: something that failed without ending the run. These are the `FAILED`
lines of `backup` and `fix-missing-thumbnails`, their `Failed files:` lists,
the `ML data` lines of `backup-metadata`, and
`[<collection>] Could not check <title>` from the thumbnail helpers.
- Warning: something that did not fail but needs your attention. The only
warning is the one `logout` prints: the cache directory still holds decrypted
data.
@@ -531,8 +518,9 @@ end of the `Failed files:` list. For `list-missing-thumbnails`, it is
An unexpected error is printed as one line, `quak: <message>`, with no stack
trace. `<message>` is the error's own message, followed by the message of each
error that caused it, separated by `: `. Some lines a command logs itself, such
as `File <id> not found` and `Login failed: <reason>`, have no prefix.
error that caused it, separated by `: `. An error a command expects, such as
`File <id> not found` or `Login failed: <reason>`, is printed without the
prefix.
`yarn quak` runs quak under Node. Node's `fetch` reports every network failure
as `fetch failed`, with the reason as its cause. A run that could not reach the
@@ -565,8 +553,8 @@ account's three keys, in base64 and unencrypted. Treat it as you would treat the
password. A missing file means "not logged in". A file that cannot be read back
is reported as corrupt, with the reason. Run `quak logout`, then `quak login`.
A command that needs a session exits 3 when the file is missing or corrupt. It
also exits 3 when the server no longer accepts the session, and prints
Both cases exit 3, and so does a session the server no longer accepts: the
command then prints
`quak: the saved session is no longer valid; run "quak login"`. `quak backup`
learns this at its first refresh. By then it has written only its lock. When the
session stops working partway through a backup, each remaining file goes into
@@ -618,8 +606,8 @@ holds a second image or video. It also refuses one whose files unpack to more
than 20 times its size plus 16 MiB. An original with no recorded hash, uploaded
by a very old client, is stored unchecked.
The cache has a size limit for originals (see "Opening a library"). Thumbnails
have none. A live photo's image and video leave the cache together.
The originals in the cache are bounded (see "Opening a library"). Thumbnails are
not. A live photo's image and video leave the cache together.
### The backup directory
@@ -693,13 +681,13 @@ with an error starting `ML data: `. Nothing is cached for it, so the next run
asks for its data again. The JSON files are rewritten on every run, so data that
arrived since the last run appears.
Every file but a video also holds `imageMetadata`: what the original itself
For an image, the file also holds `imageMetadata`: what the original itself
records. That is its `format`, `width` and `height` for a JPEG, its EXIF tags as
`exif`, and its XMP packet as `xmp`. EXIF and XMP are the two blocks of metadata
a camera or an editor writes inside an image file. For a live photo, these come
from its image. An original with none of them gets `{}`. When the original
cannot be read, the reason is in `imageMetadataError` instead, and neither the
file nor the run fails.
from its image. An original with none of them gets `{}`. A video gets no
`imageMetadata`. When the original cannot be read, the reason is in
`imageMetadataError` instead, and neither the file nor the run fails.
EXIF is read with [exifreader](https://github.com/mattiasw/ExifReader) from any
image format it reads, such as JPEG, HEIC, AVIF, PNG and WebP. `exif` holds each
@@ -776,7 +764,7 @@ touched: a file or symlink you put there stays, and a directory that still holds
one stays too, with its JSON.
A collection whose directory or JSON cannot be written does not stop the run.
For example, a file you put there may already have the directory's name. The run
One whose directory name is taken by a file you put there is an example. The run
logs `FAILED album <name>: <reason>`, lists the collection's ID in
`failedAlbums`, and exits 1. The other collections and `failures.json` are still
written.
@@ -807,8 +795,7 @@ stores the two files, which a photo viewer can open: each is
`YYYY-MM-DD.<fileID>.<ext>` with the extension it has inside the ZIP, such as
`2026-03-01.12345.heic` and `2026-03-01.12345.mov`. The file
`YYYY-MM-DD.<fileID>.livephoto.json` beside them names which is which. The live
photo counts as stored only when its `.livephoto.json` and both files it names
are present and not empty.
photo counts as stored only when both files are present and not empty.
#### `failures.json`
@@ -830,7 +817,7 @@ The kinds are:
A file leaves the list once it succeeds, or once it is no longer in the library.
A library caller that backs up only some albums (`onlyAlbumNames`) also drops a
file outside them. `failures.json` is deleted when the list is empty.
file outside them. The file is deleted when the list is empty.
#### `backup.lock`
@@ -841,19 +828,18 @@ refused run sends no request. It removes the lock when it ends, whether it
succeeded or failed.
The lock is a directory that
[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates.
While the backup runs, the backup updates the directory's modification time
every 5 seconds. A second backup of the directory fails at once: it prints
`quak: another backup of <dir> is running` and exits 2. A run that is killed, or
cut off by a crash, can leave the lock behind. Once the lock has gone 10 seconds
without an update, the next run takes it over, so nobody has to remove it by
hand.
[proper-lockfile](https://github.com/moxystudio/node-proper-lockfile) creates
and keeps touching every 5 seconds while the backup runs. A second backup of the
directory fails at once: it prints `quak: another backup of <dir> is running`
and exits 2. A run that is killed, or cut off by a crash, can leave the lock
behind. Once the lock has gone 10 seconds untouched, the next run takes it over,
so nobody has to remove it by hand.
A running backup loses its lock when someone removes `backup.lock`. Another
backup takes the lock over when this one has not updated it for 10 seconds, for
example because this process was stopped. The backup notices at its next update
of the lock, and stops before it starts another file. The originals it is
downloading at that moment are still stored. It leaves `failures.json` and
A running backup loses its lock when someone removes `backup.lock`. It also
loses it when another backup takes it over, because this one left it untouched
for 10 seconds, as a stopped process does. The backup notices the next time it
goes to touch the lock, and stops before it starts another file. The originals
it is downloading at that moment are still stored. It leaves `failures.json` and
`backup.lock` as they are, since another backup may be using them. It prints
`quak: lost the lock on <dir>: <reason>` and exits 1.
@@ -886,16 +872,13 @@ The summary and the `--json` output add the counts `verified`, `mismatched` and
`unchecked`. Each counts only originals that were already stored. A mismatch
that was replaced with matching bytes does not make the exit code non-zero.
Two kinds of failure stay in `failures.json` only until the next run:
A mismatched original whose new copy failed, and a stored original that cannot
be read, stay in `failures.json` only until the next run. A run without
`--verify` finds the original in place, counts it as skipped, and drops the
entry, so that run can exit 0. Only a `--verify` run hashes it again.
- a mismatched original whose new copy failed;
- a stored original that cannot be read.
A run without `--verify` finds the original in place, counts it as skipped, and
drops the entry, so that run can exit 0. Only a `--verify` run hashes it again.
Without `--verify`, no stored original is hashed, the summary is unchanged, and
the three counts are 0 in the JSON.
Without `--verify`, nothing is hashed, the summary is unchanged, and the three
counts are 0 in the JSON.
#### Temporary files and crash safety
@@ -917,8 +900,7 @@ scheme. Opening a library deletes such leftovers in the cache's `originals/` and
The rename replaces whatever was at the destination rather than writing through
it. A symlink there is replaced, not followed, and the new file has the
temporary file's permissions, not those of the file it replaced. Each original
is written with mode `0600`, readable by its owner only.
temporary file's permissions, not those of the file it replaced.
### The metadata dump directory
@@ -934,10 +916,9 @@ is written with mode `0600`, readable by its owner only.
the collection
```
Each run writes into `<dir>` and deletes nothing. `account.json` holds `email`
and `userID`. In `<id>-<name>`, the name has unsafe characters replaced. It is
cut short so the directory name fits in 255 bytes. The ID keeps two collections
of the same name apart.
`account.json` holds `email` and `userID`. The name after the ID in a
collection's directory name has unsafe characters replaced and is cut short so
that the whole name fits in 255 bytes; the ID keeps the directories apart.
`_collection.json` holds the fields a backup's collection JSON holds, except
`files`. Each `<fileID>.json` holds the fields a backup's JSON beside an
@@ -947,8 +928,7 @@ original holds, with these differences:
then downloaded, videos included, and read as a backup reads it (see "The JSON
beside each original").
- A file whose original holds no image metadata gets no `imageMetadata` field.
- An original that cannot be downloaded or read gives `imageMetadataError`
instead.
- An original that cannot be read gives `imageMetadataError` instead.
- A file in several collections has its original read once; its image metadata
is written with the first of them.
@@ -1026,10 +1006,10 @@ onto what the cache holds, empty or not. A refresh then starts only when
- `contentSource`: a replacement for the client's own byte source. Mainly for
tests.
In each pool, your own requests take the next free slot ahead of the two
background downloads. A file already in the cache is not fetched again. When the
originals cache grows past its limit, the least recently used originals are
evicted, but never a pinned one, and never one whose fetch is still in flight.
The two background downloads wait while your own requests run. A file already in
the cache is not fetched again. When the originals cache grows past its limit,
the least recently used originals are evicted, but never a pinned one, and never
one whose fetch is still in flight.
`onProgress` receives a `RefreshEvent`. Its `operation` names the work:
`"refresh"`, `"fetchMLData"` for a machine-learning fetch, and
@@ -1040,9 +1020,7 @@ the error message instead. In between, a refresh sends `"progress"` after each
collection whose files it fetched: `fetched` is the number of collections
fetched so far, and `total` the number it will fetch. A refresh fetches the
files of each collection that is new or changed since the cache's copy, so on an
empty cache it fetches those of every collection. The `"done"` event of a
machine-learning fetch, and of each background download, also carries `fetched`:
how many items it newly stored.
empty cache it fetches those of every collection.
`lib.status()` returns a `LibraryStatus`. It holds the collection and file
counts. `lastRefreshAt` is the time of the last refresh that succeeded, and
@@ -1080,11 +1058,11 @@ running does it start one. When the refresh fails, `fresh()` rejects. Default
reads stay silent and keep serving the last good copy.
`await lib.freshCollections()` is the fresh read of the collection list alone:
one request, answered with every collection that is not deleted and decrypts, as
`Collection` objects, in the server's order. It fetches no collection's files
and stores nothing, not even the list. It stores nothing because a refresh
fetches a collection's files only when the collection is new or changed. A
collection stored without its files would never get them.
one request, answered with every live collection that decrypts, as `Collection`
objects, in the server's order. It fetches no collection's files and stores
nothing, not even the list. It stores nothing because a refresh fetches a
collection's files only when the collection is new or changed. A collection
stored without its files would never get them.
`await lib.freshFiles(collection)` is the fresh read of one collection's files,
as `EnteFile` objects, in the server's order, each file once. `collection` is an
@@ -1111,9 +1089,8 @@ later refresh gets a copy that decrypts or learns that the server deleted it.
- `lib.timeline.groups({ groupBy, filter? })` returns `TimelineGroup[]`, newest
group first. `groupBy` is `"day"`, `"week"` or `"month"`, and the group keys
are `YYYY-MM-DD`, `YYYY-Www` (the ISO week) and `YYYY-MM`. Each group holds
`key`, `startsAt` and `fileIDs`. `key` is that group key. `startsAt` is the
start of the period, in local-time milliseconds. `fileIDs` lists its files,
newest first. Weeks start on Monday.
`startsAt`, local time in milliseconds, and `fileIDs`, newest first. Weeks
start on Monday.
A `PhotoFilter` narrows the timeline. Its fields combine with AND:
@@ -1274,9 +1251,8 @@ releases it once the backup ends.
### Request pools
`RequestPools` holds three independent pools, each with its own limit on how
many requests run at once, because Ente meters these three kinds of traffic
differently:
`RequestPools` holds three independent bounded pools, because Ente meters these
three kinds of traffic differently:
- metadata, with 10 slots;
- content, with 5;
@@ -1364,16 +1340,13 @@ A request that changes state is not blindly replayed. Every `POST` and `PUT` in
the endpoint list under "HTTP API" goes through this rule. Verifying a second
factor, for one, uses up one of a small number of attempts. Such a request is
retried only when every error code in its `cause` chain proves that no TCP
connection to the server ever existed. Three codes prove that:
- `ENOTFOUND`: name resolution found no address;
- `EAI_AGAIN`: name resolution failed for the moment;
- `ECONNREFUSED`: the peer refused the connection.
A 5xx, a reset mid-flight and a deadline are all left to the caller, because
each of them can happen after the server has already acted. These requests do
not follow redirects either: a redirect means the server already received the
request, so it is reported as an error and not retried.
connection to the server ever existed. Three codes prove that: `ENOTFOUND` and
`EAI_AGAIN`, when name resolution produced no address, and `ECONNREFUSED`, when
the peer refused the connection. A 5xx, a reset mid-flight and a deadline are
all left to the caller, because each of them can happen after the server has
already acted. These requests do not follow redirects either: a redirect means
the server already received the request, so it is reported as an error and not
retried.
The routing errors `EHOSTUNREACH`, `ENETUNREACH` and `ENETDOWN` are left out of
that rule although they look like connect-time failures. On Linux they are also
@@ -1400,11 +1373,11 @@ One imprecision is deliberate. When a body ends partway through an encrypted
chunk, the authentication check fails and carries no sign of why, so a cut
connection and corrupt bytes look the same. quak reports that as truncation,
which means it is retried. quak can tell the two apart only in a body of more
than one chunk. There, a chunk that fails while the stream carries on past it
stays an authentication failure and is not retried. Most thumbnails and every
small file are one chunk. There a wrong key, corrupt bytes and a cut connection
look alike, and all are retried. The attempt count limits the extra retries. In
return, quak never keeps a truncated file.
than one chunk: a chunk that fails while the stream carries on past it stays an
authentication failure and is not retried. Most thumbnails and every small file
are one chunk. There a wrong key, corrupt bytes and a cut connection look alike,
and all are retried. The cost is bounded by the attempt count, and it buys never
keeping a truncated file.
### The `Client`
@@ -1424,9 +1397,8 @@ field and each key's length first, and throws an error naming the bad field.
`client.logout()` clears the token and zeroes the key buffers in place. After
it, every other method on that client throws. It does not contact the server, so
the token stays valid there and in any saved snapshot. Call
`await client.logoutOnServer()` before `logout()` to end the session on the
server.
the token stays valid there and in any saved snapshot;
`await client.logoutOnServer()` ends the session on the server first.
`client.whoami()` returns `{ email, userID }`.
@@ -1434,13 +1406,13 @@ server.
`client.filesSince({ collectionID, collectionKey, sinceTime })` enumerate the
changes since a microsecond timestamp, with `0` for everything. Each returns:
- `collections` or `files`: the decrypted records that are not deleted;
- `collections` or `files`: the decrypted live records;
- `deleted`: the IDs of deleted records;
- `undecryptable`: each record that did not decrypt, as `{ id, error }`;
- `cursor`: the timestamp to pass next time.
`client.listCollections()` and `client.listFiles(collectionID, collectionKey)`
return only the decrypted records that are not deleted, from `sinceTime` `0`.
return only the live records, from `sinceTime` `0`.
`client.fetchMLData({ fileIDs, fileKeys })` fetches and decrypts the
machine-learning data of up to 200 files. `client.downloadFile(file, outPath?)`
and `client.downloadThumbnail(file, outPath?)` download and decrypt one file to
@@ -1639,9 +1611,8 @@ Login derives the keys in this order:
token, or first asks for a second factor. quak answers a TOTP request with
the code (`POST /users/two-factor/verify`); the server then returns the key
attributes and the token. quak cannot answer a passkey request. When the
account has email verification on (`isEmailMFAEnabled`), an email code
replaces SRP, and the server returns the key attributes and the token after
it.
account has email MFA on (`isEmailMFAEnabled`), an email code replaces SRP,
and the server returns the key attributes and the token after it.
5. `crypto_secretbox_open_easy` over the encrypted master key with the KEK
yields the 32-byte master key.
6. `crypto_secretbox_open_easy` over the encrypted secret key with the master
@@ -1867,13 +1838,12 @@ These apply to people and to coding agents alike.
<https://git.eeqj.de/sneak/prompts> and covers repository structure, tooling,
Makefile targets, Dockerfile conventions, dependency pinning and commit
hygiene. Never `git add -A`. Never force-push `main`.
- Dependencies are pinned:
- Every dependency is pinned:
- npm packages by hash, in `yarn.lock`;
- container images by digest;
- the nvm archive `script/bootstrap` downloads, and the Python packages
`bin/clip-keyword-vectors` installs, by SHA-256 hash;
- Node and yarn, when `script/bootstrap` installs them, by version.
- bun is not pinned: `make build-bin` uses whatever `nix-shell -p bun` provides.
- Node and yarn by version, in `script/bootstrap`.
- Follow the workflow above: branches off `next`, tests first and committed
failing, and `next` merged into `main` by sneak alone.
- Before every commit, `make lint` and `make fmt-check` must pass. `make lint`