Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2492946e9d | ||
|
|
9bd7e8f099 | ||
|
|
50e2381cad | ||
|
|
37cdbc6b9a | ||
|
|
1bf18baf00 | ||
|
|
7878ff3812 | ||
|
|
8e120a4f49 | ||
|
|
30fc349b37 |
No files matched your search
@@ -26,10 +26,8 @@ check:
|
|||||||
build:
|
build:
|
||||||
@script/build
|
@script/build
|
||||||
|
|
||||||
# Bundles the built dist/, so the binary reports the version script/build
|
build-bin:
|
||||||
# stamped.
|
@script/build-bin
|
||||||
build-bin: build
|
|
||||||
nix-shell -p bun --run "bun build dist/bin/quak.js --compile --outfile bin/quak"
|
|
||||||
|
|
||||||
install: build-bin
|
install: build-bin
|
||||||
mkdir -p ~/bin
|
mkdir -p ~/bin
|
||||||
|
|||||||
@@ -31,7 +31,8 @@ again on the next run. Every download is decrypted on its way to disk.
|
|||||||
|
|
||||||
## Getting Started
|
## Getting Started
|
||||||
|
|
||||||
The commands below need `git`, Node 22 or later and `yarn`.
|
The commands below need `git`, `yarn` and Node 22.13 or later. Node 23 needs
|
||||||
|
23.5 or later.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone https://git.eeqj.de/sneak/quak.git
|
git clone https://git.eeqj.de/sneak/quak.git
|
||||||
@@ -69,11 +70,29 @@ The library keeps a local copy of the account's album and file lists, the cache.
|
|||||||
It brings the cache up to date with the server in rounds called refreshes.
|
It brings the cache up to date with the server in rounds called refreshes.
|
||||||
|
|
||||||
quak is not published to a package registry: `package.json` marks it `private`,
|
quak is not published to a package registry: `package.json` marks it `private`,
|
||||||
and `dist/` is not committed. Clone and build it as above, then add the clone to
|
and `dist/` is not committed. Clone and build it as above, then link the clone
|
||||||
your own program's `package.json` by its path, for example
|
into your own program:
|
||||||
`yarn add file:../quak`. The package's `main`, `types` and `exports` fields
|
|
||||||
point into `dist/`, so the clone must be built first. The package is an ES
|
```bash
|
||||||
module only, so a CommonJS program cannot `require` it.
|
# In the quak clone:
|
||||||
|
yarn link
|
||||||
|
|
||||||
|
# In your program's directory:
|
||||||
|
yarn link quak
|
||||||
|
```
|
||||||
|
|
||||||
|
`yarn link` also installs the `quak` command in yarn's global bin directory, and
|
||||||
|
`yarn unlink` in the clone removes the link and the command.
|
||||||
|
|
||||||
|
A linked quak uses the dependencies that `yarn install` put in the clone, at the
|
||||||
|
versions in quak's `yarn.lock`. Avoid `yarn add file:../quak`: it resolves
|
||||||
|
quak's dependencies again, and their newer releases can need a newer Node.
|
||||||
|
`yarn link quak` does not add quak to your `package.json`, so run it again in
|
||||||
|
each new checkout of your program.
|
||||||
|
|
||||||
|
The package's `main`, `types` and `exports` fields point into `dist/`, so the
|
||||||
|
clone must be built first. The package is an ES module only, so a CommonJS
|
||||||
|
program cannot `require` it.
|
||||||
|
|
||||||
```ts
|
```ts
|
||||||
import { Client, Library } from "quak";
|
import { Client, Library } from "quak";
|
||||||
@@ -135,10 +154,15 @@ requests. The "Library reference" below covers both.
|
|||||||
## Running the backup from cron
|
## Running the backup from cron
|
||||||
|
|
||||||
`quak backup` never prompts, so cron can run it. `make install` builds quak as
|
`quak backup` never prompts, so cron can run it. `make install` builds quak as
|
||||||
one binary and copies it to `~/bin/quak`. Besides what Getting Started needs,
|
one binary and copies it to `~/bin/quak`. It uses bun, a JavaScript runtime, to
|
||||||
`make install` needs `make` and [nix](https://nixos.org/download). It uses bun,
|
bundle the built `dist/` into one executable. Besides what Getting Started
|
||||||
a JavaScript runtime, to bundle the built `dist/` into one executable. It runs
|
needs, `make install` needs `make`. It downloads bun from GitHub with `curl` and
|
||||||
bun through `nix-shell -p bun`.
|
unpacks it with `unzip`. It checks the download against a SHA-256 hash in the
|
||||||
|
repo. `make install` works on Linux and macOS, on x86-64 and arm64.
|
||||||
|
|
||||||
|
On Linux systems that use the musl C library, such as Alpine, `make install`
|
||||||
|
downloads bun's musl build. On Alpine, both that bun and `~/bin/quak` need the
|
||||||
|
`libstdc++` and `libgcc` packages.
|
||||||
|
|
||||||
Log in once with `~/bin/quak`, as the user the cron job will run as:
|
Log in once with `~/bin/quak`, as the user the cron job will run as:
|
||||||
|
|
||||||
@@ -205,7 +229,7 @@ below belong to quak itself, and go before or after the command's name.
|
|||||||
### The commands
|
### The commands
|
||||||
|
|
||||||
```
|
```
|
||||||
quak login
|
quak login [--server <url>]
|
||||||
quak whoami
|
quak whoami
|
||||||
quak logout
|
quak logout
|
||||||
quak collections [--json]
|
quak collections [--json]
|
||||||
@@ -222,13 +246,26 @@ Every command but `login` and `logout` needs a saved session. Every command but
|
|||||||
`login`, `whoami` and `logout` opens the library. How each one gets current data
|
`login`, `whoami` and `logout` opens the library. How each one gets current data
|
||||||
from the server is described under "How a command gets current data".
|
from the server is described under "How a command gets current data".
|
||||||
|
|
||||||
#### `quak login`
|
#### `quak login [--server <url>]`
|
||||||
|
|
||||||
Logs in to an account and saves the session. It reads the email from
|
Logs in to an account and saves the session. It reads the email from
|
||||||
`QUAK_EMAIL` and the password from `QUAK_PASSWORD`. For each one that is not
|
`QUAK_EMAIL` and the password from `QUAK_PASSWORD`. For each one that is not
|
||||||
set, it asks on the terminal. The password shows as `*` characters while it is
|
set, it asks on the terminal. The password shows as `*` characters while it is
|
||||||
typed.
|
typed.
|
||||||
|
|
||||||
|
quak logs in to Ente's hosted service unless you name a self-hosted Ente server.
|
||||||
|
Name it with `--server`, or with `QUAK_SERVER` when `--server` is not given.
|
||||||
|
Give the address of the server's API, for example
|
||||||
|
`quak login --server https://ente.example.com`. The address is an http or https
|
||||||
|
URL. It may include a path, but no query or fragment. quak refuses any other
|
||||||
|
address before it asks for anything. It logs
|
||||||
|
`Invalid server address "<url>": it must be an http or https URL with no query or fragment`,
|
||||||
|
and the exit status is 1.
|
||||||
|
|
||||||
|
The session file keeps the server's address. Every later command, cron runs
|
||||||
|
included, uses that server without further setting. To change server, log in
|
||||||
|
again.
|
||||||
|
|
||||||
When the account has a second factor, quak asks for the code. It asks
|
When the account has a second factor, quak asks for the code. It asks
|
||||||
`TOTP code: ` for a time-based code from an authenticator app. An account whose
|
`TOTP code: ` for a time-based code from an authenticator app. An account whose
|
||||||
only second factor is a passkey cannot log in with quak. When the account has
|
only second factor is a passkey cannot log in with quak. When the account has
|
||||||
@@ -302,7 +339,7 @@ title, with unsafe characters replaced (see "File names from the server"). With
|
|||||||
original is kept in the cache as well.
|
original is kept in the cache as well.
|
||||||
|
|
||||||
A file ID that no collection holds gives `File <id> not found`, and the exit
|
A file ID that no collection holds gives `File <id> not found`, and the exit
|
||||||
status is 1. The option `--collection <id>` is accepted and ignored.
|
status is 1.
|
||||||
|
|
||||||
A live photo is a still image and a short video that Ente stores as one file.
|
A live photo is a still image and a short video that Ente stores as one file.
|
||||||
`get` writes both, each named after the title with its own extension, as Ente's
|
`get` writes both, each named after the title with its own extension, as Ente's
|
||||||
@@ -573,9 +610,11 @@ Linux that is `$XDG_DATA_HOME/quak/session.json`, or
|
|||||||
mode `0600`, in a directory it creates with mode `0700`.
|
mode `0600`, in a directory it creates with mode `0700`.
|
||||||
|
|
||||||
The file holds the account's email, its user ID, the session token and the
|
The file holds the account's email, its user ID, the session token and the
|
||||||
account's three keys, in base64 and unencrypted. Treat it as you would treat the
|
account's three keys, in base64 and unencrypted. After a login to a self-hosted
|
||||||
password. A missing file means "not logged in". A file that cannot be read back
|
server, it also holds the server's address as `apiOrigin`. Treat it as you would
|
||||||
is reported as corrupt, with the reason. Run `quak logout`, then `quak login`.
|
treat the password. A missing file means "not logged in". A file that cannot be
|
||||||
|
read back is reported as corrupt, with the reason. Run `quak logout`, then
|
||||||
|
`quak login`.
|
||||||
|
|
||||||
A command that needs a session exits 3 when the file is missing or corrupt. It
|
A command that needs a session exits 3 when the file is missing or corrupt. It
|
||||||
also exits 3 when the server no longer accepts the session, and prints
|
also exits 3 when the server no longer accepts the session, and prints
|
||||||
@@ -593,6 +632,11 @@ Without `--cache-dir`, each account has its own cache directory:
|
|||||||
the variable is not set;
|
the variable is not set;
|
||||||
- on macOS, `~/Library/Caches/quak/<userID>`.
|
- on macOS, `~/Library/Caches/quak/<userID>`.
|
||||||
|
|
||||||
|
Two servers can give two accounts the same user ID. So an account on a
|
||||||
|
self-hosted server has the server's address, URL-encoded, between `quak` and the
|
||||||
|
user ID. For `https://ente.example.com`, that is
|
||||||
|
`quak/https%3A%2F%2Fente.example.com/<userID>`.
|
||||||
|
|
||||||
The directory is readable by its owner only (mode `0700`), because it holds
|
The directory is readable by its owner only (mode `0700`), because it holds
|
||||||
decrypted data, file keys included.
|
decrypted data, file keys included.
|
||||||
|
|
||||||
@@ -617,6 +661,13 @@ the library deletes it and `mldata/` and starts from an empty cache. Cached
|
|||||||
originals and thumbnails are kept. They are reached only through the records of
|
originals and thumbnails are kept. They are reached only through the records of
|
||||||
the current account, so another account's files are never served.
|
the current account, so another account's files are never served.
|
||||||
|
|
||||||
|
When `metadata.json` belongs to an account on another server, opening the
|
||||||
|
library also deletes the originals and thumbnails. Two servers can give two
|
||||||
|
files the same ID, so one server's cached original could be served for the
|
||||||
|
other's file. quak records the account in `metadata.json` before it stores any
|
||||||
|
original or thumbnail. It does so also when `quak get` finds a file on an empty
|
||||||
|
cache.
|
||||||
|
|
||||||
A file appears in the cache only once it is complete: each one is written to a
|
A file appears in the cache only once it is complete: each one is written to a
|
||||||
temporary file and renamed into place. Ente's clients record a hash of each
|
temporary file and renamed into place. Ente's clients record a hash of each
|
||||||
file's bytes at upload, the content hash. Every original whose metadata records
|
file's bytes at upload, the content hash. Every original whose metadata records
|
||||||
@@ -671,6 +722,9 @@ but letters and digits, or when it is longer than 16 characters.
|
|||||||
When the date or the time zone changes, the next run stores the original at its
|
When the date or the time zone changes, the next run stores the original at its
|
||||||
new path and leaves the old copy where it is.
|
new path and leaves the old copy where it is.
|
||||||
|
|
||||||
|
A file deleted in Ente keeps its original at its save path, and the JSON beside
|
||||||
|
it. The next run removes its symlinks from the collection directories.
|
||||||
|
|
||||||
The library gives the same path as `photo.savePath` and writes it with
|
The library gives the same path as `photo.savePath` and writes it with
|
||||||
`photo.download()` (see "A photo's content").
|
`photo.download()` (see "A photo's content").
|
||||||
|
|
||||||
@@ -1017,7 +1071,8 @@ onto what the cache holds, empty or not. A refresh then starts only when
|
|||||||
- `client`, required: the account's `Client`, or anything that satisfies
|
- `client`, required: the account's `Client`, or anything that satisfies
|
||||||
`LibraryClient`.
|
`LibraryClient`.
|
||||||
- `cacheDirectory`: where `metadata.json` and the content cache live. Default:
|
- `cacheDirectory`: where `metadata.json` and the content cache live. Default:
|
||||||
`quak/<userID>` under the per-user cache directory (see "The cache").
|
`quak/<userID>` under the per-user cache directory, with the URL-encoded
|
||||||
|
`apiOrigin` before `<userID>` for a client that has one (see "The cache").
|
||||||
- `downloadDirectory`: the root of the save paths. Default: `photos` in the
|
- `downloadDirectory`: the root of the save paths. Default: `photos` in the
|
||||||
working directory, resolved once when the library opens;
|
working directory, resolved once when the library opens;
|
||||||
`lib.downloadDirectory` holds the result. An original stored at its save path
|
`lib.downloadDirectory` holds the result. An original stored at its save path
|
||||||
@@ -1214,7 +1269,8 @@ do.
|
|||||||
fetch and cache as `photo.original()` and `photo.thumbnail()` do. `file` is an
|
fetch and cache as `photo.original()` and `photo.thumbnail()` do. `file` is an
|
||||||
`EnteFile` the caller holds, and the library need not hold it. `quak get` and
|
`EnteFile` the caller holds, and the library need not hold it. `quak get` and
|
||||||
`quak get-thumb` use them; on an empty cache they pass a file that
|
`quak get-thumb` use them; on an empty cache they pass a file that
|
||||||
`lib.freshFiles()` returned.
|
`lib.freshFiles()` returned. In a cache that names no account yet, they first
|
||||||
|
save the client's user ID and `apiOrigin` in `metadata.json`.
|
||||||
|
|
||||||
Lower-level reads return the decrypted model objects, which hold key material:
|
Lower-level reads return the decrypted model objects, which hold key material:
|
||||||
`lib.listCollections()`, `lib.getCollection(id)`, `lib.listFiles(collectionID)`,
|
`lib.listCollections()`, `lib.getCollection(id)`, `lib.listFiles(collectionID)`,
|
||||||
@@ -1450,6 +1506,14 @@ write wherever it likes. `Client.fromJSON(snapshot, apiOptions?)` restores a
|
|||||||
working client from that snapshot without logging in again. It checks every
|
working client from that snapshot without logging in again. It checks every
|
||||||
field and each key's length first, and throws an error naming the bad field.
|
field and each key's length first, and throws an error naming the bad field.
|
||||||
|
|
||||||
|
A client whose `apiOptions` named an `apiOrigin` puts it in its snapshot. The
|
||||||
|
restored client then uses that server, even when the `apiOptions` given to
|
||||||
|
`Client.fromJSON` name another. A snapshot without `apiOrigin` uses the
|
||||||
|
`apiOrigin` of those `apiOptions`, or the default server. `client.apiOrigin`
|
||||||
|
holds the one in use, or undefined when none was named. `Client.fromJSON`
|
||||||
|
refuses an `apiOrigin` that is not an http or https URL, or that has a query or
|
||||||
|
fragment, as `quak login --server` does.
|
||||||
|
|
||||||
`Client.fromJSON` works only once libsodium, the cryptography library quak uses,
|
`Client.fromJSON` works only once libsodium, the cryptography library quak uses,
|
||||||
is loaded. `Client.login` loads it, and `Client.fromJSON` does not. A program
|
is loaded. `Client.login` loads it, and `Client.fromJSON` does not. A program
|
||||||
that restores a session in a new process first runs `await sodium.ready`, with
|
that restores a session in a new process first runs `await sodium.ready`, with
|
||||||
@@ -1771,6 +1835,8 @@ standard. The others are additions to it.
|
|||||||
`package.json` declares (`main`, `types`, `bin`) were written, and make the
|
`package.json` declares (`main`, `types`, `bin`) were written, and make the
|
||||||
command line executable.
|
command line executable.
|
||||||
- `script/version`: print the version `script/build` stamps (see "Version").
|
- `script/version`: print the version `script/build` stamps (see "Version").
|
||||||
|
- `script/build-bin`: run `script/build`, then bundle the built `dist/` into one
|
||||||
|
binary, `bin/quak`, with bun (see "Running the backup from cron").
|
||||||
- `script/test`: run the test suite, by building the `test` phase of the
|
- `script/test`: run the test suite, by building the `test` phase of the
|
||||||
`Dockerfile`. Requires Docker.
|
`Dockerfile`. Requires Docker.
|
||||||
- `script/lint`: run eslint and a prettier check, by building the `lint` phase
|
- `script/lint`: run eslint and a prettier check, by building the `lint` phase
|
||||||
@@ -1799,8 +1865,7 @@ The Makefile targets are:
|
|||||||
- `make build`: runs `script/build`.
|
- `make build`: runs `script/build`.
|
||||||
- `make docker`: runs `script/docker`.
|
- `make docker`: runs `script/docker`.
|
||||||
- `make hooks`: runs `script/install-precommit`.
|
- `make hooks`: runs `script/install-precommit`.
|
||||||
- `make build-bin`: bundles the built `dist/` into one binary, `bin/quak`. It
|
- `make build-bin`: runs `script/build-bin`.
|
||||||
runs bun through `nix-shell -p bun`.
|
|
||||||
- `make install`: runs `make build-bin` and copies the binary to `~/bin/quak`.
|
- `make install`: runs `make build-bin` and copies the binary to `~/bin/quak`.
|
||||||
- `make dev`: runs the TypeScript compiler in watch mode.
|
- `make dev`: runs the TypeScript compiler in watch mode.
|
||||||
- `make clean`: removes `dist/`, `coverage/`, `.vitest-cache/` and the
|
- `make clean`: removes `dist/`, `coverage/`, `.vitest-cache/` and the
|
||||||
@@ -1908,9 +1973,9 @@ These apply to people and to coding agents alike.
|
|||||||
- container images by digest;
|
- container images by digest;
|
||||||
- the nvm archive `script/bootstrap` downloads, and the Python packages
|
- the nvm archive `script/bootstrap` downloads, and the Python packages
|
||||||
`bin/clip-keyword-vectors` installs, by SHA-256 hash;
|
`bin/clip-keyword-vectors` installs, by SHA-256 hash;
|
||||||
|
- the bun builds `script/build-bin` downloads, by SHA-256 hash;
|
||||||
- Node and yarn, when `script/bootstrap` installs them, by version.
|
- Node and yarn, when `script/bootstrap` installs them, by version.
|
||||||
- Not pinned:
|
- Not pinned:
|
||||||
- bun, which `make build-bin` takes from `nix-shell -p bun`;
|
|
||||||
- `make` and `git`, which `script/bootstrap` installs from the system
|
- `make` and `git`, which `script/bootstrap` installs from the system
|
||||||
package manager when they are missing, as every `Dockerfile` stage does;
|
package manager when they are missing, as every `Dockerfile` stage does;
|
||||||
- `bash` and `curl`, which `script/bootstrap` installs the same way when it
|
- `bash` and `curl`, which `script/bootstrap` installs the same way when it
|
||||||
@@ -1935,8 +2000,7 @@ These apply to people and to coding agents alike.
|
|||||||
|
|
||||||
## TODO
|
## TODO
|
||||||
|
|
||||||
`TODO.md` in the repo root is the work log: the next step, the steps done and
|
`TODO.md` in the repo root holds the next step and the steps planned. Open now:
|
||||||
the steps planned. Open now:
|
|
||||||
|
|
||||||
- [ ] Check quak against a live Ente account. This needs an account only sneak
|
- [ ] Check quak against a live Ente account. This needs an account only sneak
|
||||||
has.
|
has.
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
- branch from `next`
|
- branch from `next`
|
||||||
- do the work in Next Step
|
- do the work in Next Step
|
||||||
- move Next Step to the top of Completed Steps
|
- delete Next Step from `TODO.md`
|
||||||
- move the top item of Future Steps into Next Step
|
- move the top item of Future Steps into Next Step
|
||||||
- commit (`TODO.md` changes in the same commit as the work)
|
- commit (`TODO.md` changes in the same commit as the work)
|
||||||
- push
|
- push
|
||||||
@@ -24,741 +24,6 @@ only sneak has, so it waits on him.
|
|||||||
Tagging and releases are decided by sneak alone, and happen only when he
|
Tagging and releases are decided by sneak alone, and happen only when he
|
||||||
declares one.
|
declares one.
|
||||||
|
|
||||||
# Completed Steps
|
|
||||||
|
|
||||||
- 2026-11-10: The test suite passes on a host under heavy load
|
|
||||||
(https://git.eeqj.de/sneak/quak/issues/233). vitest runs at most four workers
|
|
||||||
instead of one per CPU core. `test/cli/bin.test.ts` compiles the sources
|
|
||||||
`bin/quak.ts` imports and starts libsodium once, before its tests run. The
|
|
||||||
fake accounts of `test/ente-server.ts` and
|
|
||||||
`test/thumbnails/thumbnails.test.ts` stretch their passwords at libsodium's
|
|
||||||
lowest Argon2id cost, since every login stretches the password again.
|
|
||||||
|
|
||||||
- 2026-10-26: On an empty cache, `quak helper fix-missing-thumbnails --file`
|
|
||||||
stops fetching collections' files once it finds every file it names
|
|
||||||
(https://git.eeqj.de/sneak/quak/issues/223). It used to wait for a refresh,
|
|
||||||
which on an empty cache fetched the files of every collection. Now it reads
|
|
||||||
the collection list, then one collection's files at a time, the most recently
|
|
||||||
updated first. It stores none of the collections or file records it reads. It
|
|
||||||
fetches each original through `lib.originalOf()`, which keeps it in the cache.
|
|
||||||
Once the cache holds the account's collections, it still waits for a refresh.
|
|
||||||
|
|
||||||
- 2026-10-26: On an empty cache, `quak get` and `quak get-thumb` stop fetching
|
|
||||||
collections' files once they find the file
|
|
||||||
(https://git.eeqj.de/sneak/quak/issues/222). They used to wait for a refresh,
|
|
||||||
which on an empty cache fetched the files of every collection. Now they read
|
|
||||||
the collection list, then one collection's files at a time. They store none of
|
|
||||||
the collections or file records they read, and fetch the original or thumbnail
|
|
||||||
through the new `lib.originalOf()` and `lib.thumbnailOf()`. Once the cache
|
|
||||||
holds the account's collections, they still wait for a refresh.
|
|
||||||
|
|
||||||
- 2026-10-26: `quak files` with a collection ID fetches that collection's files
|
|
||||||
alone (https://git.eeqj.de/sneak/quak/issues/217). It used to wait for a
|
|
||||||
refresh, which on an empty cache fetched the files of every collection. It now
|
|
||||||
reads the collection list through `lib.freshCollections()`, then the one
|
|
||||||
collection's files through the new `lib.freshFiles()`. It stores nothing, and
|
|
||||||
the files print in the server's order.
|
|
||||||
|
|
||||||
- 2026-10-22: The JSON `quak backup` and `quak backup-metadata` write for a file
|
|
||||||
holds `keywords` in place of the CLIP embedding in its ML data
|
|
||||||
(https://git.eeqj.de/sneak/quak/issues/218). The keywords are the titles of
|
|
||||||
the Discover categories of Ente's photos app that the file falls in, by the
|
|
||||||
app's rule: the cosine similarity of the file's CLIP embedding and the text
|
|
||||||
embedding of a category's query is at least the category's minimum score, and
|
|
||||||
videos stay out of the categories that leave them out. `src/clip-keywords.ts`
|
|
||||||
applies the rule. `src/clip-keyword-vectors.ts` holds the 16 categories with
|
|
||||||
their text embeddings. `bin/clip-keyword-vectors` writes it with Ente's CLIP
|
|
||||||
text model, and writes the CLIP embedding of a public-domain photo of a sunset
|
|
||||||
that the tests use.
|
|
||||||
|
|
||||||
- 2026-10-18: `-v`/`--verbose` raises the CLI's log level once for each repeat,
|
|
||||||
and `-q`/`--quiet` prints only fatal errors
|
|
||||||
(https://git.eeqj.de/sneak/quak/issues/212). The levels are fatal, error,
|
|
||||||
warning, info, debug and trace; the default is info, everything quak printed
|
|
||||||
on stderr before, `-v` gives debug and `-vv` trace. `src/cli-log.ts` holds the
|
|
||||||
level and the log the commands write stderr lines through. Debug adds each
|
|
||||||
retry, which `withRetry` reports through the new `onRetry`, and the start and
|
|
||||||
end of each refresh and ML data fetch; trace adds each request, which
|
|
||||||
`ApiClient` reports through the new `onRequest`. `--json` no longer keeps the
|
|
||||||
progress lines off stderr, so `backup` and `--json` runs log a refresh's
|
|
||||||
progress too. Every line that reports a failed file or album is at error,
|
|
||||||
which the progress callbacks of `backup`, `backup-metadata` and the thumbnail
|
|
||||||
helpers now say, and a command that exits 1 because files or albums failed
|
|
||||||
ends with one fatal line saying how many and where to look.
|
|
||||||
|
|
||||||
- 2026-10-18: `quak files` takes the collection ID as an optional argument
|
|
||||||
instead of the `--collection` option, which is gone
|
|
||||||
(https://git.eeqj.de/sneak/quak/issues/211). With no argument it lists every
|
|
||||||
file in the account once, collection by collection; a file in several
|
|
||||||
collections is listed with the first of them, which is its `collectionID` in
|
|
||||||
`--json`. The fresh read in `src/cli-read.ts` does this.
|
|
||||||
|
|
||||||
- 2026-10-18: Names quak builds from an album name or a title fit in a file name
|
|
||||||
(https://git.eeqj.de/sneak/quak/issues/205). `shortenKeepingExtension` in
|
|
||||||
`src/filename.ts` cuts a name short to a number of bytes, keeping its
|
|
||||||
extension when it fits and always its first character, and can put text right
|
|
||||||
before the extension; the backup's `withID` uses it for an ID.
|
|
||||||
`defaultFileName` there gives the name a file is saved under when the caller
|
|
||||||
gives no path: a prefix such as `thumb_` and the title with unsafe characters
|
|
||||||
replaced, cut short with it to 255 bytes, and, for a live photo's image or
|
|
||||||
video, with that part's own extension before the cut. `quak get`,
|
|
||||||
`quak get-thumb`, `downloadFile` and `downloadThumbnail` use it without
|
|
||||||
`--out` or `outPath`. `backup-metadata` cuts an album's name short so that its
|
|
||||||
folder, `<id>-<name>`, fits in 255 bytes. `safeExtension` gives `.bin` for an
|
|
||||||
extension of more than 16 letters and digits, so a save path's name always
|
|
||||||
fits.
|
|
||||||
|
|
||||||
- 2026-10-18: `quak collections` no longer waits for the files of every
|
|
||||||
collection (issue 210). On an empty cache it opened its library with a refresh
|
|
||||||
that fetched and decrypted the files of each collection in turn, printing
|
|
||||||
nothing and storing nothing until all were done, so on a large account it
|
|
||||||
seemed to hang. It now reads the collection list alone through the new
|
|
||||||
`lib.freshCollections()`, one request that stores nothing. The read commands
|
|
||||||
open their library with the new `refreshOnOpen: false`, so each makes only the
|
|
||||||
refresh it asks for. A refresh sends a `"progress"` event after each
|
|
||||||
collection whose files it fetched, and the commands that wait on one print it
|
|
||||||
on stderr, except with `--json`.
|
|
||||||
|
|
||||||
- 2026-10-16: ML data the server sends that does not decrypt, gunzip or parse is
|
|
||||||
no longer taken for no ML data (issue 198). `fetchMLDataBatch` returns why for
|
|
||||||
each such file beside the payloads; ML data that parses to something other
|
|
||||||
than a JSON object, such as `null`, is one of them. The library keeps the
|
|
||||||
reasons from its last ML data fetch that succeeded in memory, not in the
|
|
||||||
cache, and gives them in `status().mlDataErrors` and in the `fetchMLData`
|
|
||||||
event with status `"done"`; a fetch still running or one that fails does not
|
|
||||||
change them. `quak backup` writes the reason as `mlDataError` in the file's
|
|
||||||
JSON when no ML data is cached for it, and records the file in `failures.json`
|
|
||||||
with an error starting `ML data: `; `backup-metadata` writes `mlDataError`
|
|
||||||
too. Both exit 1. Tests serve ML data encrypted under a key that is not the
|
|
||||||
file's.
|
|
||||||
|
|
||||||
- 2026-10-16: With `--verify`, an original whose hash does not match is no
|
|
||||||
longer removed before its replacement is downloaded (issue 199).
|
|
||||||
`replaceOriginal` in `src/library/content.ts` downloads the replacement to a
|
|
||||||
temporary file beside it, a live photo's image and video each to its own,
|
|
||||||
hashes it and renames it over the original only when it matches. When the
|
|
||||||
download fails or does not match either, the original keeps its bytes and
|
|
||||||
links, and the file goes into `failures.json`. The replacement is always
|
|
||||||
downloaded, through the new `fetchOriginal` of the content cache, never copied
|
|
||||||
from the cache.
|
|
||||||
|
|
||||||
- 2026-10-16: The `~/bin/quak` that `make install` builds retries a refused
|
|
||||||
connection and a name that does not resolve, and the backup records either as
|
|
||||||
transient (issue 206). Bun's `fetch` reports them with codes of its own,
|
|
||||||
`ConnectionRefused` and `FailedToOpenSocket`, which `src/retry.ts` now counts
|
|
||||||
as transport failures. A `POST` or `PUT` is not replayed on them, because bun
|
|
||||||
also gives them when it resends a request on a new connection after a
|
|
||||||
connection it kept open closed, and the server may already have the request.
|
|
||||||
Bun already gave `ECONNRESET` for a reset and a `TimeoutError` for a deadline.
|
|
||||||
`classify` in `src/backup.ts` now records any `TimeoutError` as transient, so
|
|
||||||
a stalled download, whose error says only that it stalled, is no longer
|
|
||||||
`unknown`. The shapes were found by running the binary against local
|
|
||||||
listeners.
|
|
||||||
|
|
||||||
- 2026-10-15: An album whose folder or JSON `quak backup` cannot write no longer
|
|
||||||
ends the run before the other albums and `failures.json` are written (issue
|
|
||||||
197). The run prints `FAILED album <name>: <reason>`, lists the album's ID in
|
|
||||||
`failedAlbums` in the result and in `--json`, goes on, and exits 1. An album
|
|
||||||
whose folder would have the name of another album's JSON, as `x.json` beside
|
|
||||||
`x` would, gets its ID added. A folder or link name over 255 bytes is cut
|
|
||||||
short between two characters, by `shortenToBytes` in `src/filename.ts`, and
|
|
||||||
gets the ID.
|
|
||||||
|
|
||||||
- 2026-10-15: A network failure is printed and recorded with its reason, not as
|
|
||||||
just `fetch failed` (issue 196). `describeError` in `src/errors.ts` gives an
|
|
||||||
error's message followed by the message of each error on its `cause` chain,
|
|
||||||
such as `fetch failed: connect ECONNREFUSED 1.2.3.4:443`, and each address's
|
|
||||||
reason when a host has several and none could be reached. `run` prints that,
|
|
||||||
and the backup logs it and records it in `failures.json`, where `classify` now
|
|
||||||
calls the failure transient when the chain holds an errno that
|
|
||||||
`hasTransportCode` in `src/retry.ts` recognises, for a failed ML data fetch
|
|
||||||
too. The walk along the chain moved from `src/retry.ts` to `src/errors.ts`,
|
|
||||||
which both use. Tests refuse every connection of a backup, then only one
|
|
||||||
file's download, and fail the ML data fetch.
|
|
||||||
|
|
||||||
- 2026-10-15: The README says that `make install` needs nix (issue 200), in
|
|
||||||
Getting Started and in "Running the backup from cron": `make build-bin` runs
|
|
||||||
bun through `nix-shell -p bun`. The `Makefile` is unchanged.
|
|
||||||
|
|
||||||
- 2026-10-14: `lib.backup()`, and so `quak backup`, works on up to the content
|
|
||||||
pool's limit of originals at once, 5 by default, instead of one at a time
|
|
||||||
(issue 182). That many workers take files from one shared list, each doing all
|
|
||||||
the work for one original, its `--verify` hashing included, so no more
|
|
||||||
originals than that are open at once. `BackupLibrary` says the limit as
|
|
||||||
`contentConcurrency`. A test holds each download at the fake Ente server to
|
|
||||||
count those in flight. Once a worker has thrown, as one does when `onProgress`
|
|
||||||
throws, or the lock is lost, no worker takes another file; the backup fails
|
|
||||||
once the originals under way are in place, and a lost lock stops it before it
|
|
||||||
fetches ML data.
|
|
||||||
|
|
||||||
- 2026-10-14: Next Step said the cache design, issue 36, waits on sneak's
|
|
||||||
review, though that issue closed on 2026-10-03 (issue 194). It now names the
|
|
||||||
issues open on the tracker.
|
|
||||||
|
|
||||||
- 2026-10-14: A backup whose lock is lost no longer dies with a stack trace
|
|
||||||
(issue 184). `lockBackupDirectory` gives proper-lockfile an `onCompromised`
|
|
||||||
that records why the lock was lost instead of throwing from its timer, and
|
|
||||||
returns the lock, which `quak backup` passes to `lib.backup()` as `lock`. The
|
|
||||||
backup then stops before it starts another file with
|
|
||||||
`lost the lock on DIR: REASON`, which `quak backup` prints as one line before
|
|
||||||
it exits 1. Releasing a lost lock leaves `backup.lock` alone. Tests in
|
|
||||||
`test/cli/backup-end-to-end.test.ts` remove the lock, or put another backup's
|
|
||||||
in its place, while a download is held open.
|
|
||||||
|
|
||||||
- 2026-10-14: An album or file quak cannot decrypt, or whose metadata is not
|
|
||||||
JSON, no longer fails the refresh, so one bad record no longer stops
|
|
||||||
`quak backup` from storing anything (issue 181). The refresh leaves it out, an
|
|
||||||
album with its files, and the library keeps it with the reason in
|
|
||||||
`metadata.json` until the server sends a copy that decrypts or deletes it.
|
|
||||||
`quak backup` stores every other file, records each such file in
|
|
||||||
`failures.json` with an error starting `could not be decrypted:`, names each
|
|
||||||
such album's ID in a printed line, or with `--json` in `undecryptableAlbums`,
|
|
||||||
and exits 1 while any remain. A file whose copy in another album decrypts is
|
|
||||||
not recorded: its original is stored from that copy, and the album where it
|
|
||||||
does not decrypt neither links nor lists it. The read commands answer from
|
|
||||||
what decrypted.
|
|
||||||
|
|
||||||
- 2026-10-14: `quak backup` no longer asks the server for changes every 3
|
|
||||||
seconds while it runs (issue 183). It opens its library with
|
|
||||||
`refreshIntervalSeconds: Infinity`, which the library now takes as no
|
|
||||||
background timer, so the backup refreshes only as it starts and no timed
|
|
||||||
refresh starts another ML data pass during the run. A test holds a backup's
|
|
||||||
download open while two hours pass and counts its `/collections/v2` requests.
|
|
||||||
|
|
||||||
- 2026-10-14: `test/cli/backup-end-to-end.test.ts` runs `quak backup` as cron
|
|
||||||
does (issue 186): `backupCommand` and `run`, the real `loadSession` reading a
|
|
||||||
session saved after logging in, and every request answered by a fake of the
|
|
||||||
Ente HTTP API serving data encrypted with libsodium. It covers the backup tree
|
|
||||||
of an account with two albums of its own and one shared with it, a second run
|
|
||||||
that downloads nothing, a download that succeeds after two 503s, a 404 that
|
|
||||||
lands in `failures.json` with exit 1, a refresh answered 401 with exit 3, and
|
|
||||||
`--verify` putting back an original changed on disk. The fake server and the
|
|
||||||
account it logs in moved from the backup-metadata test into
|
|
||||||
`test/ente-server.ts`, unchanged.
|
|
||||||
|
|
||||||
- 2026-10-14: `files --collection`, `get`, `get-thumb` and
|
|
||||||
`helper fix-missing-thumbnails --file` take an ID only when it is written in
|
|
||||||
decimal digits and is a safe integer (issue 185). Before, `Number()` read
|
|
||||||
`0x10` as 16, `1e3` as 1000 and an empty string as 0, and `--file` dropped an
|
|
||||||
ID it could not read without a word. Anything else now prints
|
|
||||||
`Invalid file ID: <arg>` or `Invalid collection ID: <arg>` and exits 1 before
|
|
||||||
any request; one bad `--file` ID fails the whole command. `parseID` in
|
|
||||||
`src/cli-commands.ts` does the check for all four.
|
|
||||||
|
|
||||||
- 2026-10-06: The README says how to run `quak backup` from cron (issue 170):
|
|
||||||
log in once as the job's user, a crontab with a backup every night and
|
|
||||||
`--verify` on Sundays appending to a log file, and the `HOME` and
|
|
||||||
`XDG_DATA_HOME` the job needs to find the saved session. A table gives each
|
|
||||||
exit code: 0, the backup is complete; 1, files are in `failures.json` or
|
|
||||||
another error stopped the run; 2, another backup of the directory is running;
|
|
||||||
3, there is no usable session. The introduction lists everything the backup
|
|
||||||
keeps for each file.
|
|
||||||
|
|
||||||
- 2026-10-06: `quak backup --verify` and `lib.backup({ verify: true })` hash
|
|
||||||
each original already at its save path as the download check does, streamed, a
|
|
||||||
live photo as `<imageHash>:<videoHash>` (issue 168). One that does not match
|
|
||||||
the content hash its metadata records is logged, removed (both files of a live
|
|
||||||
photo) and put back in the same run, downloaded or copied from the cache, and
|
|
||||||
what is put back is hashed too. One that still does not match, or whose
|
|
||||||
download fails, goes into `failures.json`. One with no recorded hash is left
|
|
||||||
alone. The result, `--json` and the summary gain `verified`, `mismatched` and
|
|
||||||
`unchecked`. Without `--verify` nothing is hashed.
|
|
||||||
|
|
||||||
- 2026-10-06: Two backups of the same directory never run at once (issue 169).
|
|
||||||
`lib.backup()` takes a lock, `backup.lock` in its download directory, made
|
|
||||||
with `proper-lockfile`, before its refresh, and removes it when it ends,
|
|
||||||
whether it succeeds or fails. A second backup of the directory, from another
|
|
||||||
process or the same one, fails at once with an error naming the directory.
|
|
||||||
`quak backup` takes the lock before it opens its library, so a refused run
|
|
||||||
sends no request; it prints the error as one line and exits 2. A lock that has
|
|
||||||
gone 10 seconds untouched, left by a run that could not remove it, is taken
|
|
||||||
over by the next run.
|
|
||||||
|
|
||||||
- 2026-10-06: `quak backup` writes each original's EXIF, XMP and dimensions into
|
|
||||||
the file's JSON as `imageMetadata`, what `backup-metadata --exif` records
|
|
||||||
(issue 167): for a live photo from its image, for a video nothing, and `{}`
|
|
||||||
for an original with none of them. A failed read puts the reason in
|
|
||||||
`imageMetadataError` and fails neither the file nor the run. An original is
|
|
||||||
read when the run stores it or when its JSON has neither field; otherwise the
|
|
||||||
field is taken from that JSON. The hand-built JPEGs moved to
|
|
||||||
`test/exif-jpeg.ts`, beside the HEIC.
|
|
||||||
|
|
||||||
- 2026-10-06: When the server answers HTTP 401 and that ends a command that
|
|
||||||
loads the saved session, because the server no longer accepts its token, the
|
|
||||||
command prints one line,
|
|
||||||
`quak: the saved session is no longer valid; run "quak login"`, and exits 3
|
|
||||||
(issue 164). `quak backup` meets that 401 on the refresh it starts with,
|
|
||||||
before it touches any file. For the same commands, a missing or corrupt
|
|
||||||
session file keeps its message and also exits 3, so a cron job can tell that
|
|
||||||
the user must log in again. `quak logout` is unchanged. `run` in
|
|
||||||
`src/cli-run.ts` recognises the 401, which reaches it unchanged.
|
|
||||||
|
|
||||||
- 2026-10-06: `quak backup` retries a failed request for longer than the other
|
|
||||||
commands do (issue 165). `src/retry.ts` exports `UNATTENDED_RETRY_OPTIONS`
|
|
||||||
beside the unchanged default: 10 attempts, a 1 s base delay and a 60 s cap, so
|
|
||||||
a request that keeps failing waits at most 243 s before it gives up.
|
|
||||||
`bin/quak.ts` loads the backup's session with them, so its refresh, ML data
|
|
||||||
and downloads all use them. What is retried and the backoff formula are
|
|
||||||
unchanged.
|
|
||||||
|
|
||||||
- 2026-10-06: The files this repository copies from `sneak/prompts` are copied
|
|
||||||
again from its commit `dd4027b` (issue 171). `.gitignore` and `.dockerignore`
|
|
||||||
keep out more secret files, and this repository's build artifacts follow the
|
|
||||||
copied content. `script/check` runs `script/fmt-check` again, `script/cibuild`
|
|
||||||
runs `script/bootstrap` and `script/check` before the image build,
|
|
||||||
`script/fmt` and `script/fmt-check` find the pinned yarn under nvm, and the
|
|
||||||
image's last stage marks `/app` safe for git.
|
|
||||||
|
|
||||||
- 2026-10-06: `quak backup` writes the account and album records
|
|
||||||
`backup-metadata` writes (issue 166): `account.json` with the account's
|
|
||||||
`email` and `userID`, and in each album's JSON its `ownerID`, `isShared`,
|
|
||||||
`updationTime` and, when present, its three layers of magic metadata. Each
|
|
||||||
file's JSON gains its `updationTime`.
|
|
||||||
|
|
||||||
- 2026-10-05: `quak backup` writes each file's ML data (its faces and CLIP
|
|
||||||
embedding) into the file's JSON as `mlData`, the payload
|
|
||||||
`lib.mldata.forFile()` returns (issue 163). `lib.backup()` waits for an ML
|
|
||||||
data fetch before it writes the JSON files. When that fetch fails, each file
|
|
||||||
whose ML data is not cached gets the reason in `mlDataError` and counts as
|
|
||||||
failed, and the next run fetches it again.
|
|
||||||
|
|
||||||
- 2026-10-03: The download-albums example test no longer fails on a disk with
|
|
||||||
under 50 GiB free (issue 160). It opens its libraries with
|
|
||||||
`freeBelowBytes: 0`, so the free space of the disk it runs on cannot shrink
|
|
||||||
the cache, evict the original it cached and add a fetch to the count. No other
|
|
||||||
test's result depends on that free space. The 50 GiB default is unchanged.
|
|
||||||
|
|
||||||
- 2026-10-02: `docker build .` stamps the commit's tag or short commit, not
|
|
||||||
`dev` (issue 154). `script/build` writes the version `script/version` prints
|
|
||||||
into `dist/package.json`: the `VERSION` environment variable or build arg when
|
|
||||||
one is given, otherwise `git describe --tags --always`, otherwise the version
|
|
||||||
`package.json` declares. `.dockerignore` sends `.git`, and a checkout with
|
|
||||||
`.git` whose version comes out empty, `dev` or `unknown` fails the build.
|
|
||||||
`make build-bin` bundles the built `dist/`, so the single binary reports the
|
|
||||||
same version.
|
|
||||||
|
|
||||||
- 2026-10-02: `photo.exif()` returns every EXIF tag in the file as `ExifTags`,
|
|
||||||
keyed by tag name, each as exifreader decodes it, not only the thirteen common
|
|
||||||
fields (issue 156). The embedded thumbnail's tags are under `Thumbnail`,
|
|
||||||
without the thumbnail image. The thirteen typed methods stay, each picking its
|
|
||||||
field from the tags `exif()` returns, typed as in `PhotoExif`. The example
|
|
||||||
script's JSON files now carry every tag.
|
|
||||||
|
|
||||||
- 2026-10-01: The content cache no longer looks for a live photo that an earlier
|
|
||||||
version cached as one ZIP (issue 151). When the cache opens, a live photo's
|
|
||||||
file that no JSON file names is now always left alone.
|
|
||||||
|
|
||||||
- 2026-10-01: `examples/download-albums.ts` logs in, opens the library, and for
|
|
||||||
every album downloads each photo to its save path, writes the photo's record
|
|
||||||
and EXIF fields to a JSON file beside it, and writes the album's photos to
|
|
||||||
`albums/<collectionID>.json` (issue 144). The build compiles it to
|
|
||||||
`dist/examples/`; the README's "Examples" section says how to run it.
|
|
||||||
|
|
||||||
- 2026-10-01: A `Photo` has one async method for each field of `exif()`, named
|
|
||||||
and typed as in `PhotoExif`: `make()`, `model()`, `lensModel()`,
|
|
||||||
`dateTimeOriginal()`, `offsetTimeOriginal()`, `exposureTime()`, `fNumber()`,
|
|
||||||
`iso()`, `focalLength()`, `orientation()`, `gpsLatitude()`, `gpsLongitude()`
|
|
||||||
and `gpsAltitude()` (issue 148). Each calls `exif()` and returns its one
|
|
||||||
field, or undefined when the file lacks it. `Photo` implements a type with one
|
|
||||||
method per `PhotoExif` field, so the build's type check fails when a field has
|
|
||||||
no method. A test checks, on the JPEG and the HEIC, that each method gives the
|
|
||||||
same value as `exif()`.
|
|
||||||
|
|
||||||
- 2026-10-01: Each original's save path is
|
|
||||||
`YYYY/YYYY-MM/YYYY-MM-DD/YYYY-MM-DD.<fileID>.<ext>` under the library's
|
|
||||||
download directory, which defaults to `photos` in the working directory (issue
|
|
||||||
143). The date is the photo's `takenAt` in the machine's time zone.
|
|
||||||
`photo.savePath` is always a string, with or without a content cache, and
|
|
||||||
`isLocal` is true only when the original is at its save path.
|
|
||||||
`photo.download()` puts the original there, copied from the cache when the
|
|
||||||
cache holds it and fetched otherwise. `lib.backup()` does the same for each
|
|
||||||
file, writes each file's JSON beside its original, and links `collections/` to
|
|
||||||
the save paths.
|
|
||||||
|
|
||||||
- 2026-10-01: `photo.exif()` and `backup-metadata --exif` read EXIF from
|
|
||||||
HEIC/HEIF originals, a live photo's HEIC image included, as well as JPEG and
|
|
||||||
the other image formats `exifreader` reads (issue 145). `exifreader` replaces
|
|
||||||
`exif-reader` and the JPEG segment scan; `PhotoExif` is unchanged. The `exif`
|
|
||||||
field of `backup-metadata --exif` is now exifreader's tag output, and
|
|
||||||
`exifRaw` holds the whole EXIF block it could not read. The tests use a real
|
|
||||||
HEIC, `test/exif.heic`.
|
|
||||||
|
|
||||||
- 2026-10-01: A `Photo` has `savePath`, `isLocal`, `content()`, `exif()`,
|
|
||||||
`modifiedAt`, `hash` and `year` (issue 141). `savePath` is where
|
|
||||||
`lib.backup()` writes the original under the library's download directory; for
|
|
||||||
a live photo not yet stored, it carries the title's extension, and the backup
|
|
||||||
may store the image under a different one. `isLocal` says whether the whole
|
|
||||||
original is there. Both look only at the disk. `content()` returns the
|
|
||||||
original's bytes and `exif()` the common EXIF fields of a JPEG; both may
|
|
||||||
download the original, and `exif()` downloads no video. `PhotoRecord` gains
|
|
||||||
`modifiedAt` and `hash`, and the JPEG EXIF scan moved to `src/exif.ts`.
|
|
||||||
|
|
||||||
- 2026-09-29: The "Workflow" list at the top of this file now says to branch
|
|
||||||
from `next` and open a pull request that targets `next`, that the repository
|
|
||||||
manager squash-merges a reviewed pull request into `next`, and that only sneak
|
|
||||||
merges `next` into `main`, as the README does (issue 137).
|
|
||||||
|
|
||||||
- 2026-09-29: The README's "Development workflow" and "For LLMs" sections now
|
|
||||||
say work branches from `next`, every pull request targets `next`, the
|
|
||||||
repository manager squash-merges reviewed pull requests into `next`, and only
|
|
||||||
sneak merges `next` into `main` (issue 135).
|
|
||||||
|
|
||||||
- 2026-09-29: Brought the README and this file in line with `next` after the
|
|
||||||
milestone merge (issue 132). The Next Step says no implementation work is open
|
|
||||||
and the cache design (issue 36) waits on sneak's review. README corrections:
|
|
||||||
the Getting Started comments say when the library refreshes; most, not all,
|
|
||||||
Makefile targets call a script; `script/lint` and `script/test` have no host
|
|
||||||
path, though `yarn test` does; the SRP handshake uses `fast-srp-hap`, outside
|
|
||||||
`crypto/`, and a thumbnail upload's MD5 uses `node:crypto`; the SRP password
|
|
||||||
is the first 16 bytes of a 32-byte subkey; the key attributes and token come
|
|
||||||
after SRP, after the TOTP code SRP may ask for, or after the email OTP that
|
|
||||||
replaces SRP when the account has email MFA on, and quak cannot answer a
|
|
||||||
passkey; the auth token is sent as URL-safe base64 with padding; every
|
|
||||||
`TypeError` is retried; each download attempt writes its own temporary files,
|
|
||||||
two for a live photo, and only the attempt that completes renames them into
|
|
||||||
place; `runMetadataBackup` records a failed download in the file's JSON; a
|
|
||||||
second `client.logout()` does not throw; `quak logout` with no session exits
|
|
||||||
0, and names the cache directory only when it knows it; `login`, `whoami` and
|
|
||||||
`logout` open no library; `--exif` records XMP and, for a JPEG, EXIF, and no
|
|
||||||
IPTC; which commands take `--json`; a failed ML data request may not have been
|
|
||||||
retried; the thumbnail fixer is not limited to baseline JPEG; `quak backup`
|
|
||||||
still fetches ML data; a backup's JSON holds the basic metadata fields quak
|
|
||||||
keeps and the private and public magic metadata, not every decrypted field,
|
|
||||||
and the README no longer lists what the magic metadata holds; the default
|
|
||||||
cache directory is the per-user one, not an XDG path on macOS; pinned
|
|
||||||
originals can exceed `cacheOriginalsMaxBytes`; which tests cover which
|
|
||||||
operations; a default read is only as current as the last refresh whose
|
|
||||||
requests all succeeded; `fresh()` and `lib.backup()` join a refresh already
|
|
||||||
running; a `Photo` has no `thumbnailPath` or `originalPath`; and `408` and
|
|
||||||
`429` are retried.
|
|
||||||
|
|
||||||
- 2026-09-28: Tested the live-photo writer's fsyncs (issue 130). A test checks
|
|
||||||
that the image's and the video's temp files are fsynced before either is
|
|
||||||
renamed into place, and the directory after both renames, as the `writeAtomic`
|
|
||||||
test does for one file.
|
|
||||||
|
|
||||||
- 2026-09-23: Live photos are stored as their image and their video (issue 107).
|
|
||||||
A live photo, which Ente stores as one ZIP, is unpacked as it downloads into
|
|
||||||
`<fileID>.<ext>` for the image and for the video, each with its extension from
|
|
||||||
the ZIP, beside `<fileID>.livephoto.json`, which names the two. Both are
|
|
||||||
checked against the recorded hash and renamed into place only when both are
|
|
||||||
complete. A ZIP holding a second image or video, or whose two files come to
|
|
||||||
more than 20 times its size plus 16 MiB, is refused. The backup and the
|
|
||||||
content cache count the live photo as stored only with both files, the album
|
|
||||||
folder links both, `quak get` writes both, and the content result gives the
|
|
||||||
video as `videoPath`. A ZIP an earlier version stored is replaced on the next
|
|
||||||
backup run, and removed from the cache when the library opens it.
|
|
||||||
|
|
||||||
- 2026-09-23: Settled the package metadata (issue 6). quak is not published, so
|
|
||||||
`package.json` is marked `"private": true` and the `files` field is gone.
|
|
||||||
`engines.node` is `>=22`, the major version `script/bootstrap` and the
|
|
||||||
`Dockerfile` use. An `exports` map makes `.` and `./package.json` the only
|
|
||||||
importable paths; `runMetadataBackup`, the thumbnail helpers and their types
|
|
||||||
stay internal to the CLI.
|
|
||||||
|
|
||||||
- 2026-09-23: Brought the README and this file in line with the tree (issue
|
|
||||||
111). The layout lists `src/library/` and the other source files, the backup
|
|
||||||
layout names `failures.json` and the optional `thumbnails/`, "Opening a
|
|
||||||
library" says what happens when the first refresh fails, the Testing section
|
|
||||||
gives the 60-second hard cap and 20-second target for `make test` and names
|
|
||||||
the 90-second `timeout` in the `Dockerfile` as the backstop for a hung test,
|
|
||||||
and "Tag v1.0.0" is no longer listed as the next step.
|
|
||||||
|
|
||||||
- 2026-09-23: Tested `quak login` and `backup-metadata --exif` (issue 110).
|
|
||||||
`loginCommand` takes its login function and its prompts from `CliContext`, and
|
|
||||||
`bin/quak.ts` passes `Client.login` and the terminal prompts. Tests cover a
|
|
||||||
login from `QUAK_EMAIL` and `QUAK_PASSWORD` with no prompt, the TOTP prompt, a
|
|
||||||
failed login, and the saved session's modes, and show that `--exif` and
|
|
||||||
`--all` each turn on EXIF extraction and that it is off without them.
|
|
||||||
|
|
||||||
- 2026-09-23: `quak backup` writes each original once and no longer fills the
|
|
||||||
cache (issue 106). An original fetched for a backup is written by the download
|
|
||||||
writer straight into the backup's `originals/`, and the content cache records
|
|
||||||
it there instead of keeping its own copy; one the cache already held is still
|
|
||||||
copied. `quak backup` opens its library with the thumbnail and originals
|
|
||||||
precache off.
|
|
||||||
|
|
||||||
- 2026-09-23: `backup-metadata`, `helper list-missing-thumbnails` and
|
|
||||||
`helper fix-missing-thumbnails` refresh before they answer (issue 100). Each
|
|
||||||
awaits `lib.fresh()` before reading, so a file added since the cache was
|
|
||||||
written is included, and a failed refresh prints one line and exits 1 instead
|
|
||||||
of answering from a stale or empty cache. The README lists them among the
|
|
||||||
commands that refresh first.
|
|
||||||
|
|
||||||
- 2026-09-23: `quak backup` waits for the server refresh and fails when it fails
|
|
||||||
(issue 99). `lib.backup()` joins a refresh already running or starts one, as
|
|
||||||
`fresh()` does, and rejects before touching any file when it fails, leaving
|
|
||||||
`failures.json` as it was, so `quak backup` prints the error as one line and
|
|
||||||
exits 1 instead of backing up the previous run's file list, or nothing, and
|
|
||||||
exiting 0.
|
|
||||||
|
|
||||||
- 2026-09-23: CLI errors print a message instead of a stack trace (issue 102).
|
|
||||||
An error a command throws is printed as one `quak: MESSAGE` line on stderr and
|
|
||||||
the CLI exits 1 once output has drained. The wrapper that does this moved from
|
|
||||||
`bin/quak.ts` to `src/cli-run.ts`, and `bin/quak.ts` now awaits
|
|
||||||
`program.parseAsync()`.
|
|
||||||
|
|
||||||
- 2026-09-23: Opening a library no longer deletes another process's download in
|
|
||||||
progress (issue 105). The download writer's temp files are named
|
|
||||||
`.quak-<pid>-<random>.tmp`, and `removeLeftoverTempFiles`, moved from the
|
|
||||||
backup into the download module, deletes a `.quak-*.tmp` file only when the
|
|
||||||
process ID in its name is no longer running. The content cache calls it at
|
|
||||||
`open()` for `originals/` and `thumbnails/`, the backup as before.
|
|
||||||
|
|
||||||
- 2026-09-23: Re-vendored the lint and test setup from the template (issue 96).
|
|
||||||
Linting and testing are the `lint` and `test` phases of the `Dockerfile`;
|
|
||||||
`script/lint` and `script/test` each build one with `--no-cache`, and the last
|
|
||||||
stage compiles and depends on both, so `script/cibuild` is one build.
|
|
||||||
`Dockerfile.lint`, `CHECK_EPOCH`, `LINT_EPOCH` and the tests that checked them
|
|
||||||
are gone; `REPO_POLICIES.md` is re-copied.
|
|
||||||
|
|
||||||
- 2026-09-23: Stopped `helper fix-missing-thumbnails` retrying files the server
|
|
||||||
always refuses (issue 109). Both thumbnail helpers skip a file another account
|
|
||||||
owns without fetching it. The fixer skips a file whose recorded thumbnail size
|
|
||||||
is 0 or unknown before downloading it, and otherwise tries smaller encodings
|
|
||||||
(720 px quality 50 down to 160 px quality 20) until the encrypted thumbnail is
|
|
||||||
no larger than that size, skipping the file if none fits.
|
|
||||||
|
|
||||||
- 2026-09-23: Tested the live-photo hash check's error paths (issue 117). Tests
|
|
||||||
download a live photo whose ZIP names an unknown compression method, one whose
|
|
||||||
ZIP has no image entry and one with no video entry, and check that nothing is
|
|
||||||
stored and the error names the file ID; the unreadable one is not retried.
|
|
||||||
|
|
||||||
- 2026-09-23: `quak logout` ends the session on the server (issue 108). It calls
|
|
||||||
`POST /users/logout` through the new `Client.logoutOnServer()`, then deletes
|
|
||||||
`session.json` even when that call fails, says so and exits 1. It prints the
|
|
||||||
account's cache directory and says it still holds decrypted data. The default
|
|
||||||
cache path is now `defaultCacheDirectory()` in the library, shared with
|
|
||||||
`Library.open`.
|
|
||||||
|
|
||||||
- 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files
|
|
||||||
in one collection with the same title, and two collections with the same name,
|
|
||||||
each get their ID added to the name (`IMG_0001 (12345).JPG`, `Trip (10)/`), so
|
|
||||||
none replaces another's symlink or JSON. Each run removes symlinks into
|
|
||||||
`originals/` for files no longer in the collection, and the folders of deleted
|
|
||||||
or renamed collections, leaving anything else in `collections/` alone. The
|
|
||||||
README backup layout states the naming rule.
|
|
||||||
|
|
||||||
- 2026-09-23: Checked downloaded originals against their recorded content hash
|
|
||||||
(issue 68). `downloadFile`, which `quak get`, the content cache and backup all
|
|
||||||
use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and
|
|
||||||
stores nothing on a mismatch, failing with an error naming the file ID. A live
|
|
||||||
photo ZIP is unpacked as it streams with `fflate` and its image and video
|
|
||||||
hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older
|
|
||||||
clients' `imageHash` and `videoHash` fields for live photos. A file with no
|
|
||||||
recorded hash is stored unchecked.
|
|
||||||
|
|
||||||
- 2026-09-23: Kept one account's cache from mixing with another's (issue 104).
|
|
||||||
When `metadata.json` in the cache directory was written for a different,
|
|
||||||
non-zero user ID than the client's, `Library.open` deletes it and `mldata/`
|
|
||||||
and starts empty, so the first refresh enumerates from 0. This only happens
|
|
||||||
with `--cache-dir` or an explicit `cacheDirectory`; the default path already
|
|
||||||
includes the user ID. A test opens one account's cache as another account.
|
|
||||||
|
|
||||||
- 2026-09-23: `backup-metadata` no longer stops on one failed ML data request
|
|
||||||
(issue 101). Each request of up to 200 files is tried on its own; a failed one
|
|
||||||
is logged, its files are written with the reason in `mlDataError`, and the
|
|
||||||
command exits 1 once the dump is complete. `fetchMLData`, which only this
|
|
||||||
command used, is gone; the command calls `fetchMLDataBatch` per batch.
|
|
||||||
- 2026-09-23: Single-sourced the version string (issue 5). `package.json` is the
|
|
||||||
only place it is written: `src/index.ts` imports it for `VERSION` and
|
|
||||||
`bin/quak.ts` passes `VERSION` to commander. tsc copies `package.json` to
|
|
||||||
`dist/package.json`, so the import resolves from the built output too, and
|
|
||||||
`script/build` runs the built CLI with `--version` to prove it. A test checks
|
|
||||||
that `VERSION` and `quak --version` both equal the `package.json` version.
|
|
||||||
- 2026-09-23: Tested that `Library.close()` waits for the originals precache
|
|
||||||
(issue 93). The test that holds a precache fetch open while `close()` runs now
|
|
||||||
runs once with only the thumbnail fill and once with only the originals fill,
|
|
||||||
so dropping either wait from `Precache.close()` fails a test.
|
|
||||||
- 2026-09-23: Fixed two intermittently failing library tests (issue 90).
|
|
||||||
`Library.close()` now returns a promise that resolves once an in-flight
|
|
||||||
refresh (including its cache write), the ML data fetch and running precache
|
|
||||||
sweeps have finished; the library tests await it, so `afterEach` no longer
|
|
||||||
removes the cache directory while something is still writing into it. The
|
|
||||||
precache test waits for both fills to report "done" instead of for its stub
|
|
||||||
source to be called, which happened before the cache recorded the file.
|
|
||||||
- 2026-09-23: Pinned three guards reviewers found untested (issue 89). The
|
|
||||||
download idle deadline's timer is unref'd, so it can never keep the process
|
|
||||||
alive, and a test checks no timer is left after a download completes or fails.
|
|
||||||
A test covers the rejection of `#` in a request path. The EXIF scan compares
|
|
||||||
the `Exif` header only in an APP1 segment of length 8 or more, so it never
|
|
||||||
reads the next segment's bytes, with a test for a short one.
|
|
||||||
- 2026-09-23: Made the download deadline an idle deadline (issue 24).
|
|
||||||
`downloadTimeoutMs` now aborts a file or thumbnail download only after no
|
|
||||||
bytes have arrived for that long, default 60 seconds, instead of bounding the
|
|
||||||
whole transfer at 10 minutes, so a slow download that keeps making progress
|
|
||||||
completes. A download that fails before reading the whole body cancels it, so
|
|
||||||
a failed file no longer holds its connection.
|
|
||||||
- 2026-09-23: Every `ApiClient` request URL is now built by one function next to
|
|
||||||
the class (issue 18), so a self-hosted `apiOrigin` with a base path keeps it
|
|
||||||
on every request, a path works with or without a leading slash, and query
|
|
||||||
parameters are percent-encoded. A path containing `?` or `#` is rejected with
|
|
||||||
an error instead of being silently cut.
|
|
||||||
- 2026-09-23: Made the CLI testable and tested it (issue 12). The command bodies
|
|
||||||
moved from `bin/quak.ts` into `src/cli-commands.ts` as functions that take
|
|
||||||
their options and a context (output streams, session directory, cache
|
|
||||||
directory, session loader) and return an exit code; `bin/quak.ts` only wires
|
|
||||||
them to commander and exits with the code once stdout and stderr have drained,
|
|
||||||
so nothing below it calls `process.exit`. `test/cli/commands.test.ts` drives
|
|
||||||
them with a fake client: session file modes, logout, the missing and corrupt
|
|
||||||
session paths, and the output and exit code of `whoami`, `collections`,
|
|
||||||
`files`, `get`, `get-thumb`, `backup` and `helper list-missing-thumbnails`.
|
|
||||||
- 2026-09-23: Hardened the backup tree's atomic copy (issue 22). `copyAtomic`
|
|
||||||
fsyncs its temp file before the rename and the directory after it, through the
|
|
||||||
download writer's `fsyncPath`; each backup run deletes `.quak-backup-*.tmp`
|
|
||||||
files whose process is no longer running. The README backup layout names the
|
|
||||||
temp files and states that the rename replaces a symlink and takes the temp
|
|
||||||
file's permissions. Added tests for a missing and an unwritable destination
|
|
||||||
directory for `downloadFile` and `downloadThumbnail`.
|
|
||||||
- 2026-09-23: Hardened the JPEG EXIF scan behind `backup-metadata --exif` (issue
|
|
||||||
11). Every segment length is checked against the remaining bytes and lengths
|
|
||||||
under 2 stop the scan, so a truncated or corrupt original can neither throw
|
|
||||||
nor loop. A malformed or unparseable EXIF segment is recorded as
|
|
||||||
`imageMetadata.exifError`, and a failure to read the original as
|
|
||||||
`imageMetadataError` in the per-file JSON, instead of the field being left
|
|
||||||
out.
|
|
||||||
- 2026-09-22: Hardened the retry classifier (issue 80). A `POST` or `PUT` is
|
|
||||||
replayed only when every errno in the cause chain is a connect errno, and it
|
|
||||||
no longer follows redirects. `getRetryOptions()` returns a copy. Tests pin
|
|
||||||
every errno the classifier names, the cause-chain depth limit, cycle
|
|
||||||
termination, and a fresh deadline per attempt for every retrying entry point.
|
|
||||||
The README's endpoint list is the one place that names the requests the replay
|
|
||||||
rule covers.
|
|
||||||
- 2026-09-22: Stopped `make test` collecting tests from checkouts nested under
|
|
||||||
`.claude/` (issue 25). vitest ignores `.gitignore` when finding tests, so a
|
|
||||||
nested checkout ran the whole suite again; `vitest.config.ts` now adds
|
|
||||||
`.claude/**` to vitest's default excludes, and
|
|
||||||
`test/packaging/nested-checkout.test.ts` plants a nested checkout in a temp
|
|
||||||
directory and fails if vitest would collect it.
|
|
||||||
- 2026-09-22: Dropped the deprecated `@types/libsodium-wrappers-sumo` stub from
|
|
||||||
`devDependencies` (issue 27). It shipped no declarations; the types come from
|
|
||||||
`libsodium-wrappers-sumo` itself. `yarn.lock` regenerated by `yarn remove`.
|
|
||||||
- 2026-09-22: Hardened the client session lifecycle (issue 10).
|
|
||||||
`Client.fromJSON` checks every snapshot field and each key's decoded length
|
|
||||||
and names the bad field; `toJSON` reads the token through
|
|
||||||
`ApiClient.getAuthToken` and throws when there is none; `logout` zeroes the
|
|
||||||
key buffers, and `collectionsSince` re-checks for logout after its request so
|
|
||||||
it never decrypts with zeroed keys. The CLI reports a corrupt session file
|
|
||||||
separately from a missing one (`src/cli-session.ts`).
|
|
||||||
- 2026-09-22: Sanitized file names taken from server metadata (issue 9). A new
|
|
||||||
`src/filename.ts` holds the one sanitizer, used by `quak get`/`get-thumb`
|
|
||||||
without `--out`, `downloadFile`/`downloadThumbnail` without `outPath`, and the
|
|
||||||
backup and metadata backup trees; it removes separators, control characters,
|
|
||||||
leading dots and Windows device names, and falls back to a name built from the
|
|
||||||
ID for an empty title. Originals-cache extensions are letters and digits only,
|
|
||||||
else `.bin`. A user-supplied path is used as is. `decryptFile` reads a missing
|
|
||||||
or non-string title as "" and rejects metadata that is not a JSON object.
|
|
||||||
- 2026-09-22: Rewrote the README API reference (and the Getting Started / usage
|
|
||||||
snippets) to match the shipped cache/API library on `next` (issue 53, issue
|
|
||||||
13). Documented `Library.open` and its options, the default-read vs `fresh()`
|
|
||||||
distinction (and that the CLI's read commands are fresh), the record types and
|
|
||||||
`snapshot()`/`subscribe()`, the `albums`/`photos`/`timeline` read surface,
|
|
||||||
`Photo` content methods, `thumbnails.ensure`, the `mldata` search surface,
|
|
||||||
`backup()`, the three request pools (10/5/25), and the on-disk cache layout;
|
|
||||||
noted the deferred content-hash integrity check (issue 68). Docs-only; no code
|
|
||||||
changed.
|
|
||||||
- 2026-09-22: Added resumable, deletion-aware enumeration to `Client` (issue 38,
|
|
||||||
closes issue 7). `collectionsSince`/`filesSince` take a starting cursor,
|
|
||||||
decrypt live records, surface tombstoned ids in a separate `deleted` list (a
|
|
||||||
tombstone has nothing to decrypt, so it is a bare id, not a hollow record),
|
|
||||||
and return the max `updationTime` seen as the cursor to resume from.
|
|
||||||
`filesSince` refuses to loop when the diff reports `hasMore` without advancing
|
|
||||||
the cursor (issue 7). `listCollections`/`listFiles` are now thin wrappers that
|
|
||||||
enumerate from `sinceTime: 0` and drop deletions, so existing callers are
|
|
||||||
unaffected.
|
|
||||||
- 2026-09-22: Carried file size, thumbnail size, and the deletion flag through
|
|
||||||
`decryptFile` (issue 37, foundation for the cache/API design). Live files now
|
|
||||||
populate `file.size`/`thumbnail.size` from the server's `info` (left
|
|
||||||
`undefined` when the server omits it), and `isDeleted` is carried from the
|
|
||||||
diff row onto `EnteFile`. No caller change: `listFiles` still filters deleted
|
|
||||||
rows before decrypting. Surfacing a tombstone through decryption belongs to
|
|
||||||
the enumeration unit (issue 38).
|
|
||||||
- 2026-08-10: Made `lint-once.test.ts` enforce what its header claims. It walked
|
|
||||||
`make check` only, so it never read `Dockerfile` — the image CI builds through
|
|
||||||
`script/cibuild` — and a second `prettier --check .` could be added there with
|
|
||||||
the suite staying green. The walk now also starts at
|
|
||||||
`.gitea/workflows/check.yml` and follows its `run:` steps, so the graph under
|
|
||||||
test is the one CI executes rather than the one someone assumed it executes.
|
|
||||||
The lockfile assertion was a substring check against the whole of
|
|
||||||
`script/bootstrap`, which has two install sites and so reported the branch the
|
|
||||||
containers never take; the two branches are now resolved separately and every
|
|
||||||
`yarn install` in each is required to be `--frozen-lockfile`. Prettier is
|
|
||||||
counted per occurrence instead of per line, so two invocations chained with
|
|
||||||
`&&` no longer read as one, and edges are followed on counted lines instead of
|
|
||||||
being skipped. Every way for the walk to reach nothing — an unknown target, an
|
|
||||||
unknown script, a missing file, a node with no commands, an unknown node kind
|
|
||||||
— is a thrown error rather than a quiet zero. Every assertion in the file was
|
|
||||||
mutation-tested individually.
|
|
||||||
- 2026-08-10: Stopped `make check` running `prettier --check .` twice. Since
|
|
||||||
linting moved into Docker, the duplicate was one container pass and one host
|
|
||||||
pass of the same check: `script/lint` builds `Dockerfile.lint`, which runs
|
|
||||||
prettier as a build step, and `script/check` then called `script/fmt-check` as
|
|
||||||
well. The host call is gone from `script/check` and from `script/precommit`;
|
|
||||||
the container keeps checking formatting, because a successful
|
|
||||||
`Dockerfile.lint` build is what CI treats as proof of a clean tree, and it is
|
|
||||||
also what still fails the pre-commit hook on a badly formatted tree.
|
|
||||||
`script/fmt-check` survives as a standalone entrypoint, whose verdict cannot
|
|
||||||
drift from the container's. A test walks the invocation graph from each
|
|
||||||
entrypoint — through the Makefile shims, the `script/` calls and the
|
|
||||||
`docker build` — and asserts the prettier count, so the duplication cannot
|
|
||||||
come back unnoticed.
|
|
||||||
- 2026-08-10: Moved all linting into Docker. `script/lint` builds a new root
|
|
||||||
`Dockerfile.lint`, which copies the repo into the digest-pinned node image and
|
|
||||||
runs eslint and prettier as build steps, so a successful build is a clean
|
|
||||||
lint; no host lint path remains and `yarn lint` is gone from `package.json`. A
|
|
||||||
fail-closed `LINT_EPOCH` guard stops Docker serving the linter layers from
|
|
||||||
cache, which is how a lint build returns success in under a second having
|
|
||||||
linted nothing. The lint stage inside `Dockerfile` and its `COPY --from=lint`
|
|
||||||
ordering hack are gone: that image now runs `make test` and `make build` only,
|
|
||||||
because `script/check` calls `script/lint` and running it in a container would
|
|
||||||
mean docker inside docker. `script/cibuild` builds the lint image first, then
|
|
||||||
the test and build image.
|
|
||||||
- 2026-08-09: Made `make docker` green and policy-conformant. Multi-stage
|
|
||||||
Dockerfile: a lint stage runs `make fmt-check` and `make lint`, and the check
|
|
||||||
stage takes a `COPY --from=lint` dependency on it before running `make check`
|
|
||||||
and `make build`. `CHECK_EPOCH` and a fail-closed guard stop Docker serving
|
|
||||||
those two layers from cache, which is what let a build report success without
|
|
||||||
running the suite. `script/projectname` says `quak`, so the image is tagged
|
|
||||||
`quak`; `script/bootstrap` updates apt lists before installing, so a Debian
|
|
||||||
base works; `.dockerignore` no longer ships the compiled binary, the caches or
|
|
||||||
agent worktrees into the build context, and keeps `.gitignore` in it for
|
|
||||||
prettier.
|
|
||||||
- 2026-08-09: Fixed the TypeScript build. `rootDir` is the repo root, so `bin/`
|
|
||||||
compiles alongside `src/` instead of failing with TS6059; output is
|
|
||||||
`dist/src/` and `dist/bin/`, which is where `main`, `types` and `bin.quak` now
|
|
||||||
point. `script/build` verifies the declared entrypoints exist after the
|
|
||||||
compiler runs and makes the CLI executable, the Dockerfile runs `make build`
|
|
||||||
as well as `make check`, and a `quak` script makes the README's
|
|
||||||
`yarn quak <command>` examples work.
|
|
||||||
- 2026-08-09: Retry policy: no retry on 4xx (except `408` and `429`),
|
|
||||||
exponential backoff with full jitter on 5xx, transport failures and truncated
|
|
||||||
transfers, under per-attempt deadlines that cover the response body as well as
|
|
||||||
the request. Downloads retry request, stream consumption and decryption as one
|
|
||||||
unit; `postJSON` and `putJSON` are replayed only when the connection was never
|
|
||||||
established.
|
|
||||||
- 2026-08-09: Downloads verify the secretstream terminated on `TAG_FINAL` and
|
|
||||||
write output atomically: a truncated body is rejected instead of landing on
|
|
||||||
disk as a short file, and plaintext is staged in a sibling temp file and
|
|
||||||
renamed into place, so a failed download leaves the destination untouched.
|
|
||||||
- 2026-07-07 Adopted scripts-to-rule-them-all: `script/` entrypoints, Makefile
|
|
||||||
shims, README Entrypoints section
|
|
||||||
- 2026-06-10: Decrypted collections shared by other users (sealed-box keys);
|
|
||||||
listCollections drops deleted-collection tombstones.
|
|
||||||
- 2026-06-10: Login hardening: dual-2FA empty-string fields handled, TOTP
|
|
||||||
preferred when a passkey is also enrolled, interactive input via
|
|
||||||
@inquirer/prompts.
|
|
||||||
- 2026-06-10: Replaced sharp with pure JS (jpeg-js + exif-reader); added
|
|
||||||
single-binary bun build and make install.
|
|
||||||
- 2026-06-09: Added backup-metadata command (ML data always included, --exif
|
|
||||||
opt-in); rewrote README to match the implementation; added thumbnail helper
|
|
||||||
tests.
|
|
||||||
- 2026-05-13: Full CLI surface: login, backup with dedup symlink layout,
|
|
||||||
collections, files, get, get-thumb, thumbnail repair helpers.
|
|
||||||
- 2026-05-13: Client OO API with literate usage tests; file download and
|
|
||||||
decryption; all three metadata layers decrypted and persisted; renamed quack
|
|
||||||
to quak.
|
|
||||||
- 2026-05-11: SRP login flow (email OTP + TOTP) and ApiClient.
|
|
||||||
|
|
||||||
# Future Steps
|
# Future Steps
|
||||||
|
|
||||||
- Future desktop client, separate repo:
|
- Future desktop client, separate repo:
|
||||||
|
|||||||
+11
-4
@@ -74,7 +74,11 @@ const context = (): CliContext => ({
|
|||||||
sessionDir: paths.data,
|
sessionDir: paths.data,
|
||||||
cacheDir: program.opts<ProgramOptions>().cacheDir,
|
cacheDir: program.opts<ProgramOptions>().cacheDir,
|
||||||
loadSession: (path) => loadSession(path, logRequests(log())),
|
loadSession: (path) => loadSession(path, logRequests(log())),
|
||||||
login: (opts) => Client.login({ ...opts, apiOptions: logRequests(log()) }),
|
login: (opts) =>
|
||||||
|
Client.login({
|
||||||
|
...opts,
|
||||||
|
apiOptions: { ...opts.apiOptions, ...logRequests(log()) },
|
||||||
|
}),
|
||||||
prompt: (message) => input({ message }),
|
prompt: (message) => input({ message }),
|
||||||
promptSecret: (message) => password({ message, mask: true }),
|
promptSecret: (message) => password({ message, mask: true }),
|
||||||
});
|
});
|
||||||
@@ -85,7 +89,12 @@ const run = (command: Promise<number>): Promise<void> =>
|
|||||||
program
|
program
|
||||||
.command("login")
|
.command("login")
|
||||||
.description("Log in to an Ente account and save the session")
|
.description("Log in to an Ente account and save the session")
|
||||||
.action(() => run(loginCommand(context())));
|
.option(
|
||||||
|
"--server <url>",
|
||||||
|
"Address of a self-hosted Ente server's API, saved with the session " +
|
||||||
|
"(default: QUAK_SERVER, then Ente's hosted service)",
|
||||||
|
)
|
||||||
|
.action((opts: { server?: string }) => run(loginCommand(context(), opts)));
|
||||||
|
|
||||||
program
|
program
|
||||||
.command("whoami")
|
.command("whoami")
|
||||||
@@ -123,7 +132,6 @@ program
|
|||||||
.description("Download and decrypt a single file")
|
.description("Download and decrypt a single file")
|
||||||
.argument("<fileID>", "File ID (from `quak files`)")
|
.argument("<fileID>", "File ID (from `quak files`)")
|
||||||
.option("--out <path>", "Output file path")
|
.option("--out <path>", "Output file path")
|
||||||
.option("--collection <id>", "Accepted for compatibility; ignored")
|
|
||||||
.action((fileID: string, opts: { out?: string }) =>
|
.action((fileID: string, opts: { out?: string }) =>
|
||||||
run(getCommand(context(), fileID, opts)),
|
run(getCommand(context(), fileID, opts)),
|
||||||
);
|
);
|
||||||
@@ -133,7 +141,6 @@ program
|
|||||||
.description("Download and decrypt a thumbnail")
|
.description("Download and decrypt a thumbnail")
|
||||||
.argument("<fileID>", "File ID (from `quak files`)")
|
.argument("<fileID>", "File ID (from `quak files`)")
|
||||||
.option("--out <path>", "Output file path")
|
.option("--out <path>", "Output file path")
|
||||||
.option("--collection <id>", "Accepted for compatibility; ignored")
|
|
||||||
.action((fileID: string, opts: { out?: string }) =>
|
.action((fileID: string, opts: { out?: string }) =>
|
||||||
run(getThumbCommand(context(), fileID, opts)),
|
run(getThumbCommand(context(), fileID, opts)),
|
||||||
);
|
);
|
||||||
|
|||||||
+1
-1
@@ -12,7 +12,7 @@
|
|||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"engines": {
|
"engines": {
|
||||||
"node": ">=22"
|
"node": "^22.13.0 || >=23.5.0"
|
||||||
},
|
},
|
||||||
"main": "./dist/src/index.js",
|
"main": "./dist/src/index.js",
|
||||||
"types": "./dist/src/index.d.ts",
|
"types": "./dist/src/index.d.ts",
|
||||||
|
|||||||
Executable
+85
@@ -0,0 +1,85 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# script/build-bin: run script/build, then bundle the built dist/ into one
|
||||||
|
# executable, bin/quak, which reports the version script/build stamped. Our own
|
||||||
|
# extension to scripts-to-rule-them-all.
|
||||||
|
#
|
||||||
|
# bun compiles the binary from a copy of its own executable, so the binary
|
||||||
|
# loads the same libraries as the bun that built it. nix's bun loads libraries
|
||||||
|
# from the nix store: on Linux its C library, on macOS its ICU. A binary
|
||||||
|
# compiled from it loads them too. nix-collect-garbage can delete them, and the
|
||||||
|
# binary then no longer starts. This script therefore compiles with bun's own
|
||||||
|
# release build, which loads only the system's libraries. It downloads that
|
||||||
|
# build and checks it against the hash below before running it.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||||
|
|
||||||
|
# bun 1.3.13, 2026-11-11, and the SHA-256 of each release archive below. The
|
||||||
|
# x86-64 archives are the baseline builds, which also run on CPUs without AVX2.
|
||||||
|
# The musl archives are for Linux systems that use the musl C library, such as
|
||||||
|
# Alpine.
|
||||||
|
BUN_VERSION="1.3.13"
|
||||||
|
BUN_LINUX_X64_BASELINE_SHA256="9d8a24292a7068090205daac0a5a223f5f69736f5287e37bf88d3b4031edc750"
|
||||||
|
BUN_LINUX_X64_MUSL_BASELINE_SHA256="88ca7c7ad235b498f549eea2f770f434e9f0f5e9ba95168a2d3a1f235184c394"
|
||||||
|
BUN_LINUX_AARCH64_SHA256="70bae41b3908b0a120e1e58c5c8af30e74afae3b8d11b0d3fdd8e787ddfb4b22"
|
||||||
|
BUN_LINUX_AARCH64_MUSL_SHA256="5385e978107ce4934298d8d6afe9bfbb898683f6cc23e6753a0da60bc60c5b81"
|
||||||
|
BUN_DARWIN_X64_BASELINE_SHA256="a98ba6a480f22fda9b343626b906a4e26aa53618bf85d2bc5928ecf2ba45f0ed"
|
||||||
|
BUN_DARWIN_AARCH64_SHA256="5467e3f65dba526b9fea98f0cce04efafc0c63e169733ec27b876a3ad32da190"
|
||||||
|
|
||||||
|
# compile_with_downloaded_bun <archive-name> <sha256>
|
||||||
|
compile_with_downloaded_bun() {
|
||||||
|
tmp="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "$tmp"' EXIT
|
||||||
|
curl -fsSL -o "$tmp/$1.zip" \
|
||||||
|
"https://github.com/oven-sh/bun/releases/download/bun-v$BUN_VERSION/$1.zip"
|
||||||
|
# macOS always has shasum, but not every macOS has sha256sum.
|
||||||
|
if command -v sha256sum >/dev/null 2>&1; then
|
||||||
|
echo "$2 $tmp/$1.zip" | sha256sum -c -
|
||||||
|
else
|
||||||
|
echo "$2 $tmp/$1.zip" | shasum -a 256 -c -
|
||||||
|
fi
|
||||||
|
unzip -q "$tmp/$1.zip" -d "$tmp"
|
||||||
|
"$tmp/$1/bun" build dist/bin/quak.js --compile --outfile bin/quak
|
||||||
|
}
|
||||||
|
|
||||||
|
main() {
|
||||||
|
cd "$ROOT"
|
||||||
|
"$ROOT/script/build"
|
||||||
|
platform="$(uname -s)-$(uname -m)"
|
||||||
|
# musl's ldd begins its version output with "musl libc"; glibc's does not.
|
||||||
|
if ldd --version 2>&1 | grep -q '^musl libc'; then
|
||||||
|
platform="$platform-musl"
|
||||||
|
fi
|
||||||
|
case "$platform" in
|
||||||
|
Linux-x86_64)
|
||||||
|
compile_with_downloaded_bun bun-linux-x64-baseline \
|
||||||
|
"$BUN_LINUX_X64_BASELINE_SHA256"
|
||||||
|
;;
|
||||||
|
Linux-x86_64-musl)
|
||||||
|
compile_with_downloaded_bun bun-linux-x64-musl-baseline \
|
||||||
|
"$BUN_LINUX_X64_MUSL_BASELINE_SHA256"
|
||||||
|
;;
|
||||||
|
Linux-aarch64)
|
||||||
|
compile_with_downloaded_bun bun-linux-aarch64 \
|
||||||
|
"$BUN_LINUX_AARCH64_SHA256"
|
||||||
|
;;
|
||||||
|
Linux-aarch64-musl)
|
||||||
|
compile_with_downloaded_bun bun-linux-aarch64-musl \
|
||||||
|
"$BUN_LINUX_AARCH64_MUSL_SHA256"
|
||||||
|
;;
|
||||||
|
Darwin-x86_64)
|
||||||
|
compile_with_downloaded_bun bun-darwin-x64-baseline \
|
||||||
|
"$BUN_DARWIN_X64_BASELINE_SHA256"
|
||||||
|
;;
|
||||||
|
Darwin-arm64)
|
||||||
|
compile_with_downloaded_bun bun-darwin-aarch64 \
|
||||||
|
"$BUN_DARWIN_AARCH64_SHA256"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "build-bin: bun has no release build for $(uname -s) on $(uname -m)" >&2
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
main "$@"
|
||||||
+8
-1
@@ -354,7 +354,13 @@ export class ApiClient {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
async putFile(presignedURL: string, data: Uint8Array): Promise<void> {
|
// `contentMD5` is the value given to `getUploadURL`. The server signs it
|
||||||
|
// into the URL, and storage refuses a PUT without that `Content-MD5`.
|
||||||
|
async putFile(
|
||||||
|
presignedURL: string,
|
||||||
|
data: Uint8Array,
|
||||||
|
contentMD5: string,
|
||||||
|
): Promise<void> {
|
||||||
// Idempotent despite being a write: a presigned PUT stores one whole
|
// Idempotent despite being a write: a presigned PUT stores one whole
|
||||||
// object at one key in one request, so replaying it either overwrites
|
// object at one key in one request, so replaying it either overwrites
|
||||||
// the same bytes or lands them for the first time. There is no partial
|
// the same bytes or lands them for the first time. There is no partial
|
||||||
@@ -370,6 +376,7 @@ export class ApiClient {
|
|||||||
headers: {
|
headers: {
|
||||||
"Content-Type": "application/octet-stream",
|
"Content-Type": "application/octet-stream",
|
||||||
"Content-Length": String(data.length),
|
"Content-Length": String(data.length),
|
||||||
|
"Content-MD5": contentMD5,
|
||||||
},
|
},
|
||||||
body: data,
|
body: data,
|
||||||
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
||||||
|
|||||||
+25
-3
@@ -21,6 +21,7 @@ import { extname, join } from "node:path";
|
|||||||
import {
|
import {
|
||||||
type Client,
|
type Client,
|
||||||
type ClientSnapshot,
|
type ClientSnapshot,
|
||||||
|
isServerAddress,
|
||||||
type LoginOptions,
|
type LoginOptions,
|
||||||
} from "./client.js";
|
} from "./client.js";
|
||||||
import { lockBackupDirectory, type BackupLock } from "./backup.js";
|
import { lockBackupDirectory, type BackupLock } from "./backup.js";
|
||||||
@@ -56,7 +57,7 @@ export interface CliContext {
|
|||||||
// Directory holding `session.json`.
|
// Directory holding `session.json`.
|
||||||
sessionDir: string;
|
sessionDir: string;
|
||||||
// The `--cache-dir` global, or undefined to let the library pick its
|
// The `--cache-dir` global, or undefined to let the library pick its
|
||||||
// per-user default keyed by the account id.
|
// per-user default keyed by the account's server and user ID.
|
||||||
cacheDir?: string;
|
cacheDir?: string;
|
||||||
// Reads the session file into a client, or null when there is none. The
|
// Reads the session file into a client, or null when there is none. The
|
||||||
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
|
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
|
||||||
@@ -129,6 +130,7 @@ const parseID = (
|
|||||||
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
|
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
|
||||||
// originals through the on-disk cache.
|
// originals through the on-disk cache.
|
||||||
const readLibraryClient = (client: Client): LibraryClient => ({
|
const readLibraryClient = (client: Client): LibraryClient => ({
|
||||||
|
apiOrigin: client.apiOrigin,
|
||||||
whoami: () => client.whoami(),
|
whoami: () => client.whoami(),
|
||||||
collectionsSince: (args) => client.collectionsSince(args),
|
collectionsSince: (args) => client.collectionsSince(args),
|
||||||
filesSince: (args) => client.filesSince(args),
|
filesSince: (args) => client.filesSince(args),
|
||||||
@@ -155,8 +157,23 @@ const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
|
|||||||
onProgress: logRefresh(ctx.log),
|
onProgress: logRefresh(ctx.log),
|
||||||
});
|
});
|
||||||
|
|
||||||
export const loginCommand = async (ctx: CliContext): Promise<number> => {
|
// Logs in to the server `--server` or `QUAK_SERVER` names, the flag first, or
|
||||||
|
// to Ente's hosted service when neither is set. The saved session keeps the
|
||||||
|
// server, so every later command uses it.
|
||||||
|
export const loginCommand = async (
|
||||||
|
ctx: CliContext,
|
||||||
|
opts: { server?: string },
|
||||||
|
): Promise<number> => {
|
||||||
await init();
|
await init();
|
||||||
|
const server = opts.server ?? process.env.QUAK_SERVER;
|
||||||
|
if (server !== undefined && !isServerAddress(server)) {
|
||||||
|
ctx.log(
|
||||||
|
"fatal",
|
||||||
|
`Invalid server address "${server}": it must be an http or ` +
|
||||||
|
`https URL with no query or fragment`,
|
||||||
|
);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
const email = process.env.QUAK_EMAIL ?? (await ctx.prompt("Email"));
|
const email = process.env.QUAK_EMAIL ?? (await ctx.prompt("Email"));
|
||||||
const password =
|
const password =
|
||||||
process.env.QUAK_PASSWORD ?? (await ctx.promptSecret("Password"));
|
process.env.QUAK_PASSWORD ?? (await ctx.promptSecret("Password"));
|
||||||
@@ -168,6 +185,8 @@ export const loginCommand = async (ctx: CliContext): Promise<number> => {
|
|||||||
password,
|
password,
|
||||||
totp: async () => ctx.prompt("TOTP code: "),
|
totp: async () => ctx.prompt("TOTP code: "),
|
||||||
emailOTP: async () => ctx.prompt("Email verification code: "),
|
emailOTP: async () => ctx.prompt("Email verification code: "),
|
||||||
|
apiOptions:
|
||||||
|
server === undefined ? undefined : { apiOrigin: server },
|
||||||
});
|
});
|
||||||
|
|
||||||
saveSession(ctx.sessionDir, client.toJSON());
|
saveSession(ctx.sessionDir, client.toJSON());
|
||||||
@@ -208,7 +227,10 @@ export const logoutCommand = async (ctx: CliContext): Promise<number> => {
|
|||||||
try {
|
try {
|
||||||
const client = ctx.loadSession(path);
|
const client = ctx.loadSession(path);
|
||||||
if (client) {
|
if (client) {
|
||||||
cacheDir ??= defaultCacheDirectory(client.whoami().userID);
|
cacheDir ??= defaultCacheDirectory(
|
||||||
|
client.whoami().userID,
|
||||||
|
client.apiOrigin,
|
||||||
|
);
|
||||||
await client.logoutOnServer();
|
await client.logoutOnServer();
|
||||||
client.logout();
|
client.logout();
|
||||||
}
|
}
|
||||||
|
|||||||
+37
-1
@@ -40,6 +40,9 @@ export interface ClientSnapshot {
|
|||||||
masterKey: string;
|
masterKey: string;
|
||||||
secretKey: string;
|
secretKey: string;
|
||||||
publicKey: string;
|
publicKey: string;
|
||||||
|
// The `apiOrigin` the client was given, so a restored client talks to
|
||||||
|
// the server that issued its token. Absent means the default server.
|
||||||
|
apiOrigin?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
// The result of a resumable enumeration. Live decrypted records and deleted
|
// The result of a resumable enumeration. Live decrypted records and deleted
|
||||||
@@ -66,7 +69,19 @@ export interface FilesPage {
|
|||||||
const errorMessage = (err: unknown): string =>
|
const errorMessage = (err: unknown): string =>
|
||||||
err instanceof Error ? err.message : String(err);
|
err instanceof Error ? err.message : String(err);
|
||||||
|
|
||||||
|
// Whether `text` can be an Ente server's address: an http or https URL, which
|
||||||
|
// may have a path, and no query or fragment, which request URLs could not
|
||||||
|
// carry. `quak login --server` and `Client.fromJSON` both check with it.
|
||||||
|
export const isServerAddress = (text: string): boolean =>
|
||||||
|
URL.canParse(text) &&
|
||||||
|
["http:", "https:"].includes(new URL(text).protocol) &&
|
||||||
|
!/[?#]/.test(text);
|
||||||
|
|
||||||
export class Client {
|
export class Client {
|
||||||
|
// The `apiOrigin` the client was given, or undefined when it was given
|
||||||
|
// none, which means Ente's hosted service. The library keeps a cache per
|
||||||
|
// server as well as per user ID.
|
||||||
|
readonly apiOrigin: string | undefined;
|
||||||
private readonly api: ApiClient;
|
private readonly api: ApiClient;
|
||||||
private readonly email: string;
|
private readonly email: string;
|
||||||
private readonly userID: number;
|
private readonly userID: number;
|
||||||
@@ -82,6 +97,7 @@ export class Client {
|
|||||||
masterKey: Uint8Array,
|
masterKey: Uint8Array,
|
||||||
secretKey: Uint8Array,
|
secretKey: Uint8Array,
|
||||||
publicKey: Uint8Array,
|
publicKey: Uint8Array,
|
||||||
|
apiOrigin: string | undefined,
|
||||||
) {
|
) {
|
||||||
this.api = api;
|
this.api = api;
|
||||||
this.email = email;
|
this.email = email;
|
||||||
@@ -89,6 +105,7 @@ export class Client {
|
|||||||
this.masterKey = masterKey;
|
this.masterKey = masterKey;
|
||||||
this.secretKey = secretKey;
|
this.secretKey = secretKey;
|
||||||
this.publicKey = publicKey;
|
this.publicKey = publicKey;
|
||||||
|
this.apiOrigin = apiOrigin;
|
||||||
}
|
}
|
||||||
|
|
||||||
static async login(opts: LoginOptions): Promise<Client> {
|
static async login(opts: LoginOptions): Promise<Client> {
|
||||||
@@ -130,12 +147,15 @@ export class Client {
|
|||||||
unwrapped.masterKey,
|
unwrapped.masterKey,
|
||||||
unwrapped.secretKey,
|
unwrapped.secretKey,
|
||||||
unwrapped.publicKey,
|
unwrapped.publicKey,
|
||||||
|
opts.apiOptions?.apiOrigin,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Restore a client from a `toJSON()` snapshot. The snapshot usually comes
|
// Restore a client from a `toJSON()` snapshot. The snapshot usually comes
|
||||||
// straight from `JSON.parse` of a file on disk, so every field is checked
|
// straight from `JSON.parse` of a file on disk, so every field is checked
|
||||||
// before use; a bad one throws an error naming it. Needs `init()` first.
|
// before use; a bad one throws an error naming it. The snapshot's
|
||||||
|
// `apiOrigin`, when it has one, is used instead of `apiOptions.apiOrigin`.
|
||||||
|
// Needs `init()` first.
|
||||||
static fromJSON(snapshot: unknown, apiOptions?: ApiClientOptions): Client {
|
static fromJSON(snapshot: unknown, apiOptions?: ApiClientOptions): Client {
|
||||||
const invalid = (field: string, problem: string): Error =>
|
const invalid = (field: string, problem: string): Error =>
|
||||||
new Error(`Invalid session data: ${field} ${problem}`);
|
new Error(`Invalid session data: ${field} ${problem}`);
|
||||||
@@ -152,6 +172,17 @@ export class Client {
|
|||||||
if (!Number.isInteger(s.userID)) {
|
if (!Number.isInteger(s.userID)) {
|
||||||
throw invalid("userID", "must be an integer");
|
throw invalid("userID", "must be an integer");
|
||||||
}
|
}
|
||||||
|
if (
|
||||||
|
s.apiOrigin !== undefined &&
|
||||||
|
(typeof s.apiOrigin !== "string" || !isServerAddress(s.apiOrigin))
|
||||||
|
) {
|
||||||
|
throw invalid(
|
||||||
|
"apiOrigin",
|
||||||
|
"must be an http or https URL with no query or fragment",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
const apiOrigin =
|
||||||
|
(s.apiOrigin as string | undefined) ?? apiOptions?.apiOrigin;
|
||||||
const key = (field: string): Uint8Array => {
|
const key = (field: string): Uint8Array => {
|
||||||
const value = s[field];
|
const value = s[field];
|
||||||
if (typeof value !== "string") {
|
if (typeof value !== "string") {
|
||||||
@@ -175,6 +206,7 @@ export class Client {
|
|||||||
|
|
||||||
const api = new ApiClient({
|
const api = new ApiClient({
|
||||||
...apiOptions,
|
...apiOptions,
|
||||||
|
apiOrigin,
|
||||||
authToken: s.token as string,
|
authToken: s.token as string,
|
||||||
});
|
});
|
||||||
return new Client(
|
return new Client(
|
||||||
@@ -184,6 +216,7 @@ export class Client {
|
|||||||
key("masterKey"),
|
key("masterKey"),
|
||||||
key("secretKey"),
|
key("secretKey"),
|
||||||
key("publicKey"),
|
key("publicKey"),
|
||||||
|
apiOrigin,
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -222,6 +255,9 @@ export class Client {
|
|||||||
masterKey: toBase64(this.masterKey),
|
masterKey: toBase64(this.masterKey),
|
||||||
secretKey: toBase64(this.secretKey),
|
secretKey: toBase64(this.secretKey),
|
||||||
publicKey: toBase64(this.publicKey),
|
publicKey: toBase64(this.publicKey),
|
||||||
|
...(this.apiOrigin === undefined
|
||||||
|
? {}
|
||||||
|
: { apiOrigin: this.apiOrigin }),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+62
-14
@@ -119,9 +119,18 @@ export {
|
|||||||
export const DEFAULT_REFRESH_INTERVAL_SECONDS = 3;
|
export const DEFAULT_REFRESH_INTERVAL_SECONDS = 3;
|
||||||
|
|
||||||
// The account's cache directory when `cacheDirectory` is not given: the
|
// The account's cache directory when `cacheDirectory` is not given: the
|
||||||
// env-paths cache directory plus the user id, so each account has its own.
|
// env-paths cache directory plus the user id, so each account has its own. An
|
||||||
export const defaultCacheDirectory = (userID: number): string =>
|
// account on a self-hosted server has the server's address, URL-encoded, in
|
||||||
join(envPaths("quak", { suffix: "" }).cache, String(userID));
|
// between, because two servers can give two accounts the same user id.
|
||||||
|
export const defaultCacheDirectory = (
|
||||||
|
userID: number,
|
||||||
|
apiOrigin?: string,
|
||||||
|
): string => {
|
||||||
|
const cache = envPaths("quak", { suffix: "" }).cache;
|
||||||
|
return apiOrigin === undefined
|
||||||
|
? join(cache, String(userID))
|
||||||
|
: join(cache, encodeURIComponent(apiOrigin), String(userID));
|
||||||
|
};
|
||||||
|
|
||||||
// Project a metadata store into by-id records, filling each record's cache
|
// Project a metadata store into by-id records, filling each record's cache
|
||||||
// paths from the content cache when one is given. Shared by the live read
|
// paths from the content cache when one is given. Shared by the live read
|
||||||
@@ -144,6 +153,9 @@ const deriveRecordsFromStore = (
|
|||||||
// tests drive a mock with no crypto or network; the real `Client` satisfies it
|
// tests drive a mock with no crypto or network; the real `Client` satisfies it
|
||||||
// structurally.
|
// structurally.
|
||||||
export interface LibraryClient {
|
export interface LibraryClient {
|
||||||
|
// The server the client talks to, as `Client.apiOrigin` names it. Absent
|
||||||
|
// means Ente's hosted service.
|
||||||
|
readonly apiOrigin?: string;
|
||||||
whoami(): { email: string; userID: number };
|
whoami(): { email: string; userID: number };
|
||||||
collectionsSince(args: { sinceTime: number }): Promise<CollectionsPage>;
|
collectionsSince(args: { sinceTime: number }): Promise<CollectionsPage>;
|
||||||
filesSince(args: {
|
filesSince(args: {
|
||||||
@@ -196,8 +208,8 @@ export type RefreshProgressCallback = (event: RefreshEvent) => void;
|
|||||||
|
|
||||||
export interface LibraryOptions {
|
export interface LibraryOptions {
|
||||||
client: LibraryClient;
|
client: LibraryClient;
|
||||||
// Where `metadata.json` lives. Defaults to the env-paths cache directory
|
// Where `metadata.json` lives. Defaults to `defaultCacheDirectory`, so
|
||||||
// plus the user id, so each account has its own cache.
|
// each account has its own cache.
|
||||||
cacheDirectory?: string;
|
cacheDirectory?: string;
|
||||||
// The root of every photo's save path, where `Photo.download()` and
|
// The root of every photo's save path, where `Photo.download()` and
|
||||||
// `lib.backup()` put originals. Defaults to `photos` in the working
|
// `lib.backup()` put originals. Defaults to `photos` in the working
|
||||||
@@ -300,6 +312,7 @@ export class Library {
|
|||||||
// available (a metadata-only client with no explicit source).
|
// available (a metadata-only client with no explicit source).
|
||||||
private readonly cache?: ContentCache;
|
private readonly cache?: ContentCache;
|
||||||
private readonly userID: number;
|
private readonly userID: number;
|
||||||
|
private readonly apiOrigin: string | undefined;
|
||||||
private readonly intervalMs: number;
|
private readonly intervalMs: number;
|
||||||
private readonly onProgress?: RefreshProgressCallback;
|
private readonly onProgress?: RefreshProgressCallback;
|
||||||
private readonly pools: RequestPools;
|
private readonly pools: RequestPools;
|
||||||
@@ -346,6 +359,7 @@ export class Library {
|
|||||||
client: LibraryClient;
|
client: LibraryClient;
|
||||||
store: MetadataStore;
|
store: MetadataStore;
|
||||||
userID: number;
|
userID: number;
|
||||||
|
apiOrigin: string | undefined;
|
||||||
cacheDirectory: string;
|
cacheDirectory: string;
|
||||||
downloadDirectory: string;
|
downloadDirectory: string;
|
||||||
intervalMs: number;
|
intervalMs: number;
|
||||||
@@ -358,6 +372,7 @@ export class Library {
|
|||||||
this.client = args.client;
|
this.client = args.client;
|
||||||
this.store = args.store;
|
this.store = args.store;
|
||||||
this.userID = args.userID;
|
this.userID = args.userID;
|
||||||
|
this.apiOrigin = args.apiOrigin;
|
||||||
this.cacheDirectory = args.cacheDirectory;
|
this.cacheDirectory = args.cacheDirectory;
|
||||||
this.downloadDirectory = args.downloadDirectory;
|
this.downloadDirectory = args.downloadDirectory;
|
||||||
this.intervalMs = args.intervalMs;
|
this.intervalMs = args.intervalMs;
|
||||||
@@ -406,8 +421,9 @@ export class Library {
|
|||||||
// refresh: `open()` returns onto the cache as it is.
|
// refresh: `open()` returns onto the cache as it is.
|
||||||
static async open(opts: LibraryOptions): Promise<Library> {
|
static async open(opts: LibraryOptions): Promise<Library> {
|
||||||
const { userID } = opts.client.whoami();
|
const { userID } = opts.client.whoami();
|
||||||
|
const { apiOrigin } = opts.client;
|
||||||
const cacheDirectory =
|
const cacheDirectory =
|
||||||
opts.cacheDirectory ?? defaultCacheDirectory(userID);
|
opts.cacheDirectory ?? defaultCacheDirectory(userID, apiOrigin);
|
||||||
if (opts.downloadDirectory === "") {
|
if (opts.downloadDirectory === "") {
|
||||||
throw new Error(
|
throw new Error(
|
||||||
"library: downloadDirectory is empty (leave it out to save " +
|
"library: downloadDirectory is empty (leave it out to save " +
|
||||||
@@ -419,14 +435,24 @@ export class Library {
|
|||||||
let store = await MetadataStore.load(metadataPath);
|
let store = await MetadataStore.load(metadataPath);
|
||||||
// A cache directory given explicitly can hold another account's cache.
|
// A cache directory given explicitly can hold another account's cache.
|
||||||
// Its records and cursor are not this account's, so delete it and the
|
// Its records and cursor are not this account's, so delete it and the
|
||||||
// ML data beside it and start empty. A user ID of 0 means the cache
|
// ML data beside it and start empty. A cache from another server goes
|
||||||
// was never refreshed and so holds nothing to discard.
|
// whole: two servers can give two files the same ID, so its originals
|
||||||
if (store.userID !== 0 && store.userID !== userID) {
|
// and thumbnails could be served for this account's files. A user ID
|
||||||
await rm(metadataPath, { force: true });
|
// of 0 means the cache names no account, so nothing is stored in it.
|
||||||
await rm(join(cacheDirectory, "mldata"), {
|
if (
|
||||||
|
store.userID !== 0 &&
|
||||||
|
(store.userID !== userID || store.apiOrigin !== apiOrigin)
|
||||||
|
) {
|
||||||
|
const discard = ["metadata.json", "mldata"];
|
||||||
|
if (store.apiOrigin !== apiOrigin) {
|
||||||
|
discard.push("originals", "thumbnails");
|
||||||
|
}
|
||||||
|
for (const name of discard) {
|
||||||
|
await rm(join(cacheDirectory, name), {
|
||||||
recursive: true,
|
recursive: true,
|
||||||
force: true,
|
force: true,
|
||||||
});
|
});
|
||||||
|
}
|
||||||
store = await MetadataStore.load(metadataPath);
|
store = await MetadataStore.load(metadataPath);
|
||||||
}
|
}
|
||||||
const intervalMs =
|
const intervalMs =
|
||||||
@@ -490,6 +516,7 @@ export class Library {
|
|||||||
client: opts.client,
|
client: opts.client,
|
||||||
store,
|
store,
|
||||||
userID,
|
userID,
|
||||||
|
apiOrigin,
|
||||||
cacheDirectory,
|
cacheDirectory,
|
||||||
downloadDirectory,
|
downloadDirectory,
|
||||||
intervalMs,
|
intervalMs,
|
||||||
@@ -509,9 +536,11 @@ export class Library {
|
|||||||
if (opts.refreshOnOpen === false) {
|
if (opts.refreshOnOpen === false) {
|
||||||
// The caller asks for each refresh it needs.
|
// The caller asks for each refresh it needs.
|
||||||
lib.scheduleNext();
|
lib.scheduleNext();
|
||||||
} else if (store.loadedFromDisk) {
|
} else if (store.listCollections().length > 0) {
|
||||||
// An existing copy already answers reads; refresh in the background
|
// An existing copy already answers reads; refresh in the background
|
||||||
// and start the interval once that first cycle settles.
|
// and start the interval once that first cycle settles. A cache
|
||||||
|
// that names the account but holds no collection has nothing to
|
||||||
|
// answer with, such as after `originalOf` on an empty cache.
|
||||||
void lib.runRefresh().then(() => lib.scheduleNext());
|
void lib.runRefresh().then(() => lib.scheduleNext());
|
||||||
} else {
|
} else {
|
||||||
// Nothing was cached: wait for the first refresh to fill the store
|
// Nothing was cached: wait for the first refresh to fill the store
|
||||||
@@ -658,6 +687,7 @@ export class Library {
|
|||||||
"originalOf requires a library opened with a content cache",
|
"originalOf requires a library opened with a content cache",
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
await this.saveAccountInEmptyCache();
|
||||||
return this.cache.originalOf(file, opts);
|
return this.cache.originalOf(file, opts);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -673,9 +703,23 @@ export class Library {
|
|||||||
"thumbnailOf requires a library opened with a content cache",
|
"thumbnailOf requires a library opened with a content cache",
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
await this.saveAccountInEmptyCache();
|
||||||
return this.cache.thumbnailOf(file, opts);
|
return this.cache.thumbnailOf(file, opts);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Saves the account's user ID and server in a cache that names no account
|
||||||
|
// yet, before `originalOf` or `thumbnailOf` stores a file there. A later
|
||||||
|
// `open()` for an account on another server then deletes that file (see
|
||||||
|
// `open()`), although no refresh has stored any records.
|
||||||
|
private async saveAccountInEmptyCache(): Promise<void> {
|
||||||
|
if (this.store.userID !== 0) return;
|
||||||
|
this.store.userID = this.userID;
|
||||||
|
this.store.apiOrigin = this.apiOrigin;
|
||||||
|
this.unsaved = true;
|
||||||
|
await this.store.save();
|
||||||
|
this.unsaved = false;
|
||||||
|
}
|
||||||
|
|
||||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||||
// library's, each original at its save path, with a durable failure
|
// library's, each original at its save path, with a durable failure
|
||||||
// ledger (issue #51). Takes the lock in that directory first, unless the
|
// ledger (issue #51). Takes the lock in that directory first, unless the
|
||||||
@@ -857,8 +901,12 @@ export class Library {
|
|||||||
// actually changed.
|
// actually changed.
|
||||||
let changed = false;
|
let changed = false;
|
||||||
|
|
||||||
if (this.store.userID !== this.userID) {
|
if (
|
||||||
|
this.store.userID !== this.userID ||
|
||||||
|
this.store.apiOrigin !== this.apiOrigin
|
||||||
|
) {
|
||||||
this.store.userID = this.userID;
|
this.store.userID = this.userID;
|
||||||
|
this.store.apiOrigin = this.apiOrigin;
|
||||||
changed = true;
|
changed = true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+11
-12
@@ -1,11 +1,12 @@
|
|||||||
// On-disk JSON metadata store for the local cache.
|
// On-disk JSON metadata store for the local cache.
|
||||||
//
|
//
|
||||||
// The store keeps one `metadata.json` file holding the account's server
|
// The store keeps one `metadata.json` file holding the account's server
|
||||||
// state: the user id, a schema version, the cursor for the incremental
|
// state: the user id, the server's address for an account on a self-hosted
|
||||||
// collections listing, the decrypted collection and file records, and the
|
// server, a schema version, the cursor for the incremental collections
|
||||||
// collections and files the refresh could not decrypt. The whole file is read
|
// listing, the decrypted collection and file records, and the collections and
|
||||||
// into RAM on load and rewritten as a whole on save; there is no partial
|
// files the refresh could not decrypt. The whole file is read into RAM on
|
||||||
// update and no lock file. A separate refresh unit populates the store from
|
// load and rewritten as a whole on save; there is no partial update and no
|
||||||
|
// lock file. A separate refresh unit populates the store from
|
||||||
// the server — this module only stores what it is given.
|
// the server — this module only stores what it is given.
|
||||||
//
|
//
|
||||||
// The file is a cache, so it is never trusted to exist or to be intact: a
|
// The file is a cache, so it is never trusted to exist or to be intact: a
|
||||||
@@ -46,6 +47,7 @@ export interface Undecryptable {
|
|||||||
interface StoredMetadata {
|
interface StoredMetadata {
|
||||||
schemaVersion: number;
|
schemaVersion: number;
|
||||||
userID: number;
|
userID: number;
|
||||||
|
apiOrigin?: string;
|
||||||
collectionsSinceTime: Microseconds;
|
collectionsSinceTime: Microseconds;
|
||||||
collections: StoredCollection[];
|
collections: StoredCollection[];
|
||||||
files: StoredFile[];
|
files: StoredFile[];
|
||||||
@@ -68,14 +70,10 @@ export class MetadataStore {
|
|||||||
readonly path: string;
|
readonly path: string;
|
||||||
readonly schemaVersion = METADATA_SCHEMA_VERSION;
|
readonly schemaVersion = METADATA_SCHEMA_VERSION;
|
||||||
userID = 0;
|
userID = 0;
|
||||||
|
// The account's `apiOrigin`, or undefined for Ente's hosted service.
|
||||||
|
apiOrigin: string | undefined;
|
||||||
collectionsSinceTime: Microseconds = 0;
|
collectionsSinceTime: Microseconds = 0;
|
||||||
|
|
||||||
// True when `load` populated this store from a valid existing file; false
|
|
||||||
// on a first run or a missing/corrupt/wrong-version file that loaded empty.
|
|
||||||
// `Library.open` reads it to decide whether the first refresh may run in
|
|
||||||
// the background (an existing copy already serves reads) or must be awaited.
|
|
||||||
loadedFromDisk = false;
|
|
||||||
|
|
||||||
private readonly collections = new Map<number, Collection>();
|
private readonly collections = new Map<number, Collection>();
|
||||||
private readonly files = new Map<string, EnteFile>();
|
private readonly files = new Map<string, EnteFile>();
|
||||||
private undecryptable: Undecryptable[] = [];
|
private undecryptable: Undecryptable[] = [];
|
||||||
@@ -102,8 +100,8 @@ export class MetadataStore {
|
|||||||
if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) {
|
if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) {
|
||||||
return store;
|
return store;
|
||||||
}
|
}
|
||||||
store.loadedFromDisk = true;
|
|
||||||
store.userID = parsed.userID ?? 0;
|
store.userID = parsed.userID ?? 0;
|
||||||
|
store.apiOrigin = parsed.apiOrigin;
|
||||||
store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0;
|
store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0;
|
||||||
for (const stored of parsed.collections ?? []) {
|
for (const stored of parsed.collections ?? []) {
|
||||||
const collection: Collection = {
|
const collection: Collection = {
|
||||||
@@ -143,6 +141,7 @@ export class MetadataStore {
|
|||||||
const model: StoredMetadata = {
|
const model: StoredMetadata = {
|
||||||
schemaVersion: METADATA_SCHEMA_VERSION,
|
schemaVersion: METADATA_SCHEMA_VERSION,
|
||||||
userID: this.userID,
|
userID: this.userID,
|
||||||
|
apiOrigin: this.apiOrigin,
|
||||||
collectionsSinceTime: this.collectionsSinceTime,
|
collectionsSinceTime: this.collectionsSinceTime,
|
||||||
collections: [...this.collections.values()].map((c) => ({
|
collections: [...this.collections.values()].map((c) => ({
|
||||||
...c,
|
...c,
|
||||||
|
|||||||
+1
-1
@@ -325,7 +325,7 @@ export const fixMissingThumbnails = async (
|
|||||||
ciphertext.length,
|
ciphertext.length,
|
||||||
md5,
|
md5,
|
||||||
);
|
);
|
||||||
await api.putFile(url, ciphertext);
|
await api.putFile(url, ciphertext, md5);
|
||||||
await api.updateThumbnail(file.id, objectKey, toBase64(header));
|
await api.updateThumbnail(file.id, objectKey, toBase64(header));
|
||||||
|
|
||||||
log(`[${collectionName}] Thumbnail uploaded for ${title}`);
|
log(`[${collectionName}] Thumbnail uploaded for ${title}`);
|
||||||
|
|||||||
+10
-4
@@ -780,7 +780,7 @@ describe("ApiClient timeouts", () => {
|
|||||||
|
|
||||||
await client.getJSON("/a");
|
await client.getJSON("/a");
|
||||||
await client.postJSON("/b", {});
|
await client.postJSON("/b", {});
|
||||||
await client.putFile("https://s3.example/x", new Uint8Array([1]));
|
await client.putFile("https://s3.example/x", new Uint8Array([1]), "m");
|
||||||
await client.putJSON("/c", {});
|
await client.putJSON("/c", {});
|
||||||
|
|
||||||
for (const call of calls) {
|
for (const call of calls) {
|
||||||
@@ -825,7 +825,8 @@ describe("ApiClient timeouts", () => {
|
|||||||
[
|
[
|
||||||
"putFile",
|
"putFile",
|
||||||
() => new Response(null, { status: 200 }),
|
() => new Response(null, { status: 200 }),
|
||||||
(c) => c.putFile("https://s3.example/x", new Uint8Array([1])),
|
(c) =>
|
||||||
|
c.putFile("https://s3.example/x", new Uint8Array([1]), "m"),
|
||||||
],
|
],
|
||||||
[
|
[
|
||||||
"getFileStream",
|
"getFileStream",
|
||||||
@@ -1049,7 +1050,7 @@ describe("ApiClient error typing", () => {
|
|||||||
const client = new ApiClient({ fetch, retry: noWait });
|
const client = new ApiClient({ fetch, retry: noWait });
|
||||||
|
|
||||||
const err: unknown = await client
|
const err: unknown = await client
|
||||||
.putFile("https://s3.example/obj", new Uint8Array([1, 2]))
|
.putFile("https://s3.example/obj", new Uint8Array([1, 2]), "m")
|
||||||
.catch((e: unknown) => e);
|
.catch((e: unknown) => e);
|
||||||
|
|
||||||
expect(err).toBeInstanceOf(ApiError);
|
expect(err).toBeInstanceOf(ApiError);
|
||||||
@@ -1067,7 +1068,11 @@ describe("ApiClient error typing", () => {
|
|||||||
);
|
);
|
||||||
const client = new ApiClient({ fetch, retry: noWait });
|
const client = new ApiClient({ fetch, retry: noWait });
|
||||||
|
|
||||||
await client.putFile("https://s3.example/obj", new Uint8Array([1, 2]));
|
await client.putFile(
|
||||||
|
"https://s3.example/obj",
|
||||||
|
new Uint8Array([1, 2]),
|
||||||
|
"m",
|
||||||
|
);
|
||||||
expect(calls).toHaveLength(2);
|
expect(calls).toHaveLength(2);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -1277,6 +1282,7 @@ describe("ApiClient onRequest", () => {
|
|||||||
await client.putFile(
|
await client.putFile(
|
||||||
"https://s3.example/obj?X-Amz-Signature=abc",
|
"https://s3.example/obj?X-Amz-Signature=abc",
|
||||||
new Uint8Array([1, 2]),
|
new Uint8Array([1, 2]),
|
||||||
|
"m",
|
||||||
);
|
);
|
||||||
|
|
||||||
expect(requests).toEqual([
|
expect(requests).toEqual([
|
||||||
|
|||||||
+13
-2
@@ -80,13 +80,20 @@ describe("ApiClient.putFile", () => {
|
|||||||
const client = new ApiClient({ fetch, authToken: "secret-tok" });
|
const client = new ApiClient({ fetch, authToken: "secret-tok" });
|
||||||
const data = new Uint8Array([1, 2, 3, 4, 5]);
|
const data = new Uint8Array([1, 2, 3, 4, 5]);
|
||||||
|
|
||||||
await client.putFile("https://s3.example.com/presigned?sig=abc", data);
|
await client.putFile(
|
||||||
|
"https://s3.example.com/presigned?sig=abc",
|
||||||
|
data,
|
||||||
|
"md5base64==",
|
||||||
|
);
|
||||||
|
|
||||||
expect(calls[0]!.url).toBe("https://s3.example.com/presigned?sig=abc");
|
expect(calls[0]!.url).toBe("https://s3.example.com/presigned?sig=abc");
|
||||||
expect(calls[0]!.init?.method).toBe("PUT");
|
expect(calls[0]!.init?.method).toBe("PUT");
|
||||||
const headers = new Headers(calls[0]!.init?.headers as HeadersInit);
|
const headers = new Headers(calls[0]!.init?.headers as HeadersInit);
|
||||||
expect(headers.get("Content-Type")).toBe("application/octet-stream");
|
expect(headers.get("Content-Type")).toBe("application/octet-stream");
|
||||||
expect(headers.get("Content-Length")).toBe("5");
|
expect(headers.get("Content-Length")).toBe("5");
|
||||||
|
// The server signs the MD5 into the URL; storage refuses the PUT
|
||||||
|
// without it.
|
||||||
|
expect(headers.get("Content-MD5")).toBe("md5base64==");
|
||||||
// Must NOT leak auth headers to S3
|
// Must NOT leak auth headers to S3
|
||||||
expect(headers.has("X-Auth-Token")).toBe(false);
|
expect(headers.has("X-Auth-Token")).toBe(false);
|
||||||
expect(headers.has("X-Client-Package")).toBe(false);
|
expect(headers.has("X-Client-Package")).toBe(false);
|
||||||
@@ -99,7 +106,11 @@ describe("ApiClient.putFile", () => {
|
|||||||
const client = new ApiClient({ fetch });
|
const client = new ApiClient({ fetch });
|
||||||
|
|
||||||
await expect(
|
await expect(
|
||||||
client.putFile("https://s3.example.com/bad", new Uint8Array(10)),
|
client.putFile(
|
||||||
|
"https://s3.example.com/bad",
|
||||||
|
new Uint8Array(10),
|
||||||
|
"md5base64==",
|
||||||
|
),
|
||||||
).rejects.toThrow(/403/);
|
).rejects.toThrow(/403/);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
+47
-3
@@ -1,12 +1,15 @@
|
|||||||
/**
|
/**
|
||||||
* Tests for the retry options `bin/quak.ts` loads the saved session with
|
* Tests for the retry options `bin/quak.ts` loads the saved session with
|
||||||
* (`quak backup` gets the unattended ones, every other command the default),
|
* (`quak backup` gets the unattended ones, every other command the default),
|
||||||
* each with the callbacks that log its requests and retries, and for
|
* each with the callbacks that log its requests and retries, for `quak files`
|
||||||
* `quak files` taking its collection ID as an optional argument.
|
* taking its collection ID as an optional argument, for `quak get` and
|
||||||
|
* `quak get-thumb` refusing `--collection`, and for `quak login` taking
|
||||||
|
* `--server` and sending its first request to the server it names.
|
||||||
*
|
*
|
||||||
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
|
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
|
||||||
* replaced by one that records the options it is given and reports no session,
|
* replaced by one that records the options it is given and reports no session,
|
||||||
* so the command stops before it makes any request.
|
* so the command stops before it makes any request. `quak login` reads no
|
||||||
|
* session, so its test stubs `fetch` instead and records each request.
|
||||||
*
|
*
|
||||||
* The first load of the sources `bin/quak.ts` imports compiles them, and the
|
* The first load of the sources `bin/quak.ts` imports compiles them, and the
|
||||||
* first `init()` starts libsodium. On a loaded host that takes longer than a
|
* first `init()` starts libsodium. On a loaded host that takes longer than a
|
||||||
@@ -57,6 +60,8 @@ afterEach(() => {
|
|||||||
process.argv = argv;
|
process.argv = argv;
|
||||||
loaded.length = 0;
|
loaded.length = 0;
|
||||||
vi.restoreAllMocks();
|
vi.restoreAllMocks();
|
||||||
|
vi.unstubAllGlobals();
|
||||||
|
vi.unstubAllEnvs();
|
||||||
});
|
});
|
||||||
|
|
||||||
afterAll(() => {
|
afterAll(() => {
|
||||||
@@ -105,3 +110,42 @@ describe("bin/quak.ts files", () => {
|
|||||||
expect(await quak("files", "7")).toMatch(/^Not logged in/);
|
expect(await quak("files", "7")).toMatch(/^Not logged in/);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe("bin/quak.ts get and get-thumb", () => {
|
||||||
|
it.each(["get", "get-thumb"])(
|
||||||
|
"%s refuses --collection as an unknown option",
|
||||||
|
async (command) => {
|
||||||
|
expect(await quak(command, "1", "--collection", "2")).toMatch(
|
||||||
|
/^error: unknown option '--collection'/,
|
||||||
|
);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
describe("bin/quak.ts login", () => {
|
||||||
|
// The command refuses an address that is not a URL before it asks for
|
||||||
|
// anything or sends any request.
|
||||||
|
it("hands --server to the command", async () => {
|
||||||
|
expect(await quak("login", "--server", "ente.example.com")).toMatch(
|
||||||
|
/^Invalid server address "ente\.example\.com"/,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
// `fetch` answers 404, so the login fails at its first request. A login
|
||||||
|
// that succeeded would save its session in this user's real data
|
||||||
|
// directory, since env-paths is not mocked here.
|
||||||
|
it("sends its first request to the server --server names", async () => {
|
||||||
|
const requests: string[] = [];
|
||||||
|
vi.stubGlobal("fetch", async (input: string) => {
|
||||||
|
const { origin, pathname } = new URL(input);
|
||||||
|
requests.push(origin + pathname);
|
||||||
|
return new Response("not found", { status: 404 });
|
||||||
|
});
|
||||||
|
vi.stubEnv("QUAK_EMAIL", "login@example.com");
|
||||||
|
vi.stubEnv("QUAK_PASSWORD", "password");
|
||||||
|
await quak("login", "--server", "https://ente.example.com");
|
||||||
|
expect(requests).toEqual([
|
||||||
|
"https://ente.example.com/users/srp/attributes",
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
});
|
||||||
+263
-15
@@ -328,6 +328,7 @@ describe("login", () => {
|
|||||||
prompts = [];
|
prompts = [];
|
||||||
vi.stubEnv("QUAK_EMAIL", "cli@example.com");
|
vi.stubEnv("QUAK_EMAIL", "cli@example.com");
|
||||||
vi.stubEnv("QUAK_PASSWORD", "hunter2");
|
vi.stubEnv("QUAK_PASSWORD", "hunter2");
|
||||||
|
vi.stubEnv("QUAK_SERVER", undefined);
|
||||||
});
|
});
|
||||||
|
|
||||||
afterEach(() => {
|
afterEach(() => {
|
||||||
@@ -340,12 +341,14 @@ describe("login", () => {
|
|||||||
calls.push(opts);
|
calls.push(opts);
|
||||||
return loggedIn;
|
return loggedIn;
|
||||||
});
|
});
|
||||||
expect(await loginCommand(ctx)).toBe(0);
|
expect(await loginCommand(ctx, {})).toBe(0);
|
||||||
|
|
||||||
expect(prompts).toEqual([]);
|
expect(prompts).toEqual([]);
|
||||||
expect(calls).toHaveLength(1);
|
expect(calls).toHaveLength(1);
|
||||||
expect(calls[0]!.email).toBe("cli@example.com");
|
expect(calls[0]!.email).toBe("cli@example.com");
|
||||||
expect(calls[0]!.password).toBe("hunter2");
|
expect(calls[0]!.password).toBe("hunter2");
|
||||||
|
// No server named: the client uses Ente's hosted service.
|
||||||
|
expect(calls[0]!.apiOptions).toBeUndefined();
|
||||||
const path = join(ctx.sessionDir, "session.json");
|
const path = join(ctx.sessionDir, "session.json");
|
||||||
expect(stderr.text).toBe(
|
expect(stderr.text).toBe(
|
||||||
"Authenticating...\n" +
|
"Authenticating...\n" +
|
||||||
@@ -356,7 +359,7 @@ describe("login", () => {
|
|||||||
|
|
||||||
it("saves the session with mode 0600 in a directory with mode 0700", async () => {
|
it("saves the session with mode 0600 in a directory with mode 0700", async () => {
|
||||||
const ctx = loginContext(async () => loggedIn);
|
const ctx = loginContext(async () => loggedIn);
|
||||||
expect(await loginCommand(ctx)).toBe(0);
|
expect(await loginCommand(ctx, {})).toBe(0);
|
||||||
|
|
||||||
expect(statSync(ctx.sessionDir).mode & 0o777).toBe(0o700);
|
expect(statSync(ctx.sessionDir).mode & 0o777).toBe(0o700);
|
||||||
const path = join(ctx.sessionDir, "session.json");
|
const path = join(ctx.sessionDir, "session.json");
|
||||||
@@ -370,7 +373,7 @@ describe("login", () => {
|
|||||||
code = await opts.totp!();
|
code = await opts.totp!();
|
||||||
return loggedIn;
|
return loggedIn;
|
||||||
});
|
});
|
||||||
expect(await loginCommand(ctx)).toBe(0);
|
expect(await loginCommand(ctx, {})).toBe(0);
|
||||||
|
|
||||||
expect(prompts).toEqual(["TOTP code: "]);
|
expect(prompts).toEqual(["TOTP code: "]);
|
||||||
expect(code).toBe("123456");
|
expect(code).toBe("123456");
|
||||||
@@ -380,13 +383,83 @@ describe("login", () => {
|
|||||||
const ctx = loginContext(async () => {
|
const ctx = loginContext(async () => {
|
||||||
throw new Error("HTTP 401 from server");
|
throw new Error("HTTP 401 from server");
|
||||||
});
|
});
|
||||||
expect(await loginCommand(ctx)).toBe(1);
|
expect(await loginCommand(ctx, {})).toBe(1);
|
||||||
|
|
||||||
expect(stderr.text).toBe(
|
expect(stderr.text).toBe(
|
||||||
"Authenticating...\nLogin failed: HTTP 401 from server\n",
|
"Authenticating...\nLogin failed: HTTP 401 from server\n",
|
||||||
);
|
);
|
||||||
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
|
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("logs in to the server --server names, ahead of QUAK_SERVER", async () => {
|
||||||
|
vi.stubEnv("QUAK_SERVER", "https://other.example.com");
|
||||||
|
const calls: LoginOptions[] = [];
|
||||||
|
const ctx = loginContext(async (opts) => {
|
||||||
|
calls.push(opts);
|
||||||
|
return loggedIn;
|
||||||
|
});
|
||||||
|
expect(
|
||||||
|
await loginCommand(ctx, { server: "https://ente.example.com" }),
|
||||||
|
).toBe(0);
|
||||||
|
|
||||||
|
expect(calls[0]!.apiOptions).toEqual({
|
||||||
|
apiOrigin: "https://ente.example.com",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it("logs in to the server QUAK_SERVER names without --server", async () => {
|
||||||
|
vi.stubEnv("QUAK_SERVER", "http://192.0.2.1:8080/ente");
|
||||||
|
const calls: LoginOptions[] = [];
|
||||||
|
const ctx = loginContext(async (opts) => {
|
||||||
|
calls.push(opts);
|
||||||
|
return loggedIn;
|
||||||
|
});
|
||||||
|
expect(await loginCommand(ctx, {})).toBe(0);
|
||||||
|
|
||||||
|
expect(calls[0]!.apiOptions).toEqual({
|
||||||
|
apiOrigin: "http://192.0.2.1:8080/ente",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it.each([
|
||||||
|
"",
|
||||||
|
"ente.example.com",
|
||||||
|
"ftp://ente.example.com",
|
||||||
|
"https://ente.example.com/?region=eu",
|
||||||
|
"https://ente.example.com/#top",
|
||||||
|
])(
|
||||||
|
"refuses the server address %j before asking for anything",
|
||||||
|
async (server) => {
|
||||||
|
const ctx = loginContext(async () => {
|
||||||
|
throw new Error("login must not be called");
|
||||||
|
});
|
||||||
|
expect(await loginCommand(ctx, { server })).toBe(1);
|
||||||
|
|
||||||
|
expect(prompts).toEqual([]);
|
||||||
|
expect(stderr.text).toBe(
|
||||||
|
`Invalid server address "${server}": it must be an http or ` +
|
||||||
|
`https URL with no query or fragment\n`,
|
||||||
|
);
|
||||||
|
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(
|
||||||
|
false,
|
||||||
|
);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
it("refuses an empty QUAK_SERVER before asking for anything", async () => {
|
||||||
|
vi.stubEnv("QUAK_SERVER", "");
|
||||||
|
const ctx = loginContext(async () => {
|
||||||
|
throw new Error("login must not be called");
|
||||||
|
});
|
||||||
|
expect(await loginCommand(ctx, {})).toBe(1);
|
||||||
|
|
||||||
|
expect(prompts).toEqual([]);
|
||||||
|
expect(stderr.text).toBe(
|
||||||
|
`Invalid server address "": it must be an http or https URL ` +
|
||||||
|
`with no query or fragment\n`,
|
||||||
|
);
|
||||||
|
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
// These use a real client read from the session file, over a fake API that
|
// These use a real client read from the session file, over a fake API that
|
||||||
@@ -461,6 +534,16 @@ describe("logout", () => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("names a self-hosted account's default cache directory, under its server's address", async () => {
|
||||||
|
const ctx = { ...logoutContext(200), cacheDir: undefined };
|
||||||
|
const apiOrigin = "https://ente.example.com";
|
||||||
|
saveSession(ctx.sessionDir, { ...snapshot, apiOrigin });
|
||||||
|
expect(await logoutCommand(ctx)).toBe(0);
|
||||||
|
expect(stderr.text).toContain(
|
||||||
|
`Cache directory ${defaultCacheDirectory(USER_ID, apiOrigin)} still holds decrypted data`,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
it("without a session says so, calls nothing and exits 0", async () => {
|
it("without a session says so, calls nothing and exits 0", async () => {
|
||||||
expect(await logoutCommand(logoutContext(200))).toBe(0);
|
expect(await logoutCommand(logoutContext(200))).toBe(0);
|
||||||
expect(requests).toHaveLength(0);
|
expect(requests).toHaveLength(0);
|
||||||
@@ -569,14 +652,14 @@ describe("files", () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
describe("get and get-thumb", () => {
|
describe("get and get-thumb", () => {
|
||||||
it("get finds a file in any album without --collection", async () => {
|
it("get finds a file in any album", async () => {
|
||||||
const out = join(root, "diagram.png");
|
const out = join(root, "diagram.png");
|
||||||
expect(await getCommand(context(), "200", { out })).toBe(0);
|
expect(await getCommand(context(), "200", { out })).toBe(0);
|
||||||
expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
|
expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
|
||||||
expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
|
expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("get-thumb finds a file in any album without --collection", async () => {
|
it("get-thumb finds a file in any album", async () => {
|
||||||
const out = join(root, "thumb.jpg");
|
const out = join(root, "thumb.jpg");
|
||||||
expect(await getThumbCommand(context(), "200", { out })).toBe(0);
|
expect(await getThumbCommand(context(), "200", { out })).toBe(0);
|
||||||
expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
|
expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
|
||||||
@@ -1618,18 +1701,19 @@ describe("an account with many albums", () => {
|
|||||||
// encrypted as the server stores it. The original's bytes are the title.
|
// encrypted as the server stores it. The original's bytes are the title.
|
||||||
// These tests fetch no thumbnail. The server records a thumbnail size, so
|
// These tests fetch no thumbnail. The server records a thumbnail size, so
|
||||||
// `helper fix-missing-thumbnails` downloads the original before it skips
|
// `helper fix-missing-thumbnails` downloads the original before it skips
|
||||||
// the photo as not a JPEG.
|
// the photo as not a JPEG. `albumKey` is that album's key, by default the
|
||||||
const photoRow = (photo: {
|
// one in `albumKeys`.
|
||||||
|
const photoRow = (
|
||||||
|
photo: {
|
||||||
id: number;
|
id: number;
|
||||||
collectionID: number;
|
collectionID: number;
|
||||||
title: string;
|
title: string;
|
||||||
updationTime: number;
|
updationTime: number;
|
||||||
}): { row: Record<string, unknown>; original: Uint8Array } => {
|
},
|
||||||
|
albumKey = albumKeys.get(photo.collectionID)!,
|
||||||
|
): { row: Record<string, unknown>; original: Uint8Array } => {
|
||||||
const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||||
const photoKeyBox = encryptSecretbox(
|
const photoKeyBox = encryptSecretbox(photoKey, albumKey);
|
||||||
photoKey,
|
|
||||||
albumKeys.get(photo.collectionID)!,
|
|
||||||
);
|
|
||||||
const metadata = encryptStreamBlob(
|
const metadata = encryptStreamBlob(
|
||||||
new TextEncoder().encode(
|
new TextEncoder().encode(
|
||||||
JSON.stringify({
|
JSON.stringify({
|
||||||
@@ -1742,6 +1826,77 @@ describe("an account with many albums", () => {
|
|||||||
};
|
};
|
||||||
};
|
};
|
||||||
|
|
||||||
|
// The albums in the cache's `metadata.json`. A search on an empty cache
|
||||||
|
// stores none there, only the account whose original or thumbnail it
|
||||||
|
// stored.
|
||||||
|
const cachedAlbums = (): unknown[] =>
|
||||||
|
JSON.parse(readFileSync(join(root, "cache", "metadata.json"), "utf-8"))
|
||||||
|
.collections;
|
||||||
|
|
||||||
|
// A context whose login and saved session send each request to the server
|
||||||
|
// its host names: `ente.example.com` is the fake server above, and
|
||||||
|
// `photos.example.org` a second one. Every account the fake server logs in
|
||||||
|
// has user ID 42, so the second server's account has the first one's user
|
||||||
|
// ID. Its one album holds photo 200, `other-200.jpg`; the first server
|
||||||
|
// gives ID 200 to `photo-2.jpg`.
|
||||||
|
const twoServerContext = async (): Promise<CliContext> => {
|
||||||
|
const account = await buildAccount(PASSWORD);
|
||||||
|
const albumKey = sodium.crypto_secretbox_keygen();
|
||||||
|
const albumKeyBox = encryptSecretbox(albumKey, account.masterKey);
|
||||||
|
const name = encryptSecretbox(
|
||||||
|
new TextEncoder().encode("Other album"),
|
||||||
|
albumKey,
|
||||||
|
);
|
||||||
|
const { row, original } = photoRow(
|
||||||
|
{
|
||||||
|
id: 200,
|
||||||
|
collectionID: 1,
|
||||||
|
title: "other-200.jpg",
|
||||||
|
updationTime: 1,
|
||||||
|
},
|
||||||
|
albumKey,
|
||||||
|
);
|
||||||
|
const other = buildMetaFetch({
|
||||||
|
verifier: account.verifier,
|
||||||
|
srpAttributes: account.srpAttributes,
|
||||||
|
keyAttributes: account.keyAttributes,
|
||||||
|
encryptedToken: account.encryptedToken,
|
||||||
|
collections: [
|
||||||
|
{
|
||||||
|
id: 1,
|
||||||
|
owner: { id: USER_ID },
|
||||||
|
encryptedKey: toBase64(albumKeyBox.ciphertext),
|
||||||
|
keyDecryptionNonce: toBase64(albumKeyBox.nonce),
|
||||||
|
encryptedName: toBase64(name.ciphertext),
|
||||||
|
nameDecryptionNonce: toBase64(name.nonce),
|
||||||
|
type: "album",
|
||||||
|
updationTime: 1,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
filesByCollection: { 1: [row] },
|
||||||
|
encryptedMLData: {},
|
||||||
|
fileCiphertexts: { 200: original },
|
||||||
|
});
|
||||||
|
const fetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||||
|
const { host } = new URL(
|
||||||
|
input instanceof Request ? input.url : input,
|
||||||
|
);
|
||||||
|
return (host === "photos.example.org" ? other : server)(
|
||||||
|
input,
|
||||||
|
init,
|
||||||
|
);
|
||||||
|
}) as typeof globalThis.fetch;
|
||||||
|
return {
|
||||||
|
...context(),
|
||||||
|
login: (opts) =>
|
||||||
|
Client.login({
|
||||||
|
...opts,
|
||||||
|
apiOptions: { ...opts.apiOptions, fetch },
|
||||||
|
}),
|
||||||
|
loadSession: (path) => loadSession(path, { fetch }),
|
||||||
|
};
|
||||||
|
};
|
||||||
|
|
||||||
it("collections lists every album from one request, fetching no album's files", async () => {
|
it("collections lists every album from one request, fetching no album's files", async () => {
|
||||||
const ctx = accountContext();
|
const ctx = accountContext();
|
||||||
expect(await collectionsCommand(ctx, {})).toBe(0);
|
expect(await collectionsCommand(ctx, {})).toBe(0);
|
||||||
@@ -1848,7 +2003,100 @@ describe("an account with many albums", () => {
|
|||||||
"/collections/v2/diff?collectionID=2&sinceTime=0",
|
"/collections/v2/diff?collectionID=2&sinceTime=0",
|
||||||
"/?fileID=200",
|
"/?fileID=200",
|
||||||
]);
|
]);
|
||||||
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
|
expect(cachedAlbums()).toEqual([]);
|
||||||
|
});
|
||||||
|
|
||||||
|
// A self-hosted server serves downloads under its own address, at
|
||||||
|
// `/files/download/<id>`, so no request goes to Ente's hosted service.
|
||||||
|
it("login --server saves the server, and a later get sends every request to it", async () => {
|
||||||
|
vi.stubEnv("QUAK_EMAIL", "many@example.com");
|
||||||
|
vi.stubEnv("QUAK_PASSWORD", PASSWORD);
|
||||||
|
vi.stubEnv("QUAK_SERVER", undefined);
|
||||||
|
const hosts = new Set<string>();
|
||||||
|
const fetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||||
|
hosts.add(
|
||||||
|
new URL(input instanceof Request ? input.url : input).host,
|
||||||
|
);
|
||||||
|
return server(input, init);
|
||||||
|
}) as typeof globalThis.fetch;
|
||||||
|
const ctx: CliContext = {
|
||||||
|
...context(),
|
||||||
|
login: (opts) =>
|
||||||
|
Client.login({
|
||||||
|
...opts,
|
||||||
|
apiOptions: { ...opts.apiOptions, fetch },
|
||||||
|
}),
|
||||||
|
loadSession: (path) => loadSession(path, { fetch }),
|
||||||
|
};
|
||||||
|
const out = join(root, "photo.jpg");
|
||||||
|
try {
|
||||||
|
const opts = { server: "https://ente.example.com" };
|
||||||
|
expect(await loginCommand(ctx, opts)).toBe(0);
|
||||||
|
requests.length = 0;
|
||||||
|
expect(await getCommand(ctx, "200", { out })).toBe(0);
|
||||||
|
} finally {
|
||||||
|
vi.unstubAllEnvs();
|
||||||
|
}
|
||||||
|
|
||||||
|
const path = join(ctx.sessionDir, "session.json");
|
||||||
|
expect(JSON.parse(readFileSync(path, "utf-8")).apiOrigin).toBe(
|
||||||
|
"https://ente.example.com",
|
||||||
|
);
|
||||||
|
expect(readFileSync(out, "utf-8")).toBe("photo-2.jpg");
|
||||||
|
expect(requests.at(-1)).toBe("/files/download/200");
|
||||||
|
expect([...hosts]).toEqual(["ente.example.com"]);
|
||||||
|
});
|
||||||
|
|
||||||
|
// The two servers' accounts share the cache directory `--cache-dir` names.
|
||||||
|
it("after a login to another server whose account has the same user ID, files and get see none of the first server's albums and originals", async () => {
|
||||||
|
const ctx = await twoServerContext();
|
||||||
|
const first = join(root, "first.jpg");
|
||||||
|
const second = join(root, "second.jpg");
|
||||||
|
vi.stubEnv("QUAK_EMAIL", "many@example.com");
|
||||||
|
vi.stubEnv("QUAK_PASSWORD", PASSWORD);
|
||||||
|
vi.stubEnv("QUAK_SERVER", undefined);
|
||||||
|
try {
|
||||||
|
const opts = { server: "https://ente.example.com" };
|
||||||
|
expect(await loginCommand(ctx, opts)).toBe(0);
|
||||||
|
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||||
|
expect(await getCommand(ctx, "200", { out: first })).toBe(0);
|
||||||
|
|
||||||
|
const otherOpts = { server: "https://photos.example.org" };
|
||||||
|
expect(await loginCommand(ctx, otherOpts)).toBe(0);
|
||||||
|
stdout.text = "";
|
||||||
|
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||||
|
expect(await getCommand(ctx, "200", { out: second })).toBe(0);
|
||||||
|
} finally {
|
||||||
|
vi.unstubAllEnvs();
|
||||||
|
}
|
||||||
|
|
||||||
|
expect(stdout.text).toBe("200\timage\tother-200.jpg\n");
|
||||||
|
expect(readFileSync(first, "utf-8")).toBe("photo-2.jpg");
|
||||||
|
expect(readFileSync(second, "utf-8")).toBe("other-200.jpg");
|
||||||
|
});
|
||||||
|
|
||||||
|
// `get` on an empty cache stores the original but no album.
|
||||||
|
it("get on an empty cache, after the same on another server whose account has the same user ID, saves this server's photo", async () => {
|
||||||
|
const ctx = await twoServerContext();
|
||||||
|
const first = join(root, "first.jpg");
|
||||||
|
const second = join(root, "second.jpg");
|
||||||
|
vi.stubEnv("QUAK_EMAIL", "many@example.com");
|
||||||
|
vi.stubEnv("QUAK_PASSWORD", PASSWORD);
|
||||||
|
vi.stubEnv("QUAK_SERVER", undefined);
|
||||||
|
try {
|
||||||
|
const opts = { server: "https://ente.example.com" };
|
||||||
|
expect(await loginCommand(ctx, opts)).toBe(0);
|
||||||
|
expect(await getCommand(ctx, "200", { out: first })).toBe(0);
|
||||||
|
|
||||||
|
const otherOpts = { server: "https://photos.example.org" };
|
||||||
|
expect(await loginCommand(ctx, otherOpts)).toBe(0);
|
||||||
|
expect(await getCommand(ctx, "200", { out: second })).toBe(0);
|
||||||
|
} finally {
|
||||||
|
vi.unstubAllEnvs();
|
||||||
|
}
|
||||||
|
|
||||||
|
expect(readFileSync(first, "utf-8")).toBe("photo-2.jpg");
|
||||||
|
expect(readFileSync(second, "utf-8")).toBe("other-200.jpg");
|
||||||
});
|
});
|
||||||
|
|
||||||
// `files` first fills the cache, so `get` waits for a refresh. No album has
|
// `files` first fills the cache, so `get` waits for a refresh. No album has
|
||||||
@@ -1906,7 +2154,7 @@ describe("an account with many albums", () => {
|
|||||||
"/collections/v2/diff?collectionID=39&sinceTime=0",
|
"/collections/v2/diff?collectionID=39&sinceTime=0",
|
||||||
"/?fileID=3900",
|
"/?fileID=3900",
|
||||||
]);
|
]);
|
||||||
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
|
expect(cachedAlbums()).toEqual([]);
|
||||||
});
|
});
|
||||||
|
|
||||||
// `files` first fills the cache, so the helper waits for a refresh. No
|
// `files` first fills the cache, so the helper waits for a refresh. No
|
||||||
|
|||||||
@@ -47,6 +47,35 @@ describe("Client.toJSON", () => {
|
|||||||
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
|
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("round-trips a snapshot that names a server", () => {
|
||||||
|
const snapshot = {
|
||||||
|
...validSnapshot(),
|
||||||
|
apiOrigin: "https://ente.example.com",
|
||||||
|
};
|
||||||
|
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("names the server apiOptions gave fromJSON", async () => {
|
||||||
|
const apiOrigin = "https://ente.example.com";
|
||||||
|
const urls: string[] = [];
|
||||||
|
const client = Client.fromJSON(validSnapshot(), {
|
||||||
|
apiOrigin,
|
||||||
|
fetch: async (input) => {
|
||||||
|
urls.push(String(input));
|
||||||
|
return new Response(JSON.stringify({ collections: [] }), {
|
||||||
|
status: 200,
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
});
|
||||||
|
},
|
||||||
|
});
|
||||||
|
expect(client.toJSON().apiOrigin).toBe(apiOrigin);
|
||||||
|
|
||||||
|
await client.listCollections();
|
||||||
|
expect(urls).toEqual([
|
||||||
|
"https://ente.example.com/collections/v2?sinceTime=0",
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
it("throws instead of emitting a snapshot without a token", () => {
|
it("throws instead of emitting a snapshot without a token", () => {
|
||||||
const client = Client.fromJSON(validSnapshot());
|
const client = Client.fromJSON(validSnapshot());
|
||||||
client.getApiClient().clearAuthToken();
|
client.getApiClient().clearAuthToken();
|
||||||
@@ -77,6 +106,10 @@ describe("Client.fromJSON", () => {
|
|||||||
["publicKey", undefined],
|
["publicKey", undefined],
|
||||||
["publicKey", "not base64!"],
|
["publicKey", "not base64!"],
|
||||||
["publicKey", shortKey],
|
["publicKey", shortKey],
|
||||||
|
["apiOrigin", 7],
|
||||||
|
["apiOrigin", ""],
|
||||||
|
["apiOrigin", "ente.example.com"],
|
||||||
|
["apiOrigin", "ftp://ente.example.com"],
|
||||||
])("rejects %s = %j, naming the field", (field, value) => {
|
])("rejects %s = %j, naming the field", (field, value) => {
|
||||||
const snapshot: Record<string, unknown> = { ...validSnapshot() };
|
const snapshot: Record<string, unknown> = { ...validSnapshot() };
|
||||||
snapshot[field] = value;
|
snapshot[field] = value;
|
||||||
@@ -85,6 +118,30 @@ describe("Client.fromJSON", () => {
|
|||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// The token belongs to the snapshot's server, so that server wins over
|
||||||
|
// the one in `apiOptions`.
|
||||||
|
it("sends requests to the snapshot's server", async () => {
|
||||||
|
const urls: string[] = [];
|
||||||
|
const client = Client.fromJSON(
|
||||||
|
{ ...validSnapshot(), apiOrigin: "https://ente.example.com/" },
|
||||||
|
{
|
||||||
|
apiOrigin: "https://other.example.com",
|
||||||
|
fetch: async (input) => {
|
||||||
|
urls.push(String(input));
|
||||||
|
return new Response(JSON.stringify({ collections: [] }), {
|
||||||
|
status: 200,
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
});
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
await client.listCollections();
|
||||||
|
expect(urls).toEqual([
|
||||||
|
"https://ente.example.com/collections/v2?sinceTime=0",
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
it.each([null, "a string", 42])("rejects a non-object %j", (value) => {
|
it.each([null, "a string", 42])("rejects a non-object %j", (value) => {
|
||||||
expect(() => Client.fromJSON(value)).toThrow(
|
expect(() => Client.fromJSON(value)).toThrow(
|
||||||
/^Invalid session data: not a JSON object/,
|
/^Invalid session data: not a JSON object/,
|
||||||
|
|||||||
@@ -19,16 +19,22 @@
|
|||||||
* refresh fails (offline start from cache).
|
* refresh fails (offline start from cache).
|
||||||
* 5. `close()` stops the timer and is idempotent, and its promise resolves
|
* 5. `close()` stops the timer and is idempotent, and its promise resolves
|
||||||
* only once an in-flight refresh has written the cache file.
|
* only once an in-flight refresh has written the cache file.
|
||||||
* 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id.
|
* 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id,
|
||||||
* 7. `open()` branches on the cache: an empty cache awaits the first refresh
|
* with a self-hosted server's address before the user id.
|
||||||
* (it has nothing to serve yet); an existing cache serves its copy at once
|
* 7. `open()` branches on the cache: a cache that holds no collection awaits
|
||||||
* and refreshes in the background, so a slow or dead server never stalls
|
* the first refresh (it has nothing to serve yet); an existing cache serves
|
||||||
* opening.
|
* its copy at once and refreshes in the background, so a slow or dead
|
||||||
|
* server never stalls opening.
|
||||||
* 8. A save failure that leaves RAM ahead of disk keeps `status().lastError`
|
* 8. A save failure that leaves RAM ahead of disk keeps `status().lastError`
|
||||||
* set and keeps retrying the write; a later empty refresh does not clear it.
|
* set and keeps retrying the write; a later empty refresh does not clear it.
|
||||||
* 9. With `refreshOnOpen: false`, `open()` starts no refresh at all, and a
|
* 9. With `refreshOnOpen: false`, `open()` starts no refresh at all, and a
|
||||||
* refresh reports through `onProgress` each collection whose files it
|
* refresh reports through `onProgress` each collection whose files it
|
||||||
* fetched.
|
* fetched.
|
||||||
|
* 10. A cache another account filled starts empty. One filled from another
|
||||||
|
* server also loses its originals and thumbnails, since two servers can
|
||||||
|
* give two files the same ID. `thumbnailOf` on an empty cache saves the
|
||||||
|
* account's user ID and server. A refresh saves the account's server, so a
|
||||||
|
* self-hosted account keeps its own cache from one `open()` to the next.
|
||||||
*
|
*
|
||||||
* The client is a mock: no crypto, no network. It serves scripted pages and
|
* The client is a mock: no crypto, no network. It serves scripted pages and
|
||||||
* records the `sinceTime` each call carried so cursor threading is provable.
|
* records the `sinceTime` each call carried so cursor threading is provable.
|
||||||
@@ -49,7 +55,13 @@
|
|||||||
*/
|
*/
|
||||||
|
|
||||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||||
import { mkdtempSync, rmSync } from "node:fs";
|
import {
|
||||||
|
existsSync,
|
||||||
|
mkdirSync,
|
||||||
|
mkdtempSync,
|
||||||
|
rmSync,
|
||||||
|
writeFileSync,
|
||||||
|
} from "node:fs";
|
||||||
import { tmpdir } from "node:os";
|
import { tmpdir } from "node:os";
|
||||||
import { join } from "node:path";
|
import { join } from "node:path";
|
||||||
import envPaths from "env-paths";
|
import envPaths from "env-paths";
|
||||||
@@ -107,6 +119,7 @@ const file = (
|
|||||||
*/
|
*/
|
||||||
class MockClient {
|
class MockClient {
|
||||||
userID = USER_ID;
|
userID = USER_ID;
|
||||||
|
apiOrigin?: string;
|
||||||
failCollections = false;
|
failCollections = false;
|
||||||
collectionsQueue: CollectionsPage[] = [];
|
collectionsQueue: CollectionsPage[] = [];
|
||||||
filesByCollection = new Map<number, FilesPage[]>();
|
filesByCollection = new Map<number, FilesPage[]>();
|
||||||
@@ -584,6 +597,44 @@ describe("Library.open and background refresh", () => {
|
|||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// `originalOf` on an empty cache leaves such a cache behind: it names the
|
||||||
|
// account before it stores the original.
|
||||||
|
it("awaits the first refresh on a cache that names the account but holds no collection", async () => {
|
||||||
|
const seed = await MetadataStore.load(
|
||||||
|
join(cacheDirectory, "metadata.json"),
|
||||||
|
);
|
||||||
|
seed.userID = USER_ID;
|
||||||
|
await seed.save();
|
||||||
|
let releaseFirstFetch: (page: CollectionsPage) => void = () => {};
|
||||||
|
const gate = new Promise<CollectionsPage>((resolve) => {
|
||||||
|
releaseFirstFetch = resolve;
|
||||||
|
});
|
||||||
|
const client = new MockClient();
|
||||||
|
client.collectionsSince = async () => gate;
|
||||||
|
|
||||||
|
let opened = false;
|
||||||
|
const openPromise = Library.open({ client, cacheDirectory }).then(
|
||||||
|
(l) => {
|
||||||
|
opened = true;
|
||||||
|
return l;
|
||||||
|
},
|
||||||
|
);
|
||||||
|
await new Promise((r) => setTimeout(r, 20));
|
||||||
|
expect(opened).toBe(false);
|
||||||
|
|
||||||
|
releaseFirstFetch({
|
||||||
|
collections: [collection(1, 100)],
|
||||||
|
deleted: [],
|
||||||
|
cursor: 100,
|
||||||
|
});
|
||||||
|
const lib = await openPromise;
|
||||||
|
try {
|
||||||
|
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
|
||||||
|
} finally {
|
||||||
|
await lib.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
// A caller that asks for each refresh it needs, as the CLI's read commands
|
// A caller that asks for each refresh it needs, as the CLI's read commands
|
||||||
// do, opens without one: on an empty cache the refresh `open()` would
|
// do, opens without one: on an empty cache the refresh `open()` would
|
||||||
// await fetches the files of every collection, which `quak collections`
|
// await fetches the files of every collection, which `quak collections`
|
||||||
@@ -842,4 +893,165 @@ describe("Library.open and background refresh", () => {
|
|||||||
else process.env.XDG_CACHE_HOME = prev;
|
else process.env.XDG_CACHE_HOME = prev;
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("puts a self-hosted server's address, URL-encoded, before the user id in the default cacheDirectory", async () => {
|
||||||
|
const xdg = join(dir, "xdg-cache");
|
||||||
|
const prev = process.env.XDG_CACHE_HOME;
|
||||||
|
process.env.XDG_CACHE_HOME = xdg;
|
||||||
|
try {
|
||||||
|
const client = new MockClient();
|
||||||
|
client.apiOrigin = "https://ente.example.com:8443/api";
|
||||||
|
const lib = await Library.open({ client });
|
||||||
|
try {
|
||||||
|
expect(lib.cacheDirectory).toBe(
|
||||||
|
join(
|
||||||
|
envPaths("quak", { suffix: "" }).cache,
|
||||||
|
"https%3A%2F%2Fente.example.com%3A8443%2Fapi",
|
||||||
|
String(USER_ID),
|
||||||
|
),
|
||||||
|
);
|
||||||
|
expect(lib.cacheDirectory.startsWith(xdg)).toBe(true);
|
||||||
|
} finally {
|
||||||
|
await lib.close();
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
if (prev === undefined) delete process.env.XDG_CACHE_HOME;
|
||||||
|
else process.env.XDG_CACHE_HOME = prev;
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// A cache that account USER_ID on `apiOrigin` filled: its records, and an
|
||||||
|
// original and a thumbnail of file 1001.
|
||||||
|
const fillCache = async (apiOrigin?: string): Promise<void> => {
|
||||||
|
const seed = await MetadataStore.load(
|
||||||
|
join(cacheDirectory, "metadata.json"),
|
||||||
|
);
|
||||||
|
seed.userID = USER_ID;
|
||||||
|
seed.apiOrigin = apiOrigin;
|
||||||
|
seed.collectionsSinceTime = 500;
|
||||||
|
seed.putCollection(collection(1, 400));
|
||||||
|
seed.putFile(file(1001, 1, 400));
|
||||||
|
await seed.save();
|
||||||
|
for (const name of ["originals", "thumbnails"]) {
|
||||||
|
mkdirSync(join(cacheDirectory, name));
|
||||||
|
writeFileSync(join(cacheDirectory, name, "1001.jpg"), "1001");
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
it("deletes the whole cache an account on another server filled", async () => {
|
||||||
|
await fillCache("https://ente.example.com");
|
||||||
|
const client = new MockClient();
|
||||||
|
client.apiOrigin = "https://photos.example.org";
|
||||||
|
const lib = await Library.open({
|
||||||
|
client,
|
||||||
|
cacheDirectory,
|
||||||
|
refreshIntervalSeconds: 3600,
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
expect(client.collectionsSinceTimes).toEqual([0]);
|
||||||
|
expect(lib.listCollections()).toEqual([]);
|
||||||
|
for (const name of ["originals", "thumbnails"]) {
|
||||||
|
expect(existsSync(join(cacheDirectory, name, "1001.jpg"))).toBe(
|
||||||
|
false,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
await lib.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// A server gives each file its own ID, so another account's cached file
|
||||||
|
// is never served for this account's.
|
||||||
|
it("keeps the originals and thumbnails another account on the same server cached", async () => {
|
||||||
|
await fillCache();
|
||||||
|
const client = new MockClient();
|
||||||
|
client.userID = USER_ID + 1;
|
||||||
|
const lib = await Library.open({
|
||||||
|
client,
|
||||||
|
cacheDirectory,
|
||||||
|
refreshIntervalSeconds: 3600,
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
expect(client.collectionsSinceTimes).toEqual([0]);
|
||||||
|
expect(lib.listCollections()).toEqual([]);
|
||||||
|
for (const name of ["originals", "thumbnails"]) {
|
||||||
|
expect(existsSync(join(cacheDirectory, name, "1001.jpg"))).toBe(
|
||||||
|
true,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
await lib.close();
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// `quak get-thumb` on an empty cache stores a thumbnail before any refresh
|
||||||
|
// runs. `open()` for an account on another server deletes that thumbnail
|
||||||
|
// only if the cache names the account that stored it.
|
||||||
|
it("thumbnailOf on an empty cache saves the account's user ID and server in metadata.json", async () => {
|
||||||
|
const client = new MockClient();
|
||||||
|
client.apiOrigin = "https://ente.example.com";
|
||||||
|
const lib = await Library.open({
|
||||||
|
client,
|
||||||
|
cacheDirectory,
|
||||||
|
refreshOnOpen: false,
|
||||||
|
refreshIntervalSeconds: Infinity,
|
||||||
|
contentSource: {
|
||||||
|
original: async () => {
|
||||||
|
throw new Error("original not expected");
|
||||||
|
},
|
||||||
|
thumbnail: async ({ destination }) => {
|
||||||
|
writeFileSync(destination, "thumb");
|
||||||
|
return { bytesWritten: 5 };
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
await lib.thumbnailOf(file(1001, 1, 400));
|
||||||
|
} finally {
|
||||||
|
await lib.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
const saved = await MetadataStore.load(
|
||||||
|
join(cacheDirectory, "metadata.json"),
|
||||||
|
);
|
||||||
|
expect(saved.userID).toBe(USER_ID);
|
||||||
|
expect(saved.apiOrigin).toBe("https://ente.example.com");
|
||||||
|
});
|
||||||
|
|
||||||
|
// A refresh saves the account's server in `metadata.json`. Without it,
|
||||||
|
// `open()` would take a self-hosted account's own cache for another
|
||||||
|
// server's and delete it.
|
||||||
|
it("keeps a self-hosted account's own cache from one open to the next", async () => {
|
||||||
|
const client = new MockClient();
|
||||||
|
client.apiOrigin = "https://ente.example.com";
|
||||||
|
client.collectionsQueue.push({
|
||||||
|
collections: [collection(1, 100)],
|
||||||
|
deleted: [],
|
||||||
|
cursor: 100,
|
||||||
|
});
|
||||||
|
const first = await Library.open({
|
||||||
|
client,
|
||||||
|
cacheDirectory,
|
||||||
|
refreshIntervalSeconds: Infinity,
|
||||||
|
});
|
||||||
|
await first.close();
|
||||||
|
mkdirSync(join(cacheDirectory, "originals"));
|
||||||
|
writeFileSync(join(cacheDirectory, "originals", "1001.jpg"), "1001");
|
||||||
|
|
||||||
|
const lib = await Library.open({
|
||||||
|
client,
|
||||||
|
cacheDirectory,
|
||||||
|
refreshIntervalSeconds: Infinity,
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
|
||||||
|
expect(
|
||||||
|
existsSync(join(cacheDirectory, "originals", "1001.jpg")),
|
||||||
|
).toBe(true);
|
||||||
|
} finally {
|
||||||
|
await lib.close();
|
||||||
|
}
|
||||||
|
// The second open's refresh started from the saved cursor.
|
||||||
|
expect(client.collectionsSinceTimes).toEqual([0, 100]);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
@@ -0,0 +1,241 @@
|
|||||||
|
// `script/build-bin` compiles `bin/quak`, the single binary that the README's
|
||||||
|
// cron setup runs. bun compiles it from a copy of its own executable, so the
|
||||||
|
// binary loads the same libraries as the bun that built it. nix's bun loads
|
||||||
|
// libraries from the nix store, which garbage collection can delete: on Linux
|
||||||
|
// its C library, on macOS its ICU. The script therefore downloads bun's own
|
||||||
|
// release build, which loads only the system's libraries.
|
||||||
|
//
|
||||||
|
// What the script promises, and why each test matters:
|
||||||
|
//
|
||||||
|
// - Each system, machine and C library that bun publishes a build for gets its
|
||||||
|
// own archive: Linux on x86-64 or arm64, with glibc or with musl, as on
|
||||||
|
// Alpine, and macOS on Intel or Apple silicon. A bun built for another
|
||||||
|
// system, machine or C library does not start, and `make install` fails.
|
||||||
|
// - The archive's SHA-256 must match the one in the script before anything in
|
||||||
|
// the archive runs. Without that check, a tampered download would build the
|
||||||
|
// binary that cron runs.
|
||||||
|
// - Any other system or machine stops with an error, and never falls back to
|
||||||
|
// nix's bun. Building with nix's bun would bring back the dependency on the
|
||||||
|
// nix store.
|
||||||
|
//
|
||||||
|
// Each test copies the script into a fresh directory, which the script then
|
||||||
|
// treats as the checkout, and runs it there. Stubs first on PATH stand in for
|
||||||
|
// `uname`, `ldd`, `curl`, `unzip`, `nix-shell` and `shasum`, so no test
|
||||||
|
// downloads anything or runs a real bun.
|
||||||
|
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||||
|
import { execFileSync, spawnSync } from "node:child_process";
|
||||||
|
import {
|
||||||
|
chmodSync,
|
||||||
|
copyFileSync,
|
||||||
|
existsSync,
|
||||||
|
mkdirSync,
|
||||||
|
mkdtempSync,
|
||||||
|
readFileSync,
|
||||||
|
rmSync,
|
||||||
|
symlinkSync,
|
||||||
|
writeFileSync,
|
||||||
|
} from "node:fs";
|
||||||
|
import { tmpdir } from "node:os";
|
||||||
|
import { join } from "node:path";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
|
||||||
|
const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
|
||||||
|
|
||||||
|
// The stubs, keyed by the command each replaces. `uname` and `ldd` describe the
|
||||||
|
// system that runBuildBin passes in the environment. `curl`, `unzip`,
|
||||||
|
// `nix-shell` and `shasum` write the arguments they were given, one per line,
|
||||||
|
// to a file named after the command in $STUB_LOG_DIR, which also shows that
|
||||||
|
// they ran.
|
||||||
|
const stubs: Record<string, string> = {
|
||||||
|
uname: `
|
||||||
|
case "$1" in
|
||||||
|
-s) echo "$STUB_UNAME_S" ;;
|
||||||
|
-m) echo "$STUB_UNAME_M" ;;
|
||||||
|
esac`,
|
||||||
|
// The script looks only at the start of the line: "musl libc" or not.
|
||||||
|
ldd: `
|
||||||
|
if [ "$STUB_LIBC" = musl ]; then
|
||||||
|
echo "musl libc (x86_64)"
|
||||||
|
else
|
||||||
|
echo "ldd (GNU libc) 2.36"
|
||||||
|
fi`,
|
||||||
|
// Writes a file that is not the archive to the path after -o.
|
||||||
|
curl: `
|
||||||
|
printf '%s\\n' "$@" > "$STUB_LOG_DIR/curl"
|
||||||
|
while [ $# -gt 1 ]; do
|
||||||
|
if [ "$1" = -o ]; then echo "not the archive" > "$2"; fi
|
||||||
|
shift
|
||||||
|
done`,
|
||||||
|
// Unpacks a stub bun where the real archive puts bun: in a directory named
|
||||||
|
// after the archive, inside the directory after -d. That bun writes
|
||||||
|
// bin/quak, as a real compile would.
|
||||||
|
unzip: `
|
||||||
|
printf '%s\\n' "$@" > "$STUB_LOG_DIR/unzip"
|
||||||
|
for arg in "$@"; do
|
||||||
|
case "$arg" in *.zip) archive="$arg" ;; esac
|
||||||
|
if [ "$previous" = -d ]; then directory="$arg"; fi
|
||||||
|
previous="$arg"
|
||||||
|
done
|
||||||
|
bun="$directory/$(basename "$archive" .zip)/bun"
|
||||||
|
mkdir -p "$(dirname "$bun")"
|
||||||
|
echo '#!/bin/sh' > "$bun"
|
||||||
|
echo 'mkdir -p bin && echo binary > bin/quak' >> "$bun"
|
||||||
|
chmod +x "$bun"`,
|
||||||
|
"nix-shell": `
|
||||||
|
printf '%s\\n' "$@" > "$STUB_LOG_DIR/nix-shell"`,
|
||||||
|
// The script checks the hash with `shasum` where there is no `sha256sum`.
|
||||||
|
// The stub fails, as `shasum` does when the hash does not match.
|
||||||
|
shasum: `
|
||||||
|
printf '%s\\n' "$@" > "$STUB_LOG_DIR/shasum"
|
||||||
|
exit 1`,
|
||||||
|
};
|
||||||
|
|
||||||
|
let checkout = "";
|
||||||
|
|
||||||
|
const writeScript = (path: string, body: string): void => {
|
||||||
|
writeFileSync(path, `#!/bin/sh\n${body}\n`);
|
||||||
|
chmodSync(path, 0o755);
|
||||||
|
};
|
||||||
|
|
||||||
|
// A checkout holding the script, a stub script/build that does nothing, the
|
||||||
|
// stubs, and the directory they log to.
|
||||||
|
beforeEach(() => {
|
||||||
|
checkout = mkdtempSync(join(tmpdir(), "quak-build-bin-"));
|
||||||
|
mkdirSync(join(checkout, "script"));
|
||||||
|
copyFileSync(
|
||||||
|
join(repoRoot, "script/build-bin"),
|
||||||
|
join(checkout, "script/build-bin"),
|
||||||
|
);
|
||||||
|
chmodSync(join(checkout, "script/build-bin"), 0o755);
|
||||||
|
writeScript(join(checkout, "script/build"), "exit 0");
|
||||||
|
mkdirSync(join(checkout, "stubs"));
|
||||||
|
for (const [command, body] of Object.entries(stubs)) {
|
||||||
|
writeScript(join(checkout, "stubs", command), body);
|
||||||
|
}
|
||||||
|
mkdirSync(join(checkout, "log"));
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
rmSync(checkout, { recursive: true, force: true });
|
||||||
|
});
|
||||||
|
|
||||||
|
// Runs the script with nothing in its environment but the stubs' settings and a
|
||||||
|
// PATH. Unless the caller passes one, the PATH finds the stubs first and then
|
||||||
|
// the host's commands.
|
||||||
|
const runBuildBin = (
|
||||||
|
system: string,
|
||||||
|
machine: string,
|
||||||
|
{
|
||||||
|
libc = "glibc",
|
||||||
|
path = `${join(checkout, "stubs")}:${process.env.PATH}`,
|
||||||
|
} = {},
|
||||||
|
) =>
|
||||||
|
spawnSync(join(checkout, "script/build-bin"), {
|
||||||
|
cwd: checkout,
|
||||||
|
encoding: "utf-8",
|
||||||
|
env: {
|
||||||
|
PATH: path,
|
||||||
|
STUB_UNAME_S: system,
|
||||||
|
STUB_UNAME_M: machine,
|
||||||
|
STUB_LIBC: libc,
|
||||||
|
STUB_LOG_DIR: join(checkout, "log"),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
// The real commands that the script and the stubs run.
|
||||||
|
const hostCommands = [
|
||||||
|
"basename",
|
||||||
|
"chmod",
|
||||||
|
"dirname",
|
||||||
|
"grep",
|
||||||
|
"mkdir",
|
||||||
|
"mktemp",
|
||||||
|
"rm",
|
||||||
|
];
|
||||||
|
|
||||||
|
// A Mac has no `ldd`, and not every Mac has `sha256sum`. This makes a PATH that
|
||||||
|
// finds neither: the stubs other than `ldd`, then links to hostCommands.
|
||||||
|
const makeMacPath = (): string => {
|
||||||
|
rmSync(join(checkout, "stubs", "ldd"));
|
||||||
|
mkdirSync(join(checkout, "host-commands"));
|
||||||
|
for (const command of hostCommands) {
|
||||||
|
const target = execFileSync("sh", ["-c", `command -v ${command}`], {
|
||||||
|
encoding: "utf-8",
|
||||||
|
}).trim();
|
||||||
|
symlinkSync(target, join(checkout, "host-commands", command));
|
||||||
|
}
|
||||||
|
return `${join(checkout, "stubs")}:${join(checkout, "host-commands")}`;
|
||||||
|
};
|
||||||
|
|
||||||
|
const ran = (command: string): boolean =>
|
||||||
|
existsSync(join(checkout, "log", command));
|
||||||
|
|
||||||
|
const argumentsOf = (command: string): string[] =>
|
||||||
|
readFileSync(join(checkout, "log", command), "utf-8")
|
||||||
|
.trimEnd()
|
||||||
|
.split("\n");
|
||||||
|
|
||||||
|
describe("script/build-bin", () => {
|
||||||
|
// The stub curl writes a file that is not the archive, so the hash check
|
||||||
|
// must stop the script before unzip runs. A script without the check would
|
||||||
|
// run unzip, and the stub bun it unpacks would write bin/quak. On Linux the
|
||||||
|
// check must use `sha256sum`, since a stock Alpine has no `shasum`.
|
||||||
|
it.each([
|
||||||
|
["Linux", "x86_64", "glibc", "bun-linux-x64-baseline.zip"],
|
||||||
|
["Linux", "x86_64", "musl", "bun-linux-x64-musl-baseline.zip"],
|
||||||
|
["Linux", "aarch64", "glibc", "bun-linux-aarch64.zip"],
|
||||||
|
["Linux", "aarch64", "musl", "bun-linux-aarch64-musl.zip"],
|
||||||
|
])(
|
||||||
|
"on %s %s with %s, downloads %s and stops when its hash does not match",
|
||||||
|
(system, machine, libc, archive) => {
|
||||||
|
const result = runBuildBin(system, machine, { libc });
|
||||||
|
expect(argumentsOf("curl")).toContain(
|
||||||
|
"https://github.com/oven-sh/bun/releases/download/bun-v1.3.13/" +
|
||||||
|
archive,
|
||||||
|
);
|
||||||
|
expect(ran("shasum")).toBe(false);
|
||||||
|
expect(result.status).not.toBe(0);
|
||||||
|
expect(ran("unzip")).toBe(false);
|
||||||
|
expect(ran("nix-shell")).toBe(false);
|
||||||
|
expect(existsSync(join(checkout, "bin/quak"))).toBe(false);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
// On the PATH from makeMacPath the script finds no `sha256sum` and checks
|
||||||
|
// the hash with the stub `shasum`. That stub fails, so the script must stop
|
||||||
|
// before unzip runs.
|
||||||
|
// On Apple silicon `uname -m` says arm64, where bun's archive says aarch64.
|
||||||
|
it.each([
|
||||||
|
["Darwin", "x86_64", "bun-darwin-x64-baseline.zip"],
|
||||||
|
["Darwin", "arm64", "bun-darwin-aarch64.zip"],
|
||||||
|
])(
|
||||||
|
"on %s %s, downloads %s and stops when shasum rejects its hash",
|
||||||
|
(system, machine, archive) => {
|
||||||
|
const result = runBuildBin(system, machine, {
|
||||||
|
path: makeMacPath(),
|
||||||
|
});
|
||||||
|
expect(argumentsOf("curl")).toContain(
|
||||||
|
"https://github.com/oven-sh/bun/releases/download/bun-v1.3.13/" +
|
||||||
|
archive,
|
||||||
|
);
|
||||||
|
expect(argumentsOf("shasum")).toEqual(["-a", "256", "-c", "-"]);
|
||||||
|
expect(result.status).not.toBe(0);
|
||||||
|
expect(ran("unzip")).toBe(false);
|
||||||
|
expect(ran("nix-shell")).toBe(false);
|
||||||
|
expect(existsSync(join(checkout, "bin/quak"))).toBe(false);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
|
||||||
|
it.each([
|
||||||
|
["Linux", "riscv64"],
|
||||||
|
["FreeBSD", "amd64"],
|
||||||
|
])(
|
||||||
|
"stops on %s %s, which bun publishes no build for",
|
||||||
|
(system, machine) => {
|
||||||
|
const result = runBuildBin(system, machine);
|
||||||
|
expect(result.status).not.toBe(0);
|
||||||
|
expect(ran("curl")).toBe(false);
|
||||||
|
expect(ran("nix-shell")).toBe(false);
|
||||||
|
},
|
||||||
|
);
|
||||||
|
});
|
||||||
@@ -273,6 +273,7 @@ const buildThumbMock = async (opts?: {
|
|||||||
|
|
||||||
const buildThumbFetch = (m: ThumbMockState) => {
|
const buildThumbFetch = (m: ThumbMockState) => {
|
||||||
let srpServer: SrpServer;
|
let srpServer: SrpServer;
|
||||||
|
let signedMD5: string | undefined;
|
||||||
|
|
||||||
return (async (
|
return (async (
|
||||||
input: RequestInfo | URL,
|
input: RequestInfo | URL,
|
||||||
@@ -365,8 +366,9 @@ const buildThumbFetch = (m: ThumbMockState) => {
|
|||||||
return new Response("not found", { status: 404 });
|
return new Response("not found", { status: 404 });
|
||||||
}
|
}
|
||||||
|
|
||||||
// Upload URL minting
|
// Upload URL minting. The real server signs the MD5 into the URL.
|
||||||
if (path === "/files/upload-url") {
|
if (path === "/files/upload-url") {
|
||||||
|
signedMD5 = JSON.parse(init?.body as string).contentMD5;
|
||||||
return json({
|
return json({
|
||||||
objectKey: `42/thumb-${Date.now()}`,
|
objectKey: `42/thumb-${Date.now()}`,
|
||||||
url: "https://s3.mock.test/presigned-put",
|
url: "https://s3.mock.test/presigned-put",
|
||||||
@@ -375,6 +377,11 @@ const buildThumbFetch = (m: ThumbMockState) => {
|
|||||||
|
|
||||||
// Presigned PUT (S3 upload)
|
// Presigned PUT (S3 upload)
|
||||||
if (url.startsWith("https://s3.mock.test/")) {
|
if (url.startsWith("https://s3.mock.test/")) {
|
||||||
|
// Storage refuses a PUT that lacks the signed `Content-MD5`.
|
||||||
|
const headers = new Headers(init?.headers);
|
||||||
|
if (headers.get("Content-MD5") !== signedMD5) {
|
||||||
|
return new Response("AccessDenied", { status: 400 });
|
||||||
|
}
|
||||||
const body = init?.body;
|
const body = init?.body;
|
||||||
// Store the uploaded bytes for later inspection
|
// Store the uploaded bytes for later inspection
|
||||||
if (body instanceof Uint8Array) {
|
if (body instanceof Uint8Array) {
|
||||||
|
|||||||
Reference in new issue
Block a user