Author SHA1 Message Date
sneak 6f8f85fff5 quak login --server: log in to a self-hosted Ente server (closes #243)
check / check (push) Successful in 3m29s
`quak login` takes the server's API address from `--server` or
`QUAK_SERVER`. `isServerAddress` refuses anything but an http or https
URL with no query or fragment, there and in `Client.fromJSON`. The
client's snapshot keeps a non-default `apiOrigin`, so the session file
names the server and every later command uses it.

Two servers can give two accounts, and two files, the same ID. A
self-hosted account's default cache directory therefore names the
server, and `metadata.json` records it: `Library.open` deletes a cache
from another server whole. `originalOf` and `thumbnailOf` record the
account in a cache that names none before storing a file, and `open`
treats a cache with no collection as empty. This fixes
#254 (closes #254).

Model: opus-5-5
2026-11-13 21:19:24 +00:00
clawbot 9bd7e8f099 make build-bin: on macOS, build with bun's pinned release build (closes #251)
check / check (push) Successful in 2m57s
nixpkgs points its macOS bun at an ICU in the nix store, and bun build
--compile copied that reference into bin/quak, so nix-collect-garbage could
break the cron backup. script/build-bin now downloads bun 1.3.13's release
archive on macOS too, Intel (baseline) or Apple silicon, checks its SHA-256,
and compiles with it. Those builds load only /usr/lib libraries. nix-shell is
no longer used; any system without a bun release build stops with an error.
The hash check falls back to shasum where sha256sum is missing.

Model: opus-5-5
2026-11-12 02:17:25 +01:00
clawbot 50e2381cad make build-bin: on Linux, build with bun's pinned release build (closes #240)
check / check (push) Successful in 3m12s
nix's bun loads through a glibc in the nix store, and bun build --compile
copied that loader into bin/quak, so nix-collect-garbage could break the
cron backup. The build moves into script/build-bin. On Linux it downloads
bun 1.3.13's release archive for the machine from GitHub: x86-64
(baseline) or arm64, glibc or musl. It checks the archive's SHA-256
before running it, and compiles with that bun, so the binary loads
through the system loader. macOS still runs bun through nix-shell.

Model: opus-5-5
2026-11-12 00:28:48 +01:00
clawbot 37cdbc6b9a Node 22.13 minimum, and the library linked to its clone (closes #239)
check / check (push) Successful in 3m6s
`engines.node` and the README now state the minimum that quak's
`yarn.lock` needs: Node 22.13, or 23.5 on Node 23. `@inquirer/prompts`
and its packages refuse 23.0 to 23.4.

"Using the library" now links the clone with `yarn link` instead of
`yarn add file:../quak`. A linked quak runs on the dependencies the clone
installed from `yarn.lock`; `file:` resolved them afresh, so a new
upstream release could raise the Node minimum without any change here.

Model: opus-5-5
2026-11-11 23:29:01 +01:00
clawbot 1bf18baf00 Thumbnail upload PUT sends the signed Content-MD5 (closes #238)
check / check (push) Successful in 5m34s
`/files/upload-url` signs the content MD5 into the presigned URL, so
storage refused the PUT that `putFile` sent without it. `putFile` now
takes the MD5 given to `getUploadURL` and sends it. The thumbnail test's
fake storage refuses a PUT without the signed header.

Model: opus-5-5
2026-11-11 22:59:10 +01:00
clawbot 7878ff3812 README: what a backup does with a file deleted in Ente (closes #242)
check / check (push) Successful in 3m24s
A file deleted in Ente keeps its original at its save path, and the JSON
beside it. The next run removes its symlinks from the collection
directories.

Model: opus-5-5
2026-11-11 22:29:14 +01:00
clawbot 8e120a4f49 Remove the no-op --collection option from get and get-thumb (closes #241)
check / check (push) Successful in 3m6s
Both commands listed `--collection <id>` as accepted and ignored. It now
is refused as an unknown option, and the README no longer mentions it.

Model: opus-5-5
2026-11-11 21:59:55 +01:00
19 changed files with 1171 additions and 88 deletions

No files matched your search

+2 -4
View File
@@ -26,10 +26,8 @@ check:
build: build:
@script/build @script/build
# Bundles the built dist/, so the binary reports the version script/build build-bin:
# stamped. @script/build-bin
build-bin: build
nix-shell -p bun --run "bun build dist/bin/quak.js --compile --outfile bin/quak"
install: build-bin install: build-bin
mkdir -p ~/bin mkdir -p ~/bin
+85 -20
View File
@@ -31,7 +31,8 @@ again on the next run. Every download is decrypted on its way to disk.
## Getting Started ## Getting Started
The commands below need `git`, Node 22 or later and `yarn`. The commands below need `git`, `yarn` and Node 22.13 or later. Node 23 needs
23.5 or later.
```bash ```bash
git clone https://git.eeqj.de/sneak/quak.git git clone https://git.eeqj.de/sneak/quak.git
@@ -69,11 +70,29 @@ The library keeps a local copy of the account's album and file lists, the cache.
It brings the cache up to date with the server in rounds called refreshes. It brings the cache up to date with the server in rounds called refreshes.
quak is not published to a package registry: `package.json` marks it `private`, quak is not published to a package registry: `package.json` marks it `private`,
and `dist/` is not committed. Clone and build it as above, then add the clone to and `dist/` is not committed. Clone and build it as above, then link the clone
your own program's `package.json` by its path, for example into your own program:
`yarn add file:../quak`. The package's `main`, `types` and `exports` fields
point into `dist/`, so the clone must be built first. The package is an ES ```bash
module only, so a CommonJS program cannot `require` it. # In the quak clone:
yarn link
# In your program's directory:
yarn link quak
```
`yarn link` also installs the `quak` command in yarn's global bin directory, and
`yarn unlink` in the clone removes the link and the command.
A linked quak uses the dependencies that `yarn install` put in the clone, at the
versions in quak's `yarn.lock`. Avoid `yarn add file:../quak`: it resolves
quak's dependencies again, and their newer releases can need a newer Node.
`yarn link quak` does not add quak to your `package.json`, so run it again in
each new checkout of your program.
The package's `main`, `types` and `exports` fields point into `dist/`, so the
clone must be built first. The package is an ES module only, so a CommonJS
program cannot `require` it.
```ts ```ts
import { Client, Library } from "quak"; import { Client, Library } from "quak";
@@ -135,10 +154,15 @@ requests. The "Library reference" below covers both.
## Running the backup from cron ## Running the backup from cron
`quak backup` never prompts, so cron can run it. `make install` builds quak as `quak backup` never prompts, so cron can run it. `make install` builds quak as
one binary and copies it to `~/bin/quak`. Besides what Getting Started needs, one binary and copies it to `~/bin/quak`. It uses bun, a JavaScript runtime, to
`make install` needs `make` and [nix](https://nixos.org/download). It uses bun, bundle the built `dist/` into one executable. Besides what Getting Started
a JavaScript runtime, to bundle the built `dist/` into one executable. It runs needs, `make install` needs `make`. It downloads bun from GitHub with `curl` and
bun through `nix-shell -p bun`. unpacks it with `unzip`. It checks the download against a SHA-256 hash in the
repo. `make install` works on Linux and macOS, on x86-64 and arm64.
On Linux systems that use the musl C library, such as Alpine, `make install`
downloads bun's musl build. On Alpine, both that bun and `~/bin/quak` need the
`libstdc++` and `libgcc` packages.
Log in once with `~/bin/quak`, as the user the cron job will run as: Log in once with `~/bin/quak`, as the user the cron job will run as:
@@ -205,7 +229,7 @@ below belong to quak itself, and go before or after the command's name.
### The commands ### The commands
``` ```
quak login quak login [--server <url>]
quak whoami quak whoami
quak logout quak logout
quak collections [--json] quak collections [--json]
@@ -222,13 +246,26 @@ Every command but `login` and `logout` needs a saved session. Every command but
`login`, `whoami` and `logout` opens the library. How each one gets current data `login`, `whoami` and `logout` opens the library. How each one gets current data
from the server is described under "How a command gets current data". from the server is described under "How a command gets current data".
#### `quak login` #### `quak login [--server <url>]`
Logs in to an account and saves the session. It reads the email from Logs in to an account and saves the session. It reads the email from
`QUAK_EMAIL` and the password from `QUAK_PASSWORD`. For each one that is not `QUAK_EMAIL` and the password from `QUAK_PASSWORD`. For each one that is not
set, it asks on the terminal. The password shows as `*` characters while it is set, it asks on the terminal. The password shows as `*` characters while it is
typed. typed.
quak logs in to Ente's hosted service unless you name a self-hosted Ente server.
Name it with `--server`, or with `QUAK_SERVER` when `--server` is not given.
Give the address of the server's API, for example
`quak login --server https://ente.example.com`. The address is an http or https
URL. It may include a path, but no query or fragment. quak refuses any other
address before it asks for anything. It logs
`Invalid server address "<url>": it must be an http or https URL with no query or fragment`,
and the exit status is 1.
The session file keeps the server's address. Every later command, cron runs
included, uses that server without further setting. To change server, log in
again.
When the account has a second factor, quak asks for the code. It asks When the account has a second factor, quak asks for the code. It asks
`TOTP code: ` for a time-based code from an authenticator app. An account whose `TOTP code: ` for a time-based code from an authenticator app. An account whose
only second factor is a passkey cannot log in with quak. When the account has only second factor is a passkey cannot log in with quak. When the account has
@@ -573,9 +610,11 @@ Linux that is `$XDG_DATA_HOME/quak/session.json`, or
mode `0600`, in a directory it creates with mode `0700`. mode `0600`, in a directory it creates with mode `0700`.
The file holds the account's email, its user ID, the session token and the The file holds the account's email, its user ID, the session token and the
account's three keys, in base64 and unencrypted. Treat it as you would treat the account's three keys, in base64 and unencrypted. After a login to a self-hosted
password. A missing file means "not logged in". A file that cannot be read back server, it also holds the server's address as `apiOrigin`. Treat it as you would
is reported as corrupt, with the reason. Run `quak logout`, then `quak login`. treat the password. A missing file means "not logged in". A file that cannot be
read back is reported as corrupt, with the reason. Run `quak logout`, then
`quak login`.
A command that needs a session exits 3 when the file is missing or corrupt. It A command that needs a session exits 3 when the file is missing or corrupt. It
also exits 3 when the server no longer accepts the session, and prints also exits 3 when the server no longer accepts the session, and prints
@@ -593,6 +632,11 @@ Without `--cache-dir`, each account has its own cache directory:
the variable is not set; the variable is not set;
- on macOS, `~/Library/Caches/quak/<userID>`. - on macOS, `~/Library/Caches/quak/<userID>`.
Two servers can give two accounts the same user ID. So an account on a
self-hosted server has the server's address, URL-encoded, between `quak` and the
user ID. For `https://ente.example.com`, that is
`quak/https%3A%2F%2Fente.example.com/<userID>`.
The directory is readable by its owner only (mode `0700`), because it holds The directory is readable by its owner only (mode `0700`), because it holds
decrypted data, file keys included. decrypted data, file keys included.
@@ -617,6 +661,13 @@ the library deletes it and `mldata/` and starts from an empty cache. Cached
originals and thumbnails are kept. They are reached only through the records of originals and thumbnails are kept. They are reached only through the records of
the current account, so another account's files are never served. the current account, so another account's files are never served.
When `metadata.json` belongs to an account on another server, opening the
library also deletes the originals and thumbnails. Two servers can give two
files the same ID, so one server's cached original could be served for the
other's file. quak records the account in `metadata.json` before it stores any
original or thumbnail. It does so also when `quak get` finds a file on an empty
cache.
A file appears in the cache only once it is complete: each one is written to a A file appears in the cache only once it is complete: each one is written to a
temporary file and renamed into place. Ente's clients record a hash of each temporary file and renamed into place. Ente's clients record a hash of each
file's bytes at upload, the content hash. Every original whose metadata records file's bytes at upload, the content hash. Every original whose metadata records
@@ -671,6 +722,9 @@ but letters and digits, or when it is longer than 16 characters.
When the date or the time zone changes, the next run stores the original at its When the date or the time zone changes, the next run stores the original at its
new path and leaves the old copy where it is. new path and leaves the old copy where it is.
A file deleted in Ente keeps its original at its save path, and the JSON beside
it. The next run removes its symlinks from the collection directories.
The library gives the same path as `photo.savePath` and writes it with The library gives the same path as `photo.savePath` and writes it with
`photo.download()` (see "A photo's content"). `photo.download()` (see "A photo's content").
@@ -1017,7 +1071,8 @@ onto what the cache holds, empty or not. A refresh then starts only when
- `client`, required: the account's `Client`, or anything that satisfies - `client`, required: the account's `Client`, or anything that satisfies
`LibraryClient`. `LibraryClient`.
- `cacheDirectory`: where `metadata.json` and the content cache live. Default: - `cacheDirectory`: where `metadata.json` and the content cache live. Default:
`quak/<userID>` under the per-user cache directory (see "The cache"). `quak/<userID>` under the per-user cache directory, with the URL-encoded
`apiOrigin` before `<userID>` for a client that has one (see "The cache").
- `downloadDirectory`: the root of the save paths. Default: `photos` in the - `downloadDirectory`: the root of the save paths. Default: `photos` in the
working directory, resolved once when the library opens; working directory, resolved once when the library opens;
`lib.downloadDirectory` holds the result. An original stored at its save path `lib.downloadDirectory` holds the result. An original stored at its save path
@@ -1214,7 +1269,8 @@ do.
fetch and cache as `photo.original()` and `photo.thumbnail()` do. `file` is an fetch and cache as `photo.original()` and `photo.thumbnail()` do. `file` is an
`EnteFile` the caller holds, and the library need not hold it. `quak get` and `EnteFile` the caller holds, and the library need not hold it. `quak get` and
`quak get-thumb` use them; on an empty cache they pass a file that `quak get-thumb` use them; on an empty cache they pass a file that
`lib.freshFiles()` returned. `lib.freshFiles()` returned. In a cache that names no account yet, they first
save the client's user ID and `apiOrigin` in `metadata.json`.
Lower-level reads return the decrypted model objects, which hold key material: Lower-level reads return the decrypted model objects, which hold key material:
`lib.listCollections()`, `lib.getCollection(id)`, `lib.listFiles(collectionID)`, `lib.listCollections()`, `lib.getCollection(id)`, `lib.listFiles(collectionID)`,
@@ -1450,6 +1506,14 @@ write wherever it likes. `Client.fromJSON(snapshot, apiOptions?)` restores a
working client from that snapshot without logging in again. It checks every working client from that snapshot without logging in again. It checks every
field and each key's length first, and throws an error naming the bad field. field and each key's length first, and throws an error naming the bad field.
A client whose `apiOptions` named an `apiOrigin` puts it in its snapshot. The
restored client then uses that server, even when the `apiOptions` given to
`Client.fromJSON` name another. A snapshot without `apiOrigin` uses the
`apiOrigin` of those `apiOptions`, or the default server. `client.apiOrigin`
holds the one in use, or undefined when none was named. `Client.fromJSON`
refuses an `apiOrigin` that is not an http or https URL, or that has a query or
fragment, as `quak login --server` does.
`Client.fromJSON` works only once libsodium, the cryptography library quak uses, `Client.fromJSON` works only once libsodium, the cryptography library quak uses,
is loaded. `Client.login` loads it, and `Client.fromJSON` does not. A program is loaded. `Client.login` loads it, and `Client.fromJSON` does not. A program
that restores a session in a new process first runs `await sodium.ready`, with that restores a session in a new process first runs `await sodium.ready`, with
@@ -1771,6 +1835,8 @@ standard. The others are additions to it.
`package.json` declares (`main`, `types`, `bin`) were written, and make the `package.json` declares (`main`, `types`, `bin`) were written, and make the
command line executable. command line executable.
- `script/version`: print the version `script/build` stamps (see "Version"). - `script/version`: print the version `script/build` stamps (see "Version").
- `script/build-bin`: run `script/build`, then bundle the built `dist/` into one
binary, `bin/quak`, with bun (see "Running the backup from cron").
- `script/test`: run the test suite, by building the `test` phase of the - `script/test`: run the test suite, by building the `test` phase of the
`Dockerfile`. Requires Docker. `Dockerfile`. Requires Docker.
- `script/lint`: run eslint and a prettier check, by building the `lint` phase - `script/lint`: run eslint and a prettier check, by building the `lint` phase
@@ -1799,8 +1865,7 @@ The Makefile targets are:
- `make build`: runs `script/build`. - `make build`: runs `script/build`.
- `make docker`: runs `script/docker`. - `make docker`: runs `script/docker`.
- `make hooks`: runs `script/install-precommit`. - `make hooks`: runs `script/install-precommit`.
- `make build-bin`: bundles the built `dist/` into one binary, `bin/quak`. It - `make build-bin`: runs `script/build-bin`.
runs bun through `nix-shell -p bun`.
- `make install`: runs `make build-bin` and copies the binary to `~/bin/quak`. - `make install`: runs `make build-bin` and copies the binary to `~/bin/quak`.
- `make dev`: runs the TypeScript compiler in watch mode. - `make dev`: runs the TypeScript compiler in watch mode.
- `make clean`: removes `dist/`, `coverage/`, `.vitest-cache/` and the - `make clean`: removes `dist/`, `coverage/`, `.vitest-cache/` and the
@@ -1908,9 +1973,9 @@ These apply to people and to coding agents alike.
- container images by digest; - container images by digest;
- the nvm archive `script/bootstrap` downloads, and the Python packages - the nvm archive `script/bootstrap` downloads, and the Python packages
`bin/clip-keyword-vectors` installs, by SHA-256 hash; `bin/clip-keyword-vectors` installs, by SHA-256 hash;
- the bun builds `script/build-bin` downloads, by SHA-256 hash;
- Node and yarn, when `script/bootstrap` installs them, by version. - Node and yarn, when `script/bootstrap` installs them, by version.
- Not pinned: - Not pinned:
- bun, which `make build-bin` takes from `nix-shell -p bun`;
- `make` and `git`, which `script/bootstrap` installs from the system - `make` and `git`, which `script/bootstrap` installs from the system
package manager when they are missing, as every `Dockerfile` stage does; package manager when they are missing, as every `Dockerfile` stage does;
- `bash` and `curl`, which `script/bootstrap` installs the same way when it - `bash` and `curl`, which `script/bootstrap` installs the same way when it
+11 -2
View File
@@ -74,7 +74,11 @@ const context = (): CliContext => ({
sessionDir: paths.data, sessionDir: paths.data,
cacheDir: program.opts<ProgramOptions>().cacheDir, cacheDir: program.opts<ProgramOptions>().cacheDir,
loadSession: (path) => loadSession(path, logRequests(log())), loadSession: (path) => loadSession(path, logRequests(log())),
login: (opts) => Client.login({ ...opts, apiOptions: logRequests(log()) }), login: (opts) =>
Client.login({
...opts,
apiOptions: { ...opts.apiOptions, ...logRequests(log()) },
}),
prompt: (message) => input({ message }), prompt: (message) => input({ message }),
promptSecret: (message) => password({ message, mask: true }), promptSecret: (message) => password({ message, mask: true }),
}); });
@@ -85,7 +89,12 @@ const run = (command: Promise<number>): Promise<void> =>
program program
.command("login") .command("login")
.description("Log in to an Ente account and save the session") .description("Log in to an Ente account and save the session")
.action(() => run(loginCommand(context()))); .option(
"--server <url>",
"Address of a self-hosted Ente server's API, saved with the session " +
"(default: QUAK_SERVER, then Ente's hosted service)",
)
.action((opts: { server?: string }) => run(loginCommand(context(), opts)));
program program
.command("whoami") .command("whoami")
+1 -1
View File
@@ -12,7 +12,7 @@
"private": true, "private": true,
"type": "module", "type": "module",
"engines": { "engines": {
"node": ">=22" "node": "^22.13.0 || >=23.5.0"
}, },
"main": "./dist/src/index.js", "main": "./dist/src/index.js",
"types": "./dist/src/index.d.ts", "types": "./dist/src/index.d.ts",
+85
View File
@@ -0,0 +1,85 @@
#!/bin/sh
# script/build-bin: run script/build, then bundle the built dist/ into one
# executable, bin/quak, which reports the version script/build stamped. Our own
# extension to scripts-to-rule-them-all.
#
# bun compiles the binary from a copy of its own executable, so the binary
# loads the same libraries as the bun that built it. nix's bun loads libraries
# from the nix store: on Linux its C library, on macOS its ICU. A binary
# compiled from it loads them too. nix-collect-garbage can delete them, and the
# binary then no longer starts. This script therefore compiles with bun's own
# release build, which loads only the system's libraries. It downloads that
# build and checks it against the hash below before running it.
set -eu
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
# bun 1.3.13, 2026-11-11, and the SHA-256 of each release archive below. The
# x86-64 archives are the baseline builds, which also run on CPUs without AVX2.
# The musl archives are for Linux systems that use the musl C library, such as
# Alpine.
BUN_VERSION="1.3.13"
BUN_LINUX_X64_BASELINE_SHA256="9d8a24292a7068090205daac0a5a223f5f69736f5287e37bf88d3b4031edc750"
BUN_LINUX_X64_MUSL_BASELINE_SHA256="88ca7c7ad235b498f549eea2f770f434e9f0f5e9ba95168a2d3a1f235184c394"
BUN_LINUX_AARCH64_SHA256="70bae41b3908b0a120e1e58c5c8af30e74afae3b8d11b0d3fdd8e787ddfb4b22"
BUN_LINUX_AARCH64_MUSL_SHA256="5385e978107ce4934298d8d6afe9bfbb898683f6cc23e6753a0da60bc60c5b81"
BUN_DARWIN_X64_BASELINE_SHA256="a98ba6a480f22fda9b343626b906a4e26aa53618bf85d2bc5928ecf2ba45f0ed"
BUN_DARWIN_AARCH64_SHA256="5467e3f65dba526b9fea98f0cce04efafc0c63e169733ec27b876a3ad32da190"
# compile_with_downloaded_bun <archive-name> <sha256>
compile_with_downloaded_bun() {
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
curl -fsSL -o "$tmp/$1.zip" \
"https://github.com/oven-sh/bun/releases/download/bun-v$BUN_VERSION/$1.zip"
# macOS always has shasum, but not every macOS has sha256sum.
if command -v sha256sum >/dev/null 2>&1; then
echo "$2 $tmp/$1.zip" | sha256sum -c -
else
echo "$2 $tmp/$1.zip" | shasum -a 256 -c -
fi
unzip -q "$tmp/$1.zip" -d "$tmp"
"$tmp/$1/bun" build dist/bin/quak.js --compile --outfile bin/quak
}
main() {
cd "$ROOT"
"$ROOT/script/build"
platform="$(uname -s)-$(uname -m)"
# musl's ldd begins its version output with "musl libc"; glibc's does not.
if ldd --version 2>&1 | grep -q '^musl libc'; then
platform="$platform-musl"
fi
case "$platform" in
Linux-x86_64)
compile_with_downloaded_bun bun-linux-x64-baseline \
"$BUN_LINUX_X64_BASELINE_SHA256"
;;
Linux-x86_64-musl)
compile_with_downloaded_bun bun-linux-x64-musl-baseline \
"$BUN_LINUX_X64_MUSL_BASELINE_SHA256"
;;
Linux-aarch64)
compile_with_downloaded_bun bun-linux-aarch64 \
"$BUN_LINUX_AARCH64_SHA256"
;;
Linux-aarch64-musl)
compile_with_downloaded_bun bun-linux-aarch64-musl \
"$BUN_LINUX_AARCH64_MUSL_SHA256"
;;
Darwin-x86_64)
compile_with_downloaded_bun bun-darwin-x64-baseline \
"$BUN_DARWIN_X64_BASELINE_SHA256"
;;
Darwin-arm64)
compile_with_downloaded_bun bun-darwin-aarch64 \
"$BUN_DARWIN_AARCH64_SHA256"
;;
*)
echo "build-bin: bun has no release build for $(uname -s) on $(uname -m)" >&2
exit 1
;;
esac
}
main "$@"
+8 -1
View File
@@ -354,7 +354,13 @@ export class ApiClient {
}); });
} }
async putFile(presignedURL: string, data: Uint8Array): Promise<void> { // `contentMD5` is the value given to `getUploadURL`. The server signs it
// into the URL, and storage refuses a PUT without that `Content-MD5`.
async putFile(
presignedURL: string,
data: Uint8Array,
contentMD5: string,
): Promise<void> {
// Idempotent despite being a write: a presigned PUT stores one whole // Idempotent despite being a write: a presigned PUT stores one whole
// object at one key in one request, so replaying it either overwrites // object at one key in one request, so replaying it either overwrites
// the same bytes or lands them for the first time. There is no partial // the same bytes or lands them for the first time. There is no partial
@@ -370,6 +376,7 @@ export class ApiClient {
headers: { headers: {
"Content-Type": "application/octet-stream", "Content-Type": "application/octet-stream",
"Content-Length": String(data.length), "Content-Length": String(data.length),
"Content-MD5": contentMD5,
}, },
body: data, body: data,
signal: AbortSignal.timeout(this.requestTimeoutMs), signal: AbortSignal.timeout(this.requestTimeoutMs),
+25 -3
View File
@@ -21,6 +21,7 @@ import { extname, join } from "node:path";
import { import {
type Client, type Client,
type ClientSnapshot, type ClientSnapshot,
isServerAddress,
type LoginOptions, type LoginOptions,
} from "./client.js"; } from "./client.js";
import { lockBackupDirectory, type BackupLock } from "./backup.js"; import { lockBackupDirectory, type BackupLock } from "./backup.js";
@@ -56,7 +57,7 @@ export interface CliContext {
// Directory holding `session.json`. // Directory holding `session.json`.
sessionDir: string; sessionDir: string;
// The `--cache-dir` global, or undefined to let the library pick its // The `--cache-dir` global, or undefined to let the library pick its
// per-user default keyed by the account id. // per-user default keyed by the account's server and user ID.
cacheDir?: string; cacheDir?: string;
// Reads the session file into a client, or null when there is none. The // Reads the session file into a client, or null when there is none. The
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client. // CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
@@ -129,6 +130,7 @@ const parseID = (
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch // own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
// originals through the on-disk cache. // originals through the on-disk cache.
const readLibraryClient = (client: Client): LibraryClient => ({ const readLibraryClient = (client: Client): LibraryClient => ({
apiOrigin: client.apiOrigin,
whoami: () => client.whoami(), whoami: () => client.whoami(),
collectionsSince: (args) => client.collectionsSince(args), collectionsSince: (args) => client.collectionsSince(args),
filesSince: (args) => client.filesSince(args), filesSince: (args) => client.filesSince(args),
@@ -155,8 +157,23 @@ const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
onProgress: logRefresh(ctx.log), onProgress: logRefresh(ctx.log),
}); });
export const loginCommand = async (ctx: CliContext): Promise<number> => { // Logs in to the server `--server` or `QUAK_SERVER` names, the flag first, or
// to Ente's hosted service when neither is set. The saved session keeps the
// server, so every later command uses it.
export const loginCommand = async (
ctx: CliContext,
opts: { server?: string },
): Promise<number> => {
await init(); await init();
const server = opts.server ?? process.env.QUAK_SERVER;
if (server !== undefined && !isServerAddress(server)) {
ctx.log(
"fatal",
`Invalid server address "${server}": it must be an http or ` +
`https URL with no query or fragment`,
);
return 1;
}
const email = process.env.QUAK_EMAIL ?? (await ctx.prompt("Email")); const email = process.env.QUAK_EMAIL ?? (await ctx.prompt("Email"));
const password = const password =
process.env.QUAK_PASSWORD ?? (await ctx.promptSecret("Password")); process.env.QUAK_PASSWORD ?? (await ctx.promptSecret("Password"));
@@ -168,6 +185,8 @@ export const loginCommand = async (ctx: CliContext): Promise<number> => {
password, password,
totp: async () => ctx.prompt("TOTP code: "), totp: async () => ctx.prompt("TOTP code: "),
emailOTP: async () => ctx.prompt("Email verification code: "), emailOTP: async () => ctx.prompt("Email verification code: "),
apiOptions:
server === undefined ? undefined : { apiOrigin: server },
}); });
saveSession(ctx.sessionDir, client.toJSON()); saveSession(ctx.sessionDir, client.toJSON());
@@ -208,7 +227,10 @@ export const logoutCommand = async (ctx: CliContext): Promise<number> => {
try { try {
const client = ctx.loadSession(path); const client = ctx.loadSession(path);
if (client) { if (client) {
cacheDir ??= defaultCacheDirectory(client.whoami().userID); cacheDir ??= defaultCacheDirectory(
client.whoami().userID,
client.apiOrigin,
);
await client.logoutOnServer(); await client.logoutOnServer();
client.logout(); client.logout();
} }
+37 -1
View File
@@ -40,6 +40,9 @@ export interface ClientSnapshot {
masterKey: string; masterKey: string;
secretKey: string; secretKey: string;
publicKey: string; publicKey: string;
// The `apiOrigin` the client was given, so a restored client talks to
// the server that issued its token. Absent means the default server.
apiOrigin?: string;
} }
// The result of a resumable enumeration. Live decrypted records and deleted // The result of a resumable enumeration. Live decrypted records and deleted
@@ -66,7 +69,19 @@ export interface FilesPage {
const errorMessage = (err: unknown): string => const errorMessage = (err: unknown): string =>
err instanceof Error ? err.message : String(err); err instanceof Error ? err.message : String(err);
// Whether `text` can be an Ente server's address: an http or https URL, which
// may have a path, and no query or fragment, which request URLs could not
// carry. `quak login --server` and `Client.fromJSON` both check with it.
export const isServerAddress = (text: string): boolean =>
URL.canParse(text) &&
["http:", "https:"].includes(new URL(text).protocol) &&
!/[?#]/.test(text);
export class Client { export class Client {
// The `apiOrigin` the client was given, or undefined when it was given
// none, which means Ente's hosted service. The library keeps a cache per
// server as well as per user ID.
readonly apiOrigin: string | undefined;
private readonly api: ApiClient; private readonly api: ApiClient;
private readonly email: string; private readonly email: string;
private readonly userID: number; private readonly userID: number;
@@ -82,6 +97,7 @@ export class Client {
masterKey: Uint8Array, masterKey: Uint8Array,
secretKey: Uint8Array, secretKey: Uint8Array,
publicKey: Uint8Array, publicKey: Uint8Array,
apiOrigin: string | undefined,
) { ) {
this.api = api; this.api = api;
this.email = email; this.email = email;
@@ -89,6 +105,7 @@ export class Client {
this.masterKey = masterKey; this.masterKey = masterKey;
this.secretKey = secretKey; this.secretKey = secretKey;
this.publicKey = publicKey; this.publicKey = publicKey;
this.apiOrigin = apiOrigin;
} }
static async login(opts: LoginOptions): Promise<Client> { static async login(opts: LoginOptions): Promise<Client> {
@@ -130,12 +147,15 @@ export class Client {
unwrapped.masterKey, unwrapped.masterKey,
unwrapped.secretKey, unwrapped.secretKey,
unwrapped.publicKey, unwrapped.publicKey,
opts.apiOptions?.apiOrigin,
); );
} }
// Restore a client from a `toJSON()` snapshot. The snapshot usually comes // Restore a client from a `toJSON()` snapshot. The snapshot usually comes
// straight from `JSON.parse` of a file on disk, so every field is checked // straight from `JSON.parse` of a file on disk, so every field is checked
// before use; a bad one throws an error naming it. Needs `init()` first. // before use; a bad one throws an error naming it. The snapshot's
// `apiOrigin`, when it has one, is used instead of `apiOptions.apiOrigin`.
// Needs `init()` first.
static fromJSON(snapshot: unknown, apiOptions?: ApiClientOptions): Client { static fromJSON(snapshot: unknown, apiOptions?: ApiClientOptions): Client {
const invalid = (field: string, problem: string): Error => const invalid = (field: string, problem: string): Error =>
new Error(`Invalid session data: ${field} ${problem}`); new Error(`Invalid session data: ${field} ${problem}`);
@@ -152,6 +172,17 @@ export class Client {
if (!Number.isInteger(s.userID)) { if (!Number.isInteger(s.userID)) {
throw invalid("userID", "must be an integer"); throw invalid("userID", "must be an integer");
} }
if (
s.apiOrigin !== undefined &&
(typeof s.apiOrigin !== "string" || !isServerAddress(s.apiOrigin))
) {
throw invalid(
"apiOrigin",
"must be an http or https URL with no query or fragment",
);
}
const apiOrigin =
(s.apiOrigin as string | undefined) ?? apiOptions?.apiOrigin;
const key = (field: string): Uint8Array => { const key = (field: string): Uint8Array => {
const value = s[field]; const value = s[field];
if (typeof value !== "string") { if (typeof value !== "string") {
@@ -175,6 +206,7 @@ export class Client {
const api = new ApiClient({ const api = new ApiClient({
...apiOptions, ...apiOptions,
apiOrigin,
authToken: s.token as string, authToken: s.token as string,
}); });
return new Client( return new Client(
@@ -184,6 +216,7 @@ export class Client {
key("masterKey"), key("masterKey"),
key("secretKey"), key("secretKey"),
key("publicKey"), key("publicKey"),
apiOrigin,
); );
} }
@@ -222,6 +255,9 @@ export class Client {
masterKey: toBase64(this.masterKey), masterKey: toBase64(this.masterKey),
secretKey: toBase64(this.secretKey), secretKey: toBase64(this.secretKey),
publicKey: toBase64(this.publicKey), publicKey: toBase64(this.publicKey),
...(this.apiOrigin === undefined
? {}
: { apiOrigin: this.apiOrigin }),
}; };
} }
+62 -14
View File
@@ -119,9 +119,18 @@ export {
export const DEFAULT_REFRESH_INTERVAL_SECONDS = 3; export const DEFAULT_REFRESH_INTERVAL_SECONDS = 3;
// The account's cache directory when `cacheDirectory` is not given: the // The account's cache directory when `cacheDirectory` is not given: the
// env-paths cache directory plus the user id, so each account has its own. // env-paths cache directory plus the user id, so each account has its own. An
export const defaultCacheDirectory = (userID: number): string => // account on a self-hosted server has the server's address, URL-encoded, in
join(envPaths("quak", { suffix: "" }).cache, String(userID)); // between, because two servers can give two accounts the same user id.
export const defaultCacheDirectory = (
userID: number,
apiOrigin?: string,
): string => {
const cache = envPaths("quak", { suffix: "" }).cache;
return apiOrigin === undefined
? join(cache, String(userID))
: join(cache, encodeURIComponent(apiOrigin), String(userID));
};
// Project a metadata store into by-id records, filling each record's cache // Project a metadata store into by-id records, filling each record's cache
// paths from the content cache when one is given. Shared by the live read // paths from the content cache when one is given. Shared by the live read
@@ -144,6 +153,9 @@ const deriveRecordsFromStore = (
// tests drive a mock with no crypto or network; the real `Client` satisfies it // tests drive a mock with no crypto or network; the real `Client` satisfies it
// structurally. // structurally.
export interface LibraryClient { export interface LibraryClient {
// The server the client talks to, as `Client.apiOrigin` names it. Absent
// means Ente's hosted service.
readonly apiOrigin?: string;
whoami(): { email: string; userID: number }; whoami(): { email: string; userID: number };
collectionsSince(args: { sinceTime: number }): Promise<CollectionsPage>; collectionsSince(args: { sinceTime: number }): Promise<CollectionsPage>;
filesSince(args: { filesSince(args: {
@@ -196,8 +208,8 @@ export type RefreshProgressCallback = (event: RefreshEvent) => void;
export interface LibraryOptions { export interface LibraryOptions {
client: LibraryClient; client: LibraryClient;
// Where `metadata.json` lives. Defaults to the env-paths cache directory // Where `metadata.json` lives. Defaults to `defaultCacheDirectory`, so
// plus the user id, so each account has its own cache. // each account has its own cache.
cacheDirectory?: string; cacheDirectory?: string;
// The root of every photo's save path, where `Photo.download()` and // The root of every photo's save path, where `Photo.download()` and
// `lib.backup()` put originals. Defaults to `photos` in the working // `lib.backup()` put originals. Defaults to `photos` in the working
@@ -300,6 +312,7 @@ export class Library {
// available (a metadata-only client with no explicit source). // available (a metadata-only client with no explicit source).
private readonly cache?: ContentCache; private readonly cache?: ContentCache;
private readonly userID: number; private readonly userID: number;
private readonly apiOrigin: string | undefined;
private readonly intervalMs: number; private readonly intervalMs: number;
private readonly onProgress?: RefreshProgressCallback; private readonly onProgress?: RefreshProgressCallback;
private readonly pools: RequestPools; private readonly pools: RequestPools;
@@ -346,6 +359,7 @@ export class Library {
client: LibraryClient; client: LibraryClient;
store: MetadataStore; store: MetadataStore;
userID: number; userID: number;
apiOrigin: string | undefined;
cacheDirectory: string; cacheDirectory: string;
downloadDirectory: string; downloadDirectory: string;
intervalMs: number; intervalMs: number;
@@ -358,6 +372,7 @@ export class Library {
this.client = args.client; this.client = args.client;
this.store = args.store; this.store = args.store;
this.userID = args.userID; this.userID = args.userID;
this.apiOrigin = args.apiOrigin;
this.cacheDirectory = args.cacheDirectory; this.cacheDirectory = args.cacheDirectory;
this.downloadDirectory = args.downloadDirectory; this.downloadDirectory = args.downloadDirectory;
this.intervalMs = args.intervalMs; this.intervalMs = args.intervalMs;
@@ -406,8 +421,9 @@ export class Library {
// refresh: `open()` returns onto the cache as it is. // refresh: `open()` returns onto the cache as it is.
static async open(opts: LibraryOptions): Promise<Library> { static async open(opts: LibraryOptions): Promise<Library> {
const { userID } = opts.client.whoami(); const { userID } = opts.client.whoami();
const { apiOrigin } = opts.client;
const cacheDirectory = const cacheDirectory =
opts.cacheDirectory ?? defaultCacheDirectory(userID); opts.cacheDirectory ?? defaultCacheDirectory(userID, apiOrigin);
if (opts.downloadDirectory === "") { if (opts.downloadDirectory === "") {
throw new Error( throw new Error(
"library: downloadDirectory is empty (leave it out to save " + "library: downloadDirectory is empty (leave it out to save " +
@@ -419,14 +435,24 @@ export class Library {
let store = await MetadataStore.load(metadataPath); let store = await MetadataStore.load(metadataPath);
// A cache directory given explicitly can hold another account's cache. // A cache directory given explicitly can hold another account's cache.
// Its records and cursor are not this account's, so delete it and the // Its records and cursor are not this account's, so delete it and the
// ML data beside it and start empty. A user ID of 0 means the cache // ML data beside it and start empty. A cache from another server goes
// was never refreshed and so holds nothing to discard. // whole: two servers can give two files the same ID, so its originals
if (store.userID !== 0 && store.userID !== userID) { // and thumbnails could be served for this account's files. A user ID
await rm(metadataPath, { force: true }); // of 0 means the cache names no account, so nothing is stored in it.
await rm(join(cacheDirectory, "mldata"), { if (
store.userID !== 0 &&
(store.userID !== userID || store.apiOrigin !== apiOrigin)
) {
const discard = ["metadata.json", "mldata"];
if (store.apiOrigin !== apiOrigin) {
discard.push("originals", "thumbnails");
}
for (const name of discard) {
await rm(join(cacheDirectory, name), {
recursive: true, recursive: true,
force: true, force: true,
}); });
}
store = await MetadataStore.load(metadataPath); store = await MetadataStore.load(metadataPath);
} }
const intervalMs = const intervalMs =
@@ -490,6 +516,7 @@ export class Library {
client: opts.client, client: opts.client,
store, store,
userID, userID,
apiOrigin,
cacheDirectory, cacheDirectory,
downloadDirectory, downloadDirectory,
intervalMs, intervalMs,
@@ -509,9 +536,11 @@ export class Library {
if (opts.refreshOnOpen === false) { if (opts.refreshOnOpen === false) {
// The caller asks for each refresh it needs. // The caller asks for each refresh it needs.
lib.scheduleNext(); lib.scheduleNext();
} else if (store.loadedFromDisk) { } else if (store.listCollections().length > 0) {
// An existing copy already answers reads; refresh in the background // An existing copy already answers reads; refresh in the background
// and start the interval once that first cycle settles. // and start the interval once that first cycle settles. A cache
// that names the account but holds no collection has nothing to
// answer with, such as after `originalOf` on an empty cache.
void lib.runRefresh().then(() => lib.scheduleNext()); void lib.runRefresh().then(() => lib.scheduleNext());
} else { } else {
// Nothing was cached: wait for the first refresh to fill the store // Nothing was cached: wait for the first refresh to fill the store
@@ -658,6 +687,7 @@ export class Library {
"originalOf requires a library opened with a content cache", "originalOf requires a library opened with a content cache",
); );
} }
await this.saveAccountInEmptyCache();
return this.cache.originalOf(file, opts); return this.cache.originalOf(file, opts);
} }
@@ -673,9 +703,23 @@ export class Library {
"thumbnailOf requires a library opened with a content cache", "thumbnailOf requires a library opened with a content cache",
); );
} }
await this.saveAccountInEmptyCache();
return this.cache.thumbnailOf(file, opts); return this.cache.thumbnailOf(file, opts);
} }
// Saves the account's user ID and server in a cache that names no account
// yet, before `originalOf` or `thumbnailOf` stores a file there. A later
// `open()` for an account on another server then deletes that file (see
// `open()`), although no refresh has stored any records.
private async saveAccountInEmptyCache(): Promise<void> {
if (this.store.userID !== 0) return;
this.store.userID = this.userID;
this.store.apiOrigin = this.apiOrigin;
this.unsaved = true;
await this.store.save();
this.unsaved = false;
}
// Back up every in-scope file to `opts.downloadDirectory`, or else the // Back up every in-scope file to `opts.downloadDirectory`, or else the
// library's, each original at its save path, with a durable failure // library's, each original at its save path, with a durable failure
// ledger (issue #51). Takes the lock in that directory first, unless the // ledger (issue #51). Takes the lock in that directory first, unless the
@@ -857,8 +901,12 @@ export class Library {
// actually changed. // actually changed.
let changed = false; let changed = false;
if (this.store.userID !== this.userID) { if (
this.store.userID !== this.userID ||
this.store.apiOrigin !== this.apiOrigin
) {
this.store.userID = this.userID; this.store.userID = this.userID;
this.store.apiOrigin = this.apiOrigin;
changed = true; changed = true;
} }
+11 -12
View File
@@ -1,11 +1,12 @@
// On-disk JSON metadata store for the local cache. // On-disk JSON metadata store for the local cache.
// //
// The store keeps one `metadata.json` file holding the account's server // The store keeps one `metadata.json` file holding the account's server
// state: the user id, a schema version, the cursor for the incremental // state: the user id, the server's address for an account on a self-hosted
// collections listing, the decrypted collection and file records, and the // server, a schema version, the cursor for the incremental collections
// collections and files the refresh could not decrypt. The whole file is read // listing, the decrypted collection and file records, and the collections and
// into RAM on load and rewritten as a whole on save; there is no partial // files the refresh could not decrypt. The whole file is read into RAM on
// update and no lock file. A separate refresh unit populates the store from // load and rewritten as a whole on save; there is no partial update and no
// lock file. A separate refresh unit populates the store from
// the server — this module only stores what it is given. // the server — this module only stores what it is given.
// //
// The file is a cache, so it is never trusted to exist or to be intact: a // The file is a cache, so it is never trusted to exist or to be intact: a
@@ -46,6 +47,7 @@ export interface Undecryptable {
interface StoredMetadata { interface StoredMetadata {
schemaVersion: number; schemaVersion: number;
userID: number; userID: number;
apiOrigin?: string;
collectionsSinceTime: Microseconds; collectionsSinceTime: Microseconds;
collections: StoredCollection[]; collections: StoredCollection[];
files: StoredFile[]; files: StoredFile[];
@@ -68,14 +70,10 @@ export class MetadataStore {
readonly path: string; readonly path: string;
readonly schemaVersion = METADATA_SCHEMA_VERSION; readonly schemaVersion = METADATA_SCHEMA_VERSION;
userID = 0; userID = 0;
// The account's `apiOrigin`, or undefined for Ente's hosted service.
apiOrigin: string | undefined;
collectionsSinceTime: Microseconds = 0; collectionsSinceTime: Microseconds = 0;
// True when `load` populated this store from a valid existing file; false
// on a first run or a missing/corrupt/wrong-version file that loaded empty.
// `Library.open` reads it to decide whether the first refresh may run in
// the background (an existing copy already serves reads) or must be awaited.
loadedFromDisk = false;
private readonly collections = new Map<number, Collection>(); private readonly collections = new Map<number, Collection>();
private readonly files = new Map<string, EnteFile>(); private readonly files = new Map<string, EnteFile>();
private undecryptable: Undecryptable[] = []; private undecryptable: Undecryptable[] = [];
@@ -102,8 +100,8 @@ export class MetadataStore {
if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) { if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) {
return store; return store;
} }
store.loadedFromDisk = true;
store.userID = parsed.userID ?? 0; store.userID = parsed.userID ?? 0;
store.apiOrigin = parsed.apiOrigin;
store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0; store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0;
for (const stored of parsed.collections ?? []) { for (const stored of parsed.collections ?? []) {
const collection: Collection = { const collection: Collection = {
@@ -143,6 +141,7 @@ export class MetadataStore {
const model: StoredMetadata = { const model: StoredMetadata = {
schemaVersion: METADATA_SCHEMA_VERSION, schemaVersion: METADATA_SCHEMA_VERSION,
userID: this.userID, userID: this.userID,
apiOrigin: this.apiOrigin,
collectionsSinceTime: this.collectionsSinceTime, collectionsSinceTime: this.collectionsSinceTime,
collections: [...this.collections.values()].map((c) => ({ collections: [...this.collections.values()].map((c) => ({
...c, ...c,
+1 -1
View File
@@ -325,7 +325,7 @@ export const fixMissingThumbnails = async (
ciphertext.length, ciphertext.length,
md5, md5,
); );
await api.putFile(url, ciphertext); await api.putFile(url, ciphertext, md5);
await api.updateThumbnail(file.id, objectKey, toBase64(header)); await api.updateThumbnail(file.id, objectKey, toBase64(header));
log(`[${collectionName}] Thumbnail uploaded for ${title}`); log(`[${collectionName}] Thumbnail uploaded for ${title}`);
+10 -4
View File
@@ -780,7 +780,7 @@ describe("ApiClient timeouts", () => {
await client.getJSON("/a"); await client.getJSON("/a");
await client.postJSON("/b", {}); await client.postJSON("/b", {});
await client.putFile("https://s3.example/x", new Uint8Array([1])); await client.putFile("https://s3.example/x", new Uint8Array([1]), "m");
await client.putJSON("/c", {}); await client.putJSON("/c", {});
for (const call of calls) { for (const call of calls) {
@@ -825,7 +825,8 @@ describe("ApiClient timeouts", () => {
[ [
"putFile", "putFile",
() => new Response(null, { status: 200 }), () => new Response(null, { status: 200 }),
(c) => c.putFile("https://s3.example/x", new Uint8Array([1])), (c) =>
c.putFile("https://s3.example/x", new Uint8Array([1]), "m"),
], ],
[ [
"getFileStream", "getFileStream",
@@ -1049,7 +1050,7 @@ describe("ApiClient error typing", () => {
const client = new ApiClient({ fetch, retry: noWait }); const client = new ApiClient({ fetch, retry: noWait });
const err: unknown = await client const err: unknown = await client
.putFile("https://s3.example/obj", new Uint8Array([1, 2])) .putFile("https://s3.example/obj", new Uint8Array([1, 2]), "m")
.catch((e: unknown) => e); .catch((e: unknown) => e);
expect(err).toBeInstanceOf(ApiError); expect(err).toBeInstanceOf(ApiError);
@@ -1067,7 +1068,11 @@ describe("ApiClient error typing", () => {
); );
const client = new ApiClient({ fetch, retry: noWait }); const client = new ApiClient({ fetch, retry: noWait });
await client.putFile("https://s3.example/obj", new Uint8Array([1, 2])); await client.putFile(
"https://s3.example/obj",
new Uint8Array([1, 2]),
"m",
);
expect(calls).toHaveLength(2); expect(calls).toHaveLength(2);
}); });
@@ -1277,6 +1282,7 @@ describe("ApiClient onRequest", () => {
await client.putFile( await client.putFile(
"https://s3.example/obj?X-Amz-Signature=abc", "https://s3.example/obj?X-Amz-Signature=abc",
new Uint8Array([1, 2]), new Uint8Array([1, 2]),
"m",
); );
expect(requests).toEqual([ expect(requests).toEqual([
+13 -2
View File
@@ -80,13 +80,20 @@ describe("ApiClient.putFile", () => {
const client = new ApiClient({ fetch, authToken: "secret-tok" }); const client = new ApiClient({ fetch, authToken: "secret-tok" });
const data = new Uint8Array([1, 2, 3, 4, 5]); const data = new Uint8Array([1, 2, 3, 4, 5]);
await client.putFile("https://s3.example.com/presigned?sig=abc", data); await client.putFile(
"https://s3.example.com/presigned?sig=abc",
data,
"md5base64==",
);
expect(calls[0]!.url).toBe("https://s3.example.com/presigned?sig=abc"); expect(calls[0]!.url).toBe("https://s3.example.com/presigned?sig=abc");
expect(calls[0]!.init?.method).toBe("PUT"); expect(calls[0]!.init?.method).toBe("PUT");
const headers = new Headers(calls[0]!.init?.headers as HeadersInit); const headers = new Headers(calls[0]!.init?.headers as HeadersInit);
expect(headers.get("Content-Type")).toBe("application/octet-stream"); expect(headers.get("Content-Type")).toBe("application/octet-stream");
expect(headers.get("Content-Length")).toBe("5"); expect(headers.get("Content-Length")).toBe("5");
// The server signs the MD5 into the URL; storage refuses the PUT
// without it.
expect(headers.get("Content-MD5")).toBe("md5base64==");
// Must NOT leak auth headers to S3 // Must NOT leak auth headers to S3
expect(headers.has("X-Auth-Token")).toBe(false); expect(headers.has("X-Auth-Token")).toBe(false);
expect(headers.has("X-Client-Package")).toBe(false); expect(headers.has("X-Client-Package")).toBe(false);
@@ -99,7 +106,11 @@ describe("ApiClient.putFile", () => {
const client = new ApiClient({ fetch }); const client = new ApiClient({ fetch });
await expect( await expect(
client.putFile("https://s3.example.com/bad", new Uint8Array(10)), client.putFile(
"https://s3.example.com/bad",
new Uint8Array(10),
"md5base64==",
),
).rejects.toThrow(/403/); ).rejects.toThrow(/403/);
}); });
}); });
+35 -3
View File
@@ -2,12 +2,14 @@
* Tests for the retry options `bin/quak.ts` loads the saved session with * Tests for the retry options `bin/quak.ts` loads the saved session with
* (`quak backup` gets the unattended ones, every other command the default), * (`quak backup` gets the unattended ones, every other command the default),
* each with the callbacks that log its requests and retries, for `quak files` * each with the callbacks that log its requests and retries, for `quak files`
* taking its collection ID as an optional argument, and for `quak get` and * taking its collection ID as an optional argument, for `quak get` and
* `quak get-thumb` refusing `--collection`. * `quak get-thumb` refusing `--collection`, and for `quak login` taking
* `--server` and sending its first request to the server it names.
* *
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader * Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
* replaced by one that records the options it is given and reports no session, * replaced by one that records the options it is given and reports no session,
* so the command stops before it makes any request. * so the command stops before it makes any request. `quak login` reads no
* session, so its test stubs `fetch` instead and records each request.
* *
* The first load of the sources `bin/quak.ts` imports compiles them, and the * The first load of the sources `bin/quak.ts` imports compiles them, and the
* first `init()` starts libsodium. On a loaded host that takes longer than a * first `init()` starts libsodium. On a loaded host that takes longer than a
@@ -58,6 +60,8 @@ afterEach(() => {
process.argv = argv; process.argv = argv;
loaded.length = 0; loaded.length = 0;
vi.restoreAllMocks(); vi.restoreAllMocks();
vi.unstubAllGlobals();
vi.unstubAllEnvs();
}); });
afterAll(() => { afterAll(() => {
@@ -117,3 +121,31 @@ describe("bin/quak.ts get and get-thumb", () => {
}, },
); );
}); });
describe("bin/quak.ts login", () => {
// The command refuses an address that is not a URL before it asks for
// anything or sends any request.
it("hands --server to the command", async () => {
expect(await quak("login", "--server", "ente.example.com")).toMatch(
/^Invalid server address "ente\.example\.com"/,
);
});
// `fetch` answers 404, so the login fails at its first request. A login
// that succeeded would save its session in this user's real data
// directory, since env-paths is not mocked here.
it("sends its first request to the server --server names", async () => {
const requests: string[] = [];
vi.stubGlobal("fetch", async (input: string) => {
const { origin, pathname } = new URL(input);
requests.push(origin + pathname);
return new Response("not found", { status: 404 });
});
vi.stubEnv("QUAK_EMAIL", "login@example.com");
vi.stubEnv("QUAK_PASSWORD", "password");
await quak("login", "--server", "https://ente.example.com");
expect(requests).toEqual([
"https://ente.example.com/users/srp/attributes",
]);
});
});
+261 -13
View File
@@ -328,6 +328,7 @@ describe("login", () => {
prompts = []; prompts = [];
vi.stubEnv("QUAK_EMAIL", "cli@example.com"); vi.stubEnv("QUAK_EMAIL", "cli@example.com");
vi.stubEnv("QUAK_PASSWORD", "hunter2"); vi.stubEnv("QUAK_PASSWORD", "hunter2");
vi.stubEnv("QUAK_SERVER", undefined);
}); });
afterEach(() => { afterEach(() => {
@@ -340,12 +341,14 @@ describe("login", () => {
calls.push(opts); calls.push(opts);
return loggedIn; return loggedIn;
}); });
expect(await loginCommand(ctx)).toBe(0); expect(await loginCommand(ctx, {})).toBe(0);
expect(prompts).toEqual([]); expect(prompts).toEqual([]);
expect(calls).toHaveLength(1); expect(calls).toHaveLength(1);
expect(calls[0]!.email).toBe("cli@example.com"); expect(calls[0]!.email).toBe("cli@example.com");
expect(calls[0]!.password).toBe("hunter2"); expect(calls[0]!.password).toBe("hunter2");
// No server named: the client uses Ente's hosted service.
expect(calls[0]!.apiOptions).toBeUndefined();
const path = join(ctx.sessionDir, "session.json"); const path = join(ctx.sessionDir, "session.json");
expect(stderr.text).toBe( expect(stderr.text).toBe(
"Authenticating...\n" + "Authenticating...\n" +
@@ -356,7 +359,7 @@ describe("login", () => {
it("saves the session with mode 0600 in a directory with mode 0700", async () => { it("saves the session with mode 0600 in a directory with mode 0700", async () => {
const ctx = loginContext(async () => loggedIn); const ctx = loginContext(async () => loggedIn);
expect(await loginCommand(ctx)).toBe(0); expect(await loginCommand(ctx, {})).toBe(0);
expect(statSync(ctx.sessionDir).mode & 0o777).toBe(0o700); expect(statSync(ctx.sessionDir).mode & 0o777).toBe(0o700);
const path = join(ctx.sessionDir, "session.json"); const path = join(ctx.sessionDir, "session.json");
@@ -370,7 +373,7 @@ describe("login", () => {
code = await opts.totp!(); code = await opts.totp!();
return loggedIn; return loggedIn;
}); });
expect(await loginCommand(ctx)).toBe(0); expect(await loginCommand(ctx, {})).toBe(0);
expect(prompts).toEqual(["TOTP code: "]); expect(prompts).toEqual(["TOTP code: "]);
expect(code).toBe("123456"); expect(code).toBe("123456");
@@ -380,13 +383,83 @@ describe("login", () => {
const ctx = loginContext(async () => { const ctx = loginContext(async () => {
throw new Error("HTTP 401 from server"); throw new Error("HTTP 401 from server");
}); });
expect(await loginCommand(ctx)).toBe(1); expect(await loginCommand(ctx, {})).toBe(1);
expect(stderr.text).toBe( expect(stderr.text).toBe(
"Authenticating...\nLogin failed: HTTP 401 from server\n", "Authenticating...\nLogin failed: HTTP 401 from server\n",
); );
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false); expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
}); });
it("logs in to the server --server names, ahead of QUAK_SERVER", async () => {
vi.stubEnv("QUAK_SERVER", "https://other.example.com");
const calls: LoginOptions[] = [];
const ctx = loginContext(async (opts) => {
calls.push(opts);
return loggedIn;
});
expect(
await loginCommand(ctx, { server: "https://ente.example.com" }),
).toBe(0);
expect(calls[0]!.apiOptions).toEqual({
apiOrigin: "https://ente.example.com",
});
});
it("logs in to the server QUAK_SERVER names without --server", async () => {
vi.stubEnv("QUAK_SERVER", "http://192.0.2.1:8080/ente");
const calls: LoginOptions[] = [];
const ctx = loginContext(async (opts) => {
calls.push(opts);
return loggedIn;
});
expect(await loginCommand(ctx, {})).toBe(0);
expect(calls[0]!.apiOptions).toEqual({
apiOrigin: "http://192.0.2.1:8080/ente",
});
});
it.each([
"",
"ente.example.com",
"ftp://ente.example.com",
"https://ente.example.com/?region=eu",
"https://ente.example.com/#top",
])(
"refuses the server address %j before asking for anything",
async (server) => {
const ctx = loginContext(async () => {
throw new Error("login must not be called");
});
expect(await loginCommand(ctx, { server })).toBe(1);
expect(prompts).toEqual([]);
expect(stderr.text).toBe(
`Invalid server address "${server}": it must be an http or ` +
`https URL with no query or fragment\n`,
);
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(
false,
);
},
);
it("refuses an empty QUAK_SERVER before asking for anything", async () => {
vi.stubEnv("QUAK_SERVER", "");
const ctx = loginContext(async () => {
throw new Error("login must not be called");
});
expect(await loginCommand(ctx, {})).toBe(1);
expect(prompts).toEqual([]);
expect(stderr.text).toBe(
`Invalid server address "": it must be an http or https URL ` +
`with no query or fragment\n`,
);
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
});
}); });
// These use a real client read from the session file, over a fake API that // These use a real client read from the session file, over a fake API that
@@ -461,6 +534,16 @@ describe("logout", () => {
); );
}); });
it("names a self-hosted account's default cache directory, under its server's address", async () => {
const ctx = { ...logoutContext(200), cacheDir: undefined };
const apiOrigin = "https://ente.example.com";
saveSession(ctx.sessionDir, { ...snapshot, apiOrigin });
expect(await logoutCommand(ctx)).toBe(0);
expect(stderr.text).toContain(
`Cache directory ${defaultCacheDirectory(USER_ID, apiOrigin)} still holds decrypted data`,
);
});
it("without a session says so, calls nothing and exits 0", async () => { it("without a session says so, calls nothing and exits 0", async () => {
expect(await logoutCommand(logoutContext(200))).toBe(0); expect(await logoutCommand(logoutContext(200))).toBe(0);
expect(requests).toHaveLength(0); expect(requests).toHaveLength(0);
@@ -1618,18 +1701,19 @@ describe("an account with many albums", () => {
// encrypted as the server stores it. The original's bytes are the title. // encrypted as the server stores it. The original's bytes are the title.
// These tests fetch no thumbnail. The server records a thumbnail size, so // These tests fetch no thumbnail. The server records a thumbnail size, so
// `helper fix-missing-thumbnails` downloads the original before it skips // `helper fix-missing-thumbnails` downloads the original before it skips
// the photo as not a JPEG. // the photo as not a JPEG. `albumKey` is that album's key, by default the
const photoRow = (photo: { // one in `albumKeys`.
const photoRow = (
photo: {
id: number; id: number;
collectionID: number; collectionID: number;
title: string; title: string;
updationTime: number; updationTime: number;
}): { row: Record<string, unknown>; original: Uint8Array } => { },
albumKey = albumKeys.get(photo.collectionID)!,
): { row: Record<string, unknown>; original: Uint8Array } => {
const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen(); const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
const photoKeyBox = encryptSecretbox( const photoKeyBox = encryptSecretbox(photoKey, albumKey);
photoKey,
albumKeys.get(photo.collectionID)!,
);
const metadata = encryptStreamBlob( const metadata = encryptStreamBlob(
new TextEncoder().encode( new TextEncoder().encode(
JSON.stringify({ JSON.stringify({
@@ -1742,6 +1826,77 @@ describe("an account with many albums", () => {
}; };
}; };
// The albums in the cache's `metadata.json`. A search on an empty cache
// stores none there, only the account whose original or thumbnail it
// stored.
const cachedAlbums = (): unknown[] =>
JSON.parse(readFileSync(join(root, "cache", "metadata.json"), "utf-8"))
.collections;
// A context whose login and saved session send each request to the server
// its host names: `ente.example.com` is the fake server above, and
// `photos.example.org` a second one. Every account the fake server logs in
// has user ID 42, so the second server's account has the first one's user
// ID. Its one album holds photo 200, `other-200.jpg`; the first server
// gives ID 200 to `photo-2.jpg`.
const twoServerContext = async (): Promise<CliContext> => {
const account = await buildAccount(PASSWORD);
const albumKey = sodium.crypto_secretbox_keygen();
const albumKeyBox = encryptSecretbox(albumKey, account.masterKey);
const name = encryptSecretbox(
new TextEncoder().encode("Other album"),
albumKey,
);
const { row, original } = photoRow(
{
id: 200,
collectionID: 1,
title: "other-200.jpg",
updationTime: 1,
},
albumKey,
);
const other = buildMetaFetch({
verifier: account.verifier,
srpAttributes: account.srpAttributes,
keyAttributes: account.keyAttributes,
encryptedToken: account.encryptedToken,
collections: [
{
id: 1,
owner: { id: USER_ID },
encryptedKey: toBase64(albumKeyBox.ciphertext),
keyDecryptionNonce: toBase64(albumKeyBox.nonce),
encryptedName: toBase64(name.ciphertext),
nameDecryptionNonce: toBase64(name.nonce),
type: "album",
updationTime: 1,
},
],
filesByCollection: { 1: [row] },
encryptedMLData: {},
fileCiphertexts: { 200: original },
});
const fetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
const { host } = new URL(
input instanceof Request ? input.url : input,
);
return (host === "photos.example.org" ? other : server)(
input,
init,
);
}) as typeof globalThis.fetch;
return {
...context(),
login: (opts) =>
Client.login({
...opts,
apiOptions: { ...opts.apiOptions, fetch },
}),
loadSession: (path) => loadSession(path, { fetch }),
};
};
it("collections lists every album from one request, fetching no album's files", async () => { it("collections lists every album from one request, fetching no album's files", async () => {
const ctx = accountContext(); const ctx = accountContext();
expect(await collectionsCommand(ctx, {})).toBe(0); expect(await collectionsCommand(ctx, {})).toBe(0);
@@ -1848,7 +2003,100 @@ describe("an account with many albums", () => {
"/collections/v2/diff?collectionID=2&sinceTime=0", "/collections/v2/diff?collectionID=2&sinceTime=0",
"/?fileID=200", "/?fileID=200",
]); ]);
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false); expect(cachedAlbums()).toEqual([]);
});
// A self-hosted server serves downloads under its own address, at
// `/files/download/<id>`, so no request goes to Ente's hosted service.
it("login --server saves the server, and a later get sends every request to it", async () => {
vi.stubEnv("QUAK_EMAIL", "many@example.com");
vi.stubEnv("QUAK_PASSWORD", PASSWORD);
vi.stubEnv("QUAK_SERVER", undefined);
const hosts = new Set<string>();
const fetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
hosts.add(
new URL(input instanceof Request ? input.url : input).host,
);
return server(input, init);
}) as typeof globalThis.fetch;
const ctx: CliContext = {
...context(),
login: (opts) =>
Client.login({
...opts,
apiOptions: { ...opts.apiOptions, fetch },
}),
loadSession: (path) => loadSession(path, { fetch }),
};
const out = join(root, "photo.jpg");
try {
const opts = { server: "https://ente.example.com" };
expect(await loginCommand(ctx, opts)).toBe(0);
requests.length = 0;
expect(await getCommand(ctx, "200", { out })).toBe(0);
} finally {
vi.unstubAllEnvs();
}
const path = join(ctx.sessionDir, "session.json");
expect(JSON.parse(readFileSync(path, "utf-8")).apiOrigin).toBe(
"https://ente.example.com",
);
expect(readFileSync(out, "utf-8")).toBe("photo-2.jpg");
expect(requests.at(-1)).toBe("/files/download/200");
expect([...hosts]).toEqual(["ente.example.com"]);
});
// The two servers' accounts share the cache directory `--cache-dir` names.
it("after a login to another server whose account has the same user ID, files and get see none of the first server's albums and originals", async () => {
const ctx = await twoServerContext();
const first = join(root, "first.jpg");
const second = join(root, "second.jpg");
vi.stubEnv("QUAK_EMAIL", "many@example.com");
vi.stubEnv("QUAK_PASSWORD", PASSWORD);
vi.stubEnv("QUAK_SERVER", undefined);
try {
const opts = { server: "https://ente.example.com" };
expect(await loginCommand(ctx, opts)).toBe(0);
expect(await filesCommand(ctx, undefined, {})).toBe(0);
expect(await getCommand(ctx, "200", { out: first })).toBe(0);
const otherOpts = { server: "https://photos.example.org" };
expect(await loginCommand(ctx, otherOpts)).toBe(0);
stdout.text = "";
expect(await filesCommand(ctx, undefined, {})).toBe(0);
expect(await getCommand(ctx, "200", { out: second })).toBe(0);
} finally {
vi.unstubAllEnvs();
}
expect(stdout.text).toBe("200\timage\tother-200.jpg\n");
expect(readFileSync(first, "utf-8")).toBe("photo-2.jpg");
expect(readFileSync(second, "utf-8")).toBe("other-200.jpg");
});
// `get` on an empty cache stores the original but no album.
it("get on an empty cache, after the same on another server whose account has the same user ID, saves this server's photo", async () => {
const ctx = await twoServerContext();
const first = join(root, "first.jpg");
const second = join(root, "second.jpg");
vi.stubEnv("QUAK_EMAIL", "many@example.com");
vi.stubEnv("QUAK_PASSWORD", PASSWORD);
vi.stubEnv("QUAK_SERVER", undefined);
try {
const opts = { server: "https://ente.example.com" };
expect(await loginCommand(ctx, opts)).toBe(0);
expect(await getCommand(ctx, "200", { out: first })).toBe(0);
const otherOpts = { server: "https://photos.example.org" };
expect(await loginCommand(ctx, otherOpts)).toBe(0);
expect(await getCommand(ctx, "200", { out: second })).toBe(0);
} finally {
vi.unstubAllEnvs();
}
expect(readFileSync(first, "utf-8")).toBe("photo-2.jpg");
expect(readFileSync(second, "utf-8")).toBe("other-200.jpg");
}); });
// `files` first fills the cache, so `get` waits for a refresh. No album has // `files` first fills the cache, so `get` waits for a refresh. No album has
@@ -1906,7 +2154,7 @@ describe("an account with many albums", () => {
"/collections/v2/diff?collectionID=39&sinceTime=0", "/collections/v2/diff?collectionID=39&sinceTime=0",
"/?fileID=3900", "/?fileID=3900",
]); ]);
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false); expect(cachedAlbums()).toEqual([]);
}); });
// `files` first fills the cache, so the helper waits for a refresh. No // `files` first fills the cache, so the helper waits for a refresh. No
+57
View File
@@ -47,6 +47,35 @@ describe("Client.toJSON", () => {
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot); expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
}); });
it("round-trips a snapshot that names a server", () => {
const snapshot = {
...validSnapshot(),
apiOrigin: "https://ente.example.com",
};
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
});
it("names the server apiOptions gave fromJSON", async () => {
const apiOrigin = "https://ente.example.com";
const urls: string[] = [];
const client = Client.fromJSON(validSnapshot(), {
apiOrigin,
fetch: async (input) => {
urls.push(String(input));
return new Response(JSON.stringify({ collections: [] }), {
status: 200,
headers: { "content-type": "application/json" },
});
},
});
expect(client.toJSON().apiOrigin).toBe(apiOrigin);
await client.listCollections();
expect(urls).toEqual([
"https://ente.example.com/collections/v2?sinceTime=0",
]);
});
it("throws instead of emitting a snapshot without a token", () => { it("throws instead of emitting a snapshot without a token", () => {
const client = Client.fromJSON(validSnapshot()); const client = Client.fromJSON(validSnapshot());
client.getApiClient().clearAuthToken(); client.getApiClient().clearAuthToken();
@@ -77,6 +106,10 @@ describe("Client.fromJSON", () => {
["publicKey", undefined], ["publicKey", undefined],
["publicKey", "not base64!"], ["publicKey", "not base64!"],
["publicKey", shortKey], ["publicKey", shortKey],
["apiOrigin", 7],
["apiOrigin", ""],
["apiOrigin", "ente.example.com"],
["apiOrigin", "ftp://ente.example.com"],
])("rejects %s = %j, naming the field", (field, value) => { ])("rejects %s = %j, naming the field", (field, value) => {
const snapshot: Record<string, unknown> = { ...validSnapshot() }; const snapshot: Record<string, unknown> = { ...validSnapshot() };
snapshot[field] = value; snapshot[field] = value;
@@ -85,6 +118,30 @@ describe("Client.fromJSON", () => {
); );
}); });
// The token belongs to the snapshot's server, so that server wins over
// the one in `apiOptions`.
it("sends requests to the snapshot's server", async () => {
const urls: string[] = [];
const client = Client.fromJSON(
{ ...validSnapshot(), apiOrigin: "https://ente.example.com/" },
{
apiOrigin: "https://other.example.com",
fetch: async (input) => {
urls.push(String(input));
return new Response(JSON.stringify({ collections: [] }), {
status: 200,
headers: { "content-type": "application/json" },
});
},
},
);
await client.listCollections();
expect(urls).toEqual([
"https://ente.example.com/collections/v2?sinceTime=0",
]);
});
it.each([null, "a string", 42])("rejects a non-object %j", (value) => { it.each([null, "a string", 42])("rejects a non-object %j", (value) => {
expect(() => Client.fromJSON(value)).toThrow( expect(() => Client.fromJSON(value)).toThrow(
/^Invalid session data: not a JSON object/, /^Invalid session data: not a JSON object/,
+218 -6
View File
@@ -19,16 +19,22 @@
* refresh fails (offline start from cache). * refresh fails (offline start from cache).
* 5. `close()` stops the timer and is idempotent, and its promise resolves * 5. `close()` stops the timer and is idempotent, and its promise resolves
* only once an in-flight refresh has written the cache file. * only once an in-flight refresh has written the cache file.
* 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id. * 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id,
* 7. `open()` branches on the cache: an empty cache awaits the first refresh * with a self-hosted server's address before the user id.
* (it has nothing to serve yet); an existing cache serves its copy at once * 7. `open()` branches on the cache: a cache that holds no collection awaits
* and refreshes in the background, so a slow or dead server never stalls * the first refresh (it has nothing to serve yet); an existing cache serves
* opening. * its copy at once and refreshes in the background, so a slow or dead
* server never stalls opening.
* 8. A save failure that leaves RAM ahead of disk keeps `status().lastError` * 8. A save failure that leaves RAM ahead of disk keeps `status().lastError`
* set and keeps retrying the write; a later empty refresh does not clear it. * set and keeps retrying the write; a later empty refresh does not clear it.
* 9. With `refreshOnOpen: false`, `open()` starts no refresh at all, and a * 9. With `refreshOnOpen: false`, `open()` starts no refresh at all, and a
* refresh reports through `onProgress` each collection whose files it * refresh reports through `onProgress` each collection whose files it
* fetched. * fetched.
* 10. A cache another account filled starts empty. One filled from another
* server also loses its originals and thumbnails, since two servers can
* give two files the same ID. `thumbnailOf` on an empty cache saves the
* account's user ID and server. A refresh saves the account's server, so a
* self-hosted account keeps its own cache from one `open()` to the next.
* *
* The client is a mock: no crypto, no network. It serves scripted pages and * The client is a mock: no crypto, no network. It serves scripted pages and
* records the `sinceTime` each call carried so cursor threading is provable. * records the `sinceTime` each call carried so cursor threading is provable.
@@ -49,7 +55,13 @@
*/ */
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest"; import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
import { mkdtempSync, rmSync } from "node:fs"; import {
existsSync,
mkdirSync,
mkdtempSync,
rmSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { join } from "node:path"; import { join } from "node:path";
import envPaths from "env-paths"; import envPaths from "env-paths";
@@ -107,6 +119,7 @@ const file = (
*/ */
class MockClient { class MockClient {
userID = USER_ID; userID = USER_ID;
apiOrigin?: string;
failCollections = false; failCollections = false;
collectionsQueue: CollectionsPage[] = []; collectionsQueue: CollectionsPage[] = [];
filesByCollection = new Map<number, FilesPage[]>(); filesByCollection = new Map<number, FilesPage[]>();
@@ -584,6 +597,44 @@ describe("Library.open and background refresh", () => {
} }
}); });
// `originalOf` on an empty cache leaves such a cache behind: it names the
// account before it stores the original.
it("awaits the first refresh on a cache that names the account but holds no collection", async () => {
const seed = await MetadataStore.load(
join(cacheDirectory, "metadata.json"),
);
seed.userID = USER_ID;
await seed.save();
let releaseFirstFetch: (page: CollectionsPage) => void = () => {};
const gate = new Promise<CollectionsPage>((resolve) => {
releaseFirstFetch = resolve;
});
const client = new MockClient();
client.collectionsSince = async () => gate;
let opened = false;
const openPromise = Library.open({ client, cacheDirectory }).then(
(l) => {
opened = true;
return l;
},
);
await new Promise((r) => setTimeout(r, 20));
expect(opened).toBe(false);
releaseFirstFetch({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
const lib = await openPromise;
try {
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
} finally {
await lib.close();
}
});
// A caller that asks for each refresh it needs, as the CLI's read commands // A caller that asks for each refresh it needs, as the CLI's read commands
// do, opens without one: on an empty cache the refresh `open()` would // do, opens without one: on an empty cache the refresh `open()` would
// await fetches the files of every collection, which `quak collections` // await fetches the files of every collection, which `quak collections`
@@ -842,4 +893,165 @@ describe("Library.open and background refresh", () => {
else process.env.XDG_CACHE_HOME = prev; else process.env.XDG_CACHE_HOME = prev;
} }
}); });
it("puts a self-hosted server's address, URL-encoded, before the user id in the default cacheDirectory", async () => {
const xdg = join(dir, "xdg-cache");
const prev = process.env.XDG_CACHE_HOME;
process.env.XDG_CACHE_HOME = xdg;
try {
const client = new MockClient();
client.apiOrigin = "https://ente.example.com:8443/api";
const lib = await Library.open({ client });
try {
expect(lib.cacheDirectory).toBe(
join(
envPaths("quak", { suffix: "" }).cache,
"https%3A%2F%2Fente.example.com%3A8443%2Fapi",
String(USER_ID),
),
);
expect(lib.cacheDirectory.startsWith(xdg)).toBe(true);
} finally {
await lib.close();
}
} finally {
if (prev === undefined) delete process.env.XDG_CACHE_HOME;
else process.env.XDG_CACHE_HOME = prev;
}
});
// A cache that account USER_ID on `apiOrigin` filled: its records, and an
// original and a thumbnail of file 1001.
const fillCache = async (apiOrigin?: string): Promise<void> => {
const seed = await MetadataStore.load(
join(cacheDirectory, "metadata.json"),
);
seed.userID = USER_ID;
seed.apiOrigin = apiOrigin;
seed.collectionsSinceTime = 500;
seed.putCollection(collection(1, 400));
seed.putFile(file(1001, 1, 400));
await seed.save();
for (const name of ["originals", "thumbnails"]) {
mkdirSync(join(cacheDirectory, name));
writeFileSync(join(cacheDirectory, name, "1001.jpg"), "1001");
}
};
it("deletes the whole cache an account on another server filled", async () => {
await fillCache("https://ente.example.com");
const client = new MockClient();
client.apiOrigin = "https://photos.example.org";
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: 3600,
});
try {
expect(client.collectionsSinceTimes).toEqual([0]);
expect(lib.listCollections()).toEqual([]);
for (const name of ["originals", "thumbnails"]) {
expect(existsSync(join(cacheDirectory, name, "1001.jpg"))).toBe(
false,
);
}
} finally {
await lib.close();
}
});
// A server gives each file its own ID, so another account's cached file
// is never served for this account's.
it("keeps the originals and thumbnails another account on the same server cached", async () => {
await fillCache();
const client = new MockClient();
client.userID = USER_ID + 1;
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: 3600,
});
try {
expect(client.collectionsSinceTimes).toEqual([0]);
expect(lib.listCollections()).toEqual([]);
for (const name of ["originals", "thumbnails"]) {
expect(existsSync(join(cacheDirectory, name, "1001.jpg"))).toBe(
true,
);
}
} finally {
await lib.close();
}
});
// `quak get-thumb` on an empty cache stores a thumbnail before any refresh
// runs. `open()` for an account on another server deletes that thumbnail
// only if the cache names the account that stored it.
it("thumbnailOf on an empty cache saves the account's user ID and server in metadata.json", async () => {
const client = new MockClient();
client.apiOrigin = "https://ente.example.com";
const lib = await Library.open({
client,
cacheDirectory,
refreshOnOpen: false,
refreshIntervalSeconds: Infinity,
contentSource: {
original: async () => {
throw new Error("original not expected");
},
thumbnail: async ({ destination }) => {
writeFileSync(destination, "thumb");
return { bytesWritten: 5 };
},
},
});
try {
await lib.thumbnailOf(file(1001, 1, 400));
} finally {
await lib.close();
}
const saved = await MetadataStore.load(
join(cacheDirectory, "metadata.json"),
);
expect(saved.userID).toBe(USER_ID);
expect(saved.apiOrigin).toBe("https://ente.example.com");
});
// A refresh saves the account's server in `metadata.json`. Without it,
// `open()` would take a self-hosted account's own cache for another
// server's and delete it.
it("keeps a self-hosted account's own cache from one open to the next", async () => {
const client = new MockClient();
client.apiOrigin = "https://ente.example.com";
client.collectionsQueue.push({
collections: [collection(1, 100)],
deleted: [],
cursor: 100,
});
const first = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: Infinity,
});
await first.close();
mkdirSync(join(cacheDirectory, "originals"));
writeFileSync(join(cacheDirectory, "originals", "1001.jpg"), "1001");
const lib = await Library.open({
client,
cacheDirectory,
refreshIntervalSeconds: Infinity,
});
try {
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
expect(
existsSync(join(cacheDirectory, "originals", "1001.jpg")),
).toBe(true);
} finally {
await lib.close();
}
// The second open's refresh started from the saved cursor.
expect(client.collectionsSinceTimes).toEqual([0, 100]);
});
}); });
+241
View File
@@ -0,0 +1,241 @@
// `script/build-bin` compiles `bin/quak`, the single binary that the README's
// cron setup runs. bun compiles it from a copy of its own executable, so the
// binary loads the same libraries as the bun that built it. nix's bun loads
// libraries from the nix store, which garbage collection can delete: on Linux
// its C library, on macOS its ICU. The script therefore downloads bun's own
// release build, which loads only the system's libraries.
//
// What the script promises, and why each test matters:
//
// - Each system, machine and C library that bun publishes a build for gets its
// own archive: Linux on x86-64 or arm64, with glibc or with musl, as on
// Alpine, and macOS on Intel or Apple silicon. A bun built for another
// system, machine or C library does not start, and `make install` fails.
// - The archive's SHA-256 must match the one in the script before anything in
// the archive runs. Without that check, a tampered download would build the
// binary that cron runs.
// - Any other system or machine stops with an error, and never falls back to
// nix's bun. Building with nix's bun would bring back the dependency on the
// nix store.
//
// Each test copies the script into a fresh directory, which the script then
// treats as the checkout, and runs it there. Stubs first on PATH stand in for
// `uname`, `ldd`, `curl`, `unzip`, `nix-shell` and `shasum`, so no test
// downloads anything or runs a real bun.
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { execFileSync, spawnSync } from "node:child_process";
import {
chmodSync,
copyFileSync,
existsSync,
mkdirSync,
mkdtempSync,
readFileSync,
rmSync,
symlinkSync,
writeFileSync,
} from "node:fs";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileURLToPath } from "node:url";
const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
// The stubs, keyed by the command each replaces. `uname` and `ldd` describe the
// system that runBuildBin passes in the environment. `curl`, `unzip`,
// `nix-shell` and `shasum` write the arguments they were given, one per line,
// to a file named after the command in $STUB_LOG_DIR, which also shows that
// they ran.
const stubs: Record<string, string> = {
uname: `
case "$1" in
-s) echo "$STUB_UNAME_S" ;;
-m) echo "$STUB_UNAME_M" ;;
esac`,
// The script looks only at the start of the line: "musl libc" or not.
ldd: `
if [ "$STUB_LIBC" = musl ]; then
echo "musl libc (x86_64)"
else
echo "ldd (GNU libc) 2.36"
fi`,
// Writes a file that is not the archive to the path after -o.
curl: `
printf '%s\\n' "$@" > "$STUB_LOG_DIR/curl"
while [ $# -gt 1 ]; do
if [ "$1" = -o ]; then echo "not the archive" > "$2"; fi
shift
done`,
// Unpacks a stub bun where the real archive puts bun: in a directory named
// after the archive, inside the directory after -d. That bun writes
// bin/quak, as a real compile would.
unzip: `
printf '%s\\n' "$@" > "$STUB_LOG_DIR/unzip"
for arg in "$@"; do
case "$arg" in *.zip) archive="$arg" ;; esac
if [ "$previous" = -d ]; then directory="$arg"; fi
previous="$arg"
done
bun="$directory/$(basename "$archive" .zip)/bun"
mkdir -p "$(dirname "$bun")"
echo '#!/bin/sh' > "$bun"
echo 'mkdir -p bin && echo binary > bin/quak' >> "$bun"
chmod +x "$bun"`,
"nix-shell": `
printf '%s\\n' "$@" > "$STUB_LOG_DIR/nix-shell"`,
// The script checks the hash with `shasum` where there is no `sha256sum`.
// The stub fails, as `shasum` does when the hash does not match.
shasum: `
printf '%s\\n' "$@" > "$STUB_LOG_DIR/shasum"
exit 1`,
};
let checkout = "";
const writeScript = (path: string, body: string): void => {
writeFileSync(path, `#!/bin/sh\n${body}\n`);
chmodSync(path, 0o755);
};
// A checkout holding the script, a stub script/build that does nothing, the
// stubs, and the directory they log to.
beforeEach(() => {
checkout = mkdtempSync(join(tmpdir(), "quak-build-bin-"));
mkdirSync(join(checkout, "script"));
copyFileSync(
join(repoRoot, "script/build-bin"),
join(checkout, "script/build-bin"),
);
chmodSync(join(checkout, "script/build-bin"), 0o755);
writeScript(join(checkout, "script/build"), "exit 0");
mkdirSync(join(checkout, "stubs"));
for (const [command, body] of Object.entries(stubs)) {
writeScript(join(checkout, "stubs", command), body);
}
mkdirSync(join(checkout, "log"));
});
afterEach(() => {
rmSync(checkout, { recursive: true, force: true });
});
// Runs the script with nothing in its environment but the stubs' settings and a
// PATH. Unless the caller passes one, the PATH finds the stubs first and then
// the host's commands.
const runBuildBin = (
system: string,
machine: string,
{
libc = "glibc",
path = `${join(checkout, "stubs")}:${process.env.PATH}`,
} = {},
) =>
spawnSync(join(checkout, "script/build-bin"), {
cwd: checkout,
encoding: "utf-8",
env: {
PATH: path,
STUB_UNAME_S: system,
STUB_UNAME_M: machine,
STUB_LIBC: libc,
STUB_LOG_DIR: join(checkout, "log"),
},
});
// The real commands that the script and the stubs run.
const hostCommands = [
"basename",
"chmod",
"dirname",
"grep",
"mkdir",
"mktemp",
"rm",
];
// A Mac has no `ldd`, and not every Mac has `sha256sum`. This makes a PATH that
// finds neither: the stubs other than `ldd`, then links to hostCommands.
const makeMacPath = (): string => {
rmSync(join(checkout, "stubs", "ldd"));
mkdirSync(join(checkout, "host-commands"));
for (const command of hostCommands) {
const target = execFileSync("sh", ["-c", `command -v ${command}`], {
encoding: "utf-8",
}).trim();
symlinkSync(target, join(checkout, "host-commands", command));
}
return `${join(checkout, "stubs")}:${join(checkout, "host-commands")}`;
};
const ran = (command: string): boolean =>
existsSync(join(checkout, "log", command));
const argumentsOf = (command: string): string[] =>
readFileSync(join(checkout, "log", command), "utf-8")
.trimEnd()
.split("\n");
describe("script/build-bin", () => {
// The stub curl writes a file that is not the archive, so the hash check
// must stop the script before unzip runs. A script without the check would
// run unzip, and the stub bun it unpacks would write bin/quak. On Linux the
// check must use `sha256sum`, since a stock Alpine has no `shasum`.
it.each([
["Linux", "x86_64", "glibc", "bun-linux-x64-baseline.zip"],
["Linux", "x86_64", "musl", "bun-linux-x64-musl-baseline.zip"],
["Linux", "aarch64", "glibc", "bun-linux-aarch64.zip"],
["Linux", "aarch64", "musl", "bun-linux-aarch64-musl.zip"],
])(
"on %s %s with %s, downloads %s and stops when its hash does not match",
(system, machine, libc, archive) => {
const result = runBuildBin(system, machine, { libc });
expect(argumentsOf("curl")).toContain(
"https://github.com/oven-sh/bun/releases/download/bun-v1.3.13/" +
archive,
);
expect(ran("shasum")).toBe(false);
expect(result.status).not.toBe(0);
expect(ran("unzip")).toBe(false);
expect(ran("nix-shell")).toBe(false);
expect(existsSync(join(checkout, "bin/quak"))).toBe(false);
},
);
// On the PATH from makeMacPath the script finds no `sha256sum` and checks
// the hash with the stub `shasum`. That stub fails, so the script must stop
// before unzip runs.
// On Apple silicon `uname -m` says arm64, where bun's archive says aarch64.
it.each([
["Darwin", "x86_64", "bun-darwin-x64-baseline.zip"],
["Darwin", "arm64", "bun-darwin-aarch64.zip"],
])(
"on %s %s, downloads %s and stops when shasum rejects its hash",
(system, machine, archive) => {
const result = runBuildBin(system, machine, {
path: makeMacPath(),
});
expect(argumentsOf("curl")).toContain(
"https://github.com/oven-sh/bun/releases/download/bun-v1.3.13/" +
archive,
);
expect(argumentsOf("shasum")).toEqual(["-a", "256", "-c", "-"]);
expect(result.status).not.toBe(0);
expect(ran("unzip")).toBe(false);
expect(ran("nix-shell")).toBe(false);
expect(existsSync(join(checkout, "bin/quak"))).toBe(false);
},
);
it.each([
["Linux", "riscv64"],
["FreeBSD", "amd64"],
])(
"stops on %s %s, which bun publishes no build for",
(system, machine) => {
const result = runBuildBin(system, machine);
expect(result.status).not.toBe(0);
expect(ran("curl")).toBe(false);
expect(ran("nix-shell")).toBe(false);
},
);
});
+8 -1
View File
@@ -273,6 +273,7 @@ const buildThumbMock = async (opts?: {
const buildThumbFetch = (m: ThumbMockState) => { const buildThumbFetch = (m: ThumbMockState) => {
let srpServer: SrpServer; let srpServer: SrpServer;
let signedMD5: string | undefined;
return (async ( return (async (
input: RequestInfo | URL, input: RequestInfo | URL,
@@ -365,8 +366,9 @@ const buildThumbFetch = (m: ThumbMockState) => {
return new Response("not found", { status: 404 }); return new Response("not found", { status: 404 });
} }
// Upload URL minting // Upload URL minting. The real server signs the MD5 into the URL.
if (path === "/files/upload-url") { if (path === "/files/upload-url") {
signedMD5 = JSON.parse(init?.body as string).contentMD5;
return json({ return json({
objectKey: `42/thumb-${Date.now()}`, objectKey: `42/thumb-${Date.now()}`,
url: "https://s3.mock.test/presigned-put", url: "https://s3.mock.test/presigned-put",
@@ -375,6 +377,11 @@ const buildThumbFetch = (m: ThumbMockState) => {
// Presigned PUT (S3 upload) // Presigned PUT (S3 upload)
if (url.startsWith("https://s3.mock.test/")) { if (url.startsWith("https://s3.mock.test/")) {
// Storage refuses a PUT that lacks the signed `Content-MD5`.
const headers = new Headers(init?.headers);
if (headers.get("Content-MD5") !== signedMD5) {
return new Response("AccessDenied", { status: 400 });
}
const body = init?.body; const body = init?.body;
// Store the uploaded bytes for later inspection // Store the uploaded bytes for later inspection
if (body instanceof Uint8Array) { if (body instanceof Uint8Array) {