Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
6f8f85fff5 | ||
 clawbot
|
9bd7e8f099 | ||
|
clawbot
|
50e2381cad | ||
|
clawbot
|
37cdbc6b9a | ||
|
clawbot
|
1bf18baf00 | ||
|
clawbot
|
7878ff3812 | ||
|
clawbot
|
8e120a4f49 |
No files matched your search
@@ -26,10 +26,8 @@ check:
|
||||
build:
|
||||
@script/build
|
||||
|
||||
# Bundles the built dist/, so the binary reports the version script/build
|
||||
# stamped.
|
||||
build-bin: build
|
||||
nix-shell -p bun --run "bun build dist/bin/quak.js --compile --outfile bin/quak"
|
||||
build-bin:
|
||||
@script/build-bin
|
||||
|
||||
install: build-bin
|
||||
mkdir -p ~/bin
|
||||
|
||||
@@ -31,7 +31,8 @@ again on the next run. Every download is decrypted on its way to disk.
|
||||
|
||||
## Getting Started
|
||||
|
||||
The commands below need `git`, Node 22 or later and `yarn`.
|
||||
The commands below need `git`, `yarn` and Node 22.13 or later. Node 23 needs
|
||||
23.5 or later.
|
||||
|
||||
```bash
|
||||
git clone https://git.eeqj.de/sneak/quak.git
|
||||
@@ -69,11 +70,29 @@ The library keeps a local copy of the account's album and file lists, the cache.
|
||||
It brings the cache up to date with the server in rounds called refreshes.
|
||||
|
||||
quak is not published to a package registry: `package.json` marks it `private`,
|
||||
and `dist/` is not committed. Clone and build it as above, then add the clone to
|
||||
your own program's `package.json` by its path, for example
|
||||
`yarn add file:../quak`. The package's `main`, `types` and `exports` fields
|
||||
point into `dist/`, so the clone must be built first. The package is an ES
|
||||
module only, so a CommonJS program cannot `require` it.
|
||||
and `dist/` is not committed. Clone and build it as above, then link the clone
|
||||
into your own program:
|
||||
|
||||
```bash
|
||||
# In the quak clone:
|
||||
yarn link
|
||||
|
||||
# In your program's directory:
|
||||
yarn link quak
|
||||
```
|
||||
|
||||
`yarn link` also installs the `quak` command in yarn's global bin directory, and
|
||||
`yarn unlink` in the clone removes the link and the command.
|
||||
|
||||
A linked quak uses the dependencies that `yarn install` put in the clone, at the
|
||||
versions in quak's `yarn.lock`. Avoid `yarn add file:../quak`: it resolves
|
||||
quak's dependencies again, and their newer releases can need a newer Node.
|
||||
`yarn link quak` does not add quak to your `package.json`, so run it again in
|
||||
each new checkout of your program.
|
||||
|
||||
The package's `main`, `types` and `exports` fields point into `dist/`, so the
|
||||
clone must be built first. The package is an ES module only, so a CommonJS
|
||||
program cannot `require` it.
|
||||
|
||||
```ts
|
||||
import { Client, Library } from "quak";
|
||||
@@ -135,10 +154,15 @@ requests. The "Library reference" below covers both.
|
||||
## Running the backup from cron
|
||||
|
||||
`quak backup` never prompts, so cron can run it. `make install` builds quak as
|
||||
one binary and copies it to `~/bin/quak`. Besides what Getting Started needs,
|
||||
`make install` needs `make` and [nix](https://nixos.org/download). It uses bun,
|
||||
a JavaScript runtime, to bundle the built `dist/` into one executable. It runs
|
||||
bun through `nix-shell -p bun`.
|
||||
one binary and copies it to `~/bin/quak`. It uses bun, a JavaScript runtime, to
|
||||
bundle the built `dist/` into one executable. Besides what Getting Started
|
||||
needs, `make install` needs `make`. It downloads bun from GitHub with `curl` and
|
||||
unpacks it with `unzip`. It checks the download against a SHA-256 hash in the
|
||||
repo. `make install` works on Linux and macOS, on x86-64 and arm64.
|
||||
|
||||
On Linux systems that use the musl C library, such as Alpine, `make install`
|
||||
downloads bun's musl build. On Alpine, both that bun and `~/bin/quak` need the
|
||||
`libstdc++` and `libgcc` packages.
|
||||
|
||||
Log in once with `~/bin/quak`, as the user the cron job will run as:
|
||||
|
||||
@@ -205,7 +229,7 @@ below belong to quak itself, and go before or after the command's name.
|
||||
### The commands
|
||||
|
||||
```
|
||||
quak login
|
||||
quak login [--server <url>]
|
||||
quak whoami
|
||||
quak logout
|
||||
quak collections [--json]
|
||||
@@ -222,13 +246,26 @@ Every command but `login` and `logout` needs a saved session. Every command but
|
||||
`login`, `whoami` and `logout` opens the library. How each one gets current data
|
||||
from the server is described under "How a command gets current data".
|
||||
|
||||
#### `quak login`
|
||||
#### `quak login [--server <url>]`
|
||||
|
||||
Logs in to an account and saves the session. It reads the email from
|
||||
`QUAK_EMAIL` and the password from `QUAK_PASSWORD`. For each one that is not
|
||||
set, it asks on the terminal. The password shows as `*` characters while it is
|
||||
typed.
|
||||
|
||||
quak logs in to Ente's hosted service unless you name a self-hosted Ente server.
|
||||
Name it with `--server`, or with `QUAK_SERVER` when `--server` is not given.
|
||||
Give the address of the server's API, for example
|
||||
`quak login --server https://ente.example.com`. The address is an http or https
|
||||
URL. It may include a path, but no query or fragment. quak refuses any other
|
||||
address before it asks for anything. It logs
|
||||
`Invalid server address "<url>": it must be an http or https URL with no query or fragment`,
|
||||
and the exit status is 1.
|
||||
|
||||
The session file keeps the server's address. Every later command, cron runs
|
||||
included, uses that server without further setting. To change server, log in
|
||||
again.
|
||||
|
||||
When the account has a second factor, quak asks for the code. It asks
|
||||
`TOTP code: ` for a time-based code from an authenticator app. An account whose
|
||||
only second factor is a passkey cannot log in with quak. When the account has
|
||||
@@ -302,7 +339,7 @@ title, with unsafe characters replaced (see "File names from the server"). With
|
||||
original is kept in the cache as well.
|
||||
|
||||
A file ID that no collection holds gives `File <id> not found`, and the exit
|
||||
status is 1. The option `--collection <id>` is accepted and ignored.
|
||||
status is 1.
|
||||
|
||||
A live photo is a still image and a short video that Ente stores as one file.
|
||||
`get` writes both, each named after the title with its own extension, as Ente's
|
||||
@@ -573,9 +610,11 @@ Linux that is `$XDG_DATA_HOME/quak/session.json`, or
|
||||
mode `0600`, in a directory it creates with mode `0700`.
|
||||
|
||||
The file holds the account's email, its user ID, the session token and the
|
||||
account's three keys, in base64 and unencrypted. Treat it as you would treat the
|
||||
password. A missing file means "not logged in". A file that cannot be read back
|
||||
is reported as corrupt, with the reason. Run `quak logout`, then `quak login`.
|
||||
account's three keys, in base64 and unencrypted. After a login to a self-hosted
|
||||
server, it also holds the server's address as `apiOrigin`. Treat it as you would
|
||||
treat the password. A missing file means "not logged in". A file that cannot be
|
||||
read back is reported as corrupt, with the reason. Run `quak logout`, then
|
||||
`quak login`.
|
||||
|
||||
A command that needs a session exits 3 when the file is missing or corrupt. It
|
||||
also exits 3 when the server no longer accepts the session, and prints
|
||||
@@ -593,6 +632,11 @@ Without `--cache-dir`, each account has its own cache directory:
|
||||
the variable is not set;
|
||||
- on macOS, `~/Library/Caches/quak/<userID>`.
|
||||
|
||||
Two servers can give two accounts the same user ID. So an account on a
|
||||
self-hosted server has the server's address, URL-encoded, between `quak` and the
|
||||
user ID. For `https://ente.example.com`, that is
|
||||
`quak/https%3A%2F%2Fente.example.com/<userID>`.
|
||||
|
||||
The directory is readable by its owner only (mode `0700`), because it holds
|
||||
decrypted data, file keys included.
|
||||
|
||||
@@ -617,6 +661,13 @@ the library deletes it and `mldata/` and starts from an empty cache. Cached
|
||||
originals and thumbnails are kept. They are reached only through the records of
|
||||
the current account, so another account's files are never served.
|
||||
|
||||
When `metadata.json` belongs to an account on another server, opening the
|
||||
library also deletes the originals and thumbnails. Two servers can give two
|
||||
files the same ID, so one server's cached original could be served for the
|
||||
other's file. quak records the account in `metadata.json` before it stores any
|
||||
original or thumbnail. It does so also when `quak get` finds a file on an empty
|
||||
cache.
|
||||
|
||||
A file appears in the cache only once it is complete: each one is written to a
|
||||
temporary file and renamed into place. Ente's clients record a hash of each
|
||||
file's bytes at upload, the content hash. Every original whose metadata records
|
||||
@@ -1020,7 +1071,8 @@ onto what the cache holds, empty or not. A refresh then starts only when
|
||||
- `client`, required: the account's `Client`, or anything that satisfies
|
||||
`LibraryClient`.
|
||||
- `cacheDirectory`: where `metadata.json` and the content cache live. Default:
|
||||
`quak/<userID>` under the per-user cache directory (see "The cache").
|
||||
`quak/<userID>` under the per-user cache directory, with the URL-encoded
|
||||
`apiOrigin` before `<userID>` for a client that has one (see "The cache").
|
||||
- `downloadDirectory`: the root of the save paths. Default: `photos` in the
|
||||
working directory, resolved once when the library opens;
|
||||
`lib.downloadDirectory` holds the result. An original stored at its save path
|
||||
@@ -1217,7 +1269,8 @@ do.
|
||||
fetch and cache as `photo.original()` and `photo.thumbnail()` do. `file` is an
|
||||
`EnteFile` the caller holds, and the library need not hold it. `quak get` and
|
||||
`quak get-thumb` use them; on an empty cache they pass a file that
|
||||
`lib.freshFiles()` returned.
|
||||
`lib.freshFiles()` returned. In a cache that names no account yet, they first
|
||||
save the client's user ID and `apiOrigin` in `metadata.json`.
|
||||
|
||||
Lower-level reads return the decrypted model objects, which hold key material:
|
||||
`lib.listCollections()`, `lib.getCollection(id)`, `lib.listFiles(collectionID)`,
|
||||
@@ -1453,6 +1506,14 @@ write wherever it likes. `Client.fromJSON(snapshot, apiOptions?)` restores a
|
||||
working client from that snapshot without logging in again. It checks every
|
||||
field and each key's length first, and throws an error naming the bad field.
|
||||
|
||||
A client whose `apiOptions` named an `apiOrigin` puts it in its snapshot. The
|
||||
restored client then uses that server, even when the `apiOptions` given to
|
||||
`Client.fromJSON` name another. A snapshot without `apiOrigin` uses the
|
||||
`apiOrigin` of those `apiOptions`, or the default server. `client.apiOrigin`
|
||||
holds the one in use, or undefined when none was named. `Client.fromJSON`
|
||||
refuses an `apiOrigin` that is not an http or https URL, or that has a query or
|
||||
fragment, as `quak login --server` does.
|
||||
|
||||
`Client.fromJSON` works only once libsodium, the cryptography library quak uses,
|
||||
is loaded. `Client.login` loads it, and `Client.fromJSON` does not. A program
|
||||
that restores a session in a new process first runs `await sodium.ready`, with
|
||||
@@ -1774,6 +1835,8 @@ standard. The others are additions to it.
|
||||
`package.json` declares (`main`, `types`, `bin`) were written, and make the
|
||||
command line executable.
|
||||
- `script/version`: print the version `script/build` stamps (see "Version").
|
||||
- `script/build-bin`: run `script/build`, then bundle the built `dist/` into one
|
||||
binary, `bin/quak`, with bun (see "Running the backup from cron").
|
||||
- `script/test`: run the test suite, by building the `test` phase of the
|
||||
`Dockerfile`. Requires Docker.
|
||||
- `script/lint`: run eslint and a prettier check, by building the `lint` phase
|
||||
@@ -1802,8 +1865,7 @@ The Makefile targets are:
|
||||
- `make build`: runs `script/build`.
|
||||
- `make docker`: runs `script/docker`.
|
||||
- `make hooks`: runs `script/install-precommit`.
|
||||
- `make build-bin`: bundles the built `dist/` into one binary, `bin/quak`. It
|
||||
runs bun through `nix-shell -p bun`.
|
||||
- `make build-bin`: runs `script/build-bin`.
|
||||
- `make install`: runs `make build-bin` and copies the binary to `~/bin/quak`.
|
||||
- `make dev`: runs the TypeScript compiler in watch mode.
|
||||
- `make clean`: removes `dist/`, `coverage/`, `.vitest-cache/` and the
|
||||
@@ -1911,9 +1973,9 @@ These apply to people and to coding agents alike.
|
||||
- container images by digest;
|
||||
- the nvm archive `script/bootstrap` downloads, and the Python packages
|
||||
`bin/clip-keyword-vectors` installs, by SHA-256 hash;
|
||||
- the bun builds `script/build-bin` downloads, by SHA-256 hash;
|
||||
- Node and yarn, when `script/bootstrap` installs them, by version.
|
||||
- Not pinned:
|
||||
- bun, which `make build-bin` takes from `nix-shell -p bun`;
|
||||
- `make` and `git`, which `script/bootstrap` installs from the system
|
||||
package manager when they are missing, as every `Dockerfile` stage does;
|
||||
- `bash` and `curl`, which `script/bootstrap` installs the same way when it
|
||||
|
||||
+11
-4
@@ -74,7 +74,11 @@ const context = (): CliContext => ({
|
||||
sessionDir: paths.data,
|
||||
cacheDir: program.opts<ProgramOptions>().cacheDir,
|
||||
loadSession: (path) => loadSession(path, logRequests(log())),
|
||||
login: (opts) => Client.login({ ...opts, apiOptions: logRequests(log()) }),
|
||||
login: (opts) =>
|
||||
Client.login({
|
||||
...opts,
|
||||
apiOptions: { ...opts.apiOptions, ...logRequests(log()) },
|
||||
}),
|
||||
prompt: (message) => input({ message }),
|
||||
promptSecret: (message) => password({ message, mask: true }),
|
||||
});
|
||||
@@ -85,7 +89,12 @@ const run = (command: Promise<number>): Promise<void> =>
|
||||
program
|
||||
.command("login")
|
||||
.description("Log in to an Ente account and save the session")
|
||||
.action(() => run(loginCommand(context())));
|
||||
.option(
|
||||
"--server <url>",
|
||||
"Address of a self-hosted Ente server's API, saved with the session " +
|
||||
"(default: QUAK_SERVER, then Ente's hosted service)",
|
||||
)
|
||||
.action((opts: { server?: string }) => run(loginCommand(context(), opts)));
|
||||
|
||||
program
|
||||
.command("whoami")
|
||||
@@ -123,7 +132,6 @@ program
|
||||
.description("Download and decrypt a single file")
|
||||
.argument("<fileID>", "File ID (from `quak files`)")
|
||||
.option("--out <path>", "Output file path")
|
||||
.option("--collection <id>", "Accepted for compatibility; ignored")
|
||||
.action((fileID: string, opts: { out?: string }) =>
|
||||
run(getCommand(context(), fileID, opts)),
|
||||
);
|
||||
@@ -133,7 +141,6 @@ program
|
||||
.description("Download and decrypt a thumbnail")
|
||||
.argument("<fileID>", "File ID (from `quak files`)")
|
||||
.option("--out <path>", "Output file path")
|
||||
.option("--collection <id>", "Accepted for compatibility; ignored")
|
||||
.action((fileID: string, opts: { out?: string }) =>
|
||||
run(getThumbCommand(context(), fileID, opts)),
|
||||
);
|
||||
|
||||
+1
-1
@@ -12,7 +12,7 @@
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"engines": {
|
||||
"node": ">=22"
|
||||
"node": "^22.13.0 || >=23.5.0"
|
||||
},
|
||||
"main": "./dist/src/index.js",
|
||||
"types": "./dist/src/index.d.ts",
|
||||
|
||||
Executable
+85
@@ -0,0 +1,85 @@
|
||||
#!/bin/sh
|
||||
# script/build-bin: run script/build, then bundle the built dist/ into one
|
||||
# executable, bin/quak, which reports the version script/build stamped. Our own
|
||||
# extension to scripts-to-rule-them-all.
|
||||
#
|
||||
# bun compiles the binary from a copy of its own executable, so the binary
|
||||
# loads the same libraries as the bun that built it. nix's bun loads libraries
|
||||
# from the nix store: on Linux its C library, on macOS its ICU. A binary
|
||||
# compiled from it loads them too. nix-collect-garbage can delete them, and the
|
||||
# binary then no longer starts. This script therefore compiles with bun's own
|
||||
# release build, which loads only the system's libraries. It downloads that
|
||||
# build and checks it against the hash below before running it.
|
||||
set -eu
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
||||
|
||||
# bun 1.3.13, 2026-11-11, and the SHA-256 of each release archive below. The
|
||||
# x86-64 archives are the baseline builds, which also run on CPUs without AVX2.
|
||||
# The musl archives are for Linux systems that use the musl C library, such as
|
||||
# Alpine.
|
||||
BUN_VERSION="1.3.13"
|
||||
BUN_LINUX_X64_BASELINE_SHA256="9d8a24292a7068090205daac0a5a223f5f69736f5287e37bf88d3b4031edc750"
|
||||
BUN_LINUX_X64_MUSL_BASELINE_SHA256="88ca7c7ad235b498f549eea2f770f434e9f0f5e9ba95168a2d3a1f235184c394"
|
||||
BUN_LINUX_AARCH64_SHA256="70bae41b3908b0a120e1e58c5c8af30e74afae3b8d11b0d3fdd8e787ddfb4b22"
|
||||
BUN_LINUX_AARCH64_MUSL_SHA256="5385e978107ce4934298d8d6afe9bfbb898683f6cc23e6753a0da60bc60c5b81"
|
||||
BUN_DARWIN_X64_BASELINE_SHA256="a98ba6a480f22fda9b343626b906a4e26aa53618bf85d2bc5928ecf2ba45f0ed"
|
||||
BUN_DARWIN_AARCH64_SHA256="5467e3f65dba526b9fea98f0cce04efafc0c63e169733ec27b876a3ad32da190"
|
||||
|
||||
# compile_with_downloaded_bun <archive-name> <sha256>
|
||||
compile_with_downloaded_bun() {
|
||||
tmp="$(mktemp -d)"
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
curl -fsSL -o "$tmp/$1.zip" \
|
||||
"https://github.com/oven-sh/bun/releases/download/bun-v$BUN_VERSION/$1.zip"
|
||||
# macOS always has shasum, but not every macOS has sha256sum.
|
||||
if command -v sha256sum >/dev/null 2>&1; then
|
||||
echo "$2 $tmp/$1.zip" | sha256sum -c -
|
||||
else
|
||||
echo "$2 $tmp/$1.zip" | shasum -a 256 -c -
|
||||
fi
|
||||
unzip -q "$tmp/$1.zip" -d "$tmp"
|
||||
"$tmp/$1/bun" build dist/bin/quak.js --compile --outfile bin/quak
|
||||
}
|
||||
|
||||
main() {
|
||||
cd "$ROOT"
|
||||
"$ROOT/script/build"
|
||||
platform="$(uname -s)-$(uname -m)"
|
||||
# musl's ldd begins its version output with "musl libc"; glibc's does not.
|
||||
if ldd --version 2>&1 | grep -q '^musl libc'; then
|
||||
platform="$platform-musl"
|
||||
fi
|
||||
case "$platform" in
|
||||
Linux-x86_64)
|
||||
compile_with_downloaded_bun bun-linux-x64-baseline \
|
||||
"$BUN_LINUX_X64_BASELINE_SHA256"
|
||||
;;
|
||||
Linux-x86_64-musl)
|
||||
compile_with_downloaded_bun bun-linux-x64-musl-baseline \
|
||||
"$BUN_LINUX_X64_MUSL_BASELINE_SHA256"
|
||||
;;
|
||||
Linux-aarch64)
|
||||
compile_with_downloaded_bun bun-linux-aarch64 \
|
||||
"$BUN_LINUX_AARCH64_SHA256"
|
||||
;;
|
||||
Linux-aarch64-musl)
|
||||
compile_with_downloaded_bun bun-linux-aarch64-musl \
|
||||
"$BUN_LINUX_AARCH64_MUSL_SHA256"
|
||||
;;
|
||||
Darwin-x86_64)
|
||||
compile_with_downloaded_bun bun-darwin-x64-baseline \
|
||||
"$BUN_DARWIN_X64_BASELINE_SHA256"
|
||||
;;
|
||||
Darwin-arm64)
|
||||
compile_with_downloaded_bun bun-darwin-aarch64 \
|
||||
"$BUN_DARWIN_AARCH64_SHA256"
|
||||
;;
|
||||
*)
|
||||
echo "build-bin: bun has no release build for $(uname -s) on $(uname -m)" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
main "$@"
|
||||
+8
-1
@@ -354,7 +354,13 @@ export class ApiClient {
|
||||
});
|
||||
}
|
||||
|
||||
async putFile(presignedURL: string, data: Uint8Array): Promise<void> {
|
||||
// `contentMD5` is the value given to `getUploadURL`. The server signs it
|
||||
// into the URL, and storage refuses a PUT without that `Content-MD5`.
|
||||
async putFile(
|
||||
presignedURL: string,
|
||||
data: Uint8Array,
|
||||
contentMD5: string,
|
||||
): Promise<void> {
|
||||
// Idempotent despite being a write: a presigned PUT stores one whole
|
||||
// object at one key in one request, so replaying it either overwrites
|
||||
// the same bytes or lands them for the first time. There is no partial
|
||||
@@ -370,6 +376,7 @@ export class ApiClient {
|
||||
headers: {
|
||||
"Content-Type": "application/octet-stream",
|
||||
"Content-Length": String(data.length),
|
||||
"Content-MD5": contentMD5,
|
||||
},
|
||||
body: data,
|
||||
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
||||
|
||||
+25
-3
@@ -21,6 +21,7 @@ import { extname, join } from "node:path";
|
||||
import {
|
||||
type Client,
|
||||
type ClientSnapshot,
|
||||
isServerAddress,
|
||||
type LoginOptions,
|
||||
} from "./client.js";
|
||||
import { lockBackupDirectory, type BackupLock } from "./backup.js";
|
||||
@@ -56,7 +57,7 @@ export interface CliContext {
|
||||
// Directory holding `session.json`.
|
||||
sessionDir: string;
|
||||
// The `--cache-dir` global, or undefined to let the library pick its
|
||||
// per-user default keyed by the account id.
|
||||
// per-user default keyed by the account's server and user ID.
|
||||
cacheDir?: string;
|
||||
// Reads the session file into a client, or null when there is none. The
|
||||
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
|
||||
@@ -129,6 +130,7 @@ const parseID = (
|
||||
// own terms. The content source is kept so `get`/`get-thumb`/`--exif` can fetch
|
||||
// originals through the on-disk cache.
|
||||
const readLibraryClient = (client: Client): LibraryClient => ({
|
||||
apiOrigin: client.apiOrigin,
|
||||
whoami: () => client.whoami(),
|
||||
collectionsSince: (args) => client.collectionsSince(args),
|
||||
filesSince: (args) => client.filesSince(args),
|
||||
@@ -155,8 +157,23 @@ const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
|
||||
onProgress: logRefresh(ctx.log),
|
||||
});
|
||||
|
||||
export const loginCommand = async (ctx: CliContext): Promise<number> => {
|
||||
// Logs in to the server `--server` or `QUAK_SERVER` names, the flag first, or
|
||||
// to Ente's hosted service when neither is set. The saved session keeps the
|
||||
// server, so every later command uses it.
|
||||
export const loginCommand = async (
|
||||
ctx: CliContext,
|
||||
opts: { server?: string },
|
||||
): Promise<number> => {
|
||||
await init();
|
||||
const server = opts.server ?? process.env.QUAK_SERVER;
|
||||
if (server !== undefined && !isServerAddress(server)) {
|
||||
ctx.log(
|
||||
"fatal",
|
||||
`Invalid server address "${server}": it must be an http or ` +
|
||||
`https URL with no query or fragment`,
|
||||
);
|
||||
return 1;
|
||||
}
|
||||
const email = process.env.QUAK_EMAIL ?? (await ctx.prompt("Email"));
|
||||
const password =
|
||||
process.env.QUAK_PASSWORD ?? (await ctx.promptSecret("Password"));
|
||||
@@ -168,6 +185,8 @@ export const loginCommand = async (ctx: CliContext): Promise<number> => {
|
||||
password,
|
||||
totp: async () => ctx.prompt("TOTP code: "),
|
||||
emailOTP: async () => ctx.prompt("Email verification code: "),
|
||||
apiOptions:
|
||||
server === undefined ? undefined : { apiOrigin: server },
|
||||
});
|
||||
|
||||
saveSession(ctx.sessionDir, client.toJSON());
|
||||
@@ -208,7 +227,10 @@ export const logoutCommand = async (ctx: CliContext): Promise<number> => {
|
||||
try {
|
||||
const client = ctx.loadSession(path);
|
||||
if (client) {
|
||||
cacheDir ??= defaultCacheDirectory(client.whoami().userID);
|
||||
cacheDir ??= defaultCacheDirectory(
|
||||
client.whoami().userID,
|
||||
client.apiOrigin,
|
||||
);
|
||||
await client.logoutOnServer();
|
||||
client.logout();
|
||||
}
|
||||
|
||||
+37
-1
@@ -40,6 +40,9 @@ export interface ClientSnapshot {
|
||||
masterKey: string;
|
||||
secretKey: string;
|
||||
publicKey: string;
|
||||
// The `apiOrigin` the client was given, so a restored client talks to
|
||||
// the server that issued its token. Absent means the default server.
|
||||
apiOrigin?: string;
|
||||
}
|
||||
|
||||
// The result of a resumable enumeration. Live decrypted records and deleted
|
||||
@@ -66,7 +69,19 @@ export interface FilesPage {
|
||||
const errorMessage = (err: unknown): string =>
|
||||
err instanceof Error ? err.message : String(err);
|
||||
|
||||
// Whether `text` can be an Ente server's address: an http or https URL, which
|
||||
// may have a path, and no query or fragment, which request URLs could not
|
||||
// carry. `quak login --server` and `Client.fromJSON` both check with it.
|
||||
export const isServerAddress = (text: string): boolean =>
|
||||
URL.canParse(text) &&
|
||||
["http:", "https:"].includes(new URL(text).protocol) &&
|
||||
!/[?#]/.test(text);
|
||||
|
||||
export class Client {
|
||||
// The `apiOrigin` the client was given, or undefined when it was given
|
||||
// none, which means Ente's hosted service. The library keeps a cache per
|
||||
// server as well as per user ID.
|
||||
readonly apiOrigin: string | undefined;
|
||||
private readonly api: ApiClient;
|
||||
private readonly email: string;
|
||||
private readonly userID: number;
|
||||
@@ -82,6 +97,7 @@ export class Client {
|
||||
masterKey: Uint8Array,
|
||||
secretKey: Uint8Array,
|
||||
publicKey: Uint8Array,
|
||||
apiOrigin: string | undefined,
|
||||
) {
|
||||
this.api = api;
|
||||
this.email = email;
|
||||
@@ -89,6 +105,7 @@ export class Client {
|
||||
this.masterKey = masterKey;
|
||||
this.secretKey = secretKey;
|
||||
this.publicKey = publicKey;
|
||||
this.apiOrigin = apiOrigin;
|
||||
}
|
||||
|
||||
static async login(opts: LoginOptions): Promise<Client> {
|
||||
@@ -130,12 +147,15 @@ export class Client {
|
||||
unwrapped.masterKey,
|
||||
unwrapped.secretKey,
|
||||
unwrapped.publicKey,
|
||||
opts.apiOptions?.apiOrigin,
|
||||
);
|
||||
}
|
||||
|
||||
// Restore a client from a `toJSON()` snapshot. The snapshot usually comes
|
||||
// straight from `JSON.parse` of a file on disk, so every field is checked
|
||||
// before use; a bad one throws an error naming it. Needs `init()` first.
|
||||
// before use; a bad one throws an error naming it. The snapshot's
|
||||
// `apiOrigin`, when it has one, is used instead of `apiOptions.apiOrigin`.
|
||||
// Needs `init()` first.
|
||||
static fromJSON(snapshot: unknown, apiOptions?: ApiClientOptions): Client {
|
||||
const invalid = (field: string, problem: string): Error =>
|
||||
new Error(`Invalid session data: ${field} ${problem}`);
|
||||
@@ -152,6 +172,17 @@ export class Client {
|
||||
if (!Number.isInteger(s.userID)) {
|
||||
throw invalid("userID", "must be an integer");
|
||||
}
|
||||
if (
|
||||
s.apiOrigin !== undefined &&
|
||||
(typeof s.apiOrigin !== "string" || !isServerAddress(s.apiOrigin))
|
||||
) {
|
||||
throw invalid(
|
||||
"apiOrigin",
|
||||
"must be an http or https URL with no query or fragment",
|
||||
);
|
||||
}
|
||||
const apiOrigin =
|
||||
(s.apiOrigin as string | undefined) ?? apiOptions?.apiOrigin;
|
||||
const key = (field: string): Uint8Array => {
|
||||
const value = s[field];
|
||||
if (typeof value !== "string") {
|
||||
@@ -175,6 +206,7 @@ export class Client {
|
||||
|
||||
const api = new ApiClient({
|
||||
...apiOptions,
|
||||
apiOrigin,
|
||||
authToken: s.token as string,
|
||||
});
|
||||
return new Client(
|
||||
@@ -184,6 +216,7 @@ export class Client {
|
||||
key("masterKey"),
|
||||
key("secretKey"),
|
||||
key("publicKey"),
|
||||
apiOrigin,
|
||||
);
|
||||
}
|
||||
|
||||
@@ -222,6 +255,9 @@ export class Client {
|
||||
masterKey: toBase64(this.masterKey),
|
||||
secretKey: toBase64(this.secretKey),
|
||||
publicKey: toBase64(this.publicKey),
|
||||
...(this.apiOrigin === undefined
|
||||
? {}
|
||||
: { apiOrigin: this.apiOrigin }),
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
+65
-17
@@ -119,9 +119,18 @@ export {
|
||||
export const DEFAULT_REFRESH_INTERVAL_SECONDS = 3;
|
||||
|
||||
// The account's cache directory when `cacheDirectory` is not given: the
|
||||
// env-paths cache directory plus the user id, so each account has its own.
|
||||
export const defaultCacheDirectory = (userID: number): string =>
|
||||
join(envPaths("quak", { suffix: "" }).cache, String(userID));
|
||||
// env-paths cache directory plus the user id, so each account has its own. An
|
||||
// account on a self-hosted server has the server's address, URL-encoded, in
|
||||
// between, because two servers can give two accounts the same user id.
|
||||
export const defaultCacheDirectory = (
|
||||
userID: number,
|
||||
apiOrigin?: string,
|
||||
): string => {
|
||||
const cache = envPaths("quak", { suffix: "" }).cache;
|
||||
return apiOrigin === undefined
|
||||
? join(cache, String(userID))
|
||||
: join(cache, encodeURIComponent(apiOrigin), String(userID));
|
||||
};
|
||||
|
||||
// Project a metadata store into by-id records, filling each record's cache
|
||||
// paths from the content cache when one is given. Shared by the live read
|
||||
@@ -144,6 +153,9 @@ const deriveRecordsFromStore = (
|
||||
// tests drive a mock with no crypto or network; the real `Client` satisfies it
|
||||
// structurally.
|
||||
export interface LibraryClient {
|
||||
// The server the client talks to, as `Client.apiOrigin` names it. Absent
|
||||
// means Ente's hosted service.
|
||||
readonly apiOrigin?: string;
|
||||
whoami(): { email: string; userID: number };
|
||||
collectionsSince(args: { sinceTime: number }): Promise<CollectionsPage>;
|
||||
filesSince(args: {
|
||||
@@ -196,8 +208,8 @@ export type RefreshProgressCallback = (event: RefreshEvent) => void;
|
||||
|
||||
export interface LibraryOptions {
|
||||
client: LibraryClient;
|
||||
// Where `metadata.json` lives. Defaults to the env-paths cache directory
|
||||
// plus the user id, so each account has its own cache.
|
||||
// Where `metadata.json` lives. Defaults to `defaultCacheDirectory`, so
|
||||
// each account has its own cache.
|
||||
cacheDirectory?: string;
|
||||
// The root of every photo's save path, where `Photo.download()` and
|
||||
// `lib.backup()` put originals. Defaults to `photos` in the working
|
||||
@@ -300,6 +312,7 @@ export class Library {
|
||||
// available (a metadata-only client with no explicit source).
|
||||
private readonly cache?: ContentCache;
|
||||
private readonly userID: number;
|
||||
private readonly apiOrigin: string | undefined;
|
||||
private readonly intervalMs: number;
|
||||
private readonly onProgress?: RefreshProgressCallback;
|
||||
private readonly pools: RequestPools;
|
||||
@@ -346,6 +359,7 @@ export class Library {
|
||||
client: LibraryClient;
|
||||
store: MetadataStore;
|
||||
userID: number;
|
||||
apiOrigin: string | undefined;
|
||||
cacheDirectory: string;
|
||||
downloadDirectory: string;
|
||||
intervalMs: number;
|
||||
@@ -358,6 +372,7 @@ export class Library {
|
||||
this.client = args.client;
|
||||
this.store = args.store;
|
||||
this.userID = args.userID;
|
||||
this.apiOrigin = args.apiOrigin;
|
||||
this.cacheDirectory = args.cacheDirectory;
|
||||
this.downloadDirectory = args.downloadDirectory;
|
||||
this.intervalMs = args.intervalMs;
|
||||
@@ -406,8 +421,9 @@ export class Library {
|
||||
// refresh: `open()` returns onto the cache as it is.
|
||||
static async open(opts: LibraryOptions): Promise<Library> {
|
||||
const { userID } = opts.client.whoami();
|
||||
const { apiOrigin } = opts.client;
|
||||
const cacheDirectory =
|
||||
opts.cacheDirectory ?? defaultCacheDirectory(userID);
|
||||
opts.cacheDirectory ?? defaultCacheDirectory(userID, apiOrigin);
|
||||
if (opts.downloadDirectory === "") {
|
||||
throw new Error(
|
||||
"library: downloadDirectory is empty (leave it out to save " +
|
||||
@@ -419,14 +435,24 @@ export class Library {
|
||||
let store = await MetadataStore.load(metadataPath);
|
||||
// A cache directory given explicitly can hold another account's cache.
|
||||
// Its records and cursor are not this account's, so delete it and the
|
||||
// ML data beside it and start empty. A user ID of 0 means the cache
|
||||
// was never refreshed and so holds nothing to discard.
|
||||
if (store.userID !== 0 && store.userID !== userID) {
|
||||
await rm(metadataPath, { force: true });
|
||||
await rm(join(cacheDirectory, "mldata"), {
|
||||
recursive: true,
|
||||
force: true,
|
||||
});
|
||||
// ML data beside it and start empty. A cache from another server goes
|
||||
// whole: two servers can give two files the same ID, so its originals
|
||||
// and thumbnails could be served for this account's files. A user ID
|
||||
// of 0 means the cache names no account, so nothing is stored in it.
|
||||
if (
|
||||
store.userID !== 0 &&
|
||||
(store.userID !== userID || store.apiOrigin !== apiOrigin)
|
||||
) {
|
||||
const discard = ["metadata.json", "mldata"];
|
||||
if (store.apiOrigin !== apiOrigin) {
|
||||
discard.push("originals", "thumbnails");
|
||||
}
|
||||
for (const name of discard) {
|
||||
await rm(join(cacheDirectory, name), {
|
||||
recursive: true,
|
||||
force: true,
|
||||
});
|
||||
}
|
||||
store = await MetadataStore.load(metadataPath);
|
||||
}
|
||||
const intervalMs =
|
||||
@@ -490,6 +516,7 @@ export class Library {
|
||||
client: opts.client,
|
||||
store,
|
||||
userID,
|
||||
apiOrigin,
|
||||
cacheDirectory,
|
||||
downloadDirectory,
|
||||
intervalMs,
|
||||
@@ -509,9 +536,11 @@ export class Library {
|
||||
if (opts.refreshOnOpen === false) {
|
||||
// The caller asks for each refresh it needs.
|
||||
lib.scheduleNext();
|
||||
} else if (store.loadedFromDisk) {
|
||||
} else if (store.listCollections().length > 0) {
|
||||
// An existing copy already answers reads; refresh in the background
|
||||
// and start the interval once that first cycle settles.
|
||||
// and start the interval once that first cycle settles. A cache
|
||||
// that names the account but holds no collection has nothing to
|
||||
// answer with, such as after `originalOf` on an empty cache.
|
||||
void lib.runRefresh().then(() => lib.scheduleNext());
|
||||
} else {
|
||||
// Nothing was cached: wait for the first refresh to fill the store
|
||||
@@ -658,6 +687,7 @@ export class Library {
|
||||
"originalOf requires a library opened with a content cache",
|
||||
);
|
||||
}
|
||||
await this.saveAccountInEmptyCache();
|
||||
return this.cache.originalOf(file, opts);
|
||||
}
|
||||
|
||||
@@ -673,9 +703,23 @@ export class Library {
|
||||
"thumbnailOf requires a library opened with a content cache",
|
||||
);
|
||||
}
|
||||
await this.saveAccountInEmptyCache();
|
||||
return this.cache.thumbnailOf(file, opts);
|
||||
}
|
||||
|
||||
// Saves the account's user ID and server in a cache that names no account
|
||||
// yet, before `originalOf` or `thumbnailOf` stores a file there. A later
|
||||
// `open()` for an account on another server then deletes that file (see
|
||||
// `open()`), although no refresh has stored any records.
|
||||
private async saveAccountInEmptyCache(): Promise<void> {
|
||||
if (this.store.userID !== 0) return;
|
||||
this.store.userID = this.userID;
|
||||
this.store.apiOrigin = this.apiOrigin;
|
||||
this.unsaved = true;
|
||||
await this.store.save();
|
||||
this.unsaved = false;
|
||||
}
|
||||
|
||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||
// library's, each original at its save path, with a durable failure
|
||||
// ledger (issue #51). Takes the lock in that directory first, unless the
|
||||
@@ -857,8 +901,12 @@ export class Library {
|
||||
// actually changed.
|
||||
let changed = false;
|
||||
|
||||
if (this.store.userID !== this.userID) {
|
||||
if (
|
||||
this.store.userID !== this.userID ||
|
||||
this.store.apiOrigin !== this.apiOrigin
|
||||
) {
|
||||
this.store.userID = this.userID;
|
||||
this.store.apiOrigin = this.apiOrigin;
|
||||
changed = true;
|
||||
}
|
||||
|
||||
|
||||
+11
-12
@@ -1,11 +1,12 @@
|
||||
// On-disk JSON metadata store for the local cache.
|
||||
//
|
||||
// The store keeps one `metadata.json` file holding the account's server
|
||||
// state: the user id, a schema version, the cursor for the incremental
|
||||
// collections listing, the decrypted collection and file records, and the
|
||||
// collections and files the refresh could not decrypt. The whole file is read
|
||||
// into RAM on load and rewritten as a whole on save; there is no partial
|
||||
// update and no lock file. A separate refresh unit populates the store from
|
||||
// state: the user id, the server's address for an account on a self-hosted
|
||||
// server, a schema version, the cursor for the incremental collections
|
||||
// listing, the decrypted collection and file records, and the collections and
|
||||
// files the refresh could not decrypt. The whole file is read into RAM on
|
||||
// load and rewritten as a whole on save; there is no partial update and no
|
||||
// lock file. A separate refresh unit populates the store from
|
||||
// the server — this module only stores what it is given.
|
||||
//
|
||||
// The file is a cache, so it is never trusted to exist or to be intact: a
|
||||
@@ -46,6 +47,7 @@ export interface Undecryptable {
|
||||
interface StoredMetadata {
|
||||
schemaVersion: number;
|
||||
userID: number;
|
||||
apiOrigin?: string;
|
||||
collectionsSinceTime: Microseconds;
|
||||
collections: StoredCollection[];
|
||||
files: StoredFile[];
|
||||
@@ -68,14 +70,10 @@ export class MetadataStore {
|
||||
readonly path: string;
|
||||
readonly schemaVersion = METADATA_SCHEMA_VERSION;
|
||||
userID = 0;
|
||||
// The account's `apiOrigin`, or undefined for Ente's hosted service.
|
||||
apiOrigin: string | undefined;
|
||||
collectionsSinceTime: Microseconds = 0;
|
||||
|
||||
// True when `load` populated this store from a valid existing file; false
|
||||
// on a first run or a missing/corrupt/wrong-version file that loaded empty.
|
||||
// `Library.open` reads it to decide whether the first refresh may run in
|
||||
// the background (an existing copy already serves reads) or must be awaited.
|
||||
loadedFromDisk = false;
|
||||
|
||||
private readonly collections = new Map<number, Collection>();
|
||||
private readonly files = new Map<string, EnteFile>();
|
||||
private undecryptable: Undecryptable[] = [];
|
||||
@@ -102,8 +100,8 @@ export class MetadataStore {
|
||||
if (parsed.schemaVersion !== METADATA_SCHEMA_VERSION) {
|
||||
return store;
|
||||
}
|
||||
store.loadedFromDisk = true;
|
||||
store.userID = parsed.userID ?? 0;
|
||||
store.apiOrigin = parsed.apiOrigin;
|
||||
store.collectionsSinceTime = parsed.collectionsSinceTime ?? 0;
|
||||
for (const stored of parsed.collections ?? []) {
|
||||
const collection: Collection = {
|
||||
@@ -143,6 +141,7 @@ export class MetadataStore {
|
||||
const model: StoredMetadata = {
|
||||
schemaVersion: METADATA_SCHEMA_VERSION,
|
||||
userID: this.userID,
|
||||
apiOrigin: this.apiOrigin,
|
||||
collectionsSinceTime: this.collectionsSinceTime,
|
||||
collections: [...this.collections.values()].map((c) => ({
|
||||
...c,
|
||||
|
||||
+1
-1
@@ -325,7 +325,7 @@ export const fixMissingThumbnails = async (
|
||||
ciphertext.length,
|
||||
md5,
|
||||
);
|
||||
await api.putFile(url, ciphertext);
|
||||
await api.putFile(url, ciphertext, md5);
|
||||
await api.updateThumbnail(file.id, objectKey, toBase64(header));
|
||||
|
||||
log(`[${collectionName}] Thumbnail uploaded for ${title}`);
|
||||
|
||||
+10
-4
@@ -780,7 +780,7 @@ describe("ApiClient timeouts", () => {
|
||||
|
||||
await client.getJSON("/a");
|
||||
await client.postJSON("/b", {});
|
||||
await client.putFile("https://s3.example/x", new Uint8Array([1]));
|
||||
await client.putFile("https://s3.example/x", new Uint8Array([1]), "m");
|
||||
await client.putJSON("/c", {});
|
||||
|
||||
for (const call of calls) {
|
||||
@@ -825,7 +825,8 @@ describe("ApiClient timeouts", () => {
|
||||
[
|
||||
"putFile",
|
||||
() => new Response(null, { status: 200 }),
|
||||
(c) => c.putFile("https://s3.example/x", new Uint8Array([1])),
|
||||
(c) =>
|
||||
c.putFile("https://s3.example/x", new Uint8Array([1]), "m"),
|
||||
],
|
||||
[
|
||||
"getFileStream",
|
||||
@@ -1049,7 +1050,7 @@ describe("ApiClient error typing", () => {
|
||||
const client = new ApiClient({ fetch, retry: noWait });
|
||||
|
||||
const err: unknown = await client
|
||||
.putFile("https://s3.example/obj", new Uint8Array([1, 2]))
|
||||
.putFile("https://s3.example/obj", new Uint8Array([1, 2]), "m")
|
||||
.catch((e: unknown) => e);
|
||||
|
||||
expect(err).toBeInstanceOf(ApiError);
|
||||
@@ -1067,7 +1068,11 @@ describe("ApiClient error typing", () => {
|
||||
);
|
||||
const client = new ApiClient({ fetch, retry: noWait });
|
||||
|
||||
await client.putFile("https://s3.example/obj", new Uint8Array([1, 2]));
|
||||
await client.putFile(
|
||||
"https://s3.example/obj",
|
||||
new Uint8Array([1, 2]),
|
||||
"m",
|
||||
);
|
||||
expect(calls).toHaveLength(2);
|
||||
});
|
||||
|
||||
@@ -1277,6 +1282,7 @@ describe("ApiClient onRequest", () => {
|
||||
await client.putFile(
|
||||
"https://s3.example/obj?X-Amz-Signature=abc",
|
||||
new Uint8Array([1, 2]),
|
||||
"m",
|
||||
);
|
||||
|
||||
expect(requests).toEqual([
|
||||
|
||||
+13
-2
@@ -80,13 +80,20 @@ describe("ApiClient.putFile", () => {
|
||||
const client = new ApiClient({ fetch, authToken: "secret-tok" });
|
||||
const data = new Uint8Array([1, 2, 3, 4, 5]);
|
||||
|
||||
await client.putFile("https://s3.example.com/presigned?sig=abc", data);
|
||||
await client.putFile(
|
||||
"https://s3.example.com/presigned?sig=abc",
|
||||
data,
|
||||
"md5base64==",
|
||||
);
|
||||
|
||||
expect(calls[0]!.url).toBe("https://s3.example.com/presigned?sig=abc");
|
||||
expect(calls[0]!.init?.method).toBe("PUT");
|
||||
const headers = new Headers(calls[0]!.init?.headers as HeadersInit);
|
||||
expect(headers.get("Content-Type")).toBe("application/octet-stream");
|
||||
expect(headers.get("Content-Length")).toBe("5");
|
||||
// The server signs the MD5 into the URL; storage refuses the PUT
|
||||
// without it.
|
||||
expect(headers.get("Content-MD5")).toBe("md5base64==");
|
||||
// Must NOT leak auth headers to S3
|
||||
expect(headers.has("X-Auth-Token")).toBe(false);
|
||||
expect(headers.has("X-Client-Package")).toBe(false);
|
||||
@@ -99,7 +106,11 @@ describe("ApiClient.putFile", () => {
|
||||
const client = new ApiClient({ fetch });
|
||||
|
||||
await expect(
|
||||
client.putFile("https://s3.example.com/bad", new Uint8Array(10)),
|
||||
client.putFile(
|
||||
"https://s3.example.com/bad",
|
||||
new Uint8Array(10),
|
||||
"md5base64==",
|
||||
),
|
||||
).rejects.toThrow(/403/);
|
||||
});
|
||||
});
|
||||
|
||||
+47
-3
@@ -1,12 +1,15 @@
|
||||
/**
|
||||
* Tests for the retry options `bin/quak.ts` loads the saved session with
|
||||
* (`quak backup` gets the unattended ones, every other command the default),
|
||||
* each with the callbacks that log its requests and retries, and for
|
||||
* `quak files` taking its collection ID as an optional argument.
|
||||
* each with the callbacks that log its requests and retries, for `quak files`
|
||||
* taking its collection ID as an optional argument, for `quak get` and
|
||||
* `quak get-thumb` refusing `--collection`, and for `quak login` taking
|
||||
* `--server` and sending its first request to the server it names.
|
||||
*
|
||||
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
|
||||
* replaced by one that records the options it is given and reports no session,
|
||||
* so the command stops before it makes any request.
|
||||
* so the command stops before it makes any request. `quak login` reads no
|
||||
* session, so its test stubs `fetch` instead and records each request.
|
||||
*
|
||||
* The first load of the sources `bin/quak.ts` imports compiles them, and the
|
||||
* first `init()` starts libsodium. On a loaded host that takes longer than a
|
||||
@@ -57,6 +60,8 @@ afterEach(() => {
|
||||
process.argv = argv;
|
||||
loaded.length = 0;
|
||||
vi.restoreAllMocks();
|
||||
vi.unstubAllGlobals();
|
||||
vi.unstubAllEnvs();
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
@@ -105,3 +110,42 @@ describe("bin/quak.ts files", () => {
|
||||
expect(await quak("files", "7")).toMatch(/^Not logged in/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("bin/quak.ts get and get-thumb", () => {
|
||||
it.each(["get", "get-thumb"])(
|
||||
"%s refuses --collection as an unknown option",
|
||||
async (command) => {
|
||||
expect(await quak(command, "1", "--collection", "2")).toMatch(
|
||||
/^error: unknown option '--collection'/,
|
||||
);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
describe("bin/quak.ts login", () => {
|
||||
// The command refuses an address that is not a URL before it asks for
|
||||
// anything or sends any request.
|
||||
it("hands --server to the command", async () => {
|
||||
expect(await quak("login", "--server", "ente.example.com")).toMatch(
|
||||
/^Invalid server address "ente\.example\.com"/,
|
||||
);
|
||||
});
|
||||
|
||||
// `fetch` answers 404, so the login fails at its first request. A login
|
||||
// that succeeded would save its session in this user's real data
|
||||
// directory, since env-paths is not mocked here.
|
||||
it("sends its first request to the server --server names", async () => {
|
||||
const requests: string[] = [];
|
||||
vi.stubGlobal("fetch", async (input: string) => {
|
||||
const { origin, pathname } = new URL(input);
|
||||
requests.push(origin + pathname);
|
||||
return new Response("not found", { status: 404 });
|
||||
});
|
||||
vi.stubEnv("QUAK_EMAIL", "login@example.com");
|
||||
vi.stubEnv("QUAK_PASSWORD", "password");
|
||||
await quak("login", "--server", "https://ente.example.com");
|
||||
expect(requests).toEqual([
|
||||
"https://ente.example.com/users/srp/attributes",
|
||||
]);
|
||||
});
|
||||
});
|
||||
+267
-19
@@ -328,6 +328,7 @@ describe("login", () => {
|
||||
prompts = [];
|
||||
vi.stubEnv("QUAK_EMAIL", "cli@example.com");
|
||||
vi.stubEnv("QUAK_PASSWORD", "hunter2");
|
||||
vi.stubEnv("QUAK_SERVER", undefined);
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
@@ -340,12 +341,14 @@ describe("login", () => {
|
||||
calls.push(opts);
|
||||
return loggedIn;
|
||||
});
|
||||
expect(await loginCommand(ctx)).toBe(0);
|
||||
expect(await loginCommand(ctx, {})).toBe(0);
|
||||
|
||||
expect(prompts).toEqual([]);
|
||||
expect(calls).toHaveLength(1);
|
||||
expect(calls[0]!.email).toBe("cli@example.com");
|
||||
expect(calls[0]!.password).toBe("hunter2");
|
||||
// No server named: the client uses Ente's hosted service.
|
||||
expect(calls[0]!.apiOptions).toBeUndefined();
|
||||
const path = join(ctx.sessionDir, "session.json");
|
||||
expect(stderr.text).toBe(
|
||||
"Authenticating...\n" +
|
||||
@@ -356,7 +359,7 @@ describe("login", () => {
|
||||
|
||||
it("saves the session with mode 0600 in a directory with mode 0700", async () => {
|
||||
const ctx = loginContext(async () => loggedIn);
|
||||
expect(await loginCommand(ctx)).toBe(0);
|
||||
expect(await loginCommand(ctx, {})).toBe(0);
|
||||
|
||||
expect(statSync(ctx.sessionDir).mode & 0o777).toBe(0o700);
|
||||
const path = join(ctx.sessionDir, "session.json");
|
||||
@@ -370,7 +373,7 @@ describe("login", () => {
|
||||
code = await opts.totp!();
|
||||
return loggedIn;
|
||||
});
|
||||
expect(await loginCommand(ctx)).toBe(0);
|
||||
expect(await loginCommand(ctx, {})).toBe(0);
|
||||
|
||||
expect(prompts).toEqual(["TOTP code: "]);
|
||||
expect(code).toBe("123456");
|
||||
@@ -380,13 +383,83 @@ describe("login", () => {
|
||||
const ctx = loginContext(async () => {
|
||||
throw new Error("HTTP 401 from server");
|
||||
});
|
||||
expect(await loginCommand(ctx)).toBe(1);
|
||||
expect(await loginCommand(ctx, {})).toBe(1);
|
||||
|
||||
expect(stderr.text).toBe(
|
||||
"Authenticating...\nLogin failed: HTTP 401 from server\n",
|
||||
);
|
||||
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
|
||||
});
|
||||
|
||||
it("logs in to the server --server names, ahead of QUAK_SERVER", async () => {
|
||||
vi.stubEnv("QUAK_SERVER", "https://other.example.com");
|
||||
const calls: LoginOptions[] = [];
|
||||
const ctx = loginContext(async (opts) => {
|
||||
calls.push(opts);
|
||||
return loggedIn;
|
||||
});
|
||||
expect(
|
||||
await loginCommand(ctx, { server: "https://ente.example.com" }),
|
||||
).toBe(0);
|
||||
|
||||
expect(calls[0]!.apiOptions).toEqual({
|
||||
apiOrigin: "https://ente.example.com",
|
||||
});
|
||||
});
|
||||
|
||||
it("logs in to the server QUAK_SERVER names without --server", async () => {
|
||||
vi.stubEnv("QUAK_SERVER", "http://192.0.2.1:8080/ente");
|
||||
const calls: LoginOptions[] = [];
|
||||
const ctx = loginContext(async (opts) => {
|
||||
calls.push(opts);
|
||||
return loggedIn;
|
||||
});
|
||||
expect(await loginCommand(ctx, {})).toBe(0);
|
||||
|
||||
expect(calls[0]!.apiOptions).toEqual({
|
||||
apiOrigin: "http://192.0.2.1:8080/ente",
|
||||
});
|
||||
});
|
||||
|
||||
it.each([
|
||||
"",
|
||||
"ente.example.com",
|
||||
"ftp://ente.example.com",
|
||||
"https://ente.example.com/?region=eu",
|
||||
"https://ente.example.com/#top",
|
||||
])(
|
||||
"refuses the server address %j before asking for anything",
|
||||
async (server) => {
|
||||
const ctx = loginContext(async () => {
|
||||
throw new Error("login must not be called");
|
||||
});
|
||||
expect(await loginCommand(ctx, { server })).toBe(1);
|
||||
|
||||
expect(prompts).toEqual([]);
|
||||
expect(stderr.text).toBe(
|
||||
`Invalid server address "${server}": it must be an http or ` +
|
||||
`https URL with no query or fragment\n`,
|
||||
);
|
||||
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(
|
||||
false,
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
it("refuses an empty QUAK_SERVER before asking for anything", async () => {
|
||||
vi.stubEnv("QUAK_SERVER", "");
|
||||
const ctx = loginContext(async () => {
|
||||
throw new Error("login must not be called");
|
||||
});
|
||||
expect(await loginCommand(ctx, {})).toBe(1);
|
||||
|
||||
expect(prompts).toEqual([]);
|
||||
expect(stderr.text).toBe(
|
||||
`Invalid server address "": it must be an http or https URL ` +
|
||||
`with no query or fragment\n`,
|
||||
);
|
||||
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
// These use a real client read from the session file, over a fake API that
|
||||
@@ -461,6 +534,16 @@ describe("logout", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("names a self-hosted account's default cache directory, under its server's address", async () => {
|
||||
const ctx = { ...logoutContext(200), cacheDir: undefined };
|
||||
const apiOrigin = "https://ente.example.com";
|
||||
saveSession(ctx.sessionDir, { ...snapshot, apiOrigin });
|
||||
expect(await logoutCommand(ctx)).toBe(0);
|
||||
expect(stderr.text).toContain(
|
||||
`Cache directory ${defaultCacheDirectory(USER_ID, apiOrigin)} still holds decrypted data`,
|
||||
);
|
||||
});
|
||||
|
||||
it("without a session says so, calls nothing and exits 0", async () => {
|
||||
expect(await logoutCommand(logoutContext(200))).toBe(0);
|
||||
expect(requests).toHaveLength(0);
|
||||
@@ -569,14 +652,14 @@ describe("files", () => {
|
||||
});
|
||||
|
||||
describe("get and get-thumb", () => {
|
||||
it("get finds a file in any album without --collection", async () => {
|
||||
it("get finds a file in any album", async () => {
|
||||
const out = join(root, "diagram.png");
|
||||
expect(await getCommand(context(), "200", { out })).toBe(0);
|
||||
expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
|
||||
expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
|
||||
});
|
||||
|
||||
it("get-thumb finds a file in any album without --collection", async () => {
|
||||
it("get-thumb finds a file in any album", async () => {
|
||||
const out = join(root, "thumb.jpg");
|
||||
expect(await getThumbCommand(context(), "200", { out })).toBe(0);
|
||||
expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
|
||||
@@ -1618,18 +1701,19 @@ describe("an account with many albums", () => {
|
||||
// encrypted as the server stores it. The original's bytes are the title.
|
||||
// These tests fetch no thumbnail. The server records a thumbnail size, so
|
||||
// `helper fix-missing-thumbnails` downloads the original before it skips
|
||||
// the photo as not a JPEG.
|
||||
const photoRow = (photo: {
|
||||
id: number;
|
||||
collectionID: number;
|
||||
title: string;
|
||||
updationTime: number;
|
||||
}): { row: Record<string, unknown>; original: Uint8Array } => {
|
||||
// the photo as not a JPEG. `albumKey` is that album's key, by default the
|
||||
// one in `albumKeys`.
|
||||
const photoRow = (
|
||||
photo: {
|
||||
id: number;
|
||||
collectionID: number;
|
||||
title: string;
|
||||
updationTime: number;
|
||||
},
|
||||
albumKey = albumKeys.get(photo.collectionID)!,
|
||||
): { row: Record<string, unknown>; original: Uint8Array } => {
|
||||
const photoKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const photoKeyBox = encryptSecretbox(
|
||||
photoKey,
|
||||
albumKeys.get(photo.collectionID)!,
|
||||
);
|
||||
const photoKeyBox = encryptSecretbox(photoKey, albumKey);
|
||||
const metadata = encryptStreamBlob(
|
||||
new TextEncoder().encode(
|
||||
JSON.stringify({
|
||||
@@ -1742,6 +1826,77 @@ describe("an account with many albums", () => {
|
||||
};
|
||||
};
|
||||
|
||||
// The albums in the cache's `metadata.json`. A search on an empty cache
|
||||
// stores none there, only the account whose original or thumbnail it
|
||||
// stored.
|
||||
const cachedAlbums = (): unknown[] =>
|
||||
JSON.parse(readFileSync(join(root, "cache", "metadata.json"), "utf-8"))
|
||||
.collections;
|
||||
|
||||
// A context whose login and saved session send each request to the server
|
||||
// its host names: `ente.example.com` is the fake server above, and
|
||||
// `photos.example.org` a second one. Every account the fake server logs in
|
||||
// has user ID 42, so the second server's account has the first one's user
|
||||
// ID. Its one album holds photo 200, `other-200.jpg`; the first server
|
||||
// gives ID 200 to `photo-2.jpg`.
|
||||
const twoServerContext = async (): Promise<CliContext> => {
|
||||
const account = await buildAccount(PASSWORD);
|
||||
const albumKey = sodium.crypto_secretbox_keygen();
|
||||
const albumKeyBox = encryptSecretbox(albumKey, account.masterKey);
|
||||
const name = encryptSecretbox(
|
||||
new TextEncoder().encode("Other album"),
|
||||
albumKey,
|
||||
);
|
||||
const { row, original } = photoRow(
|
||||
{
|
||||
id: 200,
|
||||
collectionID: 1,
|
||||
title: "other-200.jpg",
|
||||
updationTime: 1,
|
||||
},
|
||||
albumKey,
|
||||
);
|
||||
const other = buildMetaFetch({
|
||||
verifier: account.verifier,
|
||||
srpAttributes: account.srpAttributes,
|
||||
keyAttributes: account.keyAttributes,
|
||||
encryptedToken: account.encryptedToken,
|
||||
collections: [
|
||||
{
|
||||
id: 1,
|
||||
owner: { id: USER_ID },
|
||||
encryptedKey: toBase64(albumKeyBox.ciphertext),
|
||||
keyDecryptionNonce: toBase64(albumKeyBox.nonce),
|
||||
encryptedName: toBase64(name.ciphertext),
|
||||
nameDecryptionNonce: toBase64(name.nonce),
|
||||
type: "album",
|
||||
updationTime: 1,
|
||||
},
|
||||
],
|
||||
filesByCollection: { 1: [row] },
|
||||
encryptedMLData: {},
|
||||
fileCiphertexts: { 200: original },
|
||||
});
|
||||
const fetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||
const { host } = new URL(
|
||||
input instanceof Request ? input.url : input,
|
||||
);
|
||||
return (host === "photos.example.org" ? other : server)(
|
||||
input,
|
||||
init,
|
||||
);
|
||||
}) as typeof globalThis.fetch;
|
||||
return {
|
||||
...context(),
|
||||
login: (opts) =>
|
||||
Client.login({
|
||||
...opts,
|
||||
apiOptions: { ...opts.apiOptions, fetch },
|
||||
}),
|
||||
loadSession: (path) => loadSession(path, { fetch }),
|
||||
};
|
||||
};
|
||||
|
||||
it("collections lists every album from one request, fetching no album's files", async () => {
|
||||
const ctx = accountContext();
|
||||
expect(await collectionsCommand(ctx, {})).toBe(0);
|
||||
@@ -1848,7 +2003,100 @@ describe("an account with many albums", () => {
|
||||
"/collections/v2/diff?collectionID=2&sinceTime=0",
|
||||
"/?fileID=200",
|
||||
]);
|
||||
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
|
||||
expect(cachedAlbums()).toEqual([]);
|
||||
});
|
||||
|
||||
// A self-hosted server serves downloads under its own address, at
|
||||
// `/files/download/<id>`, so no request goes to Ente's hosted service.
|
||||
it("login --server saves the server, and a later get sends every request to it", async () => {
|
||||
vi.stubEnv("QUAK_EMAIL", "many@example.com");
|
||||
vi.stubEnv("QUAK_PASSWORD", PASSWORD);
|
||||
vi.stubEnv("QUAK_SERVER", undefined);
|
||||
const hosts = new Set<string>();
|
||||
const fetch = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||
hosts.add(
|
||||
new URL(input instanceof Request ? input.url : input).host,
|
||||
);
|
||||
return server(input, init);
|
||||
}) as typeof globalThis.fetch;
|
||||
const ctx: CliContext = {
|
||||
...context(),
|
||||
login: (opts) =>
|
||||
Client.login({
|
||||
...opts,
|
||||
apiOptions: { ...opts.apiOptions, fetch },
|
||||
}),
|
||||
loadSession: (path) => loadSession(path, { fetch }),
|
||||
};
|
||||
const out = join(root, "photo.jpg");
|
||||
try {
|
||||
const opts = { server: "https://ente.example.com" };
|
||||
expect(await loginCommand(ctx, opts)).toBe(0);
|
||||
requests.length = 0;
|
||||
expect(await getCommand(ctx, "200", { out })).toBe(0);
|
||||
} finally {
|
||||
vi.unstubAllEnvs();
|
||||
}
|
||||
|
||||
const path = join(ctx.sessionDir, "session.json");
|
||||
expect(JSON.parse(readFileSync(path, "utf-8")).apiOrigin).toBe(
|
||||
"https://ente.example.com",
|
||||
);
|
||||
expect(readFileSync(out, "utf-8")).toBe("photo-2.jpg");
|
||||
expect(requests.at(-1)).toBe("/files/download/200");
|
||||
expect([...hosts]).toEqual(["ente.example.com"]);
|
||||
});
|
||||
|
||||
// The two servers' accounts share the cache directory `--cache-dir` names.
|
||||
it("after a login to another server whose account has the same user ID, files and get see none of the first server's albums and originals", async () => {
|
||||
const ctx = await twoServerContext();
|
||||
const first = join(root, "first.jpg");
|
||||
const second = join(root, "second.jpg");
|
||||
vi.stubEnv("QUAK_EMAIL", "many@example.com");
|
||||
vi.stubEnv("QUAK_PASSWORD", PASSWORD);
|
||||
vi.stubEnv("QUAK_SERVER", undefined);
|
||||
try {
|
||||
const opts = { server: "https://ente.example.com" };
|
||||
expect(await loginCommand(ctx, opts)).toBe(0);
|
||||
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||
expect(await getCommand(ctx, "200", { out: first })).toBe(0);
|
||||
|
||||
const otherOpts = { server: "https://photos.example.org" };
|
||||
expect(await loginCommand(ctx, otherOpts)).toBe(0);
|
||||
stdout.text = "";
|
||||
expect(await filesCommand(ctx, undefined, {})).toBe(0);
|
||||
expect(await getCommand(ctx, "200", { out: second })).toBe(0);
|
||||
} finally {
|
||||
vi.unstubAllEnvs();
|
||||
}
|
||||
|
||||
expect(stdout.text).toBe("200\timage\tother-200.jpg\n");
|
||||
expect(readFileSync(first, "utf-8")).toBe("photo-2.jpg");
|
||||
expect(readFileSync(second, "utf-8")).toBe("other-200.jpg");
|
||||
});
|
||||
|
||||
// `get` on an empty cache stores the original but no album.
|
||||
it("get on an empty cache, after the same on another server whose account has the same user ID, saves this server's photo", async () => {
|
||||
const ctx = await twoServerContext();
|
||||
const first = join(root, "first.jpg");
|
||||
const second = join(root, "second.jpg");
|
||||
vi.stubEnv("QUAK_EMAIL", "many@example.com");
|
||||
vi.stubEnv("QUAK_PASSWORD", PASSWORD);
|
||||
vi.stubEnv("QUAK_SERVER", undefined);
|
||||
try {
|
||||
const opts = { server: "https://ente.example.com" };
|
||||
expect(await loginCommand(ctx, opts)).toBe(0);
|
||||
expect(await getCommand(ctx, "200", { out: first })).toBe(0);
|
||||
|
||||
const otherOpts = { server: "https://photos.example.org" };
|
||||
expect(await loginCommand(ctx, otherOpts)).toBe(0);
|
||||
expect(await getCommand(ctx, "200", { out: second })).toBe(0);
|
||||
} finally {
|
||||
vi.unstubAllEnvs();
|
||||
}
|
||||
|
||||
expect(readFileSync(first, "utf-8")).toBe("photo-2.jpg");
|
||||
expect(readFileSync(second, "utf-8")).toBe("other-200.jpg");
|
||||
});
|
||||
|
||||
// `files` first fills the cache, so `get` waits for a refresh. No album has
|
||||
@@ -1906,7 +2154,7 @@ describe("an account with many albums", () => {
|
||||
"/collections/v2/diff?collectionID=39&sinceTime=0",
|
||||
"/?fileID=3900",
|
||||
]);
|
||||
expect(existsSync(join(root, "cache", "metadata.json"))).toBe(false);
|
||||
expect(cachedAlbums()).toEqual([]);
|
||||
});
|
||||
|
||||
// `files` first fills the cache, so the helper waits for a refresh. No
|
||||
|
||||
@@ -47,6 +47,35 @@ describe("Client.toJSON", () => {
|
||||
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
|
||||
});
|
||||
|
||||
it("round-trips a snapshot that names a server", () => {
|
||||
const snapshot = {
|
||||
...validSnapshot(),
|
||||
apiOrigin: "https://ente.example.com",
|
||||
};
|
||||
expect(Client.fromJSON(snapshot).toJSON()).toEqual(snapshot);
|
||||
});
|
||||
|
||||
it("names the server apiOptions gave fromJSON", async () => {
|
||||
const apiOrigin = "https://ente.example.com";
|
||||
const urls: string[] = [];
|
||||
const client = Client.fromJSON(validSnapshot(), {
|
||||
apiOrigin,
|
||||
fetch: async (input) => {
|
||||
urls.push(String(input));
|
||||
return new Response(JSON.stringify({ collections: [] }), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
},
|
||||
});
|
||||
expect(client.toJSON().apiOrigin).toBe(apiOrigin);
|
||||
|
||||
await client.listCollections();
|
||||
expect(urls).toEqual([
|
||||
"https://ente.example.com/collections/v2?sinceTime=0",
|
||||
]);
|
||||
});
|
||||
|
||||
it("throws instead of emitting a snapshot without a token", () => {
|
||||
const client = Client.fromJSON(validSnapshot());
|
||||
client.getApiClient().clearAuthToken();
|
||||
@@ -77,6 +106,10 @@ describe("Client.fromJSON", () => {
|
||||
["publicKey", undefined],
|
||||
["publicKey", "not base64!"],
|
||||
["publicKey", shortKey],
|
||||
["apiOrigin", 7],
|
||||
["apiOrigin", ""],
|
||||
["apiOrigin", "ente.example.com"],
|
||||
["apiOrigin", "ftp://ente.example.com"],
|
||||
])("rejects %s = %j, naming the field", (field, value) => {
|
||||
const snapshot: Record<string, unknown> = { ...validSnapshot() };
|
||||
snapshot[field] = value;
|
||||
@@ -85,6 +118,30 @@ describe("Client.fromJSON", () => {
|
||||
);
|
||||
});
|
||||
|
||||
// The token belongs to the snapshot's server, so that server wins over
|
||||
// the one in `apiOptions`.
|
||||
it("sends requests to the snapshot's server", async () => {
|
||||
const urls: string[] = [];
|
||||
const client = Client.fromJSON(
|
||||
{ ...validSnapshot(), apiOrigin: "https://ente.example.com/" },
|
||||
{
|
||||
apiOrigin: "https://other.example.com",
|
||||
fetch: async (input) => {
|
||||
urls.push(String(input));
|
||||
return new Response(JSON.stringify({ collections: [] }), {
|
||||
status: 200,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
},
|
||||
},
|
||||
);
|
||||
|
||||
await client.listCollections();
|
||||
expect(urls).toEqual([
|
||||
"https://ente.example.com/collections/v2?sinceTime=0",
|
||||
]);
|
||||
});
|
||||
|
||||
it.each([null, "a string", 42])("rejects a non-object %j", (value) => {
|
||||
expect(() => Client.fromJSON(value)).toThrow(
|
||||
/^Invalid session data: not a JSON object/,
|
||||
|
||||
@@ -19,16 +19,22 @@
|
||||
* refresh fails (offline start from cache).
|
||||
* 5. `close()` stops the timer and is idempotent, and its promise resolves
|
||||
* only once an in-flight refresh has written the cache file.
|
||||
* 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id.
|
||||
* 7. `open()` branches on the cache: an empty cache awaits the first refresh
|
||||
* (it has nothing to serve yet); an existing cache serves its copy at once
|
||||
* and refreshes in the background, so a slow or dead server never stalls
|
||||
* opening.
|
||||
* 6. `cacheDirectory` defaults to the env-paths cache dir plus the user id,
|
||||
* with a self-hosted server's address before the user id.
|
||||
* 7. `open()` branches on the cache: a cache that holds no collection awaits
|
||||
* the first refresh (it has nothing to serve yet); an existing cache serves
|
||||
* its copy at once and refreshes in the background, so a slow or dead
|
||||
* server never stalls opening.
|
||||
* 8. A save failure that leaves RAM ahead of disk keeps `status().lastError`
|
||||
* set and keeps retrying the write; a later empty refresh does not clear it.
|
||||
* 9. With `refreshOnOpen: false`, `open()` starts no refresh at all, and a
|
||||
* refresh reports through `onProgress` each collection whose files it
|
||||
* fetched.
|
||||
* 10. A cache another account filled starts empty. One filled from another
|
||||
* server also loses its originals and thumbnails, since two servers can
|
||||
* give two files the same ID. `thumbnailOf` on an empty cache saves the
|
||||
* account's user ID and server. A refresh saves the account's server, so a
|
||||
* self-hosted account keeps its own cache from one `open()` to the next.
|
||||
*
|
||||
* The client is a mock: no crypto, no network. It serves scripted pages and
|
||||
* records the `sinceTime` each call carried so cursor threading is provable.
|
||||
@@ -49,7 +55,13 @@
|
||||
*/
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
import { mkdtempSync, rmSync } from "node:fs";
|
||||
import {
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
rmSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import envPaths from "env-paths";
|
||||
@@ -107,6 +119,7 @@ const file = (
|
||||
*/
|
||||
class MockClient {
|
||||
userID = USER_ID;
|
||||
apiOrigin?: string;
|
||||
failCollections = false;
|
||||
collectionsQueue: CollectionsPage[] = [];
|
||||
filesByCollection = new Map<number, FilesPage[]>();
|
||||
@@ -584,6 +597,44 @@ describe("Library.open and background refresh", () => {
|
||||
}
|
||||
});
|
||||
|
||||
// `originalOf` on an empty cache leaves such a cache behind: it names the
|
||||
// account before it stores the original.
|
||||
it("awaits the first refresh on a cache that names the account but holds no collection", async () => {
|
||||
const seed = await MetadataStore.load(
|
||||
join(cacheDirectory, "metadata.json"),
|
||||
);
|
||||
seed.userID = USER_ID;
|
||||
await seed.save();
|
||||
let releaseFirstFetch: (page: CollectionsPage) => void = () => {};
|
||||
const gate = new Promise<CollectionsPage>((resolve) => {
|
||||
releaseFirstFetch = resolve;
|
||||
});
|
||||
const client = new MockClient();
|
||||
client.collectionsSince = async () => gate;
|
||||
|
||||
let opened = false;
|
||||
const openPromise = Library.open({ client, cacheDirectory }).then(
|
||||
(l) => {
|
||||
opened = true;
|
||||
return l;
|
||||
},
|
||||
);
|
||||
await new Promise((r) => setTimeout(r, 20));
|
||||
expect(opened).toBe(false);
|
||||
|
||||
releaseFirstFetch({
|
||||
collections: [collection(1, 100)],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
const lib = await openPromise;
|
||||
try {
|
||||
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// A caller that asks for each refresh it needs, as the CLI's read commands
|
||||
// do, opens without one: on an empty cache the refresh `open()` would
|
||||
// await fetches the files of every collection, which `quak collections`
|
||||
@@ -842,4 +893,165 @@ describe("Library.open and background refresh", () => {
|
||||
else process.env.XDG_CACHE_HOME = prev;
|
||||
}
|
||||
});
|
||||
|
||||
it("puts a self-hosted server's address, URL-encoded, before the user id in the default cacheDirectory", async () => {
|
||||
const xdg = join(dir, "xdg-cache");
|
||||
const prev = process.env.XDG_CACHE_HOME;
|
||||
process.env.XDG_CACHE_HOME = xdg;
|
||||
try {
|
||||
const client = new MockClient();
|
||||
client.apiOrigin = "https://ente.example.com:8443/api";
|
||||
const lib = await Library.open({ client });
|
||||
try {
|
||||
expect(lib.cacheDirectory).toBe(
|
||||
join(
|
||||
envPaths("quak", { suffix: "" }).cache,
|
||||
"https%3A%2F%2Fente.example.com%3A8443%2Fapi",
|
||||
String(USER_ID),
|
||||
),
|
||||
);
|
||||
expect(lib.cacheDirectory.startsWith(xdg)).toBe(true);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
} finally {
|
||||
if (prev === undefined) delete process.env.XDG_CACHE_HOME;
|
||||
else process.env.XDG_CACHE_HOME = prev;
|
||||
}
|
||||
});
|
||||
|
||||
// A cache that account USER_ID on `apiOrigin` filled: its records, and an
|
||||
// original and a thumbnail of file 1001.
|
||||
const fillCache = async (apiOrigin?: string): Promise<void> => {
|
||||
const seed = await MetadataStore.load(
|
||||
join(cacheDirectory, "metadata.json"),
|
||||
);
|
||||
seed.userID = USER_ID;
|
||||
seed.apiOrigin = apiOrigin;
|
||||
seed.collectionsSinceTime = 500;
|
||||
seed.putCollection(collection(1, 400));
|
||||
seed.putFile(file(1001, 1, 400));
|
||||
await seed.save();
|
||||
for (const name of ["originals", "thumbnails"]) {
|
||||
mkdirSync(join(cacheDirectory, name));
|
||||
writeFileSync(join(cacheDirectory, name, "1001.jpg"), "1001");
|
||||
}
|
||||
};
|
||||
|
||||
it("deletes the whole cache an account on another server filled", async () => {
|
||||
await fillCache("https://ente.example.com");
|
||||
const client = new MockClient();
|
||||
client.apiOrigin = "https://photos.example.org";
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: 3600,
|
||||
});
|
||||
try {
|
||||
expect(client.collectionsSinceTimes).toEqual([0]);
|
||||
expect(lib.listCollections()).toEqual([]);
|
||||
for (const name of ["originals", "thumbnails"]) {
|
||||
expect(existsSync(join(cacheDirectory, name, "1001.jpg"))).toBe(
|
||||
false,
|
||||
);
|
||||
}
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// A server gives each file its own ID, so another account's cached file
|
||||
// is never served for this account's.
|
||||
it("keeps the originals and thumbnails another account on the same server cached", async () => {
|
||||
await fillCache();
|
||||
const client = new MockClient();
|
||||
client.userID = USER_ID + 1;
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: 3600,
|
||||
});
|
||||
try {
|
||||
expect(client.collectionsSinceTimes).toEqual([0]);
|
||||
expect(lib.listCollections()).toEqual([]);
|
||||
for (const name of ["originals", "thumbnails"]) {
|
||||
expect(existsSync(join(cacheDirectory, name, "1001.jpg"))).toBe(
|
||||
true,
|
||||
);
|
||||
}
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// `quak get-thumb` on an empty cache stores a thumbnail before any refresh
|
||||
// runs. `open()` for an account on another server deletes that thumbnail
|
||||
// only if the cache names the account that stored it.
|
||||
it("thumbnailOf on an empty cache saves the account's user ID and server in metadata.json", async () => {
|
||||
const client = new MockClient();
|
||||
client.apiOrigin = "https://ente.example.com";
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshOnOpen: false,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
contentSource: {
|
||||
original: async () => {
|
||||
throw new Error("original not expected");
|
||||
},
|
||||
thumbnail: async ({ destination }) => {
|
||||
writeFileSync(destination, "thumb");
|
||||
return { bytesWritten: 5 };
|
||||
},
|
||||
},
|
||||
});
|
||||
try {
|
||||
await lib.thumbnailOf(file(1001, 1, 400));
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
|
||||
const saved = await MetadataStore.load(
|
||||
join(cacheDirectory, "metadata.json"),
|
||||
);
|
||||
expect(saved.userID).toBe(USER_ID);
|
||||
expect(saved.apiOrigin).toBe("https://ente.example.com");
|
||||
});
|
||||
|
||||
// A refresh saves the account's server in `metadata.json`. Without it,
|
||||
// `open()` would take a self-hosted account's own cache for another
|
||||
// server's and delete it.
|
||||
it("keeps a self-hosted account's own cache from one open to the next", async () => {
|
||||
const client = new MockClient();
|
||||
client.apiOrigin = "https://ente.example.com";
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 100)],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
const first = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
});
|
||||
await first.close();
|
||||
mkdirSync(join(cacheDirectory, "originals"));
|
||||
writeFileSync(join(cacheDirectory, "originals", "1001.jpg"), "1001");
|
||||
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
});
|
||||
try {
|
||||
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
|
||||
expect(
|
||||
existsSync(join(cacheDirectory, "originals", "1001.jpg")),
|
||||
).toBe(true);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
// The second open's refresh started from the saved cursor.
|
||||
expect(client.collectionsSinceTimes).toEqual([0, 100]);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,241 @@
|
||||
// `script/build-bin` compiles `bin/quak`, the single binary that the README's
|
||||
// cron setup runs. bun compiles it from a copy of its own executable, so the
|
||||
// binary loads the same libraries as the bun that built it. nix's bun loads
|
||||
// libraries from the nix store, which garbage collection can delete: on Linux
|
||||
// its C library, on macOS its ICU. The script therefore downloads bun's own
|
||||
// release build, which loads only the system's libraries.
|
||||
//
|
||||
// What the script promises, and why each test matters:
|
||||
//
|
||||
// - Each system, machine and C library that bun publishes a build for gets its
|
||||
// own archive: Linux on x86-64 or arm64, with glibc or with musl, as on
|
||||
// Alpine, and macOS on Intel or Apple silicon. A bun built for another
|
||||
// system, machine or C library does not start, and `make install` fails.
|
||||
// - The archive's SHA-256 must match the one in the script before anything in
|
||||
// the archive runs. Without that check, a tampered download would build the
|
||||
// binary that cron runs.
|
||||
// - Any other system or machine stops with an error, and never falls back to
|
||||
// nix's bun. Building with nix's bun would bring back the dependency on the
|
||||
// nix store.
|
||||
//
|
||||
// Each test copies the script into a fresh directory, which the script then
|
||||
// treats as the checkout, and runs it there. Stubs first on PATH stand in for
|
||||
// `uname`, `ldd`, `curl`, `unzip`, `nix-shell` and `shasum`, so no test
|
||||
// downloads anything or runs a real bun.
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import { execFileSync, spawnSync } from "node:child_process";
|
||||
import {
|
||||
chmodSync,
|
||||
copyFileSync,
|
||||
existsSync,
|
||||
mkdirSync,
|
||||
mkdtempSync,
|
||||
readFileSync,
|
||||
rmSync,
|
||||
symlinkSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
|
||||
const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
|
||||
|
||||
// The stubs, keyed by the command each replaces. `uname` and `ldd` describe the
|
||||
// system that runBuildBin passes in the environment. `curl`, `unzip`,
|
||||
// `nix-shell` and `shasum` write the arguments they were given, one per line,
|
||||
// to a file named after the command in $STUB_LOG_DIR, which also shows that
|
||||
// they ran.
|
||||
const stubs: Record<string, string> = {
|
||||
uname: `
|
||||
case "$1" in
|
||||
-s) echo "$STUB_UNAME_S" ;;
|
||||
-m) echo "$STUB_UNAME_M" ;;
|
||||
esac`,
|
||||
// The script looks only at the start of the line: "musl libc" or not.
|
||||
ldd: `
|
||||
if [ "$STUB_LIBC" = musl ]; then
|
||||
echo "musl libc (x86_64)"
|
||||
else
|
||||
echo "ldd (GNU libc) 2.36"
|
||||
fi`,
|
||||
// Writes a file that is not the archive to the path after -o.
|
||||
curl: `
|
||||
printf '%s\\n' "$@" > "$STUB_LOG_DIR/curl"
|
||||
while [ $# -gt 1 ]; do
|
||||
if [ "$1" = -o ]; then echo "not the archive" > "$2"; fi
|
||||
shift
|
||||
done`,
|
||||
// Unpacks a stub bun where the real archive puts bun: in a directory named
|
||||
// after the archive, inside the directory after -d. That bun writes
|
||||
// bin/quak, as a real compile would.
|
||||
unzip: `
|
||||
printf '%s\\n' "$@" > "$STUB_LOG_DIR/unzip"
|
||||
for arg in "$@"; do
|
||||
case "$arg" in *.zip) archive="$arg" ;; esac
|
||||
if [ "$previous" = -d ]; then directory="$arg"; fi
|
||||
previous="$arg"
|
||||
done
|
||||
bun="$directory/$(basename "$archive" .zip)/bun"
|
||||
mkdir -p "$(dirname "$bun")"
|
||||
echo '#!/bin/sh' > "$bun"
|
||||
echo 'mkdir -p bin && echo binary > bin/quak' >> "$bun"
|
||||
chmod +x "$bun"`,
|
||||
"nix-shell": `
|
||||
printf '%s\\n' "$@" > "$STUB_LOG_DIR/nix-shell"`,
|
||||
// The script checks the hash with `shasum` where there is no `sha256sum`.
|
||||
// The stub fails, as `shasum` does when the hash does not match.
|
||||
shasum: `
|
||||
printf '%s\\n' "$@" > "$STUB_LOG_DIR/shasum"
|
||||
exit 1`,
|
||||
};
|
||||
|
||||
let checkout = "";
|
||||
|
||||
const writeScript = (path: string, body: string): void => {
|
||||
writeFileSync(path, `#!/bin/sh\n${body}\n`);
|
||||
chmodSync(path, 0o755);
|
||||
};
|
||||
|
||||
// A checkout holding the script, a stub script/build that does nothing, the
|
||||
// stubs, and the directory they log to.
|
||||
beforeEach(() => {
|
||||
checkout = mkdtempSync(join(tmpdir(), "quak-build-bin-"));
|
||||
mkdirSync(join(checkout, "script"));
|
||||
copyFileSync(
|
||||
join(repoRoot, "script/build-bin"),
|
||||
join(checkout, "script/build-bin"),
|
||||
);
|
||||
chmodSync(join(checkout, "script/build-bin"), 0o755);
|
||||
writeScript(join(checkout, "script/build"), "exit 0");
|
||||
mkdirSync(join(checkout, "stubs"));
|
||||
for (const [command, body] of Object.entries(stubs)) {
|
||||
writeScript(join(checkout, "stubs", command), body);
|
||||
}
|
||||
mkdirSync(join(checkout, "log"));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(checkout, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// Runs the script with nothing in its environment but the stubs' settings and a
|
||||
// PATH. Unless the caller passes one, the PATH finds the stubs first and then
|
||||
// the host's commands.
|
||||
const runBuildBin = (
|
||||
system: string,
|
||||
machine: string,
|
||||
{
|
||||
libc = "glibc",
|
||||
path = `${join(checkout, "stubs")}:${process.env.PATH}`,
|
||||
} = {},
|
||||
) =>
|
||||
spawnSync(join(checkout, "script/build-bin"), {
|
||||
cwd: checkout,
|
||||
encoding: "utf-8",
|
||||
env: {
|
||||
PATH: path,
|
||||
STUB_UNAME_S: system,
|
||||
STUB_UNAME_M: machine,
|
||||
STUB_LIBC: libc,
|
||||
STUB_LOG_DIR: join(checkout, "log"),
|
||||
},
|
||||
});
|
||||
|
||||
// The real commands that the script and the stubs run.
|
||||
const hostCommands = [
|
||||
"basename",
|
||||
"chmod",
|
||||
"dirname",
|
||||
"grep",
|
||||
"mkdir",
|
||||
"mktemp",
|
||||
"rm",
|
||||
];
|
||||
|
||||
// A Mac has no `ldd`, and not every Mac has `sha256sum`. This makes a PATH that
|
||||
// finds neither: the stubs other than `ldd`, then links to hostCommands.
|
||||
const makeMacPath = (): string => {
|
||||
rmSync(join(checkout, "stubs", "ldd"));
|
||||
mkdirSync(join(checkout, "host-commands"));
|
||||
for (const command of hostCommands) {
|
||||
const target = execFileSync("sh", ["-c", `command -v ${command}`], {
|
||||
encoding: "utf-8",
|
||||
}).trim();
|
||||
symlinkSync(target, join(checkout, "host-commands", command));
|
||||
}
|
||||
return `${join(checkout, "stubs")}:${join(checkout, "host-commands")}`;
|
||||
};
|
||||
|
||||
const ran = (command: string): boolean =>
|
||||
existsSync(join(checkout, "log", command));
|
||||
|
||||
const argumentsOf = (command: string): string[] =>
|
||||
readFileSync(join(checkout, "log", command), "utf-8")
|
||||
.trimEnd()
|
||||
.split("\n");
|
||||
|
||||
describe("script/build-bin", () => {
|
||||
// The stub curl writes a file that is not the archive, so the hash check
|
||||
// must stop the script before unzip runs. A script without the check would
|
||||
// run unzip, and the stub bun it unpacks would write bin/quak. On Linux the
|
||||
// check must use `sha256sum`, since a stock Alpine has no `shasum`.
|
||||
it.each([
|
||||
["Linux", "x86_64", "glibc", "bun-linux-x64-baseline.zip"],
|
||||
["Linux", "x86_64", "musl", "bun-linux-x64-musl-baseline.zip"],
|
||||
["Linux", "aarch64", "glibc", "bun-linux-aarch64.zip"],
|
||||
["Linux", "aarch64", "musl", "bun-linux-aarch64-musl.zip"],
|
||||
])(
|
||||
"on %s %s with %s, downloads %s and stops when its hash does not match",
|
||||
(system, machine, libc, archive) => {
|
||||
const result = runBuildBin(system, machine, { libc });
|
||||
expect(argumentsOf("curl")).toContain(
|
||||
"https://github.com/oven-sh/bun/releases/download/bun-v1.3.13/" +
|
||||
archive,
|
||||
);
|
||||
expect(ran("shasum")).toBe(false);
|
||||
expect(result.status).not.toBe(0);
|
||||
expect(ran("unzip")).toBe(false);
|
||||
expect(ran("nix-shell")).toBe(false);
|
||||
expect(existsSync(join(checkout, "bin/quak"))).toBe(false);
|
||||
},
|
||||
);
|
||||
|
||||
// On the PATH from makeMacPath the script finds no `sha256sum` and checks
|
||||
// the hash with the stub `shasum`. That stub fails, so the script must stop
|
||||
// before unzip runs.
|
||||
// On Apple silicon `uname -m` says arm64, where bun's archive says aarch64.
|
||||
it.each([
|
||||
["Darwin", "x86_64", "bun-darwin-x64-baseline.zip"],
|
||||
["Darwin", "arm64", "bun-darwin-aarch64.zip"],
|
||||
])(
|
||||
"on %s %s, downloads %s and stops when shasum rejects its hash",
|
||||
(system, machine, archive) => {
|
||||
const result = runBuildBin(system, machine, {
|
||||
path: makeMacPath(),
|
||||
});
|
||||
expect(argumentsOf("curl")).toContain(
|
||||
"https://github.com/oven-sh/bun/releases/download/bun-v1.3.13/" +
|
||||
archive,
|
||||
);
|
||||
expect(argumentsOf("shasum")).toEqual(["-a", "256", "-c", "-"]);
|
||||
expect(result.status).not.toBe(0);
|
||||
expect(ran("unzip")).toBe(false);
|
||||
expect(ran("nix-shell")).toBe(false);
|
||||
expect(existsSync(join(checkout, "bin/quak"))).toBe(false);
|
||||
},
|
||||
);
|
||||
|
||||
it.each([
|
||||
["Linux", "riscv64"],
|
||||
["FreeBSD", "amd64"],
|
||||
])(
|
||||
"stops on %s %s, which bun publishes no build for",
|
||||
(system, machine) => {
|
||||
const result = runBuildBin(system, machine);
|
||||
expect(result.status).not.toBe(0);
|
||||
expect(ran("curl")).toBe(false);
|
||||
expect(ran("nix-shell")).toBe(false);
|
||||
},
|
||||
);
|
||||
});
|
||||
@@ -273,6 +273,7 @@ const buildThumbMock = async (opts?: {
|
||||
|
||||
const buildThumbFetch = (m: ThumbMockState) => {
|
||||
let srpServer: SrpServer;
|
||||
let signedMD5: string | undefined;
|
||||
|
||||
return (async (
|
||||
input: RequestInfo | URL,
|
||||
@@ -365,8 +366,9 @@ const buildThumbFetch = (m: ThumbMockState) => {
|
||||
return new Response("not found", { status: 404 });
|
||||
}
|
||||
|
||||
// Upload URL minting
|
||||
// Upload URL minting. The real server signs the MD5 into the URL.
|
||||
if (path === "/files/upload-url") {
|
||||
signedMD5 = JSON.parse(init?.body as string).contentMD5;
|
||||
return json({
|
||||
objectKey: `42/thumb-${Date.now()}`,
|
||||
url: "https://s3.mock.test/presigned-put",
|
||||
@@ -375,6 +377,11 @@ const buildThumbFetch = (m: ThumbMockState) => {
|
||||
|
||||
// Presigned PUT (S3 upload)
|
||||
if (url.startsWith("https://s3.mock.test/")) {
|
||||
// Storage refuses a PUT that lacks the signed `Content-MD5`.
|
||||
const headers = new Headers(init?.headers);
|
||||
if (headers.get("Content-MD5") !== signedMD5) {
|
||||
return new Response("AccessDenied", { status: 400 });
|
||||
}
|
||||
const body = init?.body;
|
||||
// Store the uploaded bytes for later inspection
|
||||
if (body instanceof Uint8Array) {
|
||||
|
||||
Reference in new issue
Block a user