Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
62f1f5900d |
No files matched your search
@@ -26,8 +26,22 @@ check:
|
|||||||
build:
|
build:
|
||||||
@script/build
|
@script/build
|
||||||
|
|
||||||
build-bin:
|
# Bundles the built dist/, so the binary reports the version script/build
|
||||||
@script/build-bin
|
# stamped.
|
||||||
|
#
|
||||||
|
# On Linux, nix's bun loads through a C library in the nix store, and a binary
|
||||||
|
# bun compiles from itself keeps that loader. nix-collect-garbage can delete
|
||||||
|
# that C library, and the binary then no longer starts.
|
||||||
|
# A --target other than the host's makes bun use its downloaded release build,
|
||||||
|
# which loads through the system's C library. That is tested only on x86-64,
|
||||||
|
# with the baseline build, so other Linux machines are refused.
|
||||||
|
build-bin: build
|
||||||
|
case "$$(uname -s)-$$(uname -m)" in \
|
||||||
|
Linux-x86_64) target=--target=bun-linux-x64-baseline ;; \
|
||||||
|
Linux-*) echo "make build-bin: on Linux only x86-64 is supported" >&2; exit 1 ;; \
|
||||||
|
*) target= ;; \
|
||||||
|
esac; \
|
||||||
|
nix-shell -p bun --run "bun build dist/bin/quak.js --compile $$target --outfile bin/quak"
|
||||||
|
|
||||||
install: build-bin
|
install: build-bin
|
||||||
mkdir -p ~/bin
|
mkdir -p ~/bin
|
||||||
|
|||||||
@@ -31,8 +31,7 @@ again on the next run. Every download is decrypted on its way to disk.
|
|||||||
|
|
||||||
## Getting Started
|
## Getting Started
|
||||||
|
|
||||||
The commands below need `git`, `yarn` and Node 22.13 or later. Node 23 needs
|
The commands below need `git`, Node 22 or later and `yarn`.
|
||||||
23.5 or later.
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone https://git.eeqj.de/sneak/quak.git
|
git clone https://git.eeqj.de/sneak/quak.git
|
||||||
@@ -70,29 +69,11 @@ The library keeps a local copy of the account's album and file lists, the cache.
|
|||||||
It brings the cache up to date with the server in rounds called refreshes.
|
It brings the cache up to date with the server in rounds called refreshes.
|
||||||
|
|
||||||
quak is not published to a package registry: `package.json` marks it `private`,
|
quak is not published to a package registry: `package.json` marks it `private`,
|
||||||
and `dist/` is not committed. Clone and build it as above, then link the clone
|
and `dist/` is not committed. Clone and build it as above, then add the clone to
|
||||||
into your own program:
|
your own program's `package.json` by its path, for example
|
||||||
|
`yarn add file:../quak`. The package's `main`, `types` and `exports` fields
|
||||||
```bash
|
point into `dist/`, so the clone must be built first. The package is an ES
|
||||||
# In the quak clone:
|
module only, so a CommonJS program cannot `require` it.
|
||||||
yarn link
|
|
||||||
|
|
||||||
# In your program's directory:
|
|
||||||
yarn link quak
|
|
||||||
```
|
|
||||||
|
|
||||||
`yarn link` also installs the `quak` command in yarn's global bin directory, and
|
|
||||||
`yarn unlink` in the clone removes the link and the command.
|
|
||||||
|
|
||||||
A linked quak uses the dependencies that `yarn install` put in the clone, at the
|
|
||||||
versions in quak's `yarn.lock`. Avoid `yarn add file:../quak`: it resolves
|
|
||||||
quak's dependencies again, and their newer releases can need a newer Node.
|
|
||||||
`yarn link quak` does not add quak to your `package.json`, so run it again in
|
|
||||||
each new checkout of your program.
|
|
||||||
|
|
||||||
The package's `main`, `types` and `exports` fields point into `dist/`, so the
|
|
||||||
clone must be built first. The package is an ES module only, so a CommonJS
|
|
||||||
program cannot `require` it.
|
|
||||||
|
|
||||||
```ts
|
```ts
|
||||||
import { Client, Library } from "quak";
|
import { Client, Library } from "quak";
|
||||||
@@ -154,16 +135,13 @@ requests. The "Library reference" below covers both.
|
|||||||
## Running the backup from cron
|
## Running the backup from cron
|
||||||
|
|
||||||
`quak backup` never prompts, so cron can run it. `make install` builds quak as
|
`quak backup` never prompts, so cron can run it. `make install` builds quak as
|
||||||
one binary and copies it to `~/bin/quak`. It uses bun, a JavaScript runtime, to
|
one binary and copies it to `~/bin/quak`. Besides what Getting Started needs,
|
||||||
bundle the built `dist/` into one executable. Besides what Getting Started
|
`make install` needs `make` and [nix](https://nixos.org/download). It uses bun,
|
||||||
needs, `make install` needs `make`. On Linux it downloads bun from GitHub with
|
a JavaScript runtime, to bundle the built `dist/` into one executable. It runs
|
||||||
`curl` and unpacks it with `unzip`. It checks the download against a SHA-256
|
bun through `nix-shell -p bun`. On Linux it works on x86-64 only. There, the
|
||||||
hash in the repo. On macOS it runs bun through `nix-shell -p bun`, so it needs
|
binary is built from bun's own release build rather than nix's bun, so it does
|
||||||
[nix](https://nixos.org/download).
|
not need the nix store. The first build downloads that release build into
|
||||||
|
`~/.bun`.
|
||||||
On Linux systems that use the musl C library, such as Alpine, `make install`
|
|
||||||
downloads bun's musl build. On Alpine, both that bun and `~/bin/quak` need the
|
|
||||||
`libstdc++` and `libgcc` packages.
|
|
||||||
|
|
||||||
Log in once with `~/bin/quak`, as the user the cron job will run as:
|
Log in once with `~/bin/quak`, as the user the cron job will run as:
|
||||||
|
|
||||||
@@ -327,7 +305,7 @@ title, with unsafe characters replaced (see "File names from the server"). With
|
|||||||
original is kept in the cache as well.
|
original is kept in the cache as well.
|
||||||
|
|
||||||
A file ID that no collection holds gives `File <id> not found`, and the exit
|
A file ID that no collection holds gives `File <id> not found`, and the exit
|
||||||
status is 1.
|
status is 1. The option `--collection <id>` is accepted and ignored.
|
||||||
|
|
||||||
A live photo is a still image and a short video that Ente stores as one file.
|
A live photo is a still image and a short video that Ente stores as one file.
|
||||||
`get` writes both, each named after the title with its own extension, as Ente's
|
`get` writes both, each named after the title with its own extension, as Ente's
|
||||||
@@ -696,9 +674,6 @@ but letters and digits, or when it is longer than 16 characters.
|
|||||||
When the date or the time zone changes, the next run stores the original at its
|
When the date or the time zone changes, the next run stores the original at its
|
||||||
new path and leaves the old copy where it is.
|
new path and leaves the old copy where it is.
|
||||||
|
|
||||||
A file deleted in Ente keeps its original at its save path, and the JSON beside
|
|
||||||
it. The next run removes its symlinks from the collection directories.
|
|
||||||
|
|
||||||
The library gives the same path as `photo.savePath` and writes it with
|
The library gives the same path as `photo.savePath` and writes it with
|
||||||
`photo.download()` (see "A photo's content").
|
`photo.download()` (see "A photo's content").
|
||||||
|
|
||||||
@@ -1799,8 +1774,6 @@ standard. The others are additions to it.
|
|||||||
`package.json` declares (`main`, `types`, `bin`) were written, and make the
|
`package.json` declares (`main`, `types`, `bin`) were written, and make the
|
||||||
command line executable.
|
command line executable.
|
||||||
- `script/version`: print the version `script/build` stamps (see "Version").
|
- `script/version`: print the version `script/build` stamps (see "Version").
|
||||||
- `script/build-bin`: run `script/build`, then bundle the built `dist/` into one
|
|
||||||
binary, `bin/quak`, with bun (see "Running the backup from cron").
|
|
||||||
- `script/test`: run the test suite, by building the `test` phase of the
|
- `script/test`: run the test suite, by building the `test` phase of the
|
||||||
`Dockerfile`. Requires Docker.
|
`Dockerfile`. Requires Docker.
|
||||||
- `script/lint`: run eslint and a prettier check, by building the `lint` phase
|
- `script/lint`: run eslint and a prettier check, by building the `lint` phase
|
||||||
@@ -1829,7 +1802,8 @@ The Makefile targets are:
|
|||||||
- `make build`: runs `script/build`.
|
- `make build`: runs `script/build`.
|
||||||
- `make docker`: runs `script/docker`.
|
- `make docker`: runs `script/docker`.
|
||||||
- `make hooks`: runs `script/install-precommit`.
|
- `make hooks`: runs `script/install-precommit`.
|
||||||
- `make build-bin`: runs `script/build-bin`.
|
- `make build-bin`: bundles the built `dist/` into one binary, `bin/quak`. It
|
||||||
|
runs bun through `nix-shell -p bun`.
|
||||||
- `make install`: runs `make build-bin` and copies the binary to `~/bin/quak`.
|
- `make install`: runs `make build-bin` and copies the binary to `~/bin/quak`.
|
||||||
- `make dev`: runs the TypeScript compiler in watch mode.
|
- `make dev`: runs the TypeScript compiler in watch mode.
|
||||||
- `make clean`: removes `dist/`, `coverage/`, `.vitest-cache/` and the
|
- `make clean`: removes `dist/`, `coverage/`, `.vitest-cache/` and the
|
||||||
@@ -1937,11 +1911,10 @@ These apply to people and to coding agents alike.
|
|||||||
- container images by digest;
|
- container images by digest;
|
||||||
- the nvm archive `script/bootstrap` downloads, and the Python packages
|
- the nvm archive `script/bootstrap` downloads, and the Python packages
|
||||||
`bin/clip-keyword-vectors` installs, by SHA-256 hash;
|
`bin/clip-keyword-vectors` installs, by SHA-256 hash;
|
||||||
- bun's glibc and musl builds, which `script/build-bin` downloads on Linux,
|
|
||||||
by SHA-256 hash;
|
|
||||||
- Node and yarn, when `script/bootstrap` installs them, by version.
|
- Node and yarn, when `script/bootstrap` installs them, by version.
|
||||||
- Not pinned:
|
- Not pinned:
|
||||||
- bun on macOS, which `script/build-bin` takes from `nix-shell -p bun`;
|
- bun, which `make build-bin` takes from `nix-shell -p bun`, and on Linux
|
||||||
|
the release build of the same bun version that it downloads;
|
||||||
- `make` and `git`, which `script/bootstrap` installs from the system
|
- `make` and `git`, which `script/bootstrap` installs from the system
|
||||||
package manager when they are missing, as every `Dockerfile` stage does;
|
package manager when they are missing, as every `Dockerfile` stage does;
|
||||||
- `bash` and `curl`, which `script/bootstrap` installs the same way when it
|
- `bash` and `curl`, which `script/bootstrap` installs the same way when it
|
||||||
|
|||||||
@@ -123,6 +123,7 @@ program
|
|||||||
.description("Download and decrypt a single file")
|
.description("Download and decrypt a single file")
|
||||||
.argument("<fileID>", "File ID (from `quak files`)")
|
.argument("<fileID>", "File ID (from `quak files`)")
|
||||||
.option("--out <path>", "Output file path")
|
.option("--out <path>", "Output file path")
|
||||||
|
.option("--collection <id>", "Accepted for compatibility; ignored")
|
||||||
.action((fileID: string, opts: { out?: string }) =>
|
.action((fileID: string, opts: { out?: string }) =>
|
||||||
run(getCommand(context(), fileID, opts)),
|
run(getCommand(context(), fileID, opts)),
|
||||||
);
|
);
|
||||||
@@ -132,6 +133,7 @@ program
|
|||||||
.description("Download and decrypt a thumbnail")
|
.description("Download and decrypt a thumbnail")
|
||||||
.argument("<fileID>", "File ID (from `quak files`)")
|
.argument("<fileID>", "File ID (from `quak files`)")
|
||||||
.option("--out <path>", "Output file path")
|
.option("--out <path>", "Output file path")
|
||||||
|
.option("--collection <id>", "Accepted for compatibility; ignored")
|
||||||
.action((fileID: string, opts: { out?: string }) =>
|
.action((fileID: string, opts: { out?: string }) =>
|
||||||
run(getThumbCommand(context(), fileID, opts)),
|
run(getThumbCommand(context(), fileID, opts)),
|
||||||
);
|
);
|
||||||
|
|||||||
+1
-1
@@ -12,7 +12,7 @@
|
|||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"engines": {
|
"engines": {
|
||||||
"node": "^22.13.0 || >=23.5.0"
|
"node": ">=22"
|
||||||
},
|
},
|
||||||
"main": "./dist/src/index.js",
|
"main": "./dist/src/index.js",
|
||||||
"types": "./dist/src/index.d.ts",
|
"types": "./dist/src/index.d.ts",
|
||||||
|
|||||||
@@ -1,73 +0,0 @@
|
|||||||
#!/bin/sh
|
|
||||||
# script/build-bin: run script/build, then bundle the built dist/ into one
|
|
||||||
# executable, bin/quak, which reports the version script/build stamped. Our own
|
|
||||||
# extension to scripts-to-rule-them-all.
|
|
||||||
#
|
|
||||||
# bun compiles the binary from a copy of its own executable. On Linux, nix's
|
|
||||||
# bun loads through a C library in the nix store, and so does a binary compiled
|
|
||||||
# from it. nix-collect-garbage can delete that library, and the binary then no
|
|
||||||
# longer starts. On Linux this script therefore compiles with bun's own release
|
|
||||||
# build, which loads through the system's C library. It downloads that build
|
|
||||||
# and checks it against the hash below before running it.
|
|
||||||
set -eu
|
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd -P)"
|
|
||||||
|
|
||||||
# bun 1.3.13, 2026-11-11, and the SHA-256 of each release archive below. The
|
|
||||||
# x86-64 archives are the baseline builds, which also run on CPUs without AVX2.
|
|
||||||
# The musl archives are for Linux systems that use the musl C library, such as
|
|
||||||
# Alpine.
|
|
||||||
BUN_VERSION="1.3.13"
|
|
||||||
BUN_LINUX_X64_BASELINE_SHA256="9d8a24292a7068090205daac0a5a223f5f69736f5287e37bf88d3b4031edc750"
|
|
||||||
BUN_LINUX_X64_MUSL_BASELINE_SHA256="88ca7c7ad235b498f549eea2f770f434e9f0f5e9ba95168a2d3a1f235184c394"
|
|
||||||
BUN_LINUX_AARCH64_SHA256="70bae41b3908b0a120e1e58c5c8af30e74afae3b8d11b0d3fdd8e787ddfb4b22"
|
|
||||||
BUN_LINUX_AARCH64_MUSL_SHA256="5385e978107ce4934298d8d6afe9bfbb898683f6cc23e6753a0da60bc60c5b81"
|
|
||||||
|
|
||||||
# compile_with_downloaded_bun <archive-name> <sha256>
|
|
||||||
compile_with_downloaded_bun() {
|
|
||||||
tmp="$(mktemp -d)"
|
|
||||||
trap 'rm -rf "$tmp"' EXIT
|
|
||||||
curl -fsSL -o "$tmp/$1.zip" \
|
|
||||||
"https://github.com/oven-sh/bun/releases/download/bun-v$BUN_VERSION/$1.zip"
|
|
||||||
echo "$2 $tmp/$1.zip" | sha256sum -c -
|
|
||||||
unzip -q "$tmp/$1.zip" -d "$tmp"
|
|
||||||
"$tmp/$1/bun" build dist/bin/quak.js --compile --outfile bin/quak
|
|
||||||
}
|
|
||||||
|
|
||||||
main() {
|
|
||||||
cd "$ROOT"
|
|
||||||
"$ROOT/script/build"
|
|
||||||
platform="$(uname -s)-$(uname -m)"
|
|
||||||
# musl's ldd begins its version output with "musl libc"; glibc's does not.
|
|
||||||
if ldd --version 2>&1 | grep -q '^musl libc'; then
|
|
||||||
platform="$platform-musl"
|
|
||||||
fi
|
|
||||||
case "$platform" in
|
|
||||||
Linux-x86_64)
|
|
||||||
compile_with_downloaded_bun bun-linux-x64-baseline \
|
|
||||||
"$BUN_LINUX_X64_BASELINE_SHA256"
|
|
||||||
;;
|
|
||||||
Linux-x86_64-musl)
|
|
||||||
compile_with_downloaded_bun bun-linux-x64-musl-baseline \
|
|
||||||
"$BUN_LINUX_X64_MUSL_BASELINE_SHA256"
|
|
||||||
;;
|
|
||||||
Linux-aarch64)
|
|
||||||
compile_with_downloaded_bun bun-linux-aarch64 \
|
|
||||||
"$BUN_LINUX_AARCH64_SHA256"
|
|
||||||
;;
|
|
||||||
Linux-aarch64-musl)
|
|
||||||
compile_with_downloaded_bun bun-linux-aarch64-musl \
|
|
||||||
"$BUN_LINUX_AARCH64_MUSL_SHA256"
|
|
||||||
;;
|
|
||||||
Linux-*)
|
|
||||||
echo "build-bin: bun has no release build for Linux on $(uname -m)" >&2
|
|
||||||
exit 1
|
|
||||||
;;
|
|
||||||
*)
|
|
||||||
nix-shell -p bun --run \
|
|
||||||
"bun build dist/bin/quak.js --compile --outfile bin/quak"
|
|
||||||
;;
|
|
||||||
esac
|
|
||||||
}
|
|
||||||
|
|
||||||
main "$@"
|
|
||||||
+1
-8
@@ -354,13 +354,7 @@ export class ApiClient {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
// `contentMD5` is the value given to `getUploadURL`. The server signs it
|
async putFile(presignedURL: string, data: Uint8Array): Promise<void> {
|
||||||
// into the URL, and storage refuses a PUT without that `Content-MD5`.
|
|
||||||
async putFile(
|
|
||||||
presignedURL: string,
|
|
||||||
data: Uint8Array,
|
|
||||||
contentMD5: string,
|
|
||||||
): Promise<void> {
|
|
||||||
// Idempotent despite being a write: a presigned PUT stores one whole
|
// Idempotent despite being a write: a presigned PUT stores one whole
|
||||||
// object at one key in one request, so replaying it either overwrites
|
// object at one key in one request, so replaying it either overwrites
|
||||||
// the same bytes or lands them for the first time. There is no partial
|
// the same bytes or lands them for the first time. There is no partial
|
||||||
@@ -376,7 +370,6 @@ export class ApiClient {
|
|||||||
headers: {
|
headers: {
|
||||||
"Content-Type": "application/octet-stream",
|
"Content-Type": "application/octet-stream",
|
||||||
"Content-Length": String(data.length),
|
"Content-Length": String(data.length),
|
||||||
"Content-MD5": contentMD5,
|
|
||||||
},
|
},
|
||||||
body: data,
|
body: data,
|
||||||
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
signal: AbortSignal.timeout(this.requestTimeoutMs),
|
||||||
|
|||||||
+1
-1
@@ -325,7 +325,7 @@ export const fixMissingThumbnails = async (
|
|||||||
ciphertext.length,
|
ciphertext.length,
|
||||||
md5,
|
md5,
|
||||||
);
|
);
|
||||||
await api.putFile(url, ciphertext, md5);
|
await api.putFile(url, ciphertext);
|
||||||
await api.updateThumbnail(file.id, objectKey, toBase64(header));
|
await api.updateThumbnail(file.id, objectKey, toBase64(header));
|
||||||
|
|
||||||
log(`[${collectionName}] Thumbnail uploaded for ${title}`);
|
log(`[${collectionName}] Thumbnail uploaded for ${title}`);
|
||||||
|
|||||||
+4
-10
@@ -780,7 +780,7 @@ describe("ApiClient timeouts", () => {
|
|||||||
|
|
||||||
await client.getJSON("/a");
|
await client.getJSON("/a");
|
||||||
await client.postJSON("/b", {});
|
await client.postJSON("/b", {});
|
||||||
await client.putFile("https://s3.example/x", new Uint8Array([1]), "m");
|
await client.putFile("https://s3.example/x", new Uint8Array([1]));
|
||||||
await client.putJSON("/c", {});
|
await client.putJSON("/c", {});
|
||||||
|
|
||||||
for (const call of calls) {
|
for (const call of calls) {
|
||||||
@@ -825,8 +825,7 @@ describe("ApiClient timeouts", () => {
|
|||||||
[
|
[
|
||||||
"putFile",
|
"putFile",
|
||||||
() => new Response(null, { status: 200 }),
|
() => new Response(null, { status: 200 }),
|
||||||
(c) =>
|
(c) => c.putFile("https://s3.example/x", new Uint8Array([1])),
|
||||||
c.putFile("https://s3.example/x", new Uint8Array([1]), "m"),
|
|
||||||
],
|
],
|
||||||
[
|
[
|
||||||
"getFileStream",
|
"getFileStream",
|
||||||
@@ -1050,7 +1049,7 @@ describe("ApiClient error typing", () => {
|
|||||||
const client = new ApiClient({ fetch, retry: noWait });
|
const client = new ApiClient({ fetch, retry: noWait });
|
||||||
|
|
||||||
const err: unknown = await client
|
const err: unknown = await client
|
||||||
.putFile("https://s3.example/obj", new Uint8Array([1, 2]), "m")
|
.putFile("https://s3.example/obj", new Uint8Array([1, 2]))
|
||||||
.catch((e: unknown) => e);
|
.catch((e: unknown) => e);
|
||||||
|
|
||||||
expect(err).toBeInstanceOf(ApiError);
|
expect(err).toBeInstanceOf(ApiError);
|
||||||
@@ -1068,11 +1067,7 @@ describe("ApiClient error typing", () => {
|
|||||||
);
|
);
|
||||||
const client = new ApiClient({ fetch, retry: noWait });
|
const client = new ApiClient({ fetch, retry: noWait });
|
||||||
|
|
||||||
await client.putFile(
|
await client.putFile("https://s3.example/obj", new Uint8Array([1, 2]));
|
||||||
"https://s3.example/obj",
|
|
||||||
new Uint8Array([1, 2]),
|
|
||||||
"m",
|
|
||||||
);
|
|
||||||
expect(calls).toHaveLength(2);
|
expect(calls).toHaveLength(2);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -1282,7 +1277,6 @@ describe("ApiClient onRequest", () => {
|
|||||||
await client.putFile(
|
await client.putFile(
|
||||||
"https://s3.example/obj?X-Amz-Signature=abc",
|
"https://s3.example/obj?X-Amz-Signature=abc",
|
||||||
new Uint8Array([1, 2]),
|
new Uint8Array([1, 2]),
|
||||||
"m",
|
|
||||||
);
|
);
|
||||||
|
|
||||||
expect(requests).toEqual([
|
expect(requests).toEqual([
|
||||||
|
|||||||
+2
-13
@@ -80,20 +80,13 @@ describe("ApiClient.putFile", () => {
|
|||||||
const client = new ApiClient({ fetch, authToken: "secret-tok" });
|
const client = new ApiClient({ fetch, authToken: "secret-tok" });
|
||||||
const data = new Uint8Array([1, 2, 3, 4, 5]);
|
const data = new Uint8Array([1, 2, 3, 4, 5]);
|
||||||
|
|
||||||
await client.putFile(
|
await client.putFile("https://s3.example.com/presigned?sig=abc", data);
|
||||||
"https://s3.example.com/presigned?sig=abc",
|
|
||||||
data,
|
|
||||||
"md5base64==",
|
|
||||||
);
|
|
||||||
|
|
||||||
expect(calls[0]!.url).toBe("https://s3.example.com/presigned?sig=abc");
|
expect(calls[0]!.url).toBe("https://s3.example.com/presigned?sig=abc");
|
||||||
expect(calls[0]!.init?.method).toBe("PUT");
|
expect(calls[0]!.init?.method).toBe("PUT");
|
||||||
const headers = new Headers(calls[0]!.init?.headers as HeadersInit);
|
const headers = new Headers(calls[0]!.init?.headers as HeadersInit);
|
||||||
expect(headers.get("Content-Type")).toBe("application/octet-stream");
|
expect(headers.get("Content-Type")).toBe("application/octet-stream");
|
||||||
expect(headers.get("Content-Length")).toBe("5");
|
expect(headers.get("Content-Length")).toBe("5");
|
||||||
// The server signs the MD5 into the URL; storage refuses the PUT
|
|
||||||
// without it.
|
|
||||||
expect(headers.get("Content-MD5")).toBe("md5base64==");
|
|
||||||
// Must NOT leak auth headers to S3
|
// Must NOT leak auth headers to S3
|
||||||
expect(headers.has("X-Auth-Token")).toBe(false);
|
expect(headers.has("X-Auth-Token")).toBe(false);
|
||||||
expect(headers.has("X-Client-Package")).toBe(false);
|
expect(headers.has("X-Client-Package")).toBe(false);
|
||||||
@@ -106,11 +99,7 @@ describe("ApiClient.putFile", () => {
|
|||||||
const client = new ApiClient({ fetch });
|
const client = new ApiClient({ fetch });
|
||||||
|
|
||||||
await expect(
|
await expect(
|
||||||
client.putFile(
|
client.putFile("https://s3.example.com/bad", new Uint8Array(10)),
|
||||||
"https://s3.example.com/bad",
|
|
||||||
new Uint8Array(10),
|
|
||||||
"md5base64==",
|
|
||||||
),
|
|
||||||
).rejects.toThrow(/403/);
|
).rejects.toThrow(/403/);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
+2
-14
@@ -1,9 +1,8 @@
|
|||||||
/**
|
/**
|
||||||
* Tests for the retry options `bin/quak.ts` loads the saved session with
|
* Tests for the retry options `bin/quak.ts` loads the saved session with
|
||||||
* (`quak backup` gets the unattended ones, every other command the default),
|
* (`quak backup` gets the unattended ones, every other command the default),
|
||||||
* each with the callbacks that log its requests and retries, for `quak files`
|
* each with the callbacks that log its requests and retries, and for
|
||||||
* taking its collection ID as an optional argument, and for `quak get` and
|
* `quak files` taking its collection ID as an optional argument.
|
||||||
* `quak get-thumb` refusing `--collection`.
|
|
||||||
*
|
*
|
||||||
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
|
* Each test runs `bin/quak.ts`, as the smoke test does, with its session loader
|
||||||
* replaced by one that records the options it is given and reports no session,
|
* replaced by one that records the options it is given and reports no session,
|
||||||
@@ -106,14 +105,3 @@ describe("bin/quak.ts files", () => {
|
|||||||
expect(await quak("files", "7")).toMatch(/^Not logged in/);
|
expect(await quak("files", "7")).toMatch(/^Not logged in/);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("bin/quak.ts get and get-thumb", () => {
|
|
||||||
it.each(["get", "get-thumb"])(
|
|
||||||
"%s refuses --collection as an unknown option",
|
|
||||||
async (command) => {
|
|
||||||
expect(await quak(command, "1", "--collection", "2")).toMatch(
|
|
||||||
/^error: unknown option '--collection'/,
|
|
||||||
);
|
|
||||||
},
|
|
||||||
);
|
|
||||||
});
|
|
||||||
@@ -569,14 +569,14 @@ describe("files", () => {
|
|||||||
});
|
});
|
||||||
|
|
||||||
describe("get and get-thumb", () => {
|
describe("get and get-thumb", () => {
|
||||||
it("get finds a file in any album", async () => {
|
it("get finds a file in any album without --collection", async () => {
|
||||||
const out = join(root, "diagram.png");
|
const out = join(root, "diagram.png");
|
||||||
expect(await getCommand(context(), "200", { out })).toBe(0);
|
expect(await getCommand(context(), "200", { out })).toBe(0);
|
||||||
expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
|
expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
|
||||||
expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
|
expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
|
||||||
});
|
});
|
||||||
|
|
||||||
it("get-thumb finds a file in any album", async () => {
|
it("get-thumb finds a file in any album without --collection", async () => {
|
||||||
const out = join(root, "thumb.jpg");
|
const out = join(root, "thumb.jpg");
|
||||||
expect(await getThumbCommand(context(), "200", { out })).toBe(0);
|
expect(await getThumbCommand(context(), "200", { out })).toBe(0);
|
||||||
expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
|
expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
|
||||||
|
|||||||
@@ -1,180 +0,0 @@
|
|||||||
// `script/build-bin` compiles `bin/quak`, the single binary that the README's
|
|
||||||
// cron setup runs. bun compiles it from a copy of its own executable, so the
|
|
||||||
// binary loads through the same C library as the bun that built it. nix's bun
|
|
||||||
// loads through a C library in the nix store, which garbage collection can
|
|
||||||
// delete. On Linux the script therefore downloads bun's own release build,
|
|
||||||
// which loads through the system's C library.
|
|
||||||
//
|
|
||||||
// What the script promises, and why each test matters:
|
|
||||||
//
|
|
||||||
// - On Linux, each machine and C library that bun publishes a build for gets
|
|
||||||
// its own archive: x86-64 or arm64, with glibc or with musl, as on Alpine. A
|
|
||||||
// bun built for another machine or C library does not start, and
|
|
||||||
// `make install` fails.
|
|
||||||
// - The archive's SHA-256 must match the one in the script before anything in
|
|
||||||
// the archive runs. Without that check, a tampered download would build the
|
|
||||||
// binary that cron runs.
|
|
||||||
// - Linux on any other machine stops with an error. Building there with nix's
|
|
||||||
// bun would bring back the dependency on the nix store.
|
|
||||||
// - Other systems, such as macOS, compile with nix's bun as before, and
|
|
||||||
// download nothing.
|
|
||||||
//
|
|
||||||
// Each test copies the script into a fresh directory, which the script then
|
|
||||||
// treats as the checkout, and runs it there. Stubs first on PATH stand in for
|
|
||||||
// `uname`, `ldd`, `curl`, `unzip` and `nix-shell`, so no test downloads
|
|
||||||
// anything or runs a real bun.
|
|
||||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
|
||||||
import { spawnSync } from "node:child_process";
|
|
||||||
import {
|
|
||||||
chmodSync,
|
|
||||||
copyFileSync,
|
|
||||||
existsSync,
|
|
||||||
mkdirSync,
|
|
||||||
mkdtempSync,
|
|
||||||
readFileSync,
|
|
||||||
rmSync,
|
|
||||||
writeFileSync,
|
|
||||||
} from "node:fs";
|
|
||||||
import { tmpdir } from "node:os";
|
|
||||||
import { join } from "node:path";
|
|
||||||
import { fileURLToPath } from "node:url";
|
|
||||||
|
|
||||||
const repoRoot = fileURLToPath(new URL("../../", import.meta.url));
|
|
||||||
|
|
||||||
// The stubs, keyed by the command each replaces. `uname` and `ldd` describe the
|
|
||||||
// system that runBuildBin passes in the environment. `curl`, `unzip` and
|
|
||||||
// `nix-shell` write the arguments they were given, one per line, to a file
|
|
||||||
// named after the command in $STUB_LOG_DIR, which also shows that they ran.
|
|
||||||
const stubs: Record<string, string> = {
|
|
||||||
uname: `
|
|
||||||
case "$1" in
|
|
||||||
-s) echo "$STUB_UNAME_S" ;;
|
|
||||||
-m) echo "$STUB_UNAME_M" ;;
|
|
||||||
esac`,
|
|
||||||
// The script looks only at the start of the line: "musl libc" or not.
|
|
||||||
ldd: `
|
|
||||||
if [ "$STUB_LIBC" = musl ]; then
|
|
||||||
echo "musl libc (x86_64)"
|
|
||||||
else
|
|
||||||
echo "ldd (GNU libc) 2.36"
|
|
||||||
fi`,
|
|
||||||
// Writes a file that is not the archive to the path after -o.
|
|
||||||
curl: `
|
|
||||||
printf '%s\\n' "$@" > "$STUB_LOG_DIR/curl"
|
|
||||||
while [ $# -gt 1 ]; do
|
|
||||||
if [ "$1" = -o ]; then echo "not the archive" > "$2"; fi
|
|
||||||
shift
|
|
||||||
done`,
|
|
||||||
// Unpacks a stub bun where the real archive puts bun: in a directory named
|
|
||||||
// after the archive, inside the directory after -d. That bun writes
|
|
||||||
// bin/quak, as a real compile would.
|
|
||||||
unzip: `
|
|
||||||
printf '%s\\n' "$@" > "$STUB_LOG_DIR/unzip"
|
|
||||||
for arg in "$@"; do
|
|
||||||
case "$arg" in *.zip) archive="$arg" ;; esac
|
|
||||||
if [ "$previous" = -d ]; then directory="$arg"; fi
|
|
||||||
previous="$arg"
|
|
||||||
done
|
|
||||||
bun="$directory/$(basename "$archive" .zip)/bun"
|
|
||||||
mkdir -p "$(dirname "$bun")"
|
|
||||||
echo '#!/bin/sh' > "$bun"
|
|
||||||
echo 'mkdir -p bin && echo binary > bin/quak' >> "$bun"
|
|
||||||
chmod +x "$bun"`,
|
|
||||||
"nix-shell": `
|
|
||||||
printf '%s\\n' "$@" > "$STUB_LOG_DIR/nix-shell"`,
|
|
||||||
};
|
|
||||||
|
|
||||||
let checkout = "";
|
|
||||||
|
|
||||||
const writeScript = (path: string, body: string): void => {
|
|
||||||
writeFileSync(path, `#!/bin/sh\n${body}\n`);
|
|
||||||
chmodSync(path, 0o755);
|
|
||||||
};
|
|
||||||
|
|
||||||
// A checkout holding the script, a stub script/build that does nothing, the
|
|
||||||
// stubs, and the directory they log to.
|
|
||||||
beforeEach(() => {
|
|
||||||
checkout = mkdtempSync(join(tmpdir(), "quak-build-bin-"));
|
|
||||||
mkdirSync(join(checkout, "script"));
|
|
||||||
copyFileSync(
|
|
||||||
join(repoRoot, "script/build-bin"),
|
|
||||||
join(checkout, "script/build-bin"),
|
|
||||||
);
|
|
||||||
chmodSync(join(checkout, "script/build-bin"), 0o755);
|
|
||||||
writeScript(join(checkout, "script/build"), "exit 0");
|
|
||||||
mkdirSync(join(checkout, "stubs"));
|
|
||||||
for (const [command, body] of Object.entries(stubs)) {
|
|
||||||
writeScript(join(checkout, "stubs", command), body);
|
|
||||||
}
|
|
||||||
mkdirSync(join(checkout, "log"));
|
|
||||||
});
|
|
||||||
|
|
||||||
afterEach(() => {
|
|
||||||
rmSync(checkout, { recursive: true, force: true });
|
|
||||||
});
|
|
||||||
|
|
||||||
// Runs the script with nothing in its environment but the stubs' settings and a
|
|
||||||
// PATH that finds the stubs first.
|
|
||||||
const runBuildBin = (system: string, machine: string, libc = "glibc") =>
|
|
||||||
spawnSync(join(checkout, "script/build-bin"), {
|
|
||||||
cwd: checkout,
|
|
||||||
encoding: "utf-8",
|
|
||||||
env: {
|
|
||||||
PATH: `${join(checkout, "stubs")}:${process.env.PATH}`,
|
|
||||||
STUB_UNAME_S: system,
|
|
||||||
STUB_UNAME_M: machine,
|
|
||||||
STUB_LIBC: libc,
|
|
||||||
STUB_LOG_DIR: join(checkout, "log"),
|
|
||||||
},
|
|
||||||
});
|
|
||||||
|
|
||||||
const ran = (command: string): boolean =>
|
|
||||||
existsSync(join(checkout, "log", command));
|
|
||||||
|
|
||||||
const argumentsOf = (command: string): string[] =>
|
|
||||||
readFileSync(join(checkout, "log", command), "utf-8")
|
|
||||||
.trimEnd()
|
|
||||||
.split("\n");
|
|
||||||
|
|
||||||
describe("script/build-bin", () => {
|
|
||||||
// The stub curl writes a file that is not the archive, so the hash check
|
|
||||||
// must stop the script before unzip runs. A script without the check would
|
|
||||||
// run unzip, and the stub bun it unpacks would write bin/quak.
|
|
||||||
it.each([
|
|
||||||
["x86_64", "glibc", "bun-linux-x64-baseline.zip"],
|
|
||||||
["x86_64", "musl", "bun-linux-x64-musl-baseline.zip"],
|
|
||||||
["aarch64", "glibc", "bun-linux-aarch64.zip"],
|
|
||||||
["aarch64", "musl", "bun-linux-aarch64-musl.zip"],
|
|
||||||
])(
|
|
||||||
"on Linux %s with %s, downloads %s and stops when its hash does not match",
|
|
||||||
(machine, libc, archive) => {
|
|
||||||
const result = runBuildBin("Linux", machine, libc);
|
|
||||||
expect(argumentsOf("curl")).toContain(
|
|
||||||
"https://github.com/oven-sh/bun/releases/download/bun-v1.3.13/" +
|
|
||||||
archive,
|
|
||||||
);
|
|
||||||
expect(result.status).not.toBe(0);
|
|
||||||
expect(ran("unzip")).toBe(false);
|
|
||||||
expect(existsSync(join(checkout, "bin/quak"))).toBe(false);
|
|
||||||
},
|
|
||||||
);
|
|
||||||
|
|
||||||
it("stops on a Linux machine that bun publishes no build for", () => {
|
|
||||||
const result = runBuildBin("Linux", "riscv64");
|
|
||||||
expect(result.status).not.toBe(0);
|
|
||||||
expect(ran("curl")).toBe(false);
|
|
||||||
expect(ran("nix-shell")).toBe(false);
|
|
||||||
});
|
|
||||||
|
|
||||||
it("compiles with nix's bun on macOS", () => {
|
|
||||||
const result = runBuildBin("Darwin", "arm64");
|
|
||||||
expect(result.status).toBe(0);
|
|
||||||
expect(ran("curl")).toBe(false);
|
|
||||||
expect(argumentsOf("nix-shell")).toEqual([
|
|
||||||
"-p",
|
|
||||||
"bun",
|
|
||||||
"--run",
|
|
||||||
"bun build dist/bin/quak.js --compile --outfile bin/quak",
|
|
||||||
]);
|
|
||||||
});
|
|
||||||
});
|
|
||||||
@@ -273,7 +273,6 @@ const buildThumbMock = async (opts?: {
|
|||||||
|
|
||||||
const buildThumbFetch = (m: ThumbMockState) => {
|
const buildThumbFetch = (m: ThumbMockState) => {
|
||||||
let srpServer: SrpServer;
|
let srpServer: SrpServer;
|
||||||
let signedMD5: string | undefined;
|
|
||||||
|
|
||||||
return (async (
|
return (async (
|
||||||
input: RequestInfo | URL,
|
input: RequestInfo | URL,
|
||||||
@@ -366,9 +365,8 @@ const buildThumbFetch = (m: ThumbMockState) => {
|
|||||||
return new Response("not found", { status: 404 });
|
return new Response("not found", { status: 404 });
|
||||||
}
|
}
|
||||||
|
|
||||||
// Upload URL minting. The real server signs the MD5 into the URL.
|
// Upload URL minting
|
||||||
if (path === "/files/upload-url") {
|
if (path === "/files/upload-url") {
|
||||||
signedMD5 = JSON.parse(init?.body as string).contentMD5;
|
|
||||||
return json({
|
return json({
|
||||||
objectKey: `42/thumb-${Date.now()}`,
|
objectKey: `42/thumb-${Date.now()}`,
|
||||||
url: "https://s3.mock.test/presigned-put",
|
url: "https://s3.mock.test/presigned-put",
|
||||||
@@ -377,11 +375,6 @@ const buildThumbFetch = (m: ThumbMockState) => {
|
|||||||
|
|
||||||
// Presigned PUT (S3 upload)
|
// Presigned PUT (S3 upload)
|
||||||
if (url.startsWith("https://s3.mock.test/")) {
|
if (url.startsWith("https://s3.mock.test/")) {
|
||||||
// Storage refuses a PUT that lacks the signed `Content-MD5`.
|
|
||||||
const headers = new Headers(init?.headers);
|
|
||||||
if (headers.get("Content-MD5") !== signedMD5) {
|
|
||||||
return new Response("AccessDenied", { status: 400 });
|
|
||||||
}
|
|
||||||
const body = init?.body;
|
const body = init?.body;
|
||||||
// Store the uploaded bytes for later inspection
|
// Store the uploaded bytes for later inspection
|
||||||
if (body instanceof Uint8Array) {
|
if (body instanceof Uint8Array) {
|
||||||
|
|||||||
Reference in new issue
Block a user