1 Commits
Author SHA1 Message Date
clawbot 80f691413a Make lint and test phases of the Dockerfile (closes #96)
check / check (push) Successful in 1m40s
Follows the template: Dockerfile.lint is gone; the Dockerfile has a lint
phase (eslint, prettier --check .) and a test phase (vitest, run as the
node user, which the not-writable-directory tests need), and its
last stage compiles and depends on both. script/lint and script/test
build one phase each with --no-cache; script/docker and script/cibuild
pass --no-cache, so CHECK_EPOCH and LINT_EPOCH are removed.
script/cibuild is the single image build, so CI runs lint and the tests
once each. The tests that checked the old layout are deleted,
REPO_POLICIES.md is re-copied and the README describes the new layout.

Model: opus-5-5
2026-09-23 04:50:28 +00:00
19 changed files with 198 additions and 1233 deletions
+30 -86
View File
@@ -214,23 +214,11 @@ quak/
auth/ login flow (SRP + email OTP + TOTP), key unwrap auth/ login flow (SRP + email OTP + TOTP), key unwrap
model/ decrypted Collection, File, Metadata types + decrypt fns model/ decrypted Collection, File, Metadata types + decrypt fns
download/ streaming file/thumbnail download + decryption download/ streaming file/thumbnail download + decryption
library/ the cache-backed Library: metadata store, read
surface, records, content cache, precache, ML data
and search, request pools
backup.ts resilient full-account backup with dedup backup.ts resilient full-account backup with dedup
metadata-backup.ts
backup-metadata: all decrypted metadata as JSON
mldata-fetch.ts fetch + decrypt per-file ML data
filename.ts safe file names from server metadata
errors.ts error types shared across layers errors.ts error types shared across layers
retry.ts retry classifier + exponential backoff with jitter retry.ts retry classifier + exponential backoff with jitter
thumbnails.ts detect + regenerate missing thumbnails thumbnails.ts detect + regenerate missing thumbnails
client.ts high-level Client class assembled from the above client.ts high-level Client class assembled from the above
cli-commands.ts the CLI's commands as functions returning exit codes
cli-output.ts how the CLI prints a file's title and time
cli-read.ts fresh reads for the CLI's read commands
cli-run.ts run a command, print its error, exit with its code
cli-session.ts read the saved session file back into a Client
index.ts public library exports index.ts public library exports
bin/ bin/
quak.ts CLI entrypoint (commander.js) quak.ts CLI entrypoint (commander.js)
@@ -308,7 +296,6 @@ Endpoints used:
encrypted token plus key attributes. encrypted token plus key attributes.
- `POST /users/ott` and `POST /users/verify-email`: email OTP fallback path. - `POST /users/ott` and `POST /users/verify-email`: email OTP fallback path.
- `POST /users/two-factor/verify`: TOTP second factor. - `POST /users/two-factor/verify`: TOTP second factor.
- `POST /users/logout`: end the calling token's session (`quak logout`).
- `GET /collections/v2?sinceTime=<usec>`: list collections changed since - `GET /collections/v2?sinceTime=<usec>`: list collections changed since
microsecond timestamp; pass 0 for a full enumeration. microsecond timestamp; pass 0 for a full enumeration.
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a - `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a
@@ -423,9 +410,7 @@ whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
working client from that snapshot without re-authenticating; it checks every working client from that snapshot without re-authenticating; it checks every
field and each key's length first, and throws an error naming the bad field. field and each key's length first, and throws an error naming the bad field.
`client.logout()` clears the token and zeroes the key buffers in place; every `client.logout()` clears the token and zeroes the key buffers in place; every
later call on that client throws. It does not contact the server, so the token later call on that client throws.
stays valid there and in any saved snapshot; `await client.logoutOnServer()`
first ends the session on the server (`POST /users/logout`).
The CLI stores the snapshot at the platform-appropriate data directory via The CLI stores the snapshot at the platform-appropriate data directory via
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS, `env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
@@ -435,21 +420,13 @@ you would treat the password itself. A missing file is reported as "not logged
in"; a file that exists but is corrupt is reported as such, naming the bad in"; a file that exists but is corrupt is reported as such, naming the bad
field. Both exit with status 1. field. Both exit with status 1.
`quak logout` ends the session on the server, so the token in `session.json`
stops working even in a copy of the file, and then deletes the file. If the
server call fails (or the file is corrupt), the file is still deleted, the
command says the server session could not be ended, and it exits with status 1.
It does not delete the cache: it prints the account's cache directory and says
it still holds decrypted data (file keys in `metadata.json`, cached originals
and thumbnails), for the user to delete if they want it gone.
### CLI surface ### CLI surface
``` ```
quak [--cache-dir <path>] <command> global: local metadata/content cache location quak [--cache-dir <path>] <command> global: local metadata/content cache location
quak login interactive or QUAK_EMAIL/QUAK_PASSWORD quak login interactive or QUAK_EMAIL/QUAK_PASSWORD
quak whoami print logged-in account as JSON quak whoami print logged-in account as JSON
quak logout end the session, delete it quak logout delete saved session
quak collections [--json] list all collections quak collections [--json] list all collections
quak files --collection <id> [--json] list files in a collection quak files --collection <id> [--json] list files in a collection
quak get <fileID> [--out path] [--collection] download and decrypt a file quak get <fileID> [--out path] [--collection] download and decrypt a file
@@ -461,13 +438,10 @@ quak helper fix-missing-thumbnails [--file ids] generate + upload missing thumbn
``` ```
Every command runs on the same cache-backed library. The read commands — Every command runs on the same cache-backed library. The read commands —
`collections`, `files`, `get`, `get-thumb`, `backup-metadata`, `collections`, `files`, `get`, and `get-thumb` — force a fresh server round-trip
`helper list-missing-thumbnails` and `helper fix-missing-thumbnails` — force a before they answer, so they report current account state rather than whatever
fresh server round-trip before they answer, so they report current account state the cache last held. `--cache-dir` overrides where the cache lives; without it
rather than whatever the cache last held. If that round-trip fails, the command each account gets its own directory under the per-user cache path.
prints the error on one line and exits 1. `--cache-dir` overrides where the
cache lives; without it each account gets its own directory under the per-user
cache path.
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is `get` and `get-thumb` resolve the file by ID directly, so `--collection` is
accepted for backward compatibility but ignored. `backup-metadata --exif` (alias accepted for backward compatibility but ignored. `backup-metadata --exif` (alias
@@ -484,13 +458,7 @@ on. The exit code is non-zero if any ML data request failed.
only, because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG only, because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG
image (PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept image (PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept
distinct from a `failed` repair, and does not affect the exit code; a genuine distinct from a `failed` repair, and does not affect the exit code; a genuine
failure still exits non-zero. The server accepts a new thumbnail only from the failure still exits non-zero.
file's owner and only when it is no larger than the thumbnail size it records
for the file. So a file another account owns, in an album shared with you, is
skipped by both thumbnail helpers without being fetched, and the fixer skips a
file whose recorded thumbnail size is 0 or unknown. Otherwise the fixer lowers
the quality and size of the thumbnail until it fits, and skips the file if even
the smallest does not.
### Backup layout ### Backup layout
@@ -505,15 +473,8 @@ the smallest does not.
<name>/ <name>/
<title> -> ../../originals/<fileID>.<ext> (symlink) <title> -> ../../originals/<fileID>.<ext> (symlink)
<name>.json collection metadata + file list <name>.json collection metadata + file list
failures.json files that failed and have not yet succeeded
``` ```
`failures.json` records each failed file with the kind of failure, how many
times it has been tried and when it was last tried. A file leaves it once it
succeeds, or once it is no longer in the library or in the backup's scope. The
library's `lib.backup({ includeThumbnails: true })` also writes
`thumbnails/<fileID>.jpg` beside `originals/`; `quak backup` does not.
A collection's directory and JSON are named after the collection, and a symlink A collection's directory and JSON are named after the collection, and a symlink
after the file's title, both with unsafe characters replaced. When two after the file's title, both with unsafe characters replaced. When two
collections would get the same name, or two files in one collection the same collections would get the same name, or two files in one collection the same
@@ -529,23 +490,17 @@ symlink you put there stays, and a directory that still holds one after its
symlinks are removed stays too, with its JSON. symlinks are removed stays too, with its JSON.
Each file is downloaded exactly once regardless of how many collections it Each file is downloaded exactly once regardless of how many collections it
appears in, and written once: straight into `originals/`, with no copy left in appears in. On subsequent runs, existing originals are skipped. If a download
the cache. An original the cache already held is copied from there instead. On fails, the error is logged and the backup continues with the next file. The exit
subsequent runs, existing originals are skipped. If a download fails, the error code is non-zero if any files failed.
is logged and the backup continues with the next file. The exit code is non-zero
if any files failed. `quak backup` opens its library with the thumbnail and
originals precache off, so it fetches only what the backup stores.
Each original is written to a temporary file in the same directory, synced to Each original is copied to a temporary file named
disk, and renamed into place, so an original is either complete or absent, even `.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp` in the same directory, synced
after a power cut. A downloaded original's temporary file is named to disk, and renamed into place, so an original is either complete or absent,
`.quak-<pid>-<random>.tmp`, one copied from the cache even after a power cut. A run that is killed can leave one of these temporary
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp`. A run that is killed can leave files behind; the next backup deletes those whose process is no longer running.
one of these temporary files behind; the next backup deletes those whose process Downloads and the content cache use the same scheme with `.quak-<random>.tmp`
is no longer running. The content cache uses the same scheme, and opening a names. The rename replaces whatever was at the destination rather than writing
library deletes the temporary files in the cache whose process is no longer
running, so a download another process has in progress in the same cache is left
alone. The rename replaces whatever was at the destination rather than writing
through it: a symlink there is replaced, not followed, and the new file has the through it: a symlink there is replaced, not followed, and the new file has the
temporary file's permissions, not those of the file it replaced. temporary file's permissions, not those of the file it replaced.
@@ -555,12 +510,7 @@ temporary file's permissions, not those of the file it replaced.
errors errors
- [x] Update the API reference section below to match the current implementation - [x] Update the API reference section below to match the current implementation
- [x] `make docker` green - [x] `make docker` green
- [ ] Store live photos in a form a photo viewer can open - [ ] Tag `v1.0.0`
(https://git.eeqj.de/sneak/quak/issues/107), once sneak has chosen between
keeping the ZIP and unpacking it
Tagging and releases are decided by sneak alone, and happen only when he
declares one.
Future (desktop client, separate repo): Future (desktop client, separate repo):
@@ -580,11 +530,10 @@ test suite is the canonical, executable documentation — `test/library/` and
### Opening a library ### Opening a library
`Library.open(options)` loads the on-disk cache, starts the background refresh `Library.open(options)` loads the on-disk cache, starts the background refresh
loop, and resolves to a `Library`. On an empty cache it awaits the first loop, and resolves to a `Library`. On an empty cache it awaits the first refresh
refresh, so it opens onto the account's data whenever the server is reachable; so it never opens onto empty data; on an existing cache it returns immediately
if that refresh fails, it opens with no data and records the error in and refreshes in the background, so an unreachable server does not block
`lib.status()`. On an existing cache it returns immediately and refreshes in the opening.
background, so an unreachable server does not block opening.
`LibraryOptions`: `LibraryOptions`:
@@ -695,15 +644,12 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
a query vector the caller produced elsewhere. a query vector the caller produced elsewhere.
- `await lib.backup(opts?)``BackupResult`. It refreshes, fetches every - `await lib.backup(opts?)``BackupResult`. It refreshes, fetches every
in-scope original not already in the backup (and, with `includeThumbnails`, in-scope original (and, with `includeThumbnails`, thumbnails) through the
thumbnails) through the content cache, and rebuilds the on-disk backup tree content cache, and rebuilds the on-disk backup tree with a durable failure
with a durable failure ledger. A fetched original is written straight into the ledger. `BackupOptions`: `downloadDirectory` (falls back to the one `open()`
backup's `originals/` and not into the cache, which then counts it as present; was given), `includeOriginals` (default `true`), `includeThumbnails` (default
one the cache already held is copied from there. `BackupOptions`: `false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above for the
`downloadDirectory` (falls back to the one `open()` was given), tree it writes.
`includeOriginals` (default `true`), `includeThumbnails` (default `false`),
`onlyAlbumNames`, and `onProgress`. See Backup layout above for the tree it
writes.
### Request pools ### Request pools
@@ -800,10 +746,8 @@ documents:
markdown. Use `make fmt` to format. Use `yarn` not `npm`. markdown. Use `make fmt` to format. Use `yarn` not `npm`.
- **Testing:** vitest. Tests go in `test/` mirroring the `src/` structure. - **Testing:** vitest. Tests go in `test/` mirroring the `src/` structure.
`make test` must finish in under 60 seconds (the hard cap) and should finish `make test` must complete in under 20 seconds. Use `mkdtempSync` for temporary
in under 20. The 90-second `timeout` in the `test` phase of the `Dockerfile` directories, never manual timestamp paths.
is a backstop that catches a hung test, not the time limit. Use `mkdtempSync`
for temporary directories, never manual timestamp paths.
- **Code style:** `const` for everything, `let` if reassignment is needed, never - **Code style:** `const` for everything, `let` if reassignment is needed, never
`var`. Avoid unnecessary comments. No hand-rolled crypto. The `var`. Avoid unnecessary comments. No hand-rolled crypto. The
+1 -85
View File
@@ -14,97 +14,16 @@ pre-1.0
# Next Step # Next Step
Store live photos in a form a photo viewer can open Tag v1.0.0.
(https://git.eeqj.de/sneak/quak/issues/107). This waits on sneak's choice
between keeping the ZIP and unpacking it into the image and the video.
Tagging and releases are decided by sneak alone, and happen only when he
declares one.
# Completed Steps # Completed Steps
- 2026-09-23: Settled the package metadata (issue 6). quak is not published, so
`package.json` is marked `"private": true` and the `files` field is gone.
`engines.node` is `>=22`, the major version `script/bootstrap` and the
`Dockerfile` use. An `exports` map makes `.` and `./package.json` the only
importable paths; `runMetadataBackup`, the thumbnail helpers and their types
stay internal to the CLI.
- 2026-09-23: Brought the README and this file in line with the tree (issue
111). The layout lists `src/library/` and the other source files, the backup
layout names `failures.json` and the optional `thumbnails/`, "Opening a
library" says what happens when the first refresh fails, the Testing section
gives the 60-second hard cap and 20-second target for `make test` and names
the 90-second `timeout` in the `Dockerfile` as the backstop for a hung test,
and "Tag v1.0.0" is no longer listed as the next step.
- 2026-09-23: Tested `quak login` and `backup-metadata --exif` (issue 110).
`loginCommand` takes its login function and its prompts from `CliContext`, and
`bin/quak.ts` passes `Client.login` and the terminal prompts. Tests cover a
login from `QUAK_EMAIL` and `QUAK_PASSWORD` with no prompt, the TOTP prompt, a
failed login, and the saved session's modes, and show that `--exif` and
`--all` each turn on EXIF extraction and that it is off without them.
- 2026-09-23: `quak backup` writes each original once and no longer fills the
cache (issue 106). An original fetched for a backup is written by the download
writer straight into the backup's `originals/`, and the content cache records
it there instead of keeping its own copy; one the cache already held is still
copied. `quak backup` opens its library with the thumbnail and originals
precache off.
- 2026-09-23: `backup-metadata`, `helper list-missing-thumbnails` and
`helper fix-missing-thumbnails` refresh before they answer (issue 100). Each
awaits `lib.fresh()` before reading, so a file added since the cache was
written is included, and a failed refresh prints one line and exits 1 instead
of answering from a stale or empty cache. The README lists them among the
commands that refresh first.
- 2026-09-23: `quak backup` waits for the server refresh and fails when it fails
(issue 99). `lib.backup()` joins a refresh already running or starts one, as
`fresh()` does, and rejects before touching any file when it fails, leaving
`failures.json` as it was, so `quak backup` prints the error as one line and
exits 1 instead of backing up the previous run's file list, or nothing, and
exiting 0.
- 2026-09-23: CLI errors print a message instead of a stack trace (issue 102).
An error a command throws is printed as one `quak: MESSAGE` line on stderr and
the CLI exits 1 once output has drained. The wrapper that does this moved from
`bin/quak.ts` to `src/cli-run.ts`, and `bin/quak.ts` now awaits
`program.parseAsync()`.
- 2026-09-23: Opening a library no longer deletes another process's download in
progress (issue 105). The download writer's temp files are named
`.quak-<pid>-<random>.tmp`, and `removeLeftoverTempFiles`, moved from the
backup into the download module, deletes a `.quak-*.tmp` file only when the
process ID in its name is no longer running. The content cache calls it at
`open()` for `originals/` and `thumbnails/`, the backup as before.
- 2026-09-23: Re-vendored the lint and test setup from the template (issue 96). - 2026-09-23: Re-vendored the lint and test setup from the template (issue 96).
Linting and testing are the `lint` and `test` phases of the `Dockerfile`; Linting and testing are the `lint` and `test` phases of the `Dockerfile`;
`script/lint` and `script/test` each build one with `--no-cache`, and the last `script/lint` and `script/test` each build one with `--no-cache`, and the last
stage compiles and depends on both, so `script/cibuild` is one build. stage compiles and depends on both, so `script/cibuild` is one build.
`Dockerfile.lint`, `CHECK_EPOCH`, `LINT_EPOCH` and the tests that checked them `Dockerfile.lint`, `CHECK_EPOCH`, `LINT_EPOCH` and the tests that checked them
are gone; `REPO_POLICIES.md` is re-copied. are gone; `REPO_POLICIES.md` is re-copied.
- 2026-09-23: Stopped `helper fix-missing-thumbnails` retrying files the server
always refuses (issue 109). Both thumbnail helpers skip a file another account
owns without fetching it. The fixer skips a file whose recorded thumbnail size
is 0 or unknown before downloading it, and otherwise tries smaller encodings
(720 px quality 50 down to 160 px quality 20) until the encrypted thumbnail is
no larger than that size, skipping the file if none fits.
- 2026-09-23: Tested the live-photo hash check's error paths (issue 117). Tests
download a live photo whose ZIP names an unknown compression method, one whose
ZIP has no image entry and one with no video entry, and check that nothing is
stored and the error names the file ID; the unreadable one is not retried.
- 2026-09-23: `quak logout` ends the session on the server (issue 108). It calls
`POST /users/logout` through the new `Client.logoutOnServer()`, then deletes
`session.json` even when that call fails, says so and exits 1. It prints the
account's cache directory and says it still holds decrypted data. The default
cache path is now `defaultCacheDirectory()` in the library, shared with
`Library.open`.
- 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files - 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files
in one collection with the same title, and two collections with the same name, in one collection with the same title, and two collections with the same name,
each get their ID added to the name (`IMG_0001 (12345).JPG`, `Trip (10)/`), so each get their ID added to the name (`IMG_0001 (12345).JPG`, `Trip (10)/`), so
@@ -112,7 +31,6 @@ declares one.
`originals/` for files no longer in the collection, and the folders of deleted `originals/` for files no longer in the collection, and the folders of deleted
or renamed collections, leaving anything else in `collections/` alone. The or renamed collections, leaving anything else in `collections/` alone. The
README backup layout states the naming rule. README backup layout states the naming rule.
- 2026-09-23: Checked downloaded originals against their recorded content hash - 2026-09-23: Checked downloaded originals against their recorded content hash
(issue 68). `downloadFile`, which `quak get`, the content cache and backup all (issue 68). `downloadFile`, which `quak get`, the content cache and backup all
use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and
@@ -121,14 +39,12 @@ declares one.
hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older
clients' `imageHash` and `videoHash` fields for live photos. A file with no clients' `imageHash` and `videoHash` fields for live photos. A file with no
recorded hash is stored unchecked. recorded hash is stored unchecked.
- 2026-09-23: Kept one account's cache from mixing with another's (issue 104). - 2026-09-23: Kept one account's cache from mixing with another's (issue 104).
When `metadata.json` in the cache directory was written for a different, When `metadata.json` in the cache directory was written for a different,
non-zero user ID than the client's, `Library.open` deletes it and `mldata/` non-zero user ID than the client's, `Library.open` deletes it and `mldata/`
and starts empty, so the first refresh enumerates from 0. This only happens and starts empty, so the first refresh enumerates from 0. This only happens
with `--cache-dir` or an explicit `cacheDirectory`; the default path already with `--cache-dir` or an explicit `cacheDirectory`; the default path already
includes the user ID. A test opens one account's cache as another account. includes the user ID. A test opens one account's cache as another account.
- 2026-09-23: `backup-metadata` no longer stops on one failed ML data request - 2026-09-23: `backup-metadata` no longer stops on one failed ML data request
(issue 101). Each request of up to 200 files is tried on its own; a failed one (issue 101). Each request of up to 200 files is tried on its own; a failed one
is logged, its files are written with the reason in `mlDataError`, and the is logged, its files are written with the reason in `mlDataError`, and the
+19 -10
View File
@@ -1,7 +1,6 @@
#!/usr/bin/env node #!/usr/bin/env node
import { stdout, stderr } from "node:process"; import { stdout, stderr } from "node:process";
import { input, password } from "@inquirer/prompts";
import { Command } from "commander"; import { Command } from "commander";
import envPaths from "env-paths"; import envPaths from "env-paths";
import { init } from "../src/crypto/index.js"; import { init } from "../src/crypto/index.js";
@@ -19,9 +18,7 @@ import {
listMissingThumbnailsCommand, listMissingThumbnailsCommand,
fixMissingThumbnailsCommand, fixMissingThumbnailsCommand,
} from "../src/cli-commands.js"; } from "../src/cli-commands.js";
import { run as runCommand } from "../src/cli-run.js";
import { loadSession } from "../src/cli-session.js"; import { loadSession } from "../src/cli-session.js";
import { Client } from "../src/client.js";
import { VERSION } from "../src/index.js"; import { VERSION } from "../src/index.js";
const paths = envPaths("quak", { suffix: "" }); const paths = envPaths("quak", { suffix: "" });
@@ -44,13 +41,25 @@ const context = (): CliContext => ({
sessionDir: paths.data, sessionDir: paths.data,
cacheDir: program.opts<{ cacheDir?: string }>().cacheDir, cacheDir: program.opts<{ cacheDir?: string }>().cacheDir,
loadSession, loadSession,
login: (opts) => Client.login(opts),
prompt: (message) => input({ message }),
promptSecret: (message) => password({ message, mask: true }),
}); });
const run = (command: Promise<number>): Promise<void> => // Run a command and exit with its code once stdout/stderr have drained.
runCommand(command, stdout, stderr, (code) => process.exit(code)); // Exiting before the drain can truncate piped output, and the library can keep
// the event loop alive after a command returns, so a plain return could hang.
const run = async (command: Promise<number>): Promise<void> => {
process.exitCode = await command;
const pending = [stdout, stderr].filter((s) => s.writableLength > 0);
if (pending.length === 0) {
process.exit();
return;
}
let remaining = pending.length;
for (const s of pending) {
s.once("drain", () => {
if (--remaining === 0) process.exit();
});
}
};
program program
.command("login") .command("login")
@@ -64,7 +73,7 @@ program
program program
.command("logout") .command("logout")
.description("End the session on the server and delete the saved session") .description("Delete the saved session")
.action(() => run(logoutCommand(context()))); .action(() => run(logoutCommand(context())));
program program
@@ -160,4 +169,4 @@ helper
); );
await init(); await init();
await program.parseAsync(); program.parse();
+5 -11
View File
@@ -9,23 +9,17 @@
"type": "git", "type": "git",
"url": "https://git.eeqj.de/sneak/quak.git" "url": "https://git.eeqj.de/sneak/quak.git"
}, },
"private": true,
"type": "module", "type": "module",
"engines": {
"node": ">=22"
},
"main": "./dist/src/index.js", "main": "./dist/src/index.js",
"types": "./dist/src/index.d.ts", "types": "./dist/src/index.d.ts",
"exports": {
".": {
"types": "./dist/src/index.d.ts",
"import": "./dist/src/index.js"
},
"./package.json": "./package.json"
},
"bin": { "bin": {
"quak": "./dist/bin/quak.js" "quak": "./dist/bin/quak.js"
}, },
"files": [
"dist/",
"README.md",
"LICENSE"
],
"scripts": { "scripts": {
"build": "script/build", "build": "script/build",
"quak": "node ./dist/bin/quak.js", "quak": "node ./dist/bin/quak.js",
+38 -13
View File
@@ -1,11 +1,9 @@
// The backup command, rebuilt on the library API (issue #51). // The backup command, rebuilt on the library API (issue #51).
// //
// `lib.backup()` waits for a completed refresh of the library (a failed one // `lib.backup()` refreshes the library, then, for every file in scope, gets its
// fails the backup before any file is touched), then, for every file in scope, // original bytes onto disk under `downloadDirectory` and rebuilds the derived
// gets its original bytes onto disk under `downloadDirectory` and rebuilds the // views (per-file sidecars, per-collection symlink trees, per-collection JSON)
// derived views (per-file sidecars, per-collection symlink trees, // from the model. The on-disk layout is the historical one, unchanged:
// per-collection JSON) from the model. The on-disk layout is the historical
// one, unchanged:
// //
// <downloadDirectory>/ // <downloadDirectory>/
// originals/<fileID>.<ext> the decrypted bytes // originals/<fileID>.<ext> the decrypted bytes
@@ -47,7 +45,7 @@ import {
import { copyFile, rename, rm } from "node:fs/promises"; import { copyFile, rename, rm } from "node:fs/promises";
import { basename, dirname, extname, join, relative } from "node:path"; import { basename, dirname, extname, join, relative } from "node:path";
import { fsyncPath, removeLeftoverTempFiles } from "./download/index.js"; import { fsyncPath } from "./download/index.js";
import { safeExtension, sanitizeFileName } from "./filename.js"; import { safeExtension, sanitizeFileName } from "./filename.js";
import type { Collection, EnteFile } from "./model/types.js"; import type { Collection, EnteFile } from "./model/types.js";
@@ -97,9 +95,8 @@ export interface BackupLibrary {
listCollections(): Collection[]; listCollections(): Collection[];
listFiles(collectionID: number): EnteFile[]; listFiles(collectionID: number): EnteFile[];
// Get an original's bytes onto disk through the content cache/pools, // Get an original's bytes onto disk through the content cache/pools,
// returning where they landed: `destination` when they were fetched now, // returning where they landed (the cache, or a prior backup).
// otherwise wherever they already were (the cache, or a prior backup). original(fileID: number): Promise<{ path: string }>;
original(fileID: number, destination: string): Promise<{ path: string }>;
thumbnail(fileID: number): Promise<{ path: string }>; thumbnail(fileID: number): Promise<{ path: string }>;
} }
@@ -187,6 +184,36 @@ const copyAtomic = async (src: string, dest: string): Promise<void> => {
} }
}; };
// A process-ID check: signal 0 delivers nothing and only reports whether the
// process exists. EPERM means it exists but belongs to another user.
const isRunning = (pid: number): boolean => {
try {
process.kill(pid, 0);
return true;
} catch (err) {
return (err as NodeJS.ErrnoException).code === "EPERM";
}
};
// Delete the temp files `copyAtomic` leaves behind when a backup is killed
// before its rename. Only files whose process is no longer running are
// removed, so a backup running at the same time keeps its own. A reused
// process ID can only keep a leftover a while longer, never remove a live one.
const removeLeftoverTempFiles = (dir: string): void => {
let names: string[];
try {
names = readdirSync(dir);
} catch {
return;
}
for (const name of names) {
const match = /^\.quak-backup-.*-(\d+)-[0-9a-z]*\.tmp$/.exec(name);
if (match && !isRunning(Number(match[1]))) {
rmSync(join(dir, name), { force: true });
}
}
};
// Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or // Ensure `linkPath` is a symlink to `target`, rebuilding a missing, wrong, or
// non-symlink entry. Throws on failure (a directory in the way, no permission) // non-symlink entry. Throws on failure (a directory in the way, no permission)
// so the caller records it and moves on rather than aborting the run. // so the caller records it and moves on rather than aborting the run.
@@ -426,9 +453,7 @@ export const runBackup = async (
} }
try { try {
log(`Fetching original ${file.metadata.title} (${fileID})...`); log(`Fetching original ${file.metadata.title} (${fileID})...`);
// A fetched original is written straight to `dest`; only one const { path } = await lib.original(fileID);
// that was already cached elsewhere is copied.
const { path } = await lib.original(fileID, dest);
await copyAtomic(path, dest); await copyAtomic(path, dest);
downloaded++; downloaded++;
} catch (err) { } catch (err) {
+22 -72
View File
@@ -2,10 +2,11 @@
// //
// Each command takes its options and a `CliContext` and resolves to the exit // Each command takes its options and a `CliContext` and resolves to the exit
// code; a thrown error is left to the caller. Nothing here calls // code; a thrown error is left to the caller. Nothing here calls
// `process.exit`: `bin/quak.ts` wires these to the command line, and `run` in // `process.exit`: `bin/quak.ts` wires these to the command line and exits with
// `cli-run.ts` prints a thrown error as one line and exits once output has // the returned code once output has drained. Output must stay byte-identical
// drained. Output must stay byte-identical (see `cli-output.ts`). // (see `cli-output.ts`).
import { input, password as passwordPrompt } from "@inquirer/prompts";
import { import {
copyFileSync, copyFileSync,
existsSync, existsSync,
@@ -14,17 +15,9 @@ import {
writeFileSync, writeFileSync,
} from "node:fs"; } from "node:fs";
import { join } from "node:path"; import { join } from "node:path";
import { import { Client, type ClientSnapshot } from "./client.js";
type Client,
type ClientSnapshot,
type LoginOptions,
} from "./client.js";
import { init } from "./crypto/index.js"; import { init } from "./crypto/index.js";
import { import { Library, type LibraryClient } from "./library/index.js";
defaultCacheDirectory,
Library,
type LibraryClient,
} from "./library/index.js";
import { import {
fileListRow, fileListRow,
fileListLine, fileListLine,
@@ -46,12 +39,6 @@ export interface CliContext {
// Reads the session file into a client, or null when there is none. The // Reads the session file into a client, or null when there is none. The
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client. // CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
loadSession: (path: string) => Client | null; loadSession: (path: string) => Client | null;
// Used by `login` only. The CLI passes `Client.login` and terminal
// prompts; tests pass fakes.
login: (opts: LoginOptions) => Promise<Client>;
prompt: (message: string) => Promise<string>;
// Like `prompt`, but the answer is masked as it is typed.
promptSecret: (message: string) => Promise<string>;
} }
const sessionPath = (ctx: CliContext): string => const sessionPath = (ctx: CliContext): string =>
@@ -117,19 +104,24 @@ const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
precacheOriginals: false, precacheOriginals: false,
}); });
const prompt = async (message: string): Promise<string> => input({ message });
const promptSecret = async (message: string): Promise<string> =>
passwordPrompt({ message, mask: true });
export const loginCommand = async (ctx: CliContext): Promise<number> => { export const loginCommand = async (ctx: CliContext): Promise<number> => {
await init(); await init();
const email = process.env.QUAK_EMAIL ?? (await ctx.prompt("Email")); const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
const password = const password =
process.env.QUAK_PASSWORD ?? (await ctx.promptSecret("Password")); process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
ctx.stderr.write("Authenticating...\n"); ctx.stderr.write("Authenticating...\n");
try { try {
const client = await ctx.login({ const client = await Client.login({
email, email,
password, password,
totp: async () => ctx.prompt("TOTP code: "), totp: async () => prompt("TOTP code: "),
emailOTP: async () => ctx.prompt("Email verification code: "), emailOTP: async () => prompt("Email verification code: "),
}); });
saveSession(ctx.sessionDir, client.toJSON()); saveSession(ctx.sessionDir, client.toJSON());
@@ -154,43 +146,14 @@ export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
return 0; return 0;
}; };
// Ends the session on the server, then deletes the session file even when that
// failed, and exits 1 if it did. The cache is left in place; the user is told
// where it is.
export const logoutCommand = async (ctx: CliContext): Promise<number> => { export const logoutCommand = async (ctx: CliContext): Promise<number> => {
const path = sessionPath(ctx); if (existsSync(sessionPath(ctx))) {
if (!existsSync(path)) { unlinkSync(sessionPath(ctx));
ctx.stderr.write("No session found.\n");
return 0;
}
await init();
let cacheDir = ctx.cacheDir;
let failure: string | undefined;
try {
const client = ctx.loadSession(path);
if (client) {
cacheDir ??= defaultCacheDirectory(client.whoami().userID);
await client.logoutOnServer();
client.logout();
}
} catch (err) {
failure = err instanceof Error ? err.message : String(err);
}
unlinkSync(path);
if (failure === undefined) {
ctx.stderr.write("Session ended on the server.\n");
} else {
ctx.stderr.write(
`Could not end the session on the server: ${failure}\n`,
);
}
ctx.stderr.write("Session deleted.\n"); ctx.stderr.write("Session deleted.\n");
if (cacheDir !== undefined) { } else {
ctx.stderr.write( ctx.stderr.write("No session found.\n");
`Cache directory ${cacheDir} still holds decrypted data; delete it to remove that data.\n`,
);
} }
return failure === undefined ? 0 : 1; return 0;
}; };
export const collectionsCommand = async ( export const collectionsCommand = async (
@@ -360,9 +323,6 @@ export const backupMetadataCommand = async (
if (!client) return 1; if (!client) return 1;
const lib = await openReadLibrary(ctx, client); const lib = await openReadLibrary(ctx, client);
try { try {
// Refresh first so the dump holds current account state, not what the
// cache last held; a failed refresh throws.
await lib.fresh();
const { failedMLBatches } = await runMetadataBackup(lib, client, dir, { const { failedMLBatches } = await runMetadataBackup(lib, client, dir, {
exif: opts.exif || opts.all, exif: opts.exif || opts.all,
onProgress: (msg) => ctx.stderr.write(msg + "\n"), onProgress: (msg) => ctx.stderr.write(msg + "\n"),
@@ -383,14 +343,10 @@ export const backupCommand = async (
if (!client) return 1; if (!client) return 1;
ctx.stderr.write("Starting backup...\n"); ctx.stderr.write("Starting backup...\n");
// The precache is off: the backup fetches what it needs, and must not
// also fill the cache with every thumbnail and the recent originals.
const lib = await Library.open({ const lib = await Library.open({
client, client,
downloadDirectory: dir, downloadDirectory: dir,
cacheDirectory: ctx.cacheDir, cacheDirectory: ctx.cacheDir,
precacheThumbnails: false,
precacheOriginals: false,
}); });
try { try {
const result = await lib.backup({ const result = await lib.backup({
@@ -433,9 +389,6 @@ export const listMissingThumbnailsCommand = async (
if (!client) return 1; if (!client) return 1;
const lib = await openReadLibrary(ctx, client); const lib = await openReadLibrary(ctx, client);
try { try {
// Refresh first so files added since the cache was written are
// checked; a failed refresh throws.
await lib.fresh();
const missing = await listMissingThumbnails(lib, client, (msg) => { const missing = await listMissingThumbnails(lib, client, (msg) => {
if (!opts.json) ctx.stderr.write(msg + "\n"); if (!opts.json) ctx.stderr.write(msg + "\n");
}); });
@@ -471,9 +424,6 @@ export const fixMissingThumbnailsCommand = async (
if (!client) return 1; if (!client) return 1;
const lib = await openReadLibrary(ctx, client); const lib = await openReadLibrary(ctx, client);
try { try {
// Refresh first so files added since the cache was written are found;
// a failed refresh throws.
await lib.fresh();
let fileIDs: number[]; let fileIDs: number[];
if (opts.file && opts.file.length > 0) { if (opts.file && opts.file.length > 0) {
fileIDs = opts.file.map(Number).filter(Number.isFinite); fileIDs = opts.file.map(Number).filter(Number.isFinite);
@@ -512,7 +462,7 @@ export const fixMissingThumbnailsCommand = async (
ctx.stderr.write(` Skipped: ${skipped}\n`); ctx.stderr.write(` Skipped: ${skipped}\n`);
ctx.stderr.write(` Failed: ${failed}\n`); ctx.stderr.write(` Failed: ${failed}\n`);
if (skipped > 0) { if (skipped > 0) {
ctx.stderr.write("\nSkipped:\n"); ctx.stderr.write("\nSkipped (unsupported format):\n");
for (const r of results.filter((r) => r.status === "skipped")) { for (const r of results.filter((r) => r.status === "skipped")) {
ctx.stderr.write( ctx.stderr.write(
` ${r.fileID}\t${r.title}\t${r.reason}\n`, ` ${r.fileID}\t${r.title}\t${r.reason}\n`,
-36
View File
@@ -1,36 +0,0 @@
// Runs one CLI command for `bin/quak.ts` and exits with its code.
import type { Writable } from "node:stream";
// Run a command and exit with its code once stdout/stderr have drained.
// Exiting before the drain can truncate piped output, and the library can keep
// the event loop alive after a command returns, so a plain return could hang.
// An error the command throws is printed as one `quak: MESSAGE` line, without
// the stack trace, and exits 1.
export const run = async (
command: Promise<number>,
stdout: Writable,
stderr: Writable,
exit: (code: number) => void,
): Promise<void> => {
let code: number;
try {
code = await command;
} catch (err) {
stderr.write(
`quak: ${err instanceof Error ? err.message : String(err)}\n`,
);
code = 1;
}
const pending = [stdout, stderr].filter((s) => s.writableLength > 0);
if (pending.length === 0) {
exit(code);
return;
}
let remaining = pending.length;
for (const s of pending) {
s.once("drain", () => {
if (--remaining === 0) exit(code);
});
}
};
-8
View File
@@ -217,14 +217,6 @@ export class Client {
}; };
} }
// Ends this client's session on the server (`POST /users/logout`), so the
// token stops working everywhere, including in any saved copy of it. This
// client is left as it was; call `logout()` to clear it.
async logoutOnServer(): Promise<void> {
this.assertLoggedIn();
await this.api.postJSON("/users/logout", {});
}
// Zeroes the key buffers in place, so any copy of the reference held // Zeroes the key buffers in place, so any copy of the reference held
// elsewhere is wiped too. Every method checks `assertLoggedIn` before // elsewhere is wiped too. Every method checks `assertLoggedIn` before
// touching the keys, so nothing decrypts with the zeroed keys. // touching the keys, so nothing decrypts with the zeroed keys.
+3 -40
View File
@@ -1,5 +1,4 @@
import { randomBytes } from "node:crypto"; import { randomUUID } from "node:crypto";
import { readdirSync, rmSync } from "node:fs";
import { open, rename, rm } from "node:fs/promises"; import { open, rename, rm } from "node:fs/promises";
import type { FileHandle } from "node:fs/promises"; import type { FileHandle } from "node:fs/promises";
import { dirname, join } from "node:path"; import { dirname, join } from "node:path";
@@ -181,38 +180,6 @@ export const fsyncPath = async (path: string): Promise<void> => {
} }
}; };
// A process-ID check: signal 0 delivers nothing and only reports whether the
// process exists. EPERM means it exists but belongs to another user.
const isRunning = (pid: number): boolean => {
try {
process.kill(pid, 0);
return true;
} catch (err) {
return (err as NodeJS.ErrnoException).code === "EPERM";
}
};
// Delete the temp files a killed process left in `dir`: the writer's
// `.quak-<pid>-<random>.tmp` and the backup copy's
// `.quak-backup-<name>-<pid>-<random>.tmp`. Only files whose process is no
// longer running are removed, so another process writing into the same
// directory keeps its own. A reused process ID can only keep a leftover a while
// longer, never remove a live one.
export const removeLeftoverTempFiles = (dir: string): void => {
let names: string[];
try {
names = readdirSync(dir);
} catch {
return;
}
for (const name of names) {
const match = /^\.quak-(?:.*-)?(\d+)-[0-9a-z]*\.tmp$/.exec(name);
if (match && !isRunning(Number(match[1]))) {
rmSync(join(dir, name), { force: true });
}
}
};
// Stage a write to `destination` atomically and durably, then rename it into // Stage a write to `destination` atomically and durably, then rename it into
// place. `fill` writes the contents into the open temp file handle — either the // place. `fill` writes the contents into the open temp file handle — either the
// whole buffer at once (`writeAtomic`) or chunk by chunk as they decrypt // whole buffer at once (`writeAtomic`) or chunk by chunk as they decrypt
@@ -238,12 +205,8 @@ const stageAtomic = async (
): Promise<void> => { ): Promise<void> => {
const dir = dirname(destination); const dir = dirname(destination);
// The random suffix keeps concurrent downloads of the same destination // The random suffix keeps concurrent downloads of the same destination
// from stepping on each other's temporary file; the process ID lets // from stepping on each other's temporary file.
// `removeLeftoverTempFiles` tell a leftover from a write in progress. const tmpPath = join(dir, `.quak-${randomUUID()}.tmp`);
const tmpPath = join(
dir,
`.quak-${process.pid}-${randomBytes(16).toString("hex")}.tmp`,
);
try { try {
const handle = await open(tmpPath, "w"); const handle = await open(tmpPath, "w");
try { try {
+11 -25
View File
@@ -39,7 +39,6 @@ import {
downloadFile, downloadFile,
downloadThumbnail, downloadThumbnail,
type ProgressCallback, type ProgressCallback,
removeLeftoverTempFiles,
} from "../download/index.js"; } from "../download/index.js";
import { safeExtension } from "../filename.js"; import { safeExtension } from "../filename.js";
import type { EnteFile } from "../model/types.js"; import type { EnteFile } from "../model/types.js";
@@ -47,6 +46,8 @@ import type { Priority, RequestPools } from "./pools.js";
const DIR_MODE = 0o700; const DIR_MODE = 0o700;
const FILE_MODE = 0o600; const FILE_MODE = 0o600;
const TEMP_PREFIX = ".quak-";
const TEMP_SUFFIX = ".tmp";
const GIB = 1024 * 1024 * 1024; const GIB = 1024 * 1024 * 1024;
// Owner ruling (#36): bound the originals cache at 100 GiB, but back off when // Owner ruling (#36): bound the originals cache at 100 GiB, but back off when
// the volume has under 50 GiB free so the cache never crowds the disk. // the volume has under 50 GiB free so the cache never crowds the disk.
@@ -321,23 +322,6 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress); return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
} }
// Get an original for a backup. One not present anywhere is written
// straight to `destination` and recorded there, so no second copy lands
// in the cache; one already present is returned where it is.
async backupOriginal(
fileID: number,
destination: string,
): Promise<ContentResult> {
const result = await this.acquire(
fileID,
"original",
"on-demand",
undefined,
{ destination },
);
return { path: result.path, bytes: result.bytes };
}
async ensure(args: EnsureOptions): Promise<EnsureResult[]> { async ensure(args: EnsureOptions): Promise<EnsureResult[]> {
return this.ensureThumbnails(args); return this.ensureThumbnails(args);
} }
@@ -439,14 +423,13 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
// The core: return the cached path if present, else fetch through the pool, // The core: return the cached path if present, else fetch through the pool,
// store, and return it. `cached` distinguishes a present hit (no network, // store, and return it. `cached` distinguishes a present hit (no network,
// no download event) from a fresh fetch. A fetched original is stored at // no download event) from a fresh fetch.
// `opts.destination` when given, instead of in `originalsDir`.
private async acquire( private async acquire(
fileID: number, fileID: number,
kind: Kind, kind: Kind,
priority: Priority, priority: Priority,
signal: AbortSignal | undefined, signal: AbortSignal | undefined,
opts?: { onByte?: ProgressCallback; destination?: string }, opts?: { onByte?: ProgressCallback },
): Promise<{ path: string; bytes: number; cached: boolean }> { ): Promise<{ path: string; bytes: number; cached: boolean }> {
const file = this.getFile(fileID); const file = this.getFile(fileID);
if (!file) throw new Error(`content cache: unknown file ${fileID}`); if (!file) throw new Error(`content cache: unknown file ${fileID}`);
@@ -487,7 +470,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
kind === "original" ? this.originalsDir : this.thumbnailsDir; kind === "original" ? this.originalsDir : this.thumbnailsDir;
const dest = const dest =
kind === "original" kind === "original"
? (opts?.destination ?? join(dir, originalName(file))) ? join(dir, originalName(file))
: join(dir, `${fileID}${THUMBNAIL_EXT}`); : join(dir, `${fileID}${THUMBNAIL_EXT}`);
const pool = const pool =
kind === "original" ? this.pools.content : this.pools.thumbnails; kind === "original" ? this.pools.content : this.pools.thumbnails;
@@ -671,9 +654,6 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
} }
private async scan(dir: string, into: Map<number, string>): Promise<void> { private async scan(dir: string, into: Map<number, string>): Promise<void> {
// Another process sharing this cache may still be writing its temp
// files, so only those whose process has exited are removed.
removeLeftoverTempFiles(dir);
let entries: string[]; let entries: string[];
try { try {
entries = await readdir(dir); entries = await readdir(dir);
@@ -681,6 +661,12 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
return; return;
} }
for (const name of entries) { for (const name of entries) {
if (name.startsWith(TEMP_PREFIX) && name.endsWith(TEMP_SUFFIX)) {
await rm(join(dir, name), { force: true }).catch(
() => undefined,
);
continue;
}
const id = fileIDFromName(name); const id = fileIDFromName(name);
const path = join(dir, name); const path = join(dir, name);
if (id !== undefined && existsSync(path)) into.set(id, path); if (id !== undefined && existsSync(path)) into.set(id, path);
+9 -16
View File
@@ -98,11 +98,6 @@ export {
export const DEFAULT_REFRESH_INTERVAL_SECONDS = 3; export const DEFAULT_REFRESH_INTERVAL_SECONDS = 3;
// The account's cache directory when `cacheDirectory` is not given: the
// env-paths cache directory plus the user id, so each account has its own.
export const defaultCacheDirectory = (userID: number): string =>
join(envPaths("quak", { suffix: "" }).cache, String(userID));
// Project a metadata store into by-id records, filling each record's cache // Project a metadata store into by-id records, filling each record's cache
// paths from the content cache when one is given. Shared by the live read // paths from the content cache when one is given. Shared by the live read
// projection and the precache's initial seeding at open(). // projection and the precache's initial seeding at open().
@@ -349,7 +344,8 @@ export class Library {
static async open(opts: LibraryOptions): Promise<Library> { static async open(opts: LibraryOptions): Promise<Library> {
const { userID } = opts.client.whoami(); const { userID } = opts.client.whoami();
const cacheDirectory = const cacheDirectory =
opts.cacheDirectory ?? defaultCacheDirectory(userID); opts.cacheDirectory ??
join(envPaths("quak", { suffix: "" }).cache, String(userID));
const metadataPath = join(cacheDirectory, "metadata.json"); const metadataPath = join(cacheDirectory, "metadata.json");
let store = await MetadataStore.load(metadataPath); let store = await MetadataStore.load(metadataPath);
// A cache directory given explicitly can hold another account's cache. // A cache directory given explicitly can hold another account's cache.
@@ -538,13 +534,11 @@ export class Library {
} }
// Back up every in-scope file to `downloadDirectory` in the historical // Back up every in-scope file to `downloadDirectory` in the historical
// on-disk layout, with a durable failure ledger (issue #51). Waits for a // on-disk layout, with a durable failure ledger (issue #51). Refreshes
// completed refresh first, as `fresh()` does, joining one already running, // first, fetches pending originals (and optional thumbnails) through the
// and rejects before touching any file when it fails. Then fetches pending // content cache and pools, then rebuilds the derived symlink/JSON views
// originals (and optional thumbnails) through the content cache and pools, // from the model. Throws before any network work when no download directory
// and rebuilds the derived symlink/JSON views from the model. Throws before // is available or no content cache backs the originals it must fetch.
// any network work when no download directory is available or no content
// cache backs the originals it must fetch.
backup(opts?: BackupOptions): Promise<BackupResult> { backup(opts?: BackupOptions): Promise<BackupResult> {
const downloadDirectory = const downloadDirectory =
opts?.downloadDirectory ?? this.downloadDirectory; opts?.downloadDirectory ?? this.downloadDirectory;
@@ -568,11 +562,10 @@ export class Library {
const cache = this.cache; const cache = this.cache;
return runBackup( return runBackup(
{ {
refresh: () => this.refreshNow(), refresh: () => this.runRefresh(),
listCollections: () => this.store.listCollections(), listCollections: () => this.store.listCollections(),
listFiles: (id) => this.store.listFiles(id), listFiles: (id) => this.store.listFiles(id),
original: (fileID, destination) => original: (fileID) => cache!.original(fileID),
cache!.backupOriginal(fileID, destination),
thumbnail: (fileID) => cache!.thumbnail(fileID), thumbnail: (fileID) => cache!.thumbnail(fileID),
}, },
{ ...opts, downloadDirectory }, { ...opts, downloadDirectory },
+2 -2
View File
@@ -140,8 +140,8 @@ const extractExif = async (
// Dump every decrypted metadata layer the account holds into a directory tree // Dump every decrypted metadata layer the account holds into a directory tree
// of plain JSON: account, per-collection, and per-file records including the // of plain JSON: account, per-collection, and per-file records including the
// private and public magic metadata and (by default) the ML data. Collections // private and public magic metadata and (by default) the ML data. Collections
// and files are enumerated from the library's cache, which the caller refreshes // and files are enumerated from the library's cache rather than a fresh server
// first. Returns how many ML data requests failed; their files are still // scan. Returns how many ML data requests failed; their files are still
// written, with `mlDataError` in place of `mlData`. // written, with `mlDataError` in place of `mlData`.
export const runMetadataBackup = async ( export const runMetadataBackup = async (
lib: Library, lib: Library,
+30 -88
View File
@@ -7,21 +7,8 @@ import { ApiError } from "./api/client.js";
import { encryptBlob, toBase64 } from "./crypto/index.js"; import { encryptBlob, toBase64 } from "./crypto/index.js";
import type { EnteFile } from "./model/types.js"; import type { EnteFile } from "./model/types.js";
// The server refuses a thumbnail larger than the one it already records for the const THUMB_MAX_DIMENSION = 720;
// file (`thumbnail.size`, the encrypted size), so these encodings are tried const THUMB_JPEG_QUALITY = 50;
// from largest to smallest and the first that fits is uploaded.
const THUMB_ENCODINGS = [
{ maxDimension: 720, quality: 50 },
{ maxDimension: 720, quality: 30 },
{ maxDimension: 480, quality: 30 },
{ maxDimension: 320, quality: 20 },
{ maxDimension: 160, quality: 20 },
];
// The server accepts a new thumbnail only from the file's owner, so files other
// people own in albums shared with this account are never checked or repaired.
const NOT_OWNED_REASON =
"owned by another account (only the owner can replace its thumbnail)";
export interface MissingThumbnailInfo { export interface MissingThumbnailInfo {
fileID: number; fileID: number;
@@ -32,12 +19,11 @@ export interface MissingThumbnailInfo {
// Three outcomes, not two. "fixed": a thumbnail was generated and uploaded. // Three outcomes, not two. "fixed": a thumbnail was generated and uploaded.
// "failed": something went wrong (download, encode, upload) and the file still // "failed": something went wrong (download, encode, upload) and the file still
// has no thumbnail. "skipped": the server would refuse any thumbnail for the // has no thumbnail. "skipped": the file is a format this helper cannot
// file or this helper cannot regenerate it — a file another account owns, a // regenerate — a video, or an image that is not a baseline JPEG. Skipped is a
// recorded thumbnail size nothing fits within, a video, or an image that is // deliberate, expected outcome, not an error (issue #17): the repair path is
// not a baseline JPEG. Skipped is a deliberate, expected outcome, not an error // JPEG-only because `jpeg-js` is, and a PNG or HEIC is left for a format-aware
// (issue #17): the repair path is JPEG-only because `jpeg-js` is, and a PNG or // tool rather than reported as a failure.
// HEIC is left for a format-aware tool rather than reported as a failure.
export type ThumbnailFixStatus = "fixed" | "skipped" | "failed"; export type ThumbnailFixStatus = "fixed" | "skipped" | "failed";
export interface ThumbnailFixResult { export interface ThumbnailFixResult {
@@ -59,7 +45,6 @@ export type ProgressCallback = (message: string) => void;
// exists, so it is logged and the file is left unreported. That distinction is // exists, so it is logged and the file is left unreported. That distinction is
// what stops `fix-missing-thumbnails` from regenerating and uploading over // what stops `fix-missing-thumbnails` from regenerating and uploading over
// thumbnails that were fine all along while the CDN was briefly returning 500s. // thumbnails that were fine all along while the CDN was briefly returning 500s.
// Files another account owns are logged as skipped and not checked.
export const listMissingThumbnails = async ( export const listMissingThumbnails = async (
lib: Library, lib: Library,
client: Client, client: Client,
@@ -67,7 +52,6 @@ export const listMissingThumbnails = async (
): Promise<MissingThumbnailInfo[]> => { ): Promise<MissingThumbnailInfo[]> => {
const log = onProgress ?? (() => {}); const log = onProgress ?? (() => {});
const api = client.getApiClient(); const api = client.getApiClient();
const { userID } = client.whoami();
const missing: MissingThumbnailInfo[] = []; const missing: MissingThumbnailInfo[] = [];
const seen = new Set<number>(); const seen = new Set<number>();
@@ -76,13 +60,6 @@ export const listMissingThumbnails = async (
for (const photo of album.photos.list()) { for (const photo of album.photos.list()) {
if (seen.has(photo.fileID)) continue; if (seen.has(photo.fileID)) continue;
seen.add(photo.fileID); seen.add(photo.fileID);
const file = lib.getFile(album.collectionID, photo.fileID);
if (file && file.ownerID !== userID) {
log(
`[${album.name}] Skipping ${photo.title}: ${NOT_OWNED_REASON}`,
);
continue;
}
try { try {
const stream = await api.getThumbnailStream(photo.fileID); const stream = await api.getThumbnailStream(photo.fileID);
const reader = stream.getReader(); const reader = stream.getReader();
@@ -158,13 +135,17 @@ const resizeRGBA = (
return dst; return dst;
}; };
const generateThumbnail = ( const generateThumbnail = (fileBytes: Uint8Array): Uint8Array => {
decoded: { data: Uint8Array; width: number; height: number }, const decoded = jpeg.decode(fileBytes, {
maxDimension: number, useTArray: true,
quality: number, formatAsRGBA: true,
): Uint8Array => { });
const { width: srcW, height: srcH } = decoded; const { width: srcW, height: srcH } = decoded;
const scale = Math.min(maxDimension / srcW, maxDimension / srcH, 1); const scale = Math.min(
THUMB_MAX_DIMENSION / srcW,
THUMB_MAX_DIMENSION / srcH,
1,
);
const dstW = Math.round(srcW * scale); const dstW = Math.round(srcW * scale);
const dstH = Math.round(srcH * scale); const dstH = Math.round(srcH * scale);
@@ -177,7 +158,7 @@ const generateThumbnail = (
const encoded = jpeg.encode( const encoded = jpeg.encode(
{ data: pixels, width: dstW, height: dstH }, { data: pixels, width: dstW, height: dstH },
quality, THUMB_JPEG_QUALITY,
); );
return new Uint8Array(encoded.data); return new Uint8Array(encoded.data);
}; };
@@ -190,35 +171,14 @@ const generateThumbnail = (
const isJpeg = (bytes: Uint8Array): boolean => const isJpeg = (bytes: Uint8Array): boolean =>
bytes.length >= 2 && bytes[0] === 0xff && bytes[1] === 0xd8; bytes.length >= 2 && bytes[0] === 0xff && bytes[1] === 0xd8;
// The reason a file cannot have a JPEG thumbnail regenerated for it, known from // The reason a file cannot have a JPEG thumbnail regenerated for it from its
// its record alone before any bytes are fetched, or undefined when it might. A // metadata alone, before any bytes are fetched, or undefined when it might. A
// still image still has to be checked against its actual bytes once // non-image (video, live photo) is unsupported outright; a still image still
// downloaded. // has to be checked against its actual bytes once downloaded.
const reasonToSkip = (file: EnteFile, userID: number): string | undefined => { const unsupportedByType = (file: EnteFile): string | undefined => {
if (file.ownerID !== userID) {
return NOT_OWNED_REASON;
}
if (file.metadata.fileType !== "image") { if (file.metadata.fileType !== "image") {
return `unsupported file type: ${file.metadata.fileType} (only JPEG images can be regenerated)`; return `unsupported file type: ${file.metadata.fileType} (only JPEG images can be regenerated)`;
} }
if (!file.thumbnail.size) {
return `recorded thumbnail size is ${file.thumbnail.size ?? "unknown"} (the server refuses a thumbnail larger than the one it records)`;
}
return undefined;
};
// Encrypt the largest encoding of the decoded image whose ciphertext is no
// larger than `maxSize`, or return undefined when even the smallest is larger.
const encryptThumbnailWithin = (
decoded: { data: Uint8Array; width: number; height: number },
key: Uint8Array,
maxSize: number,
): { header: Uint8Array; ciphertext: Uint8Array } | undefined => {
for (const { maxDimension, quality } of THUMB_ENCODINGS) {
const thumbJpeg = generateThumbnail(decoded, maxDimension, quality);
const encrypted = encryptBlob(thumbJpeg, key);
if (encrypted.ciphertext.length <= maxSize) return encrypted;
}
return undefined; return undefined;
}; };
@@ -237,7 +197,6 @@ export const fixMissingThumbnails = async (
const log = onProgress ?? (() => {}); const log = onProgress ?? (() => {});
const results: ThumbnailFixResult[] = []; const results: ThumbnailFixResult[] = [];
const api = client.getApiClient(); const api = client.getApiClient();
const { userID } = client.whoami();
// Resolve each requested fileID to its file record and owning album by // Resolve each requested fileID to its file record and owning album by
// enumerating the library, each file taken from the first album that holds // enumerating the library, each file taken from the first album that holds
@@ -278,19 +237,18 @@ export const fixMissingThumbnails = async (
const { file, collectionName } = entry; const { file, collectionName } = entry;
const title = file.metadata.title; const title = file.metadata.title;
const skipReason = reasonToSkip(file, userID); const typeReason = unsupportedByType(file);
if (skipReason) { if (typeReason) {
log(`[${collectionName}] Skipping ${title}: ${skipReason}`); log(`[${collectionName}] Skipping ${title}: ${typeReason}`);
results.push({ results.push({
fileID, fileID,
title, title,
collection: collectionName, collection: collectionName,
status: "skipped", status: "skipped",
reason: skipReason, reason: typeReason,
}); });
continue; continue;
} }
const maxSize = file.thumbnail.size!;
try { try {
const photo = lib.photos.byID({ fileID }); const photo = lib.photos.byID({ fileID });
@@ -319,28 +277,12 @@ export const fixMissingThumbnails = async (
} }
log(`[${collectionName}] Generating thumbnail for ${title}...`); log(`[${collectionName}] Generating thumbnail for ${title}...`);
const decoded = jpeg.decode(fileBytes, { const thumbJpeg = generateThumbnail(fileBytes);
useTArray: true,
formatAsRGBA: true,
});
const fitting = encryptThumbnailWithin(decoded, file.key, maxSize);
if (!fitting) {
const reason = `no thumbnail encoding fits the recorded thumbnail size of ${maxSize} bytes`;
log(`[${collectionName}] Skipping ${title}: ${reason}`);
results.push({
fileID,
title,
collection: collectionName,
status: "skipped",
reason,
});
continue;
}
const { header, ciphertext } = fitting;
log( log(
`[${collectionName}] Uploading thumbnail (${ciphertext.length} bytes)...`, `[${collectionName}] Encrypting and uploading thumbnail (${thumbJpeg.length} bytes)...`,
); );
const { header, ciphertext } = encryptBlob(thumbJpeg, file.key);
const md5 = createHash("md5").update(ciphertext).digest("base64"); const md5 = createHash("md5").update(ciphertext).digest("base64");
const { objectKey, url } = await api.getUploadURL( const { objectKey, url } = await api.getUploadURL(
ciphertext.length, ciphertext.length,
+1 -116
View File
@@ -562,39 +562,11 @@ describe("lib.backup", () => {
lib.close(); lib.close();
}); });
it("fetches each original once and writes it only into the backup", async () => { it("fsyncs a copied original before the rename and its directory after", async () => {
// A backup of a 500 GB account must write 500 GB, not a copy in the
// cache as well: an original fetched for the backup goes straight
// into its originals/, and the cache records it there.
const source = stubSource();
const lib = await openLibrary(source);
const outDir = join(root, "backup");
const result = await lib.backup({ downloadDirectory: outDir });
expect(result.downloaded).toBe(3);
expect(source.originalCalls).toBe(3);
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
const stored = readdirSync(join(outDir, "originals")).filter(
(name) => !name.endsWith(".json"),
);
expect(stored.sort()).toEqual(["100.jpg", "101.jpg", "200.png"]);
// The cache counts the backup's copy as present: reading the
// original afterwards fetches nothing and answers with that copy.
const read = await lib.photos.byID({ fileID: 100 })!.original();
expect(read.path).toBe(join(outDir, "originals", "100.jpg"));
expect(source.originalCalls).toBe(3);
await lib.close();
});
it("fsyncs an original copied from the cache before the rename and its directory after", async () => {
const lib = await openLibrary(stubSource()); const lib = await openLibrary(stubSource());
const outDir = join(root, "backup"); const outDir = join(root, "backup");
const originals = join(outDir, "originals"); const originals = join(outDir, "originals");
const dest = join(originals, "100.jpg"); const dest = join(originals, "100.jpg");
// Only an original already in the cache is copied into the backup;
// one fetched for the backup is written there by the download writer.
await lib.photos.byID({ fileID: 100 })!.original();
fsEvents.length = 0; fsEvents.length = 0;
await lib.backup({ downloadDirectory: outDir }); await lib.backup({ downloadDirectory: outDir });
@@ -653,93 +625,6 @@ describe("lib.backup", () => {
}); });
}); });
// Every refresh fails, as with an expired session or no network.
class FailingClient extends MockClient {
override async collectionsSince(): Promise<CollectionsPage> {
throw new Error("HTTP 401 from server");
}
}
// Holds its refresh open until `release()` is called, then reports a third
// album, so a backup can be started while that refresh is still running.
class HeldClient extends MockClient {
release!: () => void;
private held = new Promise<void>((resolve) => {
this.release = resolve;
});
override async collectionsSince(): Promise<CollectionsPage> {
await this.held;
return {
collections: [collection(3, "Later")],
deleted: [],
cursor: 2,
};
}
override async filesSince(args: {
collectionID: number;
}): Promise<FilesPage> {
if (args.collectionID !== 3) return super.filesSince(args);
return { files: [file(300, 3, "late.jpg")], deleted: [], cursor: 2 };
}
}
// Fill the library cache on disk, so the next open starts its refresh in the
// background instead of waiting for it.
const fillCache = async (): Promise<void> => {
const lib = await openLibrary(stubSource());
await lib.close();
};
describe("the refresh before a backup", () => {
it("waits for a refresh already running and backs up what it found", async () => {
await fillCache();
const client = new HeldClient();
const lib = await openLibrary(stubSource(), client);
const outDir = join(root, "backup");
const backup = lib.backup({ downloadDirectory: outDir });
client.release();
const result = await backup;
expect(result.totalFiles).toBe(4);
expect(existsSync(join(outDir, "originals", "300.jpg"))).toBe(true);
await lib.close();
});
it("fails before any download when the refresh fails, leaving failures.json as it was", async () => {
await fillCache();
const outDir = join(root, "backup");
seedLedger(outDir, 100, "beach.jpg");
const ledgerPath = join(outDir, "failures.json");
const ledgerBefore = readFileSync(ledgerPath, "utf-8");
const source = stubSource();
const lib = await openLibrary(source, new FailingClient());
await expect(lib.backup({ downloadDirectory: outDir })).rejects.toThrow(
"HTTP 401 from server",
);
expect(source.originalCalls).toBe(0);
expect(readFileSync(ledgerPath, "utf-8")).toBe(ledgerBefore);
expect(existsSync(join(outDir, "originals"))).toBe(false);
await lib.close();
});
it("fails when the refresh fails on an empty cache, instead of backing up nothing", async () => {
const source = stubSource();
const lib = await openLibrary(source, new FailingClient());
const outDir = join(root, "backup");
await expect(lib.backup({ downloadDirectory: outDir })).rejects.toThrow(
"HTTP 401 from server",
);
expect(source.originalCalls).toBe(0);
expect(existsSync(outDir)).toBe(false);
await lib.close();
});
});
// The album folders under collections/, driven through `runBackup` with a // The album folders under collections/, driven through `runBackup` with a
// stand-in library whose albums a test changes between runs. // stand-in library whose albums a test changes between runs.
describe("backup album folders", () => { describe("backup album folders", () => {
+24 -359
View File
@@ -13,7 +13,6 @@
import { import {
existsSync, existsSync,
mkdtempSync, mkdtempSync,
readdirSync,
readFileSync, readFileSync,
rmSync, rmSync,
statSync, statSync,
@@ -21,21 +20,11 @@ import {
} from "node:fs"; } from "node:fs";
import { join } from "node:path"; import { join } from "node:path";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { PassThrough } from "node:stream"; import { describe, it, expect, beforeAll, beforeEach, afterEach } from "vitest";
import {
describe,
it,
expect,
vi,
beforeAll,
beforeEach,
afterEach,
} from "vitest";
import { import {
type CliContext, type CliContext,
saveSession, saveSession,
loginCommand,
whoamiCommand, whoamiCommand,
logoutCommand, logoutCommand,
collectionsCommand, collectionsCommand,
@@ -43,17 +32,13 @@ import {
getCommand, getCommand,
getThumbCommand, getThumbCommand,
backupCommand, backupCommand,
backupMetadataCommand,
listMissingThumbnailsCommand, listMissingThumbnailsCommand,
fixMissingThumbnailsCommand,
} from "../../src/cli-commands.js"; } from "../../src/cli-commands.js";
import { run } from "../../src/cli-run.js";
import { loadSession } from "../../src/cli-session.js"; import { loadSession } from "../../src/cli-session.js";
import type { Client, ClientSnapshot, LoginOptions } from "../../src/client.js"; import type { Client, ClientSnapshot } from "../../src/client.js";
import type { ContentSource } from "../../src/library/content.js"; import type { ContentSource } from "../../src/library/content.js";
import type { Collection, EnteFile } from "../../src/model/types.js"; import type { Collection, EnteFile } from "../../src/model/types.js";
import { init, toBase64 } from "../../src/crypto/index.js"; import { init } from "../../src/crypto/index.js";
import { defaultCacheDirectory } from "../../src/library/index.js";
const USER_ID = 42; const USER_ID = 42;
@@ -96,26 +81,8 @@ const FILES: Record<number, EnteFile[]> = {
// An original is 7 bytes and a thumbnail 3. `failID` makes that file's // An original is 7 bytes and a thumbnail 3. `failID` makes that file's
// original fail; `emptyThumbID` makes the server report that file's // original fail; `emptyThumbID` makes the server report that file's
// thumbnail as empty. `withNewFile` adds new.jpg (102) to Vacation, advancing // thumbnail as empty.
// the collection's updationTime as the server does, and `refreshError` makes const fakeClient = (opts: { failID?: number; emptyThumbID?: number } = {}) => {
// listing collections fail with that message.
const fakeClient = (
opts: {
failID?: number;
emptyThumbID?: number;
withNewFile?: boolean;
refreshError?: string;
} = {},
) => {
const collections = opts.withNewFile
? [{ ...COLLECTIONS[0], updationTime: 2 }, COLLECTIONS[1]]
: COLLECTIONS;
const files = opts.withNewFile
? {
...FILES,
1: [...FILES[1], { ...file(102, 1, "new.jpg"), updationTime: 2 }],
}
: FILES;
const source: ContentSource = { const source: ContentSource = {
original: async ({ file: f, destination }) => { original: async ({ file: f, destination }) => {
if (f.id === opts.failID) throw new Error("HTTP 500 from server"); if (f.id === opts.failID) throw new Error("HTTP 500 from server");
@@ -129,19 +96,18 @@ const fakeClient = (
}; };
const fake = { const fake = {
whoami: () => ({ email: "cli@example.com", userID: USER_ID }), whoami: () => ({ email: "cli@example.com", userID: USER_ID }),
collectionsSince: async () => { collectionsSince: async () => ({
if (opts.refreshError) throw new Error(opts.refreshError); collections: COLLECTIONS,
return { collections, deleted: [], cursor: 1 }; deleted: [],
}, cursor: 1,
}),
filesSince: async (args: { collectionID: number }) => ({ filesSince: async (args: { collectionID: number }) => ({
files: files[args.collectionID] ?? [], files: FILES[args.collectionID] ?? [],
deleted: [], deleted: [],
cursor: 1, cursor: 1,
}), }),
contentSource: () => source, contentSource: () => source,
getApiClient: () => ({ getApiClient: () => ({
// The ML data request of `backup-metadata`: no file has any.
postJSON: async () => ({ data: [] }),
getThumbnailStream: async (fileID: number) => getThumbnailStream: async (fileID: number) =>
new ReadableStream<Uint8Array>({ new ReadableStream<Uint8Array>({
start(controller) { start(controller) {
@@ -175,15 +141,6 @@ const context = (client: Client | null = fakeClient()): CliContext => ({
sessionDir: join(root, "session"), sessionDir: join(root, "session"),
cacheDir: join(root, "cache"), cacheDir: join(root, "cache"),
loadSession: () => client, loadSession: () => client,
login: async () => {
throw new Error("login not expected");
},
prompt: async () => {
throw new Error("prompt not expected");
},
promptSecret: async () => {
throw new Error("prompt not expected");
},
}); });
beforeAll(async () => { beforeAll(async () => {
@@ -219,6 +176,19 @@ describe("session file", () => {
expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot); expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot);
}); });
it("is removed by logout", async () => {
const ctx = context();
saveSession(ctx.sessionDir, snapshot);
expect(await logoutCommand(ctx)).toBe(0);
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
expect(stderr.text).toBe("Session deleted.\n");
});
it("logout without a session says so and exits 0", async () => {
expect(await logoutCommand(context())).toBe(0);
expect(stderr.text).toBe("No session found.\n");
});
it("a missing session exits 1 with 'Not logged in'", async () => { it("a missing session exits 1 with 'Not logged in'", async () => {
const ctx = { ...context(), loadSession }; const ctx = { ...context(), loadSession };
expect(await whoamiCommand(ctx)).toBe(1); expect(await whoamiCommand(ctx)).toBe(1);
@@ -241,184 +211,6 @@ describe("session file", () => {
}); });
}); });
// The login function is a fake that hands back a client whose snapshot is
// `snapshot`; each prompt is recorded and answered with "123456".
describe("login", () => {
const snapshot: ClientSnapshot = {
email: "cli@example.com",
userID: USER_ID,
token: "token",
masterKey: "a",
secretKey: "b",
publicKey: "c",
};
const loggedIn = {
whoami: () => ({ email: "cli@example.com", userID: USER_ID }),
toJSON: () => snapshot,
} as unknown as Client;
let prompts: string[];
const loginContext = (
login: (opts: LoginOptions) => Promise<Client>,
): CliContext => ({
...context(),
login,
prompt: async (message) => {
prompts.push(message);
return "123456";
},
promptSecret: async (message) => {
prompts.push(message);
return "123456";
},
});
beforeEach(() => {
prompts = [];
vi.stubEnv("QUAK_EMAIL", "cli@example.com");
vi.stubEnv("QUAK_PASSWORD", "hunter2");
});
afterEach(() => {
vi.unstubAllEnvs();
});
it("takes the email and password from the environment without a prompt", async () => {
const calls: LoginOptions[] = [];
const ctx = loginContext(async (opts) => {
calls.push(opts);
return loggedIn;
});
expect(await loginCommand(ctx)).toBe(0);
expect(prompts).toEqual([]);
expect(calls).toHaveLength(1);
expect(calls[0]!.email).toBe("cli@example.com");
expect(calls[0]!.password).toBe("hunter2");
const path = join(ctx.sessionDir, "session.json");
expect(stderr.text).toBe(
"Authenticating...\n" +
`Logged in as cli@example.com (user ${USER_ID})\n` +
`Session saved to ${path}\n`,
);
});
it("saves the session with mode 0600 in a directory with mode 0700", async () => {
const ctx = loginContext(async () => loggedIn);
expect(await loginCommand(ctx)).toBe(0);
expect(statSync(ctx.sessionDir).mode & 0o777).toBe(0o700);
const path = join(ctx.sessionDir, "session.json");
expect(statSync(path).mode & 0o777).toBe(0o600);
expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot);
});
it("asks for the TOTP code when the account needs one", async () => {
let code: string | undefined;
const ctx = loginContext(async (opts) => {
code = await opts.totp!();
return loggedIn;
});
expect(await loginCommand(ctx)).toBe(0);
expect(prompts).toEqual(["TOTP code: "]);
expect(code).toBe("123456");
});
it("a failed login exits 1, says why and writes no session", async () => {
const ctx = loginContext(async () => {
throw new Error("HTTP 401 from server");
});
expect(await loginCommand(ctx)).toBe(1);
expect(stderr.text).toBe(
"Authenticating...\nLogin failed: HTTP 401 from server\n",
);
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
});
});
// These use a real client read from the session file, over a fake API that
// records each request and answers with `status`.
describe("logout", () => {
const snapshot: ClientSnapshot = {
email: "cli@example.com",
userID: USER_ID,
token: "saved-token",
masterKey: toBase64(new Uint8Array(32)),
secretKey: toBase64(new Uint8Array(32)),
publicKey: toBase64(new Uint8Array(32)),
};
const requests: Request[] = [];
const logoutContext = (status: number): CliContext => ({
...context(),
loadSession: (path) =>
loadSession(path, {
fetch: async (url, init) => {
requests.push(new Request(url, init));
return new Response(JSON.stringify({}), {
status,
headers: { "content-type": "application/json" },
});
},
}),
});
beforeEach(() => {
requests.length = 0;
});
it("ends the session on the server, then deletes the file", async () => {
const ctx = logoutContext(200);
saveSession(ctx.sessionDir, snapshot);
expect(await logoutCommand(ctx)).toBe(0);
expect(requests).toHaveLength(1);
expect(requests[0]!.method).toBe("POST");
expect(new URL(requests[0]!.url).pathname).toBe("/users/logout");
expect(requests[0]!.headers.get("X-Auth-Token")).toBe("saved-token");
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
expect(stderr.text).toBe(
"Session ended on the server.\n" +
"Session deleted.\n" +
`Cache directory ${ctx.cacheDir} still holds decrypted data; delete it to remove that data.\n`,
);
});
it("still deletes the file when the server call fails, and says so", async () => {
const ctx = logoutContext(500);
saveSession(ctx.sessionDir, snapshot);
expect(await logoutCommand(ctx)).toBe(1);
expect(requests).toHaveLength(1);
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
expect(stderr.text).toBe(
"Could not end the session on the server: HTTP 500\n" +
"Session deleted.\n" +
`Cache directory ${ctx.cacheDir} still holds decrypted data; delete it to remove that data.\n`,
);
});
it("names the account's default cache directory without --cache-dir", async () => {
const ctx = { ...logoutContext(200), cacheDir: undefined };
saveSession(ctx.sessionDir, snapshot);
expect(await logoutCommand(ctx)).toBe(0);
expect(stderr.text).toContain(
`Cache directory ${defaultCacheDirectory(USER_ID)} still holds decrypted data`,
);
});
it("without a session says so, calls nothing and exits 0", async () => {
expect(await logoutCommand(logoutContext(200))).toBe(0);
expect(requests).toHaveLength(0);
expect(stderr.text).toBe("No session found.\n");
});
});
describe("whoami", () => { describe("whoami", () => {
it("prints the account as one line of JSON", async () => { it("prints the account as one line of JSON", async () => {
expect(await whoamiCommand(context())).toBe(0); expect(await whoamiCommand(context())).toBe(0);
@@ -545,16 +337,6 @@ describe("backup", () => {
expect(stdout.text).toBe(""); expect(stdout.text).toBe("");
}); });
// The backup opens its library with the precache off: it fetches the
// originals it needs into the backup, and must not also fetch every
// thumbnail in the account, or keep originals, in the per-user cache.
it("leaves nothing in the cache's originals and thumbnails", async () => {
const dir = join(root, "backup");
expect(await backupCommand(context(), dir, {})).toBe(0);
expect(readdirSync(join(root, "cache", "thumbnails"))).toEqual([]);
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
});
it("exits 1 and lists the file when one download fails", async () => { it("exits 1 and lists the file when one download fails", async () => {
const ctx = context(fakeClient({ failID: 101 })); const ctx = context(fakeClient({ failID: 101 }));
expect(await backupCommand(ctx, join(root, "backup"), {})).toBe(1); expect(await backupCommand(ctx, join(root, "backup"), {})).toBe(1);
@@ -581,34 +363,6 @@ describe("backup", () => {
expect(result.errors[0].fileID).toBe(101); expect(result.errors[0].fileID).toBe(101);
expect(stderr.text).toBe("Starting backup...\n"); expect(stderr.text).toBe("Starting backup...\n");
}); });
it("exits 1 with the error on one line when the refresh fails", async () => {
const client = {
...fakeClient(),
collectionsSince: async () => {
throw new Error("HTTP 401 from server");
},
} as unknown as Client;
const dir = join(root, "backup");
// Through `run`, as `bin/quak.ts` does, which prints a thrown error.
const runStderr = new PassThrough();
let runText = "";
runStderr.on("data", (chunk: Buffer) => {
runText += chunk.toString();
});
const code = await new Promise<number>((resolve) => {
void run(
backupCommand(context(client), dir, {}),
new PassThrough(),
runStderr,
resolve,
);
});
expect(code).toBe(1);
expect(runText).toBe("quak: HTTP 401 from server\n");
expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n");
expect(existsSync(join(dir, "originals"))).toBe(false);
});
}); });
describe("helper list-missing-thumbnails", () => { describe("helper list-missing-thumbnails", () => {
@@ -641,92 +395,3 @@ describe("helper list-missing-thumbnails", () => {
expect(stderr.text).toBe(""); expect(stderr.text).toBe("");
}); });
}); });
describe("backup-metadata --exif", () => {
// Runs the command and returns what it printed to stderr.
const backupMetadata = async (opts: { exif?: boolean; all?: boolean }) => {
expect(
await backupMetadataCommand(context(), join(root, "dump"), opts),
).toBe(0);
return stderr.text;
};
it("--exif extracts EXIF", async () => {
expect(await backupMetadata({ exif: true })).toContain(
"[beach.jpg] Extracting EXIF...\n",
);
});
it("--all extracts EXIF", async () => {
expect(await backupMetadata({ all: true })).toContain(
"[beach.jpg] Extracting EXIF...\n",
);
});
it("without either flag extracts no EXIF", async () => {
expect(await backupMetadata({})).not.toContain("Extracting EXIF");
});
});
// Each test first runs `collections` so the cache holds the account as it was,
// then changes the server under it.
describe("backup-metadata and the thumbnail helpers refresh first", () => {
beforeEach(async () => {
expect(await collectionsCommand(context(), {})).toBe(0);
stdout.text = "";
stderr.text = "";
});
it("backup-metadata writes a file added since the cache was written", async () => {
const ctx = context(fakeClient({ withNewFile: true }));
const dir = join(root, "dump");
expect(await backupMetadataCommand(ctx, dir, {})).toBe(0);
expect(
existsSync(join(dir, "collections", "1-Vacation", "102.json")),
).toBe(true);
});
it("list-missing-thumbnails checks a file added since the cache was written", async () => {
const ctx = context(
fakeClient({ withNewFile: true, emptyThumbID: 102 }),
);
expect(await listMissingThumbnailsCommand(ctx, {})).toBe(0);
expect(stdout.text).toBe(
"102\tnew.jpg\tVacation\tempty thumbnail (0 bytes)\n",
);
});
it("fix-missing-thumbnails finds a file added since the cache was written", async () => {
const ctx = context(fakeClient({ withNewFile: true }));
expect(
await fixMissingThumbnailsCommand(ctx, {
file: ["102"],
json: true,
}),
).toBe(0);
// Found, then skipped because the server records no thumbnail size
// for it; a file missing from the cache would fail as not found.
expect(JSON.parse(stdout.text)).toMatchObject([
{ fileID: 102, title: "new.jpg", status: "skipped" },
]);
});
// `run` in `cli-run.ts` prints a thrown error as one line and exits 1.
it("all three throw when the refresh fails", async () => {
const ctx = context(
fakeClient({ refreshError: "HTTP 503 from server" }),
);
const dir = join(root, "dump");
await expect(backupMetadataCommand(ctx, dir, {})).rejects.toThrow(
"HTTP 503 from server",
);
expect(existsSync(dir)).toBe(false);
await expect(listMissingThumbnailsCommand(ctx, {})).rejects.toThrow(
"HTTP 503 from server",
);
await expect(
fixMissingThumbnailsCommand(ctx, { file: ["100"] }),
).rejects.toThrow("HTTP 503 from server");
expect(stdout.text).toBe("");
});
});
-58
View File
@@ -1,58 +0,0 @@
/**
* Tests for `run` in `src/cli-run.ts`, which every CLI command goes through.
*/
import { PassThrough } from "node:stream";
import { describe, it, expect } from "vitest";
import { run } from "../../src/cli-run.js";
// A stream whose written text is kept in `text`; writes finish at once, so
// nothing is left waiting to drain.
const collector = (): { stream: PassThrough; text: () => string } => {
const stream = new PassThrough();
const chunks: string[] = [];
stream.on("data", (chunk: Buffer) => chunks.push(chunk.toString()));
return { stream, text: () => chunks.join("") };
};
const runToExit = async (
command: Promise<number>,
): Promise<{ code: number; stdout: string; stderr: string }> => {
const stdout = collector();
const stderr = collector();
const code = await new Promise<number>((resolve) => {
void run(command, stdout.stream, stderr.stream, resolve);
});
return { code, stdout: stdout.text(), stderr: stderr.text() };
};
describe("run", () => {
it("exits with the code the command returns", async () => {
const result = await runToExit(Promise.resolve(3));
expect(result).toEqual({ code: 3, stdout: "", stderr: "" });
});
it("prints a thrown error as one line without a stack trace and exits 1", async () => {
const failing = async (): Promise<number> => {
throw new Error(
"ENOTDIR: not a directory, mkdir '/dev/null/x/originals'",
);
};
const result = await runToExit(failing());
expect(result.code).toBe(1);
expect(result.stdout).toBe("");
expect(result.stderr).toBe(
"quak: ENOTDIR: not a directory, mkdir '/dev/null/x/originals'\n",
);
});
it("prints a thrown value that is not an Error", async () => {
const result = await runToExit(Promise.reject("offline"));
expect(result).toEqual({
code: 1,
stdout: "",
stderr: "quak: offline\n",
});
});
});
+1 -45
View File
@@ -1560,13 +1560,7 @@ describe("writeAtomic", () => {
// directory so that new entry is on disk too. Do the directory fsync // directory so that new entry is on disk too. Do the directory fsync
// before the rename, or skip it, and a crash can lose the rename. // before the rename, or skip it, and a crash can lose the rename.
expect(durabilityHook.events).toHaveLength(3); expect(durabilityHook.events).toHaveLength(3);
// The temp name carries this process's ID, so a library opening the expect(durabilityHook.events[0]).toMatch(/^sync:w:.*\.tmp$/);
// same cache can tell a write in progress from a leftover.
const tempSync = durabilityHook.events[0]!;
expect(tempSync.startsWith(`sync:w:${dir}/`)).toBe(true);
expect(tempSync.slice(`sync:w:${dir}/`.length)).toMatch(
new RegExp(`^\\.quak-${process.pid}-[0-9a-f]{32}\\.tmp$`),
);
expect(durabilityHook.events[1]).toBe(`rename:${dest}`); expect(durabilityHook.events[1]).toBe(`rename:${dest}`);
expect(durabilityHook.events[2]).toBe(`sync:r:${dir}`); expect(durabilityHook.events[2]).toBe(`sync:r:${dir}`);
}); });
@@ -1750,42 +1744,4 @@ describe("downloadFile content hash", () => {
expect(readdirSync(t.dir)).toEqual([]); expect(readdirSync(t.dir)).toEqual([]);
}); });
it("rejects a live photo that is not a readable ZIP and does not retry", async () => {
// Bytes 8-9 of a ZIP entry's local header name its compression
// method; 99 is one no reader knows, so the entry cannot be read.
const zip = livePhotoZip.slice();
zip[8] = 99;
zip[9] = 0;
const t = setup(zip, { fileType: "livePhoto", hash: livePhotoHash });
await expect(t.run()).rejects.toThrow(
/file 999: live photo is not a readable ZIP/,
);
expect(readdirSync(t.dir)).toEqual([]);
expect(t.requests()).toBe(1);
});
it("rejects a live photo ZIP with no image entry", async () => {
const zip = zipSync({ "video.mov": patternBytes(900, 82) });
const t = setup(zip, { fileType: "livePhoto", hash: livePhotoHash });
await expect(t.run()).rejects.toThrow(
/file 999: live photo ZIP does not hold both an image and a video/,
);
expect(readdirSync(t.dir)).toEqual([]);
});
it("rejects a live photo ZIP with no video entry", async () => {
const zip = zipSync({ "image.heic": patternBytes(500, 81) });
const t = setup(zip, { fileType: "livePhoto", hash: livePhotoHash });
await expect(t.run()).rejects.toThrow(
/file 999: live photo ZIP does not hold both an image and a video/,
);
expect(readdirSync(t.dir)).toEqual([]);
});
}); });
+2 -20
View File
@@ -33,7 +33,6 @@ import {
mkdirSync, mkdirSync,
statSync, statSync,
} from "node:fs"; } from "node:fs";
import { spawnSync } from "node:child_process";
import { tmpdir } from "node:os"; import { tmpdir } from "node:os";
import { join } from "node:path"; import { join } from "node:path";
@@ -162,28 +161,15 @@ describe("ContentCache.open", () => {
expect(statSync(thumbnails).mode & 0o777).toBe(0o700); expect(statSync(thumbnails).mode & 0o777).toBe(0o700);
}); });
it("removes temp files of an exited process, keeping those of a running one and complete content", async () => { it("reaps orphan temp files but keeps complete content", async () => {
const originals = join(cacheDir, "originals"); const originals = join(cacheDir, "originals");
const thumbnails = join(cacheDir, "thumbnails"); const thumbnails = join(cacheDir, "thumbnails");
mkdirSync(originals, { recursive: true }); mkdirSync(originals, { recursive: true });
mkdirSync(thumbnails, { recursive: true }); mkdirSync(thumbnails, { recursive: true });
// A child that has already exited: its process ID is not running. const orphan = join(originals, ".quak-abc123.tmp");
const exitedPID = spawnSync(process.execPath, ["-e", ""]).pid;
const orphan = join(originals, `.quak-${exitedPID}-abc123.tmp`);
const orphanThumb = join(thumbnails, `.quak-${exitedPID}-abc456.tmp`);
// This test's own process stands in for another process still
// downloading into the same cache.
const inProgress = join(originals, `.quak-${process.pid}-def123.tmp`);
const inProgressThumb = join(
thumbnails,
`.quak-${process.pid}-def456.tmp`,
);
const complete = join(originals, "1.jpg"); const complete = join(originals, "1.jpg");
const thumb = join(thumbnails, "2.jpg"); const thumb = join(thumbnails, "2.jpg");
writeFileSync(orphan, "half-written"); writeFileSync(orphan, "half-written");
writeFileSync(orphanThumb, "half-written");
writeFileSync(inProgress, "half-written");
writeFileSync(inProgressThumb, "half-written");
writeFileSync(complete, "whole"); writeFileSync(complete, "whole");
writeFileSync(thumb, "whole-thumb"); writeFileSync(thumb, "whole-thumb");
@@ -191,12 +177,8 @@ describe("ContentCache.open", () => {
await cache.open(); await cache.open();
expect(existsSync(orphan)).toBe(false); expect(existsSync(orphan)).toBe(false);
expect(existsSync(orphanThumb)).toBe(false);
expect(existsSync(inProgress)).toBe(true);
expect(existsSync(inProgressThumb)).toBe(true);
expect(existsSync(complete)).toBe(true); expect(existsSync(complete)).toBe(true);
expect(existsSync(thumb)).toBe(true); expect(existsSync(thumb)).toBe(true);
expect(cache.pathsFor(1)).toEqual({ originalPath: complete });
}); });
it("records already-cached files so their paths appear in pathsFor", async () => { it("records already-cached files so their paths appear in pathsFor", async () => {
-143
View File
@@ -215,9 +215,6 @@ const buildThumbMock = async (opts?: {
thumbnail: { thumbnail: {
decryptionHeader: toBase64(sodium.randombytes_buf(24)), decryptionHeader: toBase64(sodium.randombytes_buf(24)),
}, },
// The encrypted size of the thumbnail the server records; large
// enough here that the default encoding fits.
info: { thumbSize: 1_000_000 },
updationTime: TEST_TIME, updationTime: TEST_TIME,
}; };
}; };
@@ -449,32 +446,6 @@ const openLib = (client: Client): Promise<Library> =>
precacheOriginals: false, precacheOriginals: false,
}); });
/** The mock's raw record for one file, for a test to change before login. */
const rawFile = (m: ThumbMockState, fileID: number): Record<string, unknown> =>
m.filesByCollection[1]!.find((f) => f.id === fileID)!;
/** Replace the original the mock serves for one file. */
const replaceOriginal = (
m: ThumbMockState,
fileID: number,
body: Uint8Array,
): void => {
const push = sodium.crypto_secretstream_xchacha20poly1305_init_push(
m.fileKeys[fileID]!,
);
m.fileCiphertexts[fileID] =
sodium.crypto_secretstream_xchacha20poly1305_push(
push.state,
body,
null,
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
);
rawFile(m, fileID).file = { decryptionHeader: toBase64(push.header) };
};
const isOriginalDownload = (url: string): boolean =>
url.includes("files.ente.io") || url.includes("/files/download/");
const login = (fetch: typeof globalThis.fetch, retry?: RetryOptions) => const login = (fetch: typeof globalThis.fetch, retry?: RetryOptions) =>
Client.login({ Client.login({
email: TEST_EMAIL, email: TEST_EMAIL,
@@ -610,33 +581,6 @@ describe("listMissingThumbnails", () => {
// Should still be 2, not 4 (each file checked only once) // Should still be 2, not 4 (each file checked only once)
expect(missing.length).toBe(2); expect(missing.length).toBe(2);
}); });
it("skips a file another account owns without fetching its thumbnail", async () => {
const otherMock = await buildThumbMock();
rawFile(otherMock, 102).ownerID = 7;
const logs: string[] = [];
const counted = countingFetch(
buildThumbFetch(otherMock),
(url) => url.includes("thumbnails.ente.io") && url.includes("102"),
);
const client = await login(counted.fetch);
const lib = await openLib(client);
const missing = await listMissingThumbnails(lib, client, (msg) =>
logs.push(msg),
);
lib.close();
expect(missing.map((m) => m.fileID)).toEqual([101]);
expect(counted.matched()).toBe(0);
expect(
logs.some(
(l) =>
l.includes("Skipping file-102.jpg") &&
l.includes("another account"),
),
).toBe(true);
});
}); });
describe("fixMissingThumbnails", () => { describe("fixMissingThumbnails", () => {
@@ -744,93 +688,6 @@ describe("fixMissingThumbnails", () => {
expect(fixMock.uploadedThumbnails.length).toBe(1); expect(fixMock.uploadedThumbnails.length).toBe(1);
expect(fixMock.uploadedThumbnails[0]!.fileID).toBe(101); expect(fixMock.uploadedThumbnails[0]!.fileID).toBe(101);
}); });
it("skips a file another account owns without downloading it", async () => {
// The server accepts a thumbnail only from the file's owner.
const fixMock = await buildThumbMock();
rawFile(fixMock, 101).ownerID = 7;
const counted = countingFetch(
buildThumbFetch(fixMock),
isOriginalDownload,
);
const client = await login(counted.fetch);
const lib = await openLib(client);
const results = await fixMissingThumbnails(lib, client, [101]);
lib.close();
expect(results[0]!.status).toBe("skipped");
expect(results[0]!.reason).toContain("another account");
expect(counted.matched()).toBe(0);
expect(fixMock.uploadedThumbnails.length).toBe(0);
});
it("skips a file whose recorded thumbnail size is 0 without downloading it", async () => {
// The server refuses a thumbnail larger than the one it records, and
// no thumbnail is 0 bytes.
const fixMock = await buildThumbMock();
rawFile(fixMock, 101).info = { thumbSize: 0 };
const counted = countingFetch(
buildThumbFetch(fixMock),
isOriginalDownload,
);
const client = await login(counted.fetch);
const lib = await openLib(client);
const results = await fixMissingThumbnails(lib, client, [101]);
lib.close();
expect(results[0]!.status).toBe("skipped");
expect(results[0]!.reason).toContain("recorded thumbnail size is 0");
expect(counted.matched()).toBe(0);
expect(fixMock.uploadedThumbnails.length).toBe(0);
});
it("re-encodes smaller until the thumbnail fits the recorded size", async () => {
// A noisy 400x300 JPEG, which the default encoding (quality 50, not
// resized because it is under 720 px) cannot compress below the size
// recorded here: one byte less than that encoding's ciphertext.
const fixMock = await buildThumbMock();
const w = 400;
const h = 300;
const noisy = new Uint8Array(
jpegJs.encode(
{
data: sodium.randombytes_buf(w * h * 4),
width: w,
height: h,
},
90,
).data,
);
replaceOriginal(fixMock, 101, noisy);
const decoded = jpegJs.decode(noisy, {
useTArray: true,
formatAsRGBA: true,
});
const defaultSize =
jpegJs.encode(decoded, 50).data.length +
sodium.crypto_secretstream_xchacha20poly1305_ABYTES;
const recordedSize = defaultSize - 1;
rawFile(fixMock, 101).info = { thumbSize: recordedSize };
const client = await login(buildThumbFetch(fixMock));
const lib = await openLib(client);
const results = await fixMissingThumbnails(lib, client, [101]);
lib.close();
expect(results[0]!.status).toBe("fixed");
const upload = fixMock.uploadedThumbnails[0]!;
expect(upload.ciphertext.length).toBeLessThanOrEqual(recordedSize);
const decrypted = decryptBlob(
upload.ciphertext,
fromBase64(upload.decryptionHeader),
fixMock.fileKeys[101]!,
);
expect(decrypted[0]).toBe(0xff);
expect(decrypted[1]).toBe(0xd8);
});
}); });
describe("Client.getApiClient", () => { describe("Client.getApiClient", () => {