1 Commits
Author SHA1 Message Date
sneak 4c325edc9b Refresh before backup-metadata and the thumbnail helpers answer (closes #100)
check / check (push) Successful in 1m22s
backup-metadata, helper list-missing-thumbnails and helper
fix-missing-thumbnails now await lib.fresh() before reading the library,
as collections, files, get and get-thumb already do. A file added since
the cache was written is included, and a failed refresh is thrown, so
the CLI prints one line and exits 1 instead of answering from a stale
or empty cache. The README lists them among the commands that refresh
first.

Model: opus-5-5
2026-09-23 05:49:41 +00:00
10 changed files with 68 additions and 485 deletions
+28 -61
View File
@@ -214,23 +214,11 @@ quak/
auth/ login flow (SRP + email OTP + TOTP), key unwrap
model/ decrypted Collection, File, Metadata types + decrypt fns
download/ streaming file/thumbnail download + decryption
library/ the cache-backed Library: metadata store, read
surface, records, content cache, precache, ML data
and search, request pools
backup.ts resilient full-account backup with dedup
metadata-backup.ts
backup-metadata: all decrypted metadata as JSON
mldata-fetch.ts fetch + decrypt per-file ML data
filename.ts safe file names from server metadata
errors.ts error types shared across layers
retry.ts retry classifier + exponential backoff with jitter
thumbnails.ts detect + regenerate missing thumbnails
client.ts high-level Client class assembled from the above
cli-commands.ts the CLI's commands as functions returning exit codes
cli-output.ts how the CLI prints a file's title and time
cli-read.ts fresh reads for the CLI's read commands
cli-run.ts run a command, print its error, exit with its code
cli-session.ts read the saved session file back into a Client
index.ts public library exports
bin/
quak.ts CLI entrypoint (commander.js)
@@ -505,15 +493,8 @@ the smallest does not.
<name>/
<title> -> ../../originals/<fileID>.<ext> (symlink)
<name>.json collection metadata + file list
failures.json files that failed and have not yet succeeded
```
`failures.json` records each failed file with the kind of failure, how many
times it has been tried and when it was last tried. A file leaves it once it
succeeds, or once it is no longer in the library or in the backup's scope. The
library's `lib.backup({ includeThumbnails: true })` also writes
`thumbnails/<fileID>.jpg` beside `originals/`; `quak backup` does not.
A collection's directory and JSON are named after the collection, and a symlink
after the file's title, both with unsafe characters replaced. When two
collections would get the same name, or two files in one collection the same
@@ -529,25 +510,22 @@ symlink you put there stays, and a directory that still holds one after its
symlinks are removed stays too, with its JSON.
Each file is downloaded exactly once regardless of how many collections it
appears in, and written once: straight into `originals/`, with no copy left in
the cache. An original the cache already held is copied from there instead. On
subsequent runs, existing originals are skipped. If a download fails, the error
is logged and the backup continues with the next file. The exit code is non-zero
if any files failed. `quak backup` opens its library with the thumbnail and
originals precache off, so it fetches only what the backup stores.
appears in. On subsequent runs, existing originals are skipped. If a download
fails, the error is logged and the backup continues with the next file. The exit
code is non-zero if any files failed.
Each original is written to a temporary file in the same directory, synced to
disk, and renamed into place, so an original is either complete or absent, even
after a power cut. A downloaded original's temporary file is named
`.quak-<pid>-<random>.tmp`, one copied from the cache
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp`. A run that is killed can leave
one of these temporary files behind; the next backup deletes those whose process
is no longer running. The content cache uses the same scheme, and opening a
library deletes the temporary files in the cache whose process is no longer
running, so a download another process has in progress in the same cache is left
alone. The rename replaces whatever was at the destination rather than writing
through it: a symlink there is replaced, not followed, and the new file has the
temporary file's permissions, not those of the file it replaced.
Each original is copied to a temporary file named
`.quak-backup-<fileID>.<ext>-<pid>-<random>.tmp` in the same directory, synced
to disk, and renamed into place, so an original is either complete or absent,
even after a power cut. A run that is killed can leave one of these temporary
files behind; the next backup deletes those whose process is no longer running.
Downloads and the content cache use the same scheme with
`.quak-<pid>-<random>.tmp` names, and opening a library deletes those in the
cache whose process is no longer running, so a download another process has in
progress in the same cache is left alone. The rename replaces whatever was at
the destination rather than writing through it: a symlink there is replaced, not
followed, and the new file has the temporary file's permissions, not those of
the file it replaced.
## TODO
@@ -555,12 +533,7 @@ temporary file's permissions, not those of the file it replaced.
errors
- [x] Update the API reference section below to match the current implementation
- [x] `make docker` green
- [ ] Store live photos in a form a photo viewer can open
(https://git.eeqj.de/sneak/quak/issues/107), once sneak has chosen between
keeping the ZIP and unpacking it
Tagging and releases are decided by sneak alone, and happen only when he
declares one.
- [ ] Tag `v1.0.0`
Future (desktop client, separate repo):
@@ -580,11 +553,10 @@ test suite is the canonical, executable documentation — `test/library/` and
### Opening a library
`Library.open(options)` loads the on-disk cache, starts the background refresh
loop, and resolves to a `Library`. On an empty cache it awaits the first
refresh, so it opens onto the account's data whenever the server is reachable;
if that refresh fails, it opens with no data and records the error in
`lib.status()`. On an existing cache it returns immediately and refreshes in the
background, so an unreachable server does not block opening.
loop, and resolves to a `Library`. On an empty cache it awaits the first refresh
so it never opens onto empty data; on an existing cache it returns immediately
and refreshes in the background, so an unreachable server does not block
opening.
`LibraryOptions`:
@@ -695,15 +667,12 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
default limit 20). quak bundles no text encoder, so `searchByEmbedding` takes
a query vector the caller produced elsewhere.
- `await lib.backup(opts?)` → `BackupResult`. It refreshes, fetches every
in-scope original not already in the backup (and, with `includeThumbnails`,
thumbnails) through the content cache, and rebuilds the on-disk backup tree
with a durable failure ledger. A fetched original is written straight into the
backup's `originals/` and not into the cache, which then counts it as present;
one the cache already held is copied from there. `BackupOptions`:
`downloadDirectory` (falls back to the one `open()` was given),
`includeOriginals` (default `true`), `includeThumbnails` (default `false`),
`onlyAlbumNames`, and `onProgress`. See Backup layout above for the tree it
writes.
in-scope original (and, with `includeThumbnails`, thumbnails) through the
content cache, and rebuilds the on-disk backup tree with a durable failure
ledger. `BackupOptions`: `downloadDirectory` (falls back to the one `open()`
was given), `includeOriginals` (default `true`), `includeThumbnails` (default
`false`), `onlyAlbumNames`, and `onProgress`. See Backup layout above for the
tree it writes.
### Request pools
@@ -800,10 +769,8 @@ documents:
markdown. Use `make fmt` to format. Use `yarn` not `npm`.
- **Testing:** vitest. Tests go in `test/` mirroring the `src/` structure.
`make test` must finish in under 60 seconds (the hard cap) and should finish
in under 20. The 90-second `timeout` in the `test` phase of the `Dockerfile`
is a backstop that catches a hung test, not the time limit. Use `mkdtempSync`
for temporary directories, never manual timestamp paths.
`make test` must complete in under 20 seconds. Use `mkdtempSync` for temporary
directories, never manual timestamp paths.
- **Code style:** `const` for everything, `let` if reassignment is needed, never
`var`. Avoid unnecessary comments. No hand-rolled crypto. The
+1 -42
View File
@@ -14,44 +14,10 @@ pre-1.0
# Next Step
Store live photos in a form a photo viewer can open
(https://git.eeqj.de/sneak/quak/issues/107). This waits on sneak's choice
between keeping the ZIP and unpacking it into the image and the video.
Tagging and releases are decided by sneak alone, and happen only when he
declares one.
Tag v1.0.0.
# Completed Steps
- 2026-09-23: Settled the package metadata (issue 6). quak is not published, so
`package.json` is marked `"private": true` and the `files` field is gone.
`engines.node` is `>=22`, the major version `script/bootstrap` and the
`Dockerfile` use. An `exports` map makes `.` and `./package.json` the only
importable paths; `runMetadataBackup`, the thumbnail helpers and their types
stay internal to the CLI.
- 2026-09-23: Brought the README and this file in line with the tree (issue
111). The layout lists `src/library/` and the other source files, the backup
layout names `failures.json` and the optional `thumbnails/`, "Opening a
library" says what happens when the first refresh fails, the Testing section
gives the 60-second hard cap and 20-second target for `make test` and names
the 90-second `timeout` in the `Dockerfile` as the backstop for a hung test,
and "Tag v1.0.0" is no longer listed as the next step.
- 2026-09-23: Tested `quak login` and `backup-metadata --exif` (issue 110).
`loginCommand` takes its login function and its prompts from `CliContext`, and
`bin/quak.ts` passes `Client.login` and the terminal prompts. Tests cover a
login from `QUAK_EMAIL` and `QUAK_PASSWORD` with no prompt, the TOTP prompt, a
failed login, and the saved session's modes, and show that `--exif` and
`--all` each turn on EXIF extraction and that it is off without them.
- 2026-09-23: `quak backup` writes each original once and no longer fills the
cache (issue 106). An original fetched for a backup is written by the download
writer straight into the backup's `originals/`, and the content cache records
it there instead of keeping its own copy; one the cache already held is still
copied. `quak backup` opens its library with the thumbnail and originals
precache off.
- 2026-09-23: `backup-metadata`, `helper list-missing-thumbnails` and
`helper fix-missing-thumbnails` refresh before they answer (issue 100). Each
awaits `lib.fresh()` before reading, so a file added since the cache was
@@ -59,13 +25,6 @@ declares one.
of answering from a stale or empty cache. The README lists them among the
commands that refresh first.
- 2026-09-23: `quak backup` waits for the server refresh and fails when it fails
(issue 99). `lib.backup()` joins a refresh already running or starts one, as
`fresh()` does, and rejects before touching any file when it fails, leaving
`failures.json` as it was, so `quak backup` prints the error as one line and
exits 1 instead of backing up the previous run's file list, or nothing, and
exiting 0.
- 2026-09-23: CLI errors print a message instead of a stack trace (issue 102).
An error a command throws is printed as one `quak: MESSAGE` line on stderr and
the CLI exits 1 once output has drained. The wrapper that does this moved from
-5
View File
@@ -1,7 +1,6 @@
#!/usr/bin/env node
import { stdout, stderr } from "node:process";
import { input, password } from "@inquirer/prompts";
import { Command } from "commander";
import envPaths from "env-paths";
import { init } from "../src/crypto/index.js";
@@ -21,7 +20,6 @@ import {
} from "../src/cli-commands.js";
import { run as runCommand } from "../src/cli-run.js";
import { loadSession } from "../src/cli-session.js";
import { Client } from "../src/client.js";
import { VERSION } from "../src/index.js";
const paths = envPaths("quak", { suffix: "" });
@@ -44,9 +42,6 @@ const context = (): CliContext => ({
sessionDir: paths.data,
cacheDir: program.opts<{ cacheDir?: string }>().cacheDir,
loadSession,
login: (opts) => Client.login(opts),
prompt: (message) => input({ message }),
promptSecret: (message) => password({ message, mask: true }),
});
const run = (command: Promise<number>): Promise<void> =>
+5 -11
View File
@@ -9,23 +9,17 @@
"type": "git",
"url": "https://git.eeqj.de/sneak/quak.git"
},
"private": true,
"type": "module",
"engines": {
"node": ">=22"
},
"main": "./dist/src/index.js",
"types": "./dist/src/index.d.ts",
"exports": {
".": {
"types": "./dist/src/index.d.ts",
"import": "./dist/src/index.js"
},
"./package.json": "./package.json"
},
"bin": {
"quak": "./dist/bin/quak.js"
},
"files": [
"dist/",
"README.md",
"LICENSE"
],
"scripts": {
"build": "script/build",
"quak": "node ./dist/bin/quak.js",
+7 -12
View File
@@ -1,11 +1,9 @@
// The backup command, rebuilt on the library API (issue #51).
//
// `lib.backup()` waits for a completed refresh of the library (a failed one
// fails the backup before any file is touched), then, for every file in scope,
// gets its original bytes onto disk under `downloadDirectory` and rebuilds the
// derived views (per-file sidecars, per-collection symlink trees,
// per-collection JSON) from the model. The on-disk layout is the historical
// one, unchanged:
// `lib.backup()` refreshes the library, then, for every file in scope, gets its
// original bytes onto disk under `downloadDirectory` and rebuilds the derived
// views (per-file sidecars, per-collection symlink trees, per-collection JSON)
// from the model. The on-disk layout is the historical one, unchanged:
//
// <downloadDirectory>/
// originals/<fileID>.<ext> the decrypted bytes
@@ -97,9 +95,8 @@ export interface BackupLibrary {
listCollections(): Collection[];
listFiles(collectionID: number): EnteFile[];
// Get an original's bytes onto disk through the content cache/pools,
// returning where they landed: `destination` when they were fetched now,
// otherwise wherever they already were (the cache, or a prior backup).
original(fileID: number, destination: string): Promise<{ path: string }>;
// returning where they landed (the cache, or a prior backup).
original(fileID: number): Promise<{ path: string }>;
thumbnail(fileID: number): Promise<{ path: string }>;
}
@@ -426,9 +423,7 @@ export const runBackup = async (
}
try {
log(`Fetching original ${file.metadata.title} (${fileID})...`);
// A fetched original is written straight to `dest`; only one
// that was already cached elsewhere is copied.
const { path } = await lib.original(fileID, dest);
const { path } = await lib.original(fileID);
await copyAtomic(path, dest);
downloaded++;
} catch (err) {
+14 -21
View File
@@ -4,8 +4,10 @@
// code; a thrown error is left to the caller. Nothing here calls
// `process.exit`: `bin/quak.ts` wires these to the command line, and `run` in
// `cli-run.ts` prints a thrown error as one line and exits once output has
// drained. Output must stay byte-identical (see `cli-output.ts`).
// drained. Output must stay byte-identical
// (see `cli-output.ts`).
import { input, password as passwordPrompt } from "@inquirer/prompts";
import {
copyFileSync,
existsSync,
@@ -14,11 +16,7 @@ import {
writeFileSync,
} from "node:fs";
import { join } from "node:path";
import {
type Client,
type ClientSnapshot,
type LoginOptions,
} from "./client.js";
import { Client, type ClientSnapshot } from "./client.js";
import { init } from "./crypto/index.js";
import {
defaultCacheDirectory,
@@ -46,12 +44,6 @@ export interface CliContext {
// Reads the session file into a client, or null when there is none. The
// CLI passes `loadSession` from `cli-session.ts`; tests pass a fake client.
loadSession: (path: string) => Client | null;
// Used by `login` only. The CLI passes `Client.login` and terminal
// prompts; tests pass fakes.
login: (opts: LoginOptions) => Promise<Client>;
prompt: (message: string) => Promise<string>;
// Like `prompt`, but the answer is masked as it is typed.
promptSecret: (message: string) => Promise<string>;
}
const sessionPath = (ctx: CliContext): string =>
@@ -117,19 +109,24 @@ const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
precacheOriginals: false,
});
const prompt = async (message: string): Promise<string> => input({ message });
const promptSecret = async (message: string): Promise<string> =>
passwordPrompt({ message, mask: true });
export const loginCommand = async (ctx: CliContext): Promise<number> => {
await init();
const email = process.env.QUAK_EMAIL ?? (await ctx.prompt("Email"));
const email = process.env.QUAK_EMAIL ?? (await prompt("Email"));
const password =
process.env.QUAK_PASSWORD ?? (await ctx.promptSecret("Password"));
process.env.QUAK_PASSWORD ?? (await promptSecret("Password"));
ctx.stderr.write("Authenticating...\n");
try {
const client = await ctx.login({
const client = await Client.login({
email,
password,
totp: async () => ctx.prompt("TOTP code: "),
emailOTP: async () => ctx.prompt("Email verification code: "),
totp: async () => prompt("TOTP code: "),
emailOTP: async () => prompt("Email verification code: "),
});
saveSession(ctx.sessionDir, client.toJSON());
@@ -383,14 +380,10 @@ export const backupCommand = async (
if (!client) return 1;
ctx.stderr.write("Starting backup...\n");
// The precache is off: the backup fetches what it needs, and must not
// also fill the cache with every thumbnail and the recent originals.
const lib = await Library.open({
client,
downloadDirectory: dir,
cacheDirectory: ctx.cacheDir,
precacheThumbnails: false,
precacheOriginals: false,
});
try {
const result = await lib.backup({
+3 -21
View File
@@ -321,23 +321,6 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
return this.get(fileID, "thumbnail", "on-demand", opts?.onProgress);
}
// Get an original for a backup. One not present anywhere is written
// straight to `destination` and recorded there, so no second copy lands
// in the cache; one already present is returned where it is.
async backupOriginal(
fileID: number,
destination: string,
): Promise<ContentResult> {
const result = await this.acquire(
fileID,
"original",
"on-demand",
undefined,
{ destination },
);
return { path: result.path, bytes: result.bytes };
}
async ensure(args: EnsureOptions): Promise<EnsureResult[]> {
return this.ensureThumbnails(args);
}
@@ -439,14 +422,13 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
// The core: return the cached path if present, else fetch through the pool,
// store, and return it. `cached` distinguishes a present hit (no network,
// no download event) from a fresh fetch. A fetched original is stored at
// `opts.destination` when given, instead of in `originalsDir`.
// no download event) from a fresh fetch.
private async acquire(
fileID: number,
kind: Kind,
priority: Priority,
signal: AbortSignal | undefined,
opts?: { onByte?: ProgressCallback; destination?: string },
opts?: { onByte?: ProgressCallback },
): Promise<{ path: string; bytes: number; cached: boolean }> {
const file = this.getFile(fileID);
if (!file) throw new Error(`content cache: unknown file ${fileID}`);
@@ -487,7 +469,7 @@ export class ContentCache implements PhotoContent, ThumbnailsAPI {
kind === "original" ? this.originalsDir : this.thumbnailsDir;
const dest =
kind === "original"
? (opts?.destination ?? join(dir, originalName(file)))
? join(dir, originalName(file))
: join(dir, `${fileID}${THUMBNAIL_EXT}`);
const pool =
kind === "original" ? this.pools.content : this.pools.thumbnails;
+7 -10
View File
@@ -538,13 +538,11 @@ export class Library {
}
// Back up every in-scope file to `downloadDirectory` in the historical
// on-disk layout, with a durable failure ledger (issue #51). Waits for a
// completed refresh first, as `fresh()` does, joining one already running,
// and rejects before touching any file when it fails. Then fetches pending
// originals (and optional thumbnails) through the content cache and pools,
// and rebuilds the derived symlink/JSON views from the model. Throws before
// any network work when no download directory is available or no content
// cache backs the originals it must fetch.
// on-disk layout, with a durable failure ledger (issue #51). Refreshes
// first, fetches pending originals (and optional thumbnails) through the
// content cache and pools, then rebuilds the derived symlink/JSON views
// from the model. Throws before any network work when no download directory
// is available or no content cache backs the originals it must fetch.
backup(opts?: BackupOptions): Promise<BackupResult> {
const downloadDirectory =
opts?.downloadDirectory ?? this.downloadDirectory;
@@ -568,11 +566,10 @@ export class Library {
const cache = this.cache;
return runBackup(
{
refresh: () => this.refreshNow(),
refresh: () => this.runRefresh(),
listCollections: () => this.store.listCollections(),
listFiles: (id) => this.store.listFiles(id),
original: (fileID, destination) =>
cache!.backupOriginal(fileID, destination),
original: (fileID) => cache!.original(fileID),
thumbnail: (fileID) => cache!.thumbnail(fileID),
},
{ ...opts, downloadDirectory },
+1 -116
View File
@@ -562,39 +562,11 @@ describe("lib.backup", () => {
lib.close();
});
it("fetches each original once and writes it only into the backup", async () => {
// A backup of a 500 GB account must write 500 GB, not a copy in the
// cache as well: an original fetched for the backup goes straight
// into its originals/, and the cache records it there.
const source = stubSource();
const lib = await openLibrary(source);
const outDir = join(root, "backup");
const result = await lib.backup({ downloadDirectory: outDir });
expect(result.downloaded).toBe(3);
expect(source.originalCalls).toBe(3);
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
const stored = readdirSync(join(outDir, "originals")).filter(
(name) => !name.endsWith(".json"),
);
expect(stored.sort()).toEqual(["100.jpg", "101.jpg", "200.png"]);
// The cache counts the backup's copy as present: reading the
// original afterwards fetches nothing and answers with that copy.
const read = await lib.photos.byID({ fileID: 100 })!.original();
expect(read.path).toBe(join(outDir, "originals", "100.jpg"));
expect(source.originalCalls).toBe(3);
await lib.close();
});
it("fsyncs an original copied from the cache before the rename and its directory after", async () => {
it("fsyncs a copied original before the rename and its directory after", async () => {
const lib = await openLibrary(stubSource());
const outDir = join(root, "backup");
const originals = join(outDir, "originals");
const dest = join(originals, "100.jpg");
// Only an original already in the cache is copied into the backup;
// one fetched for the backup is written there by the download writer.
await lib.photos.byID({ fileID: 100 })!.original();
fsEvents.length = 0;
await lib.backup({ downloadDirectory: outDir });
@@ -653,93 +625,6 @@ describe("lib.backup", () => {
});
});
// Every refresh fails, as with an expired session or no network.
class FailingClient extends MockClient {
override async collectionsSince(): Promise<CollectionsPage> {
throw new Error("HTTP 401 from server");
}
}
// Holds its refresh open until `release()` is called, then reports a third
// album, so a backup can be started while that refresh is still running.
class HeldClient extends MockClient {
release!: () => void;
private held = new Promise<void>((resolve) => {
this.release = resolve;
});
override async collectionsSince(): Promise<CollectionsPage> {
await this.held;
return {
collections: [collection(3, "Later")],
deleted: [],
cursor: 2,
};
}
override async filesSince(args: {
collectionID: number;
}): Promise<FilesPage> {
if (args.collectionID !== 3) return super.filesSince(args);
return { files: [file(300, 3, "late.jpg")], deleted: [], cursor: 2 };
}
}
// Fill the library cache on disk, so the next open starts its refresh in the
// background instead of waiting for it.
const fillCache = async (): Promise<void> => {
const lib = await openLibrary(stubSource());
await lib.close();
};
describe("the refresh before a backup", () => {
it("waits for a refresh already running and backs up what it found", async () => {
await fillCache();
const client = new HeldClient();
const lib = await openLibrary(stubSource(), client);
const outDir = join(root, "backup");
const backup = lib.backup({ downloadDirectory: outDir });
client.release();
const result = await backup;
expect(result.totalFiles).toBe(4);
expect(existsSync(join(outDir, "originals", "300.jpg"))).toBe(true);
await lib.close();
});
it("fails before any download when the refresh fails, leaving failures.json as it was", async () => {
await fillCache();
const outDir = join(root, "backup");
seedLedger(outDir, 100, "beach.jpg");
const ledgerPath = join(outDir, "failures.json");
const ledgerBefore = readFileSync(ledgerPath, "utf-8");
const source = stubSource();
const lib = await openLibrary(source, new FailingClient());
await expect(lib.backup({ downloadDirectory: outDir })).rejects.toThrow(
"HTTP 401 from server",
);
expect(source.originalCalls).toBe(0);
expect(readFileSync(ledgerPath, "utf-8")).toBe(ledgerBefore);
expect(existsSync(join(outDir, "originals"))).toBe(false);
await lib.close();
});
it("fails when the refresh fails on an empty cache, instead of backing up nothing", async () => {
const source = stubSource();
const lib = await openLibrary(source, new FailingClient());
const outDir = join(root, "backup");
await expect(lib.backup({ downloadDirectory: outDir })).rejects.toThrow(
"HTTP 401 from server",
);
expect(source.originalCalls).toBe(0);
expect(existsSync(outDir)).toBe(false);
await lib.close();
});
});
// The album folders under collections/, driven through `runBackup` with a
// stand-in library whose albums a test changes between runs.
describe("backup album folders", () => {
+2 -186
View File
@@ -13,7 +13,6 @@
import {
existsSync,
mkdtempSync,
readdirSync,
readFileSync,
rmSync,
statSync,
@@ -21,21 +20,11 @@ import {
} from "node:fs";
import { join } from "node:path";
import { tmpdir } from "node:os";
import { PassThrough } from "node:stream";
import {
describe,
it,
expect,
vi,
beforeAll,
beforeEach,
afterEach,
} from "vitest";
import { describe, it, expect, beforeAll, beforeEach, afterEach } from "vitest";
import {
type CliContext,
saveSession,
loginCommand,
whoamiCommand,
logoutCommand,
collectionsCommand,
@@ -47,9 +36,8 @@ import {
listMissingThumbnailsCommand,
fixMissingThumbnailsCommand,
} from "../../src/cli-commands.js";
import { run } from "../../src/cli-run.js";
import { loadSession } from "../../src/cli-session.js";
import type { Client, ClientSnapshot, LoginOptions } from "../../src/client.js";
import type { Client, ClientSnapshot } from "../../src/client.js";
import type { ContentSource } from "../../src/library/content.js";
import type { Collection, EnteFile } from "../../src/model/types.js";
import { init, toBase64 } from "../../src/crypto/index.js";
@@ -175,15 +163,6 @@ const context = (client: Client | null = fakeClient()): CliContext => ({
sessionDir: join(root, "session"),
cacheDir: join(root, "cache"),
loadSession: () => client,
login: async () => {
throw new Error("login not expected");
},
prompt: async () => {
throw new Error("prompt not expected");
},
promptSecret: async () => {
throw new Error("prompt not expected");
},
});
beforeAll(async () => {
@@ -241,105 +220,6 @@ describe("session file", () => {
});
});
// The login function is a fake that hands back a client whose snapshot is
// `snapshot`; each prompt is recorded and answered with "123456".
describe("login", () => {
const snapshot: ClientSnapshot = {
email: "cli@example.com",
userID: USER_ID,
token: "token",
masterKey: "a",
secretKey: "b",
publicKey: "c",
};
const loggedIn = {
whoami: () => ({ email: "cli@example.com", userID: USER_ID }),
toJSON: () => snapshot,
} as unknown as Client;
let prompts: string[];
const loginContext = (
login: (opts: LoginOptions) => Promise<Client>,
): CliContext => ({
...context(),
login,
prompt: async (message) => {
prompts.push(message);
return "123456";
},
promptSecret: async (message) => {
prompts.push(message);
return "123456";
},
});
beforeEach(() => {
prompts = [];
vi.stubEnv("QUAK_EMAIL", "cli@example.com");
vi.stubEnv("QUAK_PASSWORD", "hunter2");
});
afterEach(() => {
vi.unstubAllEnvs();
});
it("takes the email and password from the environment without a prompt", async () => {
const calls: LoginOptions[] = [];
const ctx = loginContext(async (opts) => {
calls.push(opts);
return loggedIn;
});
expect(await loginCommand(ctx)).toBe(0);
expect(prompts).toEqual([]);
expect(calls).toHaveLength(1);
expect(calls[0]!.email).toBe("cli@example.com");
expect(calls[0]!.password).toBe("hunter2");
const path = join(ctx.sessionDir, "session.json");
expect(stderr.text).toBe(
"Authenticating...\n" +
`Logged in as cli@example.com (user ${USER_ID})\n` +
`Session saved to ${path}\n`,
);
});
it("saves the session with mode 0600 in a directory with mode 0700", async () => {
const ctx = loginContext(async () => loggedIn);
expect(await loginCommand(ctx)).toBe(0);
expect(statSync(ctx.sessionDir).mode & 0o777).toBe(0o700);
const path = join(ctx.sessionDir, "session.json");
expect(statSync(path).mode & 0o777).toBe(0o600);
expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot);
});
it("asks for the TOTP code when the account needs one", async () => {
let code: string | undefined;
const ctx = loginContext(async (opts) => {
code = await opts.totp!();
return loggedIn;
});
expect(await loginCommand(ctx)).toBe(0);
expect(prompts).toEqual(["TOTP code: "]);
expect(code).toBe("123456");
});
it("a failed login exits 1, says why and writes no session", async () => {
const ctx = loginContext(async () => {
throw new Error("HTTP 401 from server");
});
expect(await loginCommand(ctx)).toBe(1);
expect(stderr.text).toBe(
"Authenticating...\nLogin failed: HTTP 401 from server\n",
);
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
});
});
// These use a real client read from the session file, over a fake API that
// records each request and answers with `status`.
describe("logout", () => {
@@ -545,16 +425,6 @@ describe("backup", () => {
expect(stdout.text).toBe("");
});
// The backup opens its library with the precache off: it fetches the
// originals it needs into the backup, and must not also fetch every
// thumbnail in the account, or keep originals, in the per-user cache.
it("leaves nothing in the cache's originals and thumbnails", async () => {
const dir = join(root, "backup");
expect(await backupCommand(context(), dir, {})).toBe(0);
expect(readdirSync(join(root, "cache", "thumbnails"))).toEqual([]);
expect(readdirSync(join(root, "cache", "originals"))).toEqual([]);
});
it("exits 1 and lists the file when one download fails", async () => {
const ctx = context(fakeClient({ failID: 101 }));
expect(await backupCommand(ctx, join(root, "backup"), {})).toBe(1);
@@ -581,34 +451,6 @@ describe("backup", () => {
expect(result.errors[0].fileID).toBe(101);
expect(stderr.text).toBe("Starting backup...\n");
});
it("exits 1 with the error on one line when the refresh fails", async () => {
const client = {
...fakeClient(),
collectionsSince: async () => {
throw new Error("HTTP 401 from server");
},
} as unknown as Client;
const dir = join(root, "backup");
// Through `run`, as `bin/quak.ts` does, which prints a thrown error.
const runStderr = new PassThrough();
let runText = "";
runStderr.on("data", (chunk: Buffer) => {
runText += chunk.toString();
});
const code = await new Promise<number>((resolve) => {
void run(
backupCommand(context(client), dir, {}),
new PassThrough(),
runStderr,
resolve,
);
});
expect(code).toBe(1);
expect(runText).toBe("quak: HTTP 401 from server\n");
expect(stderr.text).toBe("Starting backup...\nRefreshing library...\n");
expect(existsSync(join(dir, "originals"))).toBe(false);
});
});
describe("helper list-missing-thumbnails", () => {
@@ -642,32 +484,6 @@ describe("helper list-missing-thumbnails", () => {
});
});
describe("backup-metadata --exif", () => {
// Runs the command and returns what it printed to stderr.
const backupMetadata = async (opts: { exif?: boolean; all?: boolean }) => {
expect(
await backupMetadataCommand(context(), join(root, "dump"), opts),
).toBe(0);
return stderr.text;
};
it("--exif extracts EXIF", async () => {
expect(await backupMetadata({ exif: true })).toContain(
"[beach.jpg] Extracting EXIF...\n",
);
});
it("--all extracts EXIF", async () => {
expect(await backupMetadata({ all: true })).toContain(
"[beach.jpg] Extracting EXIF...\n",
);
});
it("without either flag extracts no EXIF", async () => {
expect(await backupMetadata({})).not.toContain("Extracting EXIF");
});
});
// Each test first runs `collections` so the cache holds the account as it was,
// then changes the server under it.
describe("backup-metadata and the thumbnail helpers refresh first", () => {