Compare commits
4
Commits
c005978ee7
...
4e7d629b56
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4e7d629b56 | ||
|
|
c24c4dda4f | ||
|
|
4bb75ca323 | ||
|
|
cd05a458dc |
@@ -318,6 +318,7 @@ Endpoints used:
|
||||
encrypted token plus key attributes.
|
||||
- `POST /users/ott` and `POST /users/verify-email`: email OTP fallback path.
|
||||
- `POST /users/two-factor/verify`: TOTP second factor.
|
||||
- `POST /users/logout`: end the calling token's session (`quak logout`).
|
||||
- `GET /collections/v2?sinceTime=<usec>`: list collections changed since
|
||||
microsecond timestamp; pass 0 for a full enumeration.
|
||||
- `GET /collections/v2/diff?collectionID=<id>&sinceTime=<usec>`: list files in a
|
||||
@@ -432,7 +433,9 @@ whatever else fits their use case. `Client.fromJSON(snapshot)` restores a
|
||||
working client from that snapshot without re-authenticating; it checks every
|
||||
field and each key's length first, and throws an error naming the bad field.
|
||||
`client.logout()` clears the token and zeroes the key buffers in place; every
|
||||
later call on that client throws.
|
||||
later call on that client throws. It does not contact the server, so the token
|
||||
stays valid there and in any saved snapshot; `await client.logoutOnServer()`
|
||||
first ends the session on the server (`POST /users/logout`).
|
||||
|
||||
The CLI stores the snapshot at the platform-appropriate data directory via
|
||||
`env-paths`: `~/Library/Application Support/quak/session.json` on macOS,
|
||||
@@ -442,13 +445,21 @@ you would treat the password itself. A missing file is reported as "not logged
|
||||
in"; a file that exists but is corrupt is reported as such, naming the bad
|
||||
field. Both exit with status 1.
|
||||
|
||||
`quak logout` ends the session on the server, so the token in `session.json`
|
||||
stops working even in a copy of the file, and then deletes the file. If the
|
||||
server call fails (or the file is corrupt), the file is still deleted, the
|
||||
command says the server session could not be ended, and it exits with status 1.
|
||||
It does not delete the cache: it prints the account's cache directory and says
|
||||
it still holds decrypted data (file keys in `metadata.json`, cached originals
|
||||
and thumbnails), for the user to delete if they want it gone.
|
||||
|
||||
### CLI surface
|
||||
|
||||
```
|
||||
quak [--cache-dir <path>] <command> global: local metadata/content cache location
|
||||
quak login interactive or QUAK_EMAIL/QUAK_PASSWORD
|
||||
quak whoami print logged-in account as JSON
|
||||
quak logout delete saved session
|
||||
quak logout end the session, delete it
|
||||
quak collections [--json] list all collections
|
||||
quak files --collection <id> [--json] list files in a collection
|
||||
quak get <fileID> [--out path] [--collection] download and decrypt a file
|
||||
@@ -480,7 +491,13 @@ on. The exit code is non-zero if any ML data request failed.
|
||||
only, because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG
|
||||
image (PNG, HEIC) or a video is reported as `skipped` (unsupported format), kept
|
||||
distinct from a `failed` repair, and does not affect the exit code; a genuine
|
||||
failure still exits non-zero.
|
||||
failure still exits non-zero. The server accepts a new thumbnail only from the
|
||||
file's owner and only when it is no larger than the thumbnail size it records
|
||||
for the file. So a file another account owns, in an album shared with you, is
|
||||
skipped by both thumbnail helpers without being fetched, and the fixer skips a
|
||||
file whose recorded thumbnail size is 0 or unknown. Otherwise the fixer lowers
|
||||
the quality and size of the thumbnail until it fits, and skips the file if even
|
||||
the smallest does not.
|
||||
|
||||
### Backup layout
|
||||
|
||||
@@ -497,6 +514,20 @@ failure still exits non-zero.
|
||||
<name>.json collection metadata + file list
|
||||
```
|
||||
|
||||
A collection's directory and JSON are named after the collection, and a symlink
|
||||
after the file's title, both with unsafe characters replaced. When two
|
||||
collections would get the same name, or two files in one collection the same
|
||||
title (ignoring case in both), each of them gets its ID added: two albums named
|
||||
`Trip` become `Trip (10)/` and `Trip (11)/`, and two files titled `IMG_0001.JPG`
|
||||
become `IMG_0001 (12345).JPG` and `IMG_0001 (12346).JPG`. IDs never change, so a
|
||||
name stays the same from run to run until such a clash appears or goes away.
|
||||
|
||||
Each run removes the symlinks into `originals/` that no longer belong in their
|
||||
collection's directory, and the directories (and JSON) of collections that were
|
||||
deleted or renamed. Nothing else in `collections/` is touched: a file or a
|
||||
symlink you put there stays, and a directory that still holds one after its
|
||||
symlinks are removed stays too, with its JSON.
|
||||
|
||||
Each file is downloaded exactly once regardless of how many collections it
|
||||
appears in. On subsequent runs, existing originals are skipped. If a download
|
||||
fails, the error is logged and the backup continues with the next file. The exit
|
||||
|
||||
@@ -18,6 +18,33 @@ Tag v1.0.0.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-09-23: Stopped `helper fix-missing-thumbnails` retrying files the server
|
||||
always refuses (issue 109). Both thumbnail helpers skip a file another account
|
||||
owns without fetching it. The fixer skips a file whose recorded thumbnail size
|
||||
is 0 or unknown before downloading it, and otherwise tries smaller encodings
|
||||
(720 px quality 50 down to 160 px quality 20) until the encrypted thumbnail is
|
||||
no larger than that size, skipping the file if none fits.
|
||||
|
||||
- 2026-09-23: Tested the live-photo hash check's error paths (issue 117). Tests
|
||||
download a live photo whose ZIP names an unknown compression method, one whose
|
||||
ZIP has no image entry and one with no video entry, and check that nothing is
|
||||
stored and the error names the file ID; the unreadable one is not retried.
|
||||
|
||||
- 2026-09-23: `quak logout` ends the session on the server (issue 108). It calls
|
||||
`POST /users/logout` through the new `Client.logoutOnServer()`, then deletes
|
||||
`session.json` even when that call fails, says so and exits 1. It prints the
|
||||
account's cache directory and says it still holds decrypted data. The default
|
||||
cache path is now `defaultCacheDirectory()` in the library, shared with
|
||||
`Library.open`.
|
||||
|
||||
- 2026-09-23: Fixed the backup's per-collection folders (issue 103). Two files
|
||||
in one collection with the same title, and two collections with the same name,
|
||||
each get their ID added to the name (`IMG_0001 (12345).JPG`, `Trip (10)/`), so
|
||||
none replaces another's symlink or JSON. Each run removes symlinks into
|
||||
`originals/` for files no longer in the collection, and the folders of deleted
|
||||
or renamed collections, leaving anything else in `collections/` alone. The
|
||||
README backup layout states the naming rule.
|
||||
|
||||
- 2026-09-23: Checked downloaded originals against their recorded content hash
|
||||
(issue 68). `downloadFile`, which `quak get`, the content cache and backup all
|
||||
use, hashes the decrypted bytes (unkeyed BLAKE2b-512, standard base64) and
|
||||
@@ -26,12 +53,14 @@ Tag v1.0.0.
|
||||
hashed separately as `<imageHash>:<videoHash>`. `decryptFile` reads older
|
||||
clients' `imageHash` and `videoHash` fields for live photos. A file with no
|
||||
recorded hash is stored unchecked.
|
||||
|
||||
- 2026-09-23: Kept one account's cache from mixing with another's (issue 104).
|
||||
When `metadata.json` in the cache directory was written for a different,
|
||||
non-zero user ID than the client's, `Library.open` deletes it and `mldata/`
|
||||
and starts empty, so the first refresh enumerates from 0. This only happens
|
||||
with `--cache-dir` or an explicit `cacheDirectory`; the default path already
|
||||
includes the user ID. A test opens one account's cache as another account.
|
||||
|
||||
- 2026-09-23: `backup-metadata` no longer stops on one failed ML data request
|
||||
(issue 101). Each request of up to 200 files is tried on its own; a failed one
|
||||
is logged, its files are written with the reason in `mlDataError`, and the
|
||||
|
||||
+1
-1
@@ -73,7 +73,7 @@ program
|
||||
|
||||
program
|
||||
.command("logout")
|
||||
.description("Delete the saved session")
|
||||
.description("End the session on the server and delete the saved session")
|
||||
.action(() => run(logoutCommand(context())));
|
||||
|
||||
program
|
||||
|
||||
+134
-11
@@ -17,7 +17,9 @@
|
||||
// temp-then-rename, so a file that exists is whole and is never re-fetched — an
|
||||
// interrupted run resumes by listing the directory. The derived views hold no
|
||||
// unique state, so they are rebuilt every run; that repairs stale sidecars and
|
||||
// missing or broken symlinks left by an earlier crash.
|
||||
// missing or broken symlinks left by an earlier crash. A rebuild also removes
|
||||
// the symlinks into originals/ that no longer belong to an album, and the
|
||||
// directories of albums that no longer exist.
|
||||
//
|
||||
// Resilience (issue #8): no per-file condition aborts the run. A failed
|
||||
// download or a failed symlink is caught, recorded in `failures.json` with a
|
||||
@@ -34,13 +36,14 @@ import {
|
||||
readdirSync,
|
||||
readFileSync,
|
||||
readlinkSync,
|
||||
rmdirSync,
|
||||
rmSync,
|
||||
statSync,
|
||||
symlinkSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { copyFile, rename, rm } from "node:fs/promises";
|
||||
import { basename, dirname, join, relative } from "node:path";
|
||||
import { basename, dirname, extname, join, relative } from "node:path";
|
||||
|
||||
import { fsyncPath } from "./download/index.js";
|
||||
import { safeExtension, sanitizeFileName } from "./filename.js";
|
||||
@@ -227,6 +230,93 @@ const rebuildSymlink = (linkPath: string, target: string): void => {
|
||||
symlinkSync(target, linkPath);
|
||||
};
|
||||
|
||||
// The on-disk names for the entries of one directory, keyed by ID. Each name
|
||||
// is used as is unless another entry would get the same name, ignoring case
|
||||
// (two names that differ only in case are one entry on a case-insensitive
|
||||
// file system); then every entry sharing it gets ` (<id>)`, before the
|
||||
// extension when `beforeExtension` is set. A name with an ID added can match
|
||||
// another entry's own name (`IMG (6).JPG`), so this repeats until no name is
|
||||
// shared. IDs are stable, so the names are too.
|
||||
const namesByID = (
|
||||
entries: { id: number; name: string }[],
|
||||
beforeExtension: boolean,
|
||||
): Map<number, string> => {
|
||||
const withID = (id: number, name: string): string => {
|
||||
const ext = beforeExtension ? extname(name) : "";
|
||||
const stem = name.slice(0, name.length - ext.length);
|
||||
return `${stem} (${id})${ext}`;
|
||||
};
|
||||
const names = new Map<number, string>();
|
||||
for (const { id, name } of entries) names.set(id, name);
|
||||
const suffixed = new Set<number>();
|
||||
for (;;) {
|
||||
const counts = new Map<string, number>();
|
||||
for (const name of names.values()) {
|
||||
const key = name.toLowerCase();
|
||||
counts.set(key, (counts.get(key) ?? 0) + 1);
|
||||
}
|
||||
let changed = false;
|
||||
for (const { id, name } of entries) {
|
||||
if (suffixed.has(id)) continue;
|
||||
if (counts.get(name.toLowerCase()) === 1) continue;
|
||||
names.set(id, withID(id, name));
|
||||
suffixed.add(id);
|
||||
changed = true;
|
||||
}
|
||||
if (!changed) return names;
|
||||
}
|
||||
};
|
||||
|
||||
// Remove the symlinks in the album directory `dir` that point into
|
||||
// `originalsDir` and are not named in `keep`. Nothing else in the directory
|
||||
// is touched: anything else there was put there by the user.
|
||||
const removeStaleLinks = (
|
||||
dir: string,
|
||||
keep: Set<string>,
|
||||
originalsDir: string,
|
||||
): void => {
|
||||
const target = relative(dir, originalsDir);
|
||||
for (const name of readdirSync(dir)) {
|
||||
if (keep.has(name)) continue;
|
||||
const path = join(dir, name);
|
||||
if (
|
||||
lstatSync(path).isSymbolicLink() &&
|
||||
dirname(readlinkSync(path)) === target
|
||||
) {
|
||||
rmSync(path);
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
// Remove the directories under `collectionsDir` that an earlier run wrote for
|
||||
// an album that is gone or renamed: a directory not named in `current` with a
|
||||
// `<name>.json` beside it holding an album ID, which is what a run writes. Its
|
||||
// symlinks into originals/ are removed; if that leaves it empty, it and its
|
||||
// JSON are deleted, otherwise both stay for what the user put there.
|
||||
const removeStaleAlbumDirs = (
|
||||
collectionsDir: string,
|
||||
current: Set<string>,
|
||||
originalsDir: string,
|
||||
): void => {
|
||||
for (const entry of readdirSync(collectionsDir, { withFileTypes: true })) {
|
||||
if (!entry.isDirectory() || current.has(entry.name)) continue;
|
||||
const jsonPath = join(collectionsDir, `${entry.name}.json`);
|
||||
try {
|
||||
const album = JSON.parse(readFileSync(jsonPath, "utf-8")) as {
|
||||
id?: unknown;
|
||||
};
|
||||
if (typeof album.id !== "number") continue;
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
const dir = join(collectionsDir, entry.name);
|
||||
removeStaleLinks(dir, new Set(), originalsDir);
|
||||
if (readdirSync(dir).length > 0) continue;
|
||||
rmdirSync(dir);
|
||||
rmSync(jsonPath);
|
||||
}
|
||||
};
|
||||
|
||||
const loadLedger = (path: string): Map<number, FailureEntry> => {
|
||||
const ledger = new Map<number, FailureEntry>();
|
||||
try {
|
||||
@@ -303,9 +393,10 @@ export const runBackup = async (
|
||||
|
||||
// Collections in scope, and the distinct files across them (a file shared
|
||||
// by two albums is one original).
|
||||
const collections = lib
|
||||
.listCollections()
|
||||
.filter((c) => (only ? only.has(c.name) : true));
|
||||
const allCollections = lib.listCollections();
|
||||
const collections = allCollections.filter((c) =>
|
||||
only ? only.has(c.name) : true,
|
||||
);
|
||||
const collectionName = new Map<number, string>();
|
||||
for (const c of collections) collectionName.set(c.id, c.name);
|
||||
|
||||
@@ -406,23 +497,55 @@ export const runBackup = async (
|
||||
}
|
||||
}
|
||||
|
||||
// Then the per-collection symlink trees and JSON.
|
||||
// Then the per-collection symlink trees and JSON. Directory names are
|
||||
// chosen across every album, not just those in scope, so a scoped run
|
||||
// names an album the same as a full one and never takes the directory of
|
||||
// an album it skipped. Stale entries are removed before anything is
|
||||
// rebuilt, so on a case-insensitive file system removing an old name can
|
||||
// never remove the new one.
|
||||
const albumDirNames = namesByID(
|
||||
allCollections.map((c) => ({
|
||||
id: c.id,
|
||||
name: sanitizeFileName(c.name, `collection-${c.id}`),
|
||||
})),
|
||||
false,
|
||||
);
|
||||
try {
|
||||
removeStaleAlbumDirs(
|
||||
collectionsDir,
|
||||
new Set(albumDirNames.values()),
|
||||
originalsDir,
|
||||
);
|
||||
} catch (err) {
|
||||
log(`FAILED removing old album directories: ${errorMessage(err)}`);
|
||||
}
|
||||
|
||||
for (const c of collections) {
|
||||
const colDirName = sanitizeFileName(c.name, `collection-${c.id}`);
|
||||
const colDirName = albumDirNames.get(c.id)!;
|
||||
const colDir = join(collectionsDir, colDirName);
|
||||
mkdirSync(colDir, { recursive: true });
|
||||
|
||||
const files = filesByCollection.get(c.id) ?? [];
|
||||
const linkNames = namesByID(
|
||||
files.map((f) => ({
|
||||
id: f.id,
|
||||
name: sanitizeFileName(f.metadata.title, `file-${f.id}`),
|
||||
})),
|
||||
true,
|
||||
);
|
||||
try {
|
||||
removeStaleLinks(colDir, new Set(linkNames.values()), originalsDir);
|
||||
} catch (err) {
|
||||
log(`FAILED removing old links in ${c.name}: ${errorMessage(err)}`);
|
||||
}
|
||||
|
||||
const metaFiles: { id: number; metadata: EnteFile["metadata"] }[] = [];
|
||||
for (const file of files) {
|
||||
metaFiles.push({ id: file.id, metadata: file.metadata });
|
||||
if (!includeOriginals) continue;
|
||||
const orig = join(originalsDir, originalName(file));
|
||||
if (!isPresent(orig)) continue;
|
||||
const linkName = sanitizeFileName(
|
||||
file.metadata.title,
|
||||
`file-${file.id}`,
|
||||
);
|
||||
const linkName = linkNames.get(file.id)!;
|
||||
const linkPath = join(colDir, linkName);
|
||||
try {
|
||||
rebuildSymlink(linkPath, relative(colDir, orig));
|
||||
|
||||
+40
-7
@@ -17,7 +17,11 @@ import {
|
||||
import { join } from "node:path";
|
||||
import { Client, type ClientSnapshot } from "./client.js";
|
||||
import { init } from "./crypto/index.js";
|
||||
import { Library, type LibraryClient } from "./library/index.js";
|
||||
import {
|
||||
defaultCacheDirectory,
|
||||
Library,
|
||||
type LibraryClient,
|
||||
} from "./library/index.js";
|
||||
import {
|
||||
fileListRow,
|
||||
fileListLine,
|
||||
@@ -146,14 +150,43 @@ export const whoamiCommand = async (ctx: CliContext): Promise<number> => {
|
||||
return 0;
|
||||
};
|
||||
|
||||
// Ends the session on the server, then deletes the session file even when that
|
||||
// failed, and exits 1 if it did. The cache is left in place; the user is told
|
||||
// where it is.
|
||||
export const logoutCommand = async (ctx: CliContext): Promise<number> => {
|
||||
if (existsSync(sessionPath(ctx))) {
|
||||
unlinkSync(sessionPath(ctx));
|
||||
ctx.stderr.write("Session deleted.\n");
|
||||
} else {
|
||||
const path = sessionPath(ctx);
|
||||
if (!existsSync(path)) {
|
||||
ctx.stderr.write("No session found.\n");
|
||||
return 0;
|
||||
}
|
||||
return 0;
|
||||
await init();
|
||||
let cacheDir = ctx.cacheDir;
|
||||
let failure: string | undefined;
|
||||
try {
|
||||
const client = ctx.loadSession(path);
|
||||
if (client) {
|
||||
cacheDir ??= defaultCacheDirectory(client.whoami().userID);
|
||||
await client.logoutOnServer();
|
||||
client.logout();
|
||||
}
|
||||
} catch (err) {
|
||||
failure = err instanceof Error ? err.message : String(err);
|
||||
}
|
||||
unlinkSync(path);
|
||||
if (failure === undefined) {
|
||||
ctx.stderr.write("Session ended on the server.\n");
|
||||
} else {
|
||||
ctx.stderr.write(
|
||||
`Could not end the session on the server: ${failure}\n`,
|
||||
);
|
||||
}
|
||||
ctx.stderr.write("Session deleted.\n");
|
||||
if (cacheDir !== undefined) {
|
||||
ctx.stderr.write(
|
||||
`Cache directory ${cacheDir} still holds decrypted data; delete it to remove that data.\n`,
|
||||
);
|
||||
}
|
||||
return failure === undefined ? 0 : 1;
|
||||
};
|
||||
|
||||
export const collectionsCommand = async (
|
||||
@@ -462,7 +495,7 @@ export const fixMissingThumbnailsCommand = async (
|
||||
ctx.stderr.write(` Skipped: ${skipped}\n`);
|
||||
ctx.stderr.write(` Failed: ${failed}\n`);
|
||||
if (skipped > 0) {
|
||||
ctx.stderr.write("\nSkipped (unsupported format):\n");
|
||||
ctx.stderr.write("\nSkipped:\n");
|
||||
for (const r of results.filter((r) => r.status === "skipped")) {
|
||||
ctx.stderr.write(
|
||||
` ${r.fileID}\t${r.title}\t${r.reason}\n`,
|
||||
|
||||
@@ -217,6 +217,14 @@ export class Client {
|
||||
};
|
||||
}
|
||||
|
||||
// Ends this client's session on the server (`POST /users/logout`), so the
|
||||
// token stops working everywhere, including in any saved copy of it. This
|
||||
// client is left as it was; call `logout()` to clear it.
|
||||
async logoutOnServer(): Promise<void> {
|
||||
this.assertLoggedIn();
|
||||
await this.api.postJSON("/users/logout", {});
|
||||
}
|
||||
|
||||
// Zeroes the key buffers in place, so any copy of the reference held
|
||||
// elsewhere is wiped too. Every method checks `assertLoggedIn` before
|
||||
// touching the keys, so nothing decrypts with the zeroed keys.
|
||||
|
||||
@@ -98,6 +98,11 @@ export {
|
||||
|
||||
export const DEFAULT_REFRESH_INTERVAL_SECONDS = 3;
|
||||
|
||||
// The account's cache directory when `cacheDirectory` is not given: the
|
||||
// env-paths cache directory plus the user id, so each account has its own.
|
||||
export const defaultCacheDirectory = (userID: number): string =>
|
||||
join(envPaths("quak", { suffix: "" }).cache, String(userID));
|
||||
|
||||
// Project a metadata store into by-id records, filling each record's cache
|
||||
// paths from the content cache when one is given. Shared by the live read
|
||||
// projection and the precache's initial seeding at open().
|
||||
@@ -344,8 +349,7 @@ export class Library {
|
||||
static async open(opts: LibraryOptions): Promise<Library> {
|
||||
const { userID } = opts.client.whoami();
|
||||
const cacheDirectory =
|
||||
opts.cacheDirectory ??
|
||||
join(envPaths("quak", { suffix: "" }).cache, String(userID));
|
||||
opts.cacheDirectory ?? defaultCacheDirectory(userID);
|
||||
const metadataPath = join(cacheDirectory, "metadata.json");
|
||||
let store = await MetadataStore.load(metadataPath);
|
||||
// A cache directory given explicitly can hold another account's cache.
|
||||
|
||||
+88
-30
@@ -7,8 +7,21 @@ import { ApiError } from "./api/client.js";
|
||||
import { encryptBlob, toBase64 } from "./crypto/index.js";
|
||||
import type { EnteFile } from "./model/types.js";
|
||||
|
||||
const THUMB_MAX_DIMENSION = 720;
|
||||
const THUMB_JPEG_QUALITY = 50;
|
||||
// The server refuses a thumbnail larger than the one it already records for the
|
||||
// file (`thumbnail.size`, the encrypted size), so these encodings are tried
|
||||
// from largest to smallest and the first that fits is uploaded.
|
||||
const THUMB_ENCODINGS = [
|
||||
{ maxDimension: 720, quality: 50 },
|
||||
{ maxDimension: 720, quality: 30 },
|
||||
{ maxDimension: 480, quality: 30 },
|
||||
{ maxDimension: 320, quality: 20 },
|
||||
{ maxDimension: 160, quality: 20 },
|
||||
];
|
||||
|
||||
// The server accepts a new thumbnail only from the file's owner, so files other
|
||||
// people own in albums shared with this account are never checked or repaired.
|
||||
const NOT_OWNED_REASON =
|
||||
"owned by another account (only the owner can replace its thumbnail)";
|
||||
|
||||
export interface MissingThumbnailInfo {
|
||||
fileID: number;
|
||||
@@ -19,11 +32,12 @@ export interface MissingThumbnailInfo {
|
||||
|
||||
// Three outcomes, not two. "fixed": a thumbnail was generated and uploaded.
|
||||
// "failed": something went wrong (download, encode, upload) and the file still
|
||||
// has no thumbnail. "skipped": the file is a format this helper cannot
|
||||
// regenerate — a video, or an image that is not a baseline JPEG. Skipped is a
|
||||
// deliberate, expected outcome, not an error (issue #17): the repair path is
|
||||
// JPEG-only because `jpeg-js` is, and a PNG or HEIC is left for a format-aware
|
||||
// tool rather than reported as a failure.
|
||||
// has no thumbnail. "skipped": the server would refuse any thumbnail for the
|
||||
// file or this helper cannot regenerate it — a file another account owns, a
|
||||
// recorded thumbnail size nothing fits within, a video, or an image that is
|
||||
// not a baseline JPEG. Skipped is a deliberate, expected outcome, not an error
|
||||
// (issue #17): the repair path is JPEG-only because `jpeg-js` is, and a PNG or
|
||||
// HEIC is left for a format-aware tool rather than reported as a failure.
|
||||
export type ThumbnailFixStatus = "fixed" | "skipped" | "failed";
|
||||
|
||||
export interface ThumbnailFixResult {
|
||||
@@ -45,6 +59,7 @@ export type ProgressCallback = (message: string) => void;
|
||||
// exists, so it is logged and the file is left unreported. That distinction is
|
||||
// what stops `fix-missing-thumbnails` from regenerating and uploading over
|
||||
// thumbnails that were fine all along while the CDN was briefly returning 500s.
|
||||
// Files another account owns are logged as skipped and not checked.
|
||||
export const listMissingThumbnails = async (
|
||||
lib: Library,
|
||||
client: Client,
|
||||
@@ -52,6 +67,7 @@ export const listMissingThumbnails = async (
|
||||
): Promise<MissingThumbnailInfo[]> => {
|
||||
const log = onProgress ?? (() => {});
|
||||
const api = client.getApiClient();
|
||||
const { userID } = client.whoami();
|
||||
const missing: MissingThumbnailInfo[] = [];
|
||||
const seen = new Set<number>();
|
||||
|
||||
@@ -60,6 +76,13 @@ export const listMissingThumbnails = async (
|
||||
for (const photo of album.photos.list()) {
|
||||
if (seen.has(photo.fileID)) continue;
|
||||
seen.add(photo.fileID);
|
||||
const file = lib.getFile(album.collectionID, photo.fileID);
|
||||
if (file && file.ownerID !== userID) {
|
||||
log(
|
||||
`[${album.name}] Skipping ${photo.title}: ${NOT_OWNED_REASON}`,
|
||||
);
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
const stream = await api.getThumbnailStream(photo.fileID);
|
||||
const reader = stream.getReader();
|
||||
@@ -135,17 +158,13 @@ const resizeRGBA = (
|
||||
return dst;
|
||||
};
|
||||
|
||||
const generateThumbnail = (fileBytes: Uint8Array): Uint8Array => {
|
||||
const decoded = jpeg.decode(fileBytes, {
|
||||
useTArray: true,
|
||||
formatAsRGBA: true,
|
||||
});
|
||||
const generateThumbnail = (
|
||||
decoded: { data: Uint8Array; width: number; height: number },
|
||||
maxDimension: number,
|
||||
quality: number,
|
||||
): Uint8Array => {
|
||||
const { width: srcW, height: srcH } = decoded;
|
||||
const scale = Math.min(
|
||||
THUMB_MAX_DIMENSION / srcW,
|
||||
THUMB_MAX_DIMENSION / srcH,
|
||||
1,
|
||||
);
|
||||
const scale = Math.min(maxDimension / srcW, maxDimension / srcH, 1);
|
||||
const dstW = Math.round(srcW * scale);
|
||||
const dstH = Math.round(srcH * scale);
|
||||
|
||||
@@ -158,7 +177,7 @@ const generateThumbnail = (fileBytes: Uint8Array): Uint8Array => {
|
||||
|
||||
const encoded = jpeg.encode(
|
||||
{ data: pixels, width: dstW, height: dstH },
|
||||
THUMB_JPEG_QUALITY,
|
||||
quality,
|
||||
);
|
||||
return new Uint8Array(encoded.data);
|
||||
};
|
||||
@@ -171,14 +190,35 @@ const generateThumbnail = (fileBytes: Uint8Array): Uint8Array => {
|
||||
const isJpeg = (bytes: Uint8Array): boolean =>
|
||||
bytes.length >= 2 && bytes[0] === 0xff && bytes[1] === 0xd8;
|
||||
|
||||
// The reason a file cannot have a JPEG thumbnail regenerated for it from its
|
||||
// metadata alone, before any bytes are fetched, or undefined when it might. A
|
||||
// non-image (video, live photo) is unsupported outright; a still image still
|
||||
// has to be checked against its actual bytes once downloaded.
|
||||
const unsupportedByType = (file: EnteFile): string | undefined => {
|
||||
// The reason a file cannot have a JPEG thumbnail regenerated for it, known from
|
||||
// its record alone before any bytes are fetched, or undefined when it might. A
|
||||
// still image still has to be checked against its actual bytes once
|
||||
// downloaded.
|
||||
const reasonToSkip = (file: EnteFile, userID: number): string | undefined => {
|
||||
if (file.ownerID !== userID) {
|
||||
return NOT_OWNED_REASON;
|
||||
}
|
||||
if (file.metadata.fileType !== "image") {
|
||||
return `unsupported file type: ${file.metadata.fileType} (only JPEG images can be regenerated)`;
|
||||
}
|
||||
if (!file.thumbnail.size) {
|
||||
return `recorded thumbnail size is ${file.thumbnail.size ?? "unknown"} (the server refuses a thumbnail larger than the one it records)`;
|
||||
}
|
||||
return undefined;
|
||||
};
|
||||
|
||||
// Encrypt the largest encoding of the decoded image whose ciphertext is no
|
||||
// larger than `maxSize`, or return undefined when even the smallest is larger.
|
||||
const encryptThumbnailWithin = (
|
||||
decoded: { data: Uint8Array; width: number; height: number },
|
||||
key: Uint8Array,
|
||||
maxSize: number,
|
||||
): { header: Uint8Array; ciphertext: Uint8Array } | undefined => {
|
||||
for (const { maxDimension, quality } of THUMB_ENCODINGS) {
|
||||
const thumbJpeg = generateThumbnail(decoded, maxDimension, quality);
|
||||
const encrypted = encryptBlob(thumbJpeg, key);
|
||||
if (encrypted.ciphertext.length <= maxSize) return encrypted;
|
||||
}
|
||||
return undefined;
|
||||
};
|
||||
|
||||
@@ -197,6 +237,7 @@ export const fixMissingThumbnails = async (
|
||||
const log = onProgress ?? (() => {});
|
||||
const results: ThumbnailFixResult[] = [];
|
||||
const api = client.getApiClient();
|
||||
const { userID } = client.whoami();
|
||||
|
||||
// Resolve each requested fileID to its file record and owning album by
|
||||
// enumerating the library, each file taken from the first album that holds
|
||||
@@ -237,18 +278,19 @@ export const fixMissingThumbnails = async (
|
||||
const { file, collectionName } = entry;
|
||||
const title = file.metadata.title;
|
||||
|
||||
const typeReason = unsupportedByType(file);
|
||||
if (typeReason) {
|
||||
log(`[${collectionName}] Skipping ${title}: ${typeReason}`);
|
||||
const skipReason = reasonToSkip(file, userID);
|
||||
if (skipReason) {
|
||||
log(`[${collectionName}] Skipping ${title}: ${skipReason}`);
|
||||
results.push({
|
||||
fileID,
|
||||
title,
|
||||
collection: collectionName,
|
||||
status: "skipped",
|
||||
reason: typeReason,
|
||||
reason: skipReason,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const maxSize = file.thumbnail.size!;
|
||||
|
||||
try {
|
||||
const photo = lib.photos.byID({ fileID });
|
||||
@@ -277,12 +319,28 @@ export const fixMissingThumbnails = async (
|
||||
}
|
||||
|
||||
log(`[${collectionName}] Generating thumbnail for ${title}...`);
|
||||
const thumbJpeg = generateThumbnail(fileBytes);
|
||||
const decoded = jpeg.decode(fileBytes, {
|
||||
useTArray: true,
|
||||
formatAsRGBA: true,
|
||||
});
|
||||
const fitting = encryptThumbnailWithin(decoded, file.key, maxSize);
|
||||
if (!fitting) {
|
||||
const reason = `no thumbnail encoding fits the recorded thumbnail size of ${maxSize} bytes`;
|
||||
log(`[${collectionName}] Skipping ${title}: ${reason}`);
|
||||
results.push({
|
||||
fileID,
|
||||
title,
|
||||
collection: collectionName,
|
||||
status: "skipped",
|
||||
reason,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const { header, ciphertext } = fitting;
|
||||
|
||||
log(
|
||||
`[${collectionName}] Encrypting and uploading thumbnail (${thumbJpeg.length} bytes)...`,
|
||||
`[${collectionName}] Uploading thumbnail (${ciphertext.length} bytes)...`,
|
||||
);
|
||||
const { header, ciphertext } = encryptBlob(thumbJpeg, file.key);
|
||||
const md5 = createHash("md5").update(ciphertext).digest("base64");
|
||||
const { objectKey, url } = await api.getUploadURL(
|
||||
ciphertext.length,
|
||||
|
||||
@@ -40,6 +40,7 @@ import {
|
||||
readFileSync,
|
||||
readlinkSync,
|
||||
rmSync,
|
||||
symlinkSync,
|
||||
writeFileSync,
|
||||
} from "node:fs";
|
||||
import { spawnSync } from "node:child_process";
|
||||
@@ -47,6 +48,7 @@ import { join } from "node:path";
|
||||
import { tmpdir } from "node:os";
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from "vitest";
|
||||
|
||||
import { runBackup, type BackupLibrary } from "../../src/backup.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import type { ContentSource } from "../../src/library/content.js";
|
||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||
@@ -622,3 +624,318 @@ describe("lib.backup", () => {
|
||||
lib.close();
|
||||
});
|
||||
});
|
||||
|
||||
// The album folders under collections/, driven through `runBackup` with a
|
||||
// stand-in library whose albums a test changes between runs.
|
||||
describe("backup album folders", () => {
|
||||
interface Album {
|
||||
collection: Collection;
|
||||
files: EnteFile[];
|
||||
}
|
||||
|
||||
const libraryOf = (albums: Album[]): BackupLibrary => ({
|
||||
refresh: async () => {},
|
||||
listCollections: () => albums.map((a) => a.collection),
|
||||
listFiles: (id) =>
|
||||
albums.find((a) => a.collection.id === id)?.files ?? [],
|
||||
original: async (fileID) => {
|
||||
const path = join(root, `source-${fileID}`);
|
||||
writeFileSync(path, `original ${fileID}`);
|
||||
return { path };
|
||||
},
|
||||
thumbnail: async () => {
|
||||
throw new Error("no thumbnails in this stand-in");
|
||||
},
|
||||
});
|
||||
|
||||
// Every entry under collections/, one level of directories deep, with each
|
||||
// symlink's target.
|
||||
const tree = (outDir: string): string[] => {
|
||||
const lines: string[] = [];
|
||||
const list = (dir: string, prefix: string): void => {
|
||||
for (const name of readdirSync(dir).sort()) {
|
||||
const path = join(dir, name);
|
||||
const st = lstatSync(path);
|
||||
if (st.isSymbolicLink()) {
|
||||
lines.push(`${prefix}${name} -> ${readlinkSync(path)}`);
|
||||
} else if (st.isDirectory() && prefix === "") {
|
||||
lines.push(`${name}/`);
|
||||
list(path, `${name}/`);
|
||||
} else {
|
||||
lines.push(`${prefix}${name}`);
|
||||
}
|
||||
}
|
||||
};
|
||||
list(join(outDir, "collections"), "");
|
||||
return lines;
|
||||
};
|
||||
|
||||
const albumID = (outDir: string, jsonName: string): number =>
|
||||
JSON.parse(readFileSync(join(outDir, "collections", jsonName), "utf-8"))
|
||||
.id;
|
||||
|
||||
it("gives every file and every album its own name when names repeat", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf([
|
||||
{
|
||||
collection: collection(10, "Trip"),
|
||||
files: [
|
||||
file(1, 10, "IMG_0001.JPG"),
|
||||
file(2, 10, "IMG_0001.JPG"),
|
||||
file(4, 10, "img_0001.jpg"),
|
||||
file(3, 10, "other.jpg"),
|
||||
],
|
||||
},
|
||||
{
|
||||
collection: collection(11, "Trip"),
|
||||
files: [file(3, 11, "other.jpg")],
|
||||
},
|
||||
]);
|
||||
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(tree(outDir)).toEqual([
|
||||
"Trip (10)/",
|
||||
"Trip (10)/IMG_0001 (1).JPG -> ../../originals/1.JPG",
|
||||
"Trip (10)/IMG_0001 (2).JPG -> ../../originals/2.JPG",
|
||||
"Trip (10)/img_0001 (4).jpg -> ../../originals/4.jpg",
|
||||
"Trip (10)/other.jpg -> ../../originals/3.jpg",
|
||||
"Trip (10).json",
|
||||
"Trip (11)/",
|
||||
"Trip (11)/other.jpg -> ../../originals/3.jpg",
|
||||
"Trip (11).json",
|
||||
]);
|
||||
expect(albumID(outDir, "Trip (10).json")).toBe(10);
|
||||
expect(albumID(outDir, "Trip (11).json")).toBe(11);
|
||||
});
|
||||
|
||||
it("keeps names unique when a name with an ID added is another entry's own name", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf([
|
||||
{
|
||||
collection: collection(10, "Trip"),
|
||||
files: [
|
||||
file(5, 10, "IMG (6).JPG"),
|
||||
file(6, 10, "IMG.JPG"),
|
||||
file(7, 10, "IMG.JPG"),
|
||||
],
|
||||
},
|
||||
{
|
||||
collection: collection(11, "Trip"),
|
||||
files: [file(8, 11, "a.jpg")],
|
||||
},
|
||||
{
|
||||
collection: collection(12, "Trip (11)"),
|
||||
files: [file(9, 12, "b.jpg")],
|
||||
},
|
||||
]);
|
||||
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(tree(outDir)).toEqual([
|
||||
"Trip (10)/",
|
||||
"Trip (10)/IMG (6) (5).JPG -> ../../originals/5.JPG",
|
||||
"Trip (10)/IMG (6).JPG -> ../../originals/6.JPG",
|
||||
"Trip (10)/IMG (7).JPG -> ../../originals/7.JPG",
|
||||
"Trip (10).json",
|
||||
"Trip (11)/",
|
||||
"Trip (11)/a.jpg -> ../../originals/8.jpg",
|
||||
"Trip (11) (12)/",
|
||||
"Trip (11) (12)/b.jpg -> ../../originals/9.jpg",
|
||||
"Trip (11) (12).json",
|
||||
"Trip (11).json",
|
||||
]);
|
||||
expect(albumID(outDir, "Trip (10).json")).toBe(10);
|
||||
expect(albumID(outDir, "Trip (11).json")).toBe(11);
|
||||
expect(albumID(outDir, "Trip (11) (12).json")).toBe(12);
|
||||
});
|
||||
|
||||
it("changes nothing on a second run over an unchanged account", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf([
|
||||
{
|
||||
collection: collection(10, "Trip"),
|
||||
files: [
|
||||
file(1, 10, "IMG_0001.JPG"),
|
||||
file(2, 10, "IMG_0001.JPG"),
|
||||
],
|
||||
},
|
||||
{
|
||||
collection: collection(11, "Trip"),
|
||||
files: [file(3, 11, "other.jpg")],
|
||||
},
|
||||
]);
|
||||
|
||||
await runBackup(lib, { downloadDirectory: outDir });
|
||||
const before = tree(outDir);
|
||||
const second = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
expect(second.downloaded).toBe(0);
|
||||
expect(second.failed).toBe(0);
|
||||
expect(tree(outDir)).toEqual(before);
|
||||
});
|
||||
|
||||
it("leaves the albums an onlyAlbumNames run skips as they were", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
// "trip" is skipped by the scoped run but its name clashes with the
|
||||
// in-scope "Trip", so "Trip" must keep its ID suffix.
|
||||
const lib = libraryOf([
|
||||
{
|
||||
collection: collection(10, "Trip"),
|
||||
files: [file(1, 10, "a.jpg")],
|
||||
},
|
||||
{
|
||||
collection: collection(11, "trip"),
|
||||
files: [file(2, 11, "b.jpg")],
|
||||
},
|
||||
{
|
||||
collection: collection(12, "Work"),
|
||||
files: [file(3, 12, "c.jpg")],
|
||||
},
|
||||
]);
|
||||
const json = (name: string): string =>
|
||||
readFileSync(join(outDir, "collections", name), "utf-8");
|
||||
|
||||
await runBackup(lib, { downloadDirectory: outDir });
|
||||
const before = tree(outDir);
|
||||
const skippedJSON = [json("trip (11).json"), json("Work.json")];
|
||||
const scoped = await runBackup(lib, {
|
||||
downloadDirectory: outDir,
|
||||
onlyAlbumNames: ["Trip"],
|
||||
});
|
||||
|
||||
expect(scoped.failed).toBe(0);
|
||||
expect(before).toEqual([
|
||||
"Trip (10)/",
|
||||
"Trip (10)/a.jpg -> ../../originals/1.jpg",
|
||||
"Trip (10).json",
|
||||
"Work/",
|
||||
"Work/c.jpg -> ../../originals/3.jpg",
|
||||
"Work.json",
|
||||
"trip (11)/",
|
||||
"trip (11)/b.jpg -> ../../originals/2.jpg",
|
||||
"trip (11).json",
|
||||
]);
|
||||
expect(tree(outDir)).toEqual(before);
|
||||
expect([json("trip (11).json"), json("Work.json")]).toEqual(
|
||||
skippedJSON,
|
||||
);
|
||||
});
|
||||
|
||||
it("removes links and album folders that are gone, and nothing the user added", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const albums: Album[] = [
|
||||
{
|
||||
collection: collection(10, "Trip"),
|
||||
files: [
|
||||
file(1, 10, "IMG_0001.JPG"),
|
||||
file(2, 10, "IMG_0001.JPG"),
|
||||
file(3, 10, "other.jpg"),
|
||||
],
|
||||
},
|
||||
{
|
||||
collection: collection(12, "Work"),
|
||||
files: [file(5, 12, "a.jpg")],
|
||||
},
|
||||
{
|
||||
collection: collection(13, "Old"),
|
||||
files: [file(5, 13, "a.jpg")],
|
||||
},
|
||||
];
|
||||
const lib = libraryOf(albums);
|
||||
await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
// What the user put in the tree: a note and a symlink of their own in
|
||||
// an album, a note in an album about to be renamed, and a folder quak
|
||||
// did not create.
|
||||
const collectionsDir = join(outDir, "collections");
|
||||
writeFileSync(join(collectionsDir, "Trip", "notes.txt"), "mine");
|
||||
symlinkSync("../elsewhere", join(collectionsDir, "Trip", "mine"));
|
||||
writeFileSync(join(collectionsDir, "Work", "keep.txt"), "mine");
|
||||
mkdirSync(join(collectionsDir, "Mine"));
|
||||
writeFileSync(join(collectionsDir, "Mine", "keep.txt"), "mine");
|
||||
|
||||
// File 2 leaves Trip, Work is renamed Office, Old is deleted.
|
||||
albums[0]!.files.splice(1, 1);
|
||||
albums[1]!.collection = collection(12, "Office");
|
||||
albums.splice(2, 1);
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(tree(outDir)).toEqual([
|
||||
"Mine/",
|
||||
"Mine/keep.txt",
|
||||
"Office/",
|
||||
"Office/a.jpg -> ../../originals/5.jpg",
|
||||
"Office.json",
|
||||
"Trip/",
|
||||
"Trip/IMG_0001.JPG -> ../../originals/1.JPG",
|
||||
"Trip/mine -> ../elsewhere",
|
||||
"Trip/notes.txt",
|
||||
"Trip/other.jpg -> ../../originals/3.jpg",
|
||||
"Trip.json",
|
||||
"Work/",
|
||||
"Work/keep.txt",
|
||||
"Work.json",
|
||||
]);
|
||||
});
|
||||
|
||||
// One album backed up, then a folder the user made beside it holding a
|
||||
// symlink into originals/, with `json` (if given) as its sibling JSON.
|
||||
const backupWithUserFolder = async (
|
||||
json: string | undefined,
|
||||
): Promise<{ outDir: string; failed: number }> => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf([
|
||||
{
|
||||
collection: collection(10, "Trip"),
|
||||
files: [file(1, 10, "a.jpg")],
|
||||
},
|
||||
]);
|
||||
await runBackup(lib, { downloadDirectory: outDir });
|
||||
const collectionsDir = join(outDir, "collections");
|
||||
mkdirSync(join(collectionsDir, "Mine"));
|
||||
symlinkSync(
|
||||
"../../originals/1.jpg",
|
||||
join(collectionsDir, "Mine", "a.jpg"),
|
||||
);
|
||||
if (json !== undefined) {
|
||||
writeFileSync(join(collectionsDir, "Mine.json"), json);
|
||||
}
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
return { outDir, failed: result.failed };
|
||||
};
|
||||
|
||||
it("leaves a user folder with no JSON beside it as it was", async () => {
|
||||
const { outDir, failed } = await backupWithUserFolder(undefined);
|
||||
|
||||
expect(failed).toBe(0);
|
||||
expect(tree(outDir)).toEqual([
|
||||
"Mine/",
|
||||
"Mine/a.jpg -> ../../originals/1.jpg",
|
||||
"Trip/",
|
||||
"Trip/a.jpg -> ../../originals/1.jpg",
|
||||
"Trip.json",
|
||||
]);
|
||||
});
|
||||
|
||||
it("leaves a user folder whose JSON has no album ID as it was", async () => {
|
||||
const json = '{"name":"Mine"}';
|
||||
const { outDir, failed } = await backupWithUserFolder(json);
|
||||
|
||||
expect(failed).toBe(0);
|
||||
expect(tree(outDir)).toEqual([
|
||||
"Mine/",
|
||||
"Mine/a.jpg -> ../../originals/1.jpg",
|
||||
"Mine.json",
|
||||
"Trip/",
|
||||
"Trip/a.jpg -> ../../originals/1.jpg",
|
||||
"Trip.json",
|
||||
]);
|
||||
expect(
|
||||
readFileSync(join(outDir, "collections", "Mine.json"), "utf-8"),
|
||||
).toBe(json);
|
||||
});
|
||||
});
|
||||
|
||||
+81
-14
@@ -38,7 +38,8 @@ import { loadSession } from "../../src/cli-session.js";
|
||||
import type { Client, ClientSnapshot } from "../../src/client.js";
|
||||
import type { ContentSource } from "../../src/library/content.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
import { init } from "../../src/crypto/index.js";
|
||||
import { init, toBase64 } from "../../src/crypto/index.js";
|
||||
import { defaultCacheDirectory } from "../../src/library/index.js";
|
||||
|
||||
const USER_ID = 42;
|
||||
|
||||
@@ -176,19 +177,6 @@ describe("session file", () => {
|
||||
expect(JSON.parse(readFileSync(path, "utf-8"))).toEqual(snapshot);
|
||||
});
|
||||
|
||||
it("is removed by logout", async () => {
|
||||
const ctx = context();
|
||||
saveSession(ctx.sessionDir, snapshot);
|
||||
expect(await logoutCommand(ctx)).toBe(0);
|
||||
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
|
||||
expect(stderr.text).toBe("Session deleted.\n");
|
||||
});
|
||||
|
||||
it("logout without a session says so and exits 0", async () => {
|
||||
expect(await logoutCommand(context())).toBe(0);
|
||||
expect(stderr.text).toBe("No session found.\n");
|
||||
});
|
||||
|
||||
it("a missing session exits 1 with 'Not logged in'", async () => {
|
||||
const ctx = { ...context(), loadSession };
|
||||
expect(await whoamiCommand(ctx)).toBe(1);
|
||||
@@ -211,6 +199,85 @@ describe("session file", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// These use a real client read from the session file, over a fake API that
|
||||
// records each request and answers with `status`.
|
||||
describe("logout", () => {
|
||||
const snapshot: ClientSnapshot = {
|
||||
email: "cli@example.com",
|
||||
userID: USER_ID,
|
||||
token: "saved-token",
|
||||
masterKey: toBase64(new Uint8Array(32)),
|
||||
secretKey: toBase64(new Uint8Array(32)),
|
||||
publicKey: toBase64(new Uint8Array(32)),
|
||||
};
|
||||
|
||||
const requests: Request[] = [];
|
||||
|
||||
const logoutContext = (status: number): CliContext => ({
|
||||
...context(),
|
||||
loadSession: (path) =>
|
||||
loadSession(path, {
|
||||
fetch: async (url, init) => {
|
||||
requests.push(new Request(url, init));
|
||||
return new Response(JSON.stringify({}), {
|
||||
status,
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
},
|
||||
}),
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
requests.length = 0;
|
||||
});
|
||||
|
||||
it("ends the session on the server, then deletes the file", async () => {
|
||||
const ctx = logoutContext(200);
|
||||
saveSession(ctx.sessionDir, snapshot);
|
||||
expect(await logoutCommand(ctx)).toBe(0);
|
||||
|
||||
expect(requests).toHaveLength(1);
|
||||
expect(requests[0]!.method).toBe("POST");
|
||||
expect(new URL(requests[0]!.url).pathname).toBe("/users/logout");
|
||||
expect(requests[0]!.headers.get("X-Auth-Token")).toBe("saved-token");
|
||||
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
|
||||
expect(stderr.text).toBe(
|
||||
"Session ended on the server.\n" +
|
||||
"Session deleted.\n" +
|
||||
`Cache directory ${ctx.cacheDir} still holds decrypted data; delete it to remove that data.\n`,
|
||||
);
|
||||
});
|
||||
|
||||
it("still deletes the file when the server call fails, and says so", async () => {
|
||||
const ctx = logoutContext(500);
|
||||
saveSession(ctx.sessionDir, snapshot);
|
||||
expect(await logoutCommand(ctx)).toBe(1);
|
||||
|
||||
expect(requests).toHaveLength(1);
|
||||
expect(existsSync(join(ctx.sessionDir, "session.json"))).toBe(false);
|
||||
expect(stderr.text).toBe(
|
||||
"Could not end the session on the server: HTTP 500\n" +
|
||||
"Session deleted.\n" +
|
||||
`Cache directory ${ctx.cacheDir} still holds decrypted data; delete it to remove that data.\n`,
|
||||
);
|
||||
});
|
||||
|
||||
it("names the account's default cache directory without --cache-dir", async () => {
|
||||
const ctx = { ...logoutContext(200), cacheDir: undefined };
|
||||
saveSession(ctx.sessionDir, snapshot);
|
||||
expect(await logoutCommand(ctx)).toBe(0);
|
||||
expect(stderr.text).toContain(
|
||||
`Cache directory ${defaultCacheDirectory(USER_ID)} still holds decrypted data`,
|
||||
);
|
||||
});
|
||||
|
||||
it("without a session says so, calls nothing and exits 0", async () => {
|
||||
expect(await logoutCommand(logoutContext(200))).toBe(0);
|
||||
expect(requests).toHaveLength(0);
|
||||
expect(stderr.text).toBe("No session found.\n");
|
||||
});
|
||||
});
|
||||
|
||||
describe("whoami", () => {
|
||||
it("prints the account as one line of JSON", async () => {
|
||||
expect(await whoamiCommand(context())).toBe(0);
|
||||
|
||||
@@ -1747,4 +1747,42 @@ describe("downloadFile content hash", () => {
|
||||
|
||||
expect(readdirSync(t.dir)).toEqual([]);
|
||||
});
|
||||
|
||||
it("rejects a live photo that is not a readable ZIP and does not retry", async () => {
|
||||
// Bytes 8-9 of a ZIP entry's local header name its compression
|
||||
// method; 99 is one no reader knows, so the entry cannot be read.
|
||||
const zip = livePhotoZip.slice();
|
||||
zip[8] = 99;
|
||||
zip[9] = 0;
|
||||
const t = setup(zip, { fileType: "livePhoto", hash: livePhotoHash });
|
||||
|
||||
await expect(t.run()).rejects.toThrow(
|
||||
/file 999: live photo is not a readable ZIP/,
|
||||
);
|
||||
|
||||
expect(readdirSync(t.dir)).toEqual([]);
|
||||
expect(t.requests()).toBe(1);
|
||||
});
|
||||
|
||||
it("rejects a live photo ZIP with no image entry", async () => {
|
||||
const zip = zipSync({ "video.mov": patternBytes(900, 82) });
|
||||
const t = setup(zip, { fileType: "livePhoto", hash: livePhotoHash });
|
||||
|
||||
await expect(t.run()).rejects.toThrow(
|
||||
/file 999: live photo ZIP does not hold both an image and a video/,
|
||||
);
|
||||
|
||||
expect(readdirSync(t.dir)).toEqual([]);
|
||||
});
|
||||
|
||||
it("rejects a live photo ZIP with no video entry", async () => {
|
||||
const zip = zipSync({ "image.heic": patternBytes(500, 81) });
|
||||
const t = setup(zip, { fileType: "livePhoto", hash: livePhotoHash });
|
||||
|
||||
await expect(t.run()).rejects.toThrow(
|
||||
/file 999: live photo ZIP does not hold both an image and a video/,
|
||||
);
|
||||
|
||||
expect(readdirSync(t.dir)).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -215,6 +215,9 @@ const buildThumbMock = async (opts?: {
|
||||
thumbnail: {
|
||||
decryptionHeader: toBase64(sodium.randombytes_buf(24)),
|
||||
},
|
||||
// The encrypted size of the thumbnail the server records; large
|
||||
// enough here that the default encoding fits.
|
||||
info: { thumbSize: 1_000_000 },
|
||||
updationTime: TEST_TIME,
|
||||
};
|
||||
};
|
||||
@@ -446,6 +449,32 @@ const openLib = (client: Client): Promise<Library> =>
|
||||
precacheOriginals: false,
|
||||
});
|
||||
|
||||
/** The mock's raw record for one file, for a test to change before login. */
|
||||
const rawFile = (m: ThumbMockState, fileID: number): Record<string, unknown> =>
|
||||
m.filesByCollection[1]!.find((f) => f.id === fileID)!;
|
||||
|
||||
/** Replace the original the mock serves for one file. */
|
||||
const replaceOriginal = (
|
||||
m: ThumbMockState,
|
||||
fileID: number,
|
||||
body: Uint8Array,
|
||||
): void => {
|
||||
const push = sodium.crypto_secretstream_xchacha20poly1305_init_push(
|
||||
m.fileKeys[fileID]!,
|
||||
);
|
||||
m.fileCiphertexts[fileID] =
|
||||
sodium.crypto_secretstream_xchacha20poly1305_push(
|
||||
push.state,
|
||||
body,
|
||||
null,
|
||||
sodium.crypto_secretstream_xchacha20poly1305_TAG_FINAL,
|
||||
);
|
||||
rawFile(m, fileID).file = { decryptionHeader: toBase64(push.header) };
|
||||
};
|
||||
|
||||
const isOriginalDownload = (url: string): boolean =>
|
||||
url.includes("files.ente.io") || url.includes("/files/download/");
|
||||
|
||||
const login = (fetch: typeof globalThis.fetch, retry?: RetryOptions) =>
|
||||
Client.login({
|
||||
email: TEST_EMAIL,
|
||||
@@ -581,6 +610,33 @@ describe("listMissingThumbnails", () => {
|
||||
// Should still be 2, not 4 (each file checked only once)
|
||||
expect(missing.length).toBe(2);
|
||||
});
|
||||
|
||||
it("skips a file another account owns without fetching its thumbnail", async () => {
|
||||
const otherMock = await buildThumbMock();
|
||||
rawFile(otherMock, 102).ownerID = 7;
|
||||
const logs: string[] = [];
|
||||
const counted = countingFetch(
|
||||
buildThumbFetch(otherMock),
|
||||
(url) => url.includes("thumbnails.ente.io") && url.includes("102"),
|
||||
);
|
||||
const client = await login(counted.fetch);
|
||||
const lib = await openLib(client);
|
||||
|
||||
const missing = await listMissingThumbnails(lib, client, (msg) =>
|
||||
logs.push(msg),
|
||||
);
|
||||
lib.close();
|
||||
|
||||
expect(missing.map((m) => m.fileID)).toEqual([101]);
|
||||
expect(counted.matched()).toBe(0);
|
||||
expect(
|
||||
logs.some(
|
||||
(l) =>
|
||||
l.includes("Skipping file-102.jpg") &&
|
||||
l.includes("another account"),
|
||||
),
|
||||
).toBe(true);
|
||||
});
|
||||
});
|
||||
|
||||
describe("fixMissingThumbnails", () => {
|
||||
@@ -688,6 +744,93 @@ describe("fixMissingThumbnails", () => {
|
||||
expect(fixMock.uploadedThumbnails.length).toBe(1);
|
||||
expect(fixMock.uploadedThumbnails[0]!.fileID).toBe(101);
|
||||
});
|
||||
|
||||
it("skips a file another account owns without downloading it", async () => {
|
||||
// The server accepts a thumbnail only from the file's owner.
|
||||
const fixMock = await buildThumbMock();
|
||||
rawFile(fixMock, 101).ownerID = 7;
|
||||
const counted = countingFetch(
|
||||
buildThumbFetch(fixMock),
|
||||
isOriginalDownload,
|
||||
);
|
||||
const client = await login(counted.fetch);
|
||||
const lib = await openLib(client);
|
||||
|
||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
||||
lib.close();
|
||||
|
||||
expect(results[0]!.status).toBe("skipped");
|
||||
expect(results[0]!.reason).toContain("another account");
|
||||
expect(counted.matched()).toBe(0);
|
||||
expect(fixMock.uploadedThumbnails.length).toBe(0);
|
||||
});
|
||||
|
||||
it("skips a file whose recorded thumbnail size is 0 without downloading it", async () => {
|
||||
// The server refuses a thumbnail larger than the one it records, and
|
||||
// no thumbnail is 0 bytes.
|
||||
const fixMock = await buildThumbMock();
|
||||
rawFile(fixMock, 101).info = { thumbSize: 0 };
|
||||
const counted = countingFetch(
|
||||
buildThumbFetch(fixMock),
|
||||
isOriginalDownload,
|
||||
);
|
||||
const client = await login(counted.fetch);
|
||||
const lib = await openLib(client);
|
||||
|
||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
||||
lib.close();
|
||||
|
||||
expect(results[0]!.status).toBe("skipped");
|
||||
expect(results[0]!.reason).toContain("recorded thumbnail size is 0");
|
||||
expect(counted.matched()).toBe(0);
|
||||
expect(fixMock.uploadedThumbnails.length).toBe(0);
|
||||
});
|
||||
|
||||
it("re-encodes smaller until the thumbnail fits the recorded size", async () => {
|
||||
// A noisy 400x300 JPEG, which the default encoding (quality 50, not
|
||||
// resized because it is under 720 px) cannot compress below the size
|
||||
// recorded here: one byte less than that encoding's ciphertext.
|
||||
const fixMock = await buildThumbMock();
|
||||
const w = 400;
|
||||
const h = 300;
|
||||
const noisy = new Uint8Array(
|
||||
jpegJs.encode(
|
||||
{
|
||||
data: sodium.randombytes_buf(w * h * 4),
|
||||
width: w,
|
||||
height: h,
|
||||
},
|
||||
90,
|
||||
).data,
|
||||
);
|
||||
replaceOriginal(fixMock, 101, noisy);
|
||||
const decoded = jpegJs.decode(noisy, {
|
||||
useTArray: true,
|
||||
formatAsRGBA: true,
|
||||
});
|
||||
const defaultSize =
|
||||
jpegJs.encode(decoded, 50).data.length +
|
||||
sodium.crypto_secretstream_xchacha20poly1305_ABYTES;
|
||||
const recordedSize = defaultSize - 1;
|
||||
rawFile(fixMock, 101).info = { thumbSize: recordedSize };
|
||||
|
||||
const client = await login(buildThumbFetch(fixMock));
|
||||
const lib = await openLib(client);
|
||||
|
||||
const results = await fixMissingThumbnails(lib, client, [101]);
|
||||
lib.close();
|
||||
|
||||
expect(results[0]!.status).toBe("fixed");
|
||||
const upload = fixMock.uploadedThumbnails[0]!;
|
||||
expect(upload.ciphertext.length).toBeLessThanOrEqual(recordedSize);
|
||||
const decrypted = decryptBlob(
|
||||
upload.ciphertext,
|
||||
fromBase64(upload.decryptionHeader),
|
||||
fixMock.fileKeys[101]!,
|
||||
);
|
||||
expect(decrypted[0]).toBe(0xff);
|
||||
expect(decrypted[1]).toBe(0xd8);
|
||||
});
|
||||
});
|
||||
|
||||
describe("Client.getApiClient", () => {
|
||||
|
||||
Reference in New Issue
Block a user