Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e03e99b486 | ||
|
|
fd40714a6d | ||
|
|
982b878d69 |
@@ -661,7 +661,15 @@ library. The read commands — `collections`, `files`, `get`, `get-thumb`,
|
||||
`backup-metadata`, `helper list-missing-thumbnails` and
|
||||
`helper fix-missing-thumbnails` — force a fresh server round-trip before they
|
||||
answer, so they report current account state rather than whatever the cache last
|
||||
held. If that round-trip fails, the command prints the error on one line and
|
||||
held. `collections` asks the server for the collection list alone, in one
|
||||
request, and neither fetches any collection's files nor changes the cache. The
|
||||
others wait for a refresh. With an empty cache, as on a first run, that refresh
|
||||
fetches the files of every collection, one collection after another, and stores
|
||||
nothing until it has them all, which on a large account takes a while. Meanwhile
|
||||
the command prints a line on stderr for each collection whose files it has
|
||||
fetched, such as `Fetched the files of 3 of 120 collections`; with `--json`,
|
||||
`files` and the two helpers print no such line. stdout holds only the command's
|
||||
output. If that round-trip fails, the command prints the error on one line and
|
||||
exits 1, or 3 when the server no longer accepts the saved session (see "Session
|
||||
handling"). An album or file quak cannot decrypt does not fail it; the command
|
||||
answers from what did decrypt (see "Default reads vs. fresh reads").
|
||||
@@ -679,24 +687,26 @@ Without `--out`, `get` writes the file to the working directory under its title,
|
||||
and `get-thumb` writes the thumbnail there under `thumb_` followed by the title,
|
||||
each with unsafe characters replaced. A name over 255 bytes, which most file
|
||||
systems refuse, is cut short between two characters before its extension, so the
|
||||
extension is kept whenever it fits.
|
||||
extension is kept whenever it fits. An extension too long to leave room for the
|
||||
name's first character is cut short too, so the name never starts with a dot.
|
||||
|
||||
`get` and `get-thumb` resolve the file by ID directly, so `--collection` is
|
||||
accepted for backward compatibility but ignored. For a live photo, `get` writes
|
||||
its image and its video, each named after the title with its own extension, as
|
||||
Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`). With
|
||||
`--out PATH`, the image is written to `PATH` and the video beside it, with
|
||||
`PATH`'s name and the video's extension; a `PATH` with the video's extension is
|
||||
refused. `backup-metadata --exif` (alias `--all`) additionally fetches each
|
||||
file's original through the cache and records, from it or a live photo's image,
|
||||
its XMP metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is
|
||||
read with [exifreader](https://github.com/mattiasw/ExifReader) from any image
|
||||
format it reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's
|
||||
`exif` field is exifreader's EXIF tag output: each tag by name, with its
|
||||
`value`, `description` and `computed` value. An EXIF block exifreader finds but
|
||||
reads no tag from is recorded, base64, as `exifRaw`, with the reason in
|
||||
`exifError`. `collections`, `files`, `backup`, `helper list-missing-thumbnails`
|
||||
and `helper fix-missing-thumbnails` take `--json` for machine-readable output.
|
||||
Ente's clients name them (`IMG_0001.heic` and `IMG_0001.mov`), and each name is
|
||||
cut short as above once its own extension is in it. With `--out PATH`, the image
|
||||
is written to `PATH` and the video beside it, with `PATH`'s name and the video's
|
||||
extension; a `PATH` with the video's extension is refused.
|
||||
`backup-metadata --exif` (alias `--all`) additionally fetches each file's
|
||||
original through the cache and records, from it or a live photo's image, its XMP
|
||||
metadata, its EXIF metadata and, for a JPEG, its dimensions. EXIF is read with
|
||||
[exifreader](https://github.com/mattiasw/ExifReader) from any image format it
|
||||
reads, JPEG, HEIC/HEIF, AVIF, PNG and WebP among them. The record's `exif` field
|
||||
is exifreader's EXIF tag output: each tag by name, with its `value`,
|
||||
`description` and `computed` value. An EXIF block exifreader finds but reads no
|
||||
tag from is recorded, base64, as `exifRaw`, with the reason in `exifError`.
|
||||
`collections`, `files`, `backup`, `helper list-missing-thumbnails` and
|
||||
`helper fix-missing-thumbnails` take `--json` for machine-readable output.
|
||||
|
||||
`backup --verify` also hashes the originals already in the backup and replaces
|
||||
any that do not match the content hash Ente records; one it cannot replace stays
|
||||
@@ -856,11 +866,12 @@ gets its ID added as well: beside an album `x`, whose JSON is `x.json`, an album
|
||||
named `x.json` gets the directory `x.json (12)/`. A name too long for a file
|
||||
name, which most file systems limit to 255 bytes, gets its ID added too. A name
|
||||
with its ID added is cut short between two characters in the part before the ID,
|
||||
just far enough that it fits. When a symlink's extension alone leaves no room
|
||||
for the ID, the extension is cut short and the part before the ID keeps only the
|
||||
bytes that cut leaves over. A collection's directory name also leaves room for
|
||||
the `.json` its JSON adds. IDs never change, so a name stays the same from run
|
||||
to run until such a clash appears or goes away.
|
||||
just far enough that it fits. When a symlink's extension leaves no room for the
|
||||
ID and the first character of the name, the extension is cut short, and the part
|
||||
before the ID keeps that character and any bytes the cut leaves over. A
|
||||
collection's directory name also leaves room for the `.json` its JSON adds. IDs
|
||||
never change, so a name stays the same from run to run until such a clash
|
||||
appears or goes away.
|
||||
|
||||
A collection whose directory or JSON cannot be written, such as one whose
|
||||
directory name is taken by a file you put in `collections/`, does not stop the
|
||||
@@ -974,7 +985,11 @@ loop, and resolves to a `Library`. On an empty cache it awaits the first
|
||||
refresh, so it opens onto the account's data whenever the server is reachable;
|
||||
if that refresh fails, it opens with no data and records the error in
|
||||
`lib.status()`. On an existing cache it returns immediately and refreshes in the
|
||||
background, so an unreachable server does not block opening.
|
||||
background, so an unreachable server does not block opening. With
|
||||
`refreshOnOpen: false` it starts no refresh and returns at once onto what the
|
||||
cache holds, empty or not; a refresh then starts only when `fresh()` or
|
||||
`lib.backup()` asks, or the background timer fires. The CLI's read commands open
|
||||
it so (see "CLI surface").
|
||||
|
||||
`LibraryOptions`:
|
||||
|
||||
@@ -984,6 +999,7 @@ background, so an unreachable server does not block opening.
|
||||
| `cacheDirectory` | `quak/<userID>` under the per-user cache directory | where `metadata.json` and the content cache live |
|
||||
| `downloadDirectory` | `photos` in the working directory | root of the save paths; an original stored there counts as cached |
|
||||
| `refreshIntervalSeconds` | `3` | background refresh cadence; `Infinity` for no background timer |
|
||||
| `refreshOnOpen` | `true` | start a refresh as the library opens |
|
||||
| `precacheThumbnails` | `true` | prefetch every thumbnail, newest first |
|
||||
| `precacheOriginals` | `true` | prefetch the favorites album and the latest-window originals |
|
||||
| `precacheOriginalsDays` | `7` | length in days of that latest window |
|
||||
@@ -997,6 +1013,14 @@ background, so an unreachable server does not block opening.
|
||||
The default `downloadDirectory` is resolved against the working directory once,
|
||||
when the library opens; `lib.downloadDirectory` holds the result.
|
||||
|
||||
`onProgress` receives a `RefreshEvent` with status `"started"` as a refresh, an
|
||||
ML data fetch or a precache sweep starts, and `"done"` or `"failed"` as it ends.
|
||||
In between, a refresh sends `"progress"` after each collection whose files it
|
||||
fetched, with `fetched`, how many collections it has fetched the files of so
|
||||
far, and `total`, how many it will. A refresh fetches the files of each
|
||||
collection that is new or changed since the cache's copy, so on an empty cache
|
||||
it fetches those of every collection.
|
||||
|
||||
Concurrency is set through `pools`: construct
|
||||
`new RequestPools({ metadataConcurrency, contentConcurrency, thumbnailConcurrency })`
|
||||
and pass it. The three pools default to 10 / 5 / 25 (see Request pools below).
|
||||
@@ -1036,6 +1060,15 @@ rejects the caller (default reads stay silent and keep serving the last good
|
||||
copy). The CLI's read commands use fresh reads (issue
|
||||
https://git.eeqj.de/sneak/quak/issues/75).
|
||||
|
||||
`await lib.freshCollections()` is the fresh read of the collection list alone:
|
||||
one collections request, answered with every live collection that decrypts, as
|
||||
the `Collection` objects `listCollections()` returns, in the server's order. It
|
||||
fetches no collection's files and stores nothing, not even the list: a refresh
|
||||
fetches a collection's files only when the collection is new or has changed
|
||||
since the stored copy, so a collection stored without its files would never get
|
||||
them. `quak collections` uses it (issue
|
||||
https://git.eeqj.de/sneak/quak/issues/210).
|
||||
|
||||
An album or file in the server's answer that quak cannot decrypt, or whose
|
||||
metadata is not JSON, does not fail the refresh. The refresh leaves it out, an
|
||||
album with all its files, and drops any older copy the cache held, so reads
|
||||
|
||||
@@ -17,9 +17,11 @@ pre-1.0
|
||||
|
||||
# Next Step
|
||||
|
||||
No bug is open. The check of quak against a live Ente account,
|
||||
https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak has, so
|
||||
it waits on him.
|
||||
No bug is open. Two changes sneak asked for are open:
|
||||
https://git.eeqj.de/sneak/quak/issues/211 and
|
||||
https://git.eeqj.de/sneak/quak/issues/212. The check of quak against a live Ente
|
||||
account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
|
||||
has, so it waits on him.
|
||||
|
||||
Tagging and releases are decided by sneak alone, and happen only when he
|
||||
declares one.
|
||||
@@ -27,16 +29,31 @@ declares one.
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-18: Names quak builds from an album name or a title fit in a file name
|
||||
(issue 205). `shortenKeepingExtension` in `src/filename.ts` cuts a name short
|
||||
to a number of bytes, keeping its extension when it fits, and can put text
|
||||
right before the extension; the backup's `withID` uses it for an ID. The
|
||||
default names of `quak get`, `quak get-thumb`, `downloadFile` and
|
||||
`downloadThumbnail` are cut short with it to 255 bytes, `thumb_` included.
|
||||
`backup-metadata` cuts an album's name short so that its folder,
|
||||
`<id>-<name>`, fits in 255 bytes. `safeExtension` gives `.bin` for an
|
||||
(https://git.eeqj.de/sneak/quak/issues/205). `shortenKeepingExtension` in
|
||||
`src/filename.ts` cuts a name short to a number of bytes, keeping its
|
||||
extension when it fits and always its first character, and can put text right
|
||||
before the extension; the backup's `withID` uses it for an ID.
|
||||
`defaultFileName` there gives the name a file is saved under when the caller
|
||||
gives no path: a prefix such as `thumb_` and the title with unsafe characters
|
||||
replaced, cut short with it to 255 bytes, and, for a live photo's image or
|
||||
video, with that part's own extension before the cut. `quak get`,
|
||||
`quak get-thumb`, `downloadFile` and `downloadThumbnail` use it without
|
||||
`--out` or `outPath`. `backup-metadata` cuts an album's name short so that its
|
||||
folder, `<id>-<name>`, fits in 255 bytes. `safeExtension` gives `.bin` for an
|
||||
extension of more than 16 letters and digits, so a save path's name always
|
||||
fits.
|
||||
|
||||
- 2026-10-18: `quak collections` no longer waits for the files of every
|
||||
collection (issue 210). On an empty cache it opened its library with a refresh
|
||||
that fetched and decrypted the files of each collection in turn, printing
|
||||
nothing and storing nothing until all were done, so on a large account it
|
||||
seemed to hang. It now reads the collection list alone through the new
|
||||
`lib.freshCollections()`, one request that stores nothing. The read commands
|
||||
open their library with the new `refreshOnOpen: false`, so each makes only the
|
||||
refresh it asks for. A refresh sends a `"progress"` event after each
|
||||
collection whose files it fetched, and the commands that wait on one print it
|
||||
on stderr, except with `--json`.
|
||||
|
||||
- 2026-10-16: ML data the server sends that does not decrypt, gunzip or parse is
|
||||
no longer taken for no ML data (issue 198). `fetchMLDataBatch` returns why for
|
||||
each such file beside the payloads; ML data that parses to something other
|
||||
|
||||
+4
-3
@@ -290,9 +290,10 @@ const rebuildSymlink = (linkPath: string, target: string): void => {
|
||||
// case (two names that differ only in case are one entry on a case-insensitive
|
||||
// file system), or it is too long for a file name; then it gets ` (<id>)`,
|
||||
// right before the extension when `beforeExtension` is set. To fit, the part
|
||||
// before the ID is cut short as far as it must be. When the extension alone
|
||||
// leaves no room for the ID, the extension is cut short and the part before
|
||||
// the ID keeps only the bytes that cut leaves over. With `withJSON`, each
|
||||
// before the ID is cut short as far as it must be. When the extension leaves
|
||||
// no room for the ID and the first character of the name, the extension is
|
||||
// cut short, and the part before the ID keeps that character and any bytes
|
||||
// the cut leaves over (see `shortenKeepingExtension`). With `withJSON`, each
|
||||
// entry also has the JSON file `<name>.json` beside it, as an album does: a
|
||||
// name then leaves room for `.json`, and one that is another entry's JSON name
|
||||
// gets its ID added too.
|
||||
|
||||
+33
-14
@@ -125,15 +125,29 @@ const readLibraryClient = (client: Client): LibraryClient => ({
|
||||
|
||||
// Open a library for a single point command: the aggressive background precache
|
||||
// (issue #48) is off — a one-shot `collections` or `get` must not start
|
||||
// downloading the whole account — and the refresh interval is long so no second
|
||||
// refresh fires mid-command.
|
||||
const openReadLibrary = (ctx: CliContext, client: Client): Promise<Library> =>
|
||||
// downloading the whole account — and no refresh starts as it opens or on a
|
||||
// timer: each command asks for the one it needs, and `collections` needs none
|
||||
// (issue #210). While a refresh runs, each collection whose files it has
|
||||
// fetched is counted on stderr, unless `json` is set, so a large account shows
|
||||
// progress rather than nothing.
|
||||
const openReadLibrary = (
|
||||
ctx: CliContext,
|
||||
client: Client,
|
||||
opts: { json?: boolean } = {},
|
||||
): Promise<Library> =>
|
||||
Library.open({
|
||||
client: readLibraryClient(client),
|
||||
cacheDirectory: ctx.cacheDir,
|
||||
refreshIntervalSeconds: 3600,
|
||||
refreshOnOpen: false,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
precacheThumbnails: false,
|
||||
precacheOriginals: false,
|
||||
onProgress: (event) => {
|
||||
if (opts.json || event.status !== "progress") return;
|
||||
ctx.stderr.write(
|
||||
`Fetched the files of ${event.fetched} of ${event.total} collections\n`,
|
||||
);
|
||||
},
|
||||
});
|
||||
|
||||
export const loginCommand = async (ctx: CliContext): Promise<number> => {
|
||||
@@ -221,9 +235,10 @@ export const collectionsCommand = async (
|
||||
if (!client) return 3;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
try {
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52): the pre-library CLI printed current state in
|
||||
// this order, not the albums projection's newest-first order.
|
||||
// Ask the server for the collection list alone and list it in its
|
||||
// order (issue #36 amendment, issues #52 and #210): the pre-library
|
||||
// CLI printed current state in this order, not the albums
|
||||
// projection's newest-first order.
|
||||
const collections = await freshCollections(lib);
|
||||
|
||||
if (opts.json) {
|
||||
@@ -264,7 +279,7 @@ export const filesCommand = async (
|
||||
const collectionID = parseID(ctx, "collection", opts.collection);
|
||||
if (collectionID === undefined) return 1;
|
||||
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
const lib = await openReadLibrary(ctx, client, opts);
|
||||
try {
|
||||
// Force a server round-trip and list in enumeration order (issue #36
|
||||
// amendment, issue #52). Each file prints from its own decrypted
|
||||
@@ -317,8 +332,8 @@ export const getCommand = async (
|
||||
const result = await photo.original();
|
||||
// Default name is the file's own title, as the pre-library CLI used
|
||||
// (not the editedName-preferring projection title) (issue #52).
|
||||
const outPath = opts.out ?? originalName(file);
|
||||
if (result.videoPath === undefined) {
|
||||
const outPath = opts.out ?? originalName(file);
|
||||
copyFileSync(result.path, outPath);
|
||||
ctx.stderr.write(`${result.bytes} bytes -> ${outPath}\n`);
|
||||
return 0;
|
||||
@@ -326,9 +341,13 @@ export const getCommand = async (
|
||||
// A live photo is written as its image and its video, each named after
|
||||
// the title with its own extension, as Ente's clients name them. With
|
||||
// --out, the image goes there and the video beside it.
|
||||
const imageOut =
|
||||
opts.out ?? withExtension(outPath, extname(result.path));
|
||||
const videoOut = withExtension(outPath, extname(result.videoPath));
|
||||
const imageExt = extname(result.path);
|
||||
const videoExt = extname(result.videoPath);
|
||||
const imageOut = opts.out ?? originalName(file, imageExt);
|
||||
const videoOut =
|
||||
opts.out === undefined
|
||||
? originalName(file, videoExt)
|
||||
: withExtension(opts.out, videoExt);
|
||||
if (imageOut.toLowerCase() === videoOut.toLowerCase()) {
|
||||
ctx.stderr.write(
|
||||
`File ${fileID} is a live photo, and its video would also be written to ${imageOut}\n`,
|
||||
@@ -490,7 +509,7 @@ export const listMissingThumbnailsCommand = async (
|
||||
await init();
|
||||
const client = requireSession(ctx);
|
||||
if (!client) return 3;
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
const lib = await openReadLibrary(ctx, client, opts);
|
||||
try {
|
||||
// Refresh first so files added since the cache was written are
|
||||
// checked; a failed refresh throws.
|
||||
@@ -535,7 +554,7 @@ export const fixMissingThumbnailsCommand = async (
|
||||
if (fileID === undefined) return 1;
|
||||
fileIDs.push(fileID);
|
||||
}
|
||||
const lib = await openReadLibrary(ctx, client);
|
||||
const lib = await openReadLibrary(ctx, client, opts);
|
||||
try {
|
||||
// Refresh first so files added since the cache was written are found;
|
||||
// a failed refresh throws.
|
||||
|
||||
+8
-19
@@ -9,11 +9,7 @@
|
||||
// `metadata.title`, and issue #52 requires that output stay byte-identical, so
|
||||
// the commands shape their output from the raw `EnteFile` through here.
|
||||
|
||||
import {
|
||||
MAX_NAME_BYTES,
|
||||
sanitizeFileName,
|
||||
shortenKeepingExtension,
|
||||
} from "./filename.js";
|
||||
import { defaultFileName } from "./filename.js";
|
||||
import type { EnteFile, FileType, Microseconds } from "./model/types.js";
|
||||
|
||||
// One row of `quak files --json`.
|
||||
@@ -37,20 +33,13 @@ export const fileListRow = (file: EnteFile): FileListRow => ({
|
||||
export const fileListLine = (file: EnteFile): string =>
|
||||
`${file.id}\t${file.metadata.fileType}\t${file.metadata.title}`;
|
||||
|
||||
// Default output path for `quak get` when `--out` is not given. The title comes
|
||||
// from the server, so it is sanitized, and cut short to fit in a file name,
|
||||
// keeping its extension; `--out` is the user's and is used as is.
|
||||
export const originalName = (file: EnteFile): string =>
|
||||
shortenKeepingExtension(
|
||||
sanitizeFileName(file.metadata.title, `file-${file.id}`),
|
||||
MAX_NAME_BYTES,
|
||||
);
|
||||
// Default output path for `quak get` when `--out` is not given, made from the
|
||||
// title (see `defaultFileName`), with `ext` in place of its extension for a
|
||||
// live photo's image or video; `--out` is the user's and is used as is.
|
||||
export const originalName = (file: EnteFile, ext?: string): string =>
|
||||
defaultFileName(file.metadata.title, file.id, { ext });
|
||||
|
||||
// Default output path for `quak get-thumb` when `--out` is not given: `thumb_`
|
||||
// and the title, cut short so that the two fit in a file name.
|
||||
// and the title.
|
||||
export const thumbnailName = (file: EnteFile): string =>
|
||||
"thumb_" +
|
||||
shortenKeepingExtension(
|
||||
sanitizeFileName(file.metadata.title, `file-${file.id}`),
|
||||
MAX_NAME_BYTES - "thumb_".length,
|
||||
);
|
||||
defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
|
||||
|
||||
+16
-13
@@ -2,12 +2,16 @@
|
||||
//
|
||||
// `collections`, `files --collection`, `get`, and `get-thumb` must answer for
|
||||
// the account's state at the moment the command runs, not for whatever the
|
||||
// local cache last happened to hold (owner amendment, issue #36). Each helper
|
||||
// therefore forces a server round-trip through `Library.fresh()` and only then
|
||||
// reads — so a collection, file, or metadata change made elsewhere is visible.
|
||||
// local cache last happened to hold (owner amendment, issue #36). `files`,
|
||||
// `get` and `get-thumb` therefore force a server round-trip through
|
||||
// `Library.fresh()` and only then read — so a collection, file, or metadata
|
||||
// change made elsewhere is visible. `collections` asks the library for the
|
||||
// collection list alone through `Library.freshCollections()` (issue #210): a
|
||||
// refresh would also fetch the files of every collection, which on a large
|
||||
// account takes long and which `collections` does not print.
|
||||
//
|
||||
// `collections` and `files` also list in the library's own enumeration order —
|
||||
// `listCollections()`/`listFiles()`, the order the pre-library CLI printed —
|
||||
// `collections` lists in the server's order and `files` in the library's own
|
||||
// enumeration order (`listFiles()`), the orders the pre-library CLI printed,
|
||||
// rather than the `albums`/`photos` projection's newest-first order, which
|
||||
// re-sorts the rows. The field values still come from each record's raw
|
||||
// metadata via `cli-output.ts`.
|
||||
@@ -17,23 +21,22 @@ import type { Photo, PhotosAPI } from "./library/index.js";
|
||||
|
||||
// The slice of `Library` these helpers read. `Library` satisfies it
|
||||
// structurally; a test can drive them with a stand-in that records the
|
||||
// `fresh()` call and serves records in a known enumeration order.
|
||||
// `fresh()` and `freshCollections()` calls and serves records in a known
|
||||
// enumeration order.
|
||||
export interface FreshReadLibrary {
|
||||
fresh(): Promise<unknown>;
|
||||
listCollections(): Collection[];
|
||||
freshCollections(): Promise<Collection[]>;
|
||||
getCollection(id: number): Collection | undefined;
|
||||
listFiles(collectionID: number): EnteFile[];
|
||||
getFileByID(fileID: number): EnteFile | undefined;
|
||||
photos: Pick<PhotosAPI, "byID">;
|
||||
}
|
||||
|
||||
// Every live collection, current as of a forced refresh, in enumeration order.
|
||||
export const freshCollections = async (
|
||||
// Every live collection, current as of one collections request, in the
|
||||
// server's order.
|
||||
export const freshCollections = (
|
||||
lib: FreshReadLibrary,
|
||||
): Promise<Collection[]> => {
|
||||
await lib.fresh();
|
||||
return lib.listCollections();
|
||||
};
|
||||
): Promise<Collection[]> => lib.freshCollections();
|
||||
|
||||
// The files of one collection, current as of a forced refresh, in enumeration
|
||||
// order. `undefined` (not an empty list) when the collection does not exist, so
|
||||
|
||||
+27
-26
@@ -16,13 +16,7 @@ import {
|
||||
streamTagFinal,
|
||||
} from "../crypto/index.js";
|
||||
import { TruncatedStreamError } from "../errors.js";
|
||||
import {
|
||||
MAX_NAME_BYTES,
|
||||
safeExtension,
|
||||
sanitizeFileName,
|
||||
shortenKeepingExtension,
|
||||
withExtension,
|
||||
} from "../filename.js";
|
||||
import { defaultFileName, safeExtension, withExtension } from "../filename.js";
|
||||
import { withRetry } from "../retry.js";
|
||||
import type { ApiClient } from "../api/client.js";
|
||||
import type { EnteFile } from "../model/types.js";
|
||||
@@ -371,10 +365,14 @@ const openPart = async (
|
||||
// name starts with `image` as the image and the one whose name starts with
|
||||
// `video` as the video, and refuses a ZIP holding a second of either. It is
|
||||
// written unpacked: each part is named `destination` with the extension
|
||||
// replaced by its own entry's, and the two must differ ignoring case. When the
|
||||
// file records a hash, `<imageHash>:<videoHash>` must match it, each over that
|
||||
// part's own bytes. Only then are the image, then the video, renamed into
|
||||
// place; on any failure neither is stored.
|
||||
// replaced by its own entry's, and the two must differ ignoring case. When
|
||||
// `defaultName` is set, `destination` is the name `defaultFileName` made from
|
||||
// the title rather than a path the caller gave, and each part is named as
|
||||
// `defaultFileName` names it with its own extension, so that the name is cut
|
||||
// short to fit with that extension. When the file records a hash,
|
||||
// `<imageHash>:<videoHash>` must match it, each over that part's own bytes.
|
||||
// Only then are the image, then the video, renamed into place; on any failure
|
||||
// neither is stored.
|
||||
//
|
||||
// The ZIP is chosen by its uploader and may expand enormously, so each part is
|
||||
// written as it decompresses and never held, and the ZIP is refused once the
|
||||
@@ -394,6 +392,7 @@ const decryptLivePhoto = async (
|
||||
key: Uint8Array,
|
||||
onProgress: ProgressCallback | undefined,
|
||||
file: EnteFile,
|
||||
defaultName: boolean,
|
||||
): Promise<DownloadResult> => {
|
||||
const sliceSize = 4096;
|
||||
const dir = dirname(destination);
|
||||
@@ -492,8 +491,12 @@ const decryptLivePhoto = async (
|
||||
`live photo's image and video have the same extension, ${video.ext}`,
|
||||
);
|
||||
}
|
||||
const path = withExtension(destination, image.ext);
|
||||
const videoPath = withExtension(destination, video.ext);
|
||||
const partPath = (ext: string): string =>
|
||||
defaultName
|
||||
? defaultFileName(file.metadata.title, file.id, { ext })
|
||||
: withExtension(destination, ext);
|
||||
const path = partPath(image.ext);
|
||||
const videoPath = partPath(video.ext);
|
||||
for (const part of parts) {
|
||||
await part.handle.sync();
|
||||
await part.handle.close();
|
||||
@@ -537,6 +540,9 @@ const decryptLivePhoto = async (
|
||||
// and then removes its own temp file, and only the attempt that reaches
|
||||
// TAG_FINAL renames one into place, so a download that needed three tries still
|
||||
// performs exactly one rename over the destination.
|
||||
//
|
||||
// `defaultName` says `destination` is the name `defaultFileName` made, not a
|
||||
// path the caller gave (see `decryptLivePhoto`).
|
||||
const fetchAndDecrypt = async (
|
||||
api: ApiClient,
|
||||
openStream: () => Promise<ReadableStream<Uint8Array>>,
|
||||
@@ -545,6 +551,7 @@ const fetchAndDecrypt = async (
|
||||
destination: string,
|
||||
onProgress?: ProgressCallback,
|
||||
original?: EnteFile,
|
||||
defaultName = false,
|
||||
): Promise<DownloadResult> =>
|
||||
withRetry(async () => {
|
||||
const stream = await openStream();
|
||||
@@ -557,6 +564,7 @@ const fetchAndDecrypt = async (
|
||||
key,
|
||||
onProgress,
|
||||
original,
|
||||
defaultName,
|
||||
);
|
||||
}
|
||||
const bytesWritten = await decryptToTemp(
|
||||
@@ -588,15 +596,11 @@ export const downloadFile = async (
|
||||
outPath?: string,
|
||||
onProgress?: ProgressCallback,
|
||||
): Promise<DownloadResult> => {
|
||||
// `outPath` is the caller's and is used as is; the title is the server's
|
||||
// and is sanitized so it can only name a file in the current directory,
|
||||
// and cut short to fit in a file name, keeping its extension.
|
||||
// `outPath` is the caller's and is used as is. Without it, the file is
|
||||
// written in the current directory under the name `defaultFileName` makes
|
||||
// from the title, which is the server's.
|
||||
const resolvedPath =
|
||||
outPath ??
|
||||
shortenKeepingExtension(
|
||||
sanitizeFileName(file.metadata.title, `file-${file.id}`),
|
||||
MAX_NAME_BYTES,
|
||||
);
|
||||
outPath ?? defaultFileName(file.metadata.title, file.id);
|
||||
const header = fromBase64(file.file.decryptionHeader);
|
||||
return fetchAndDecrypt(
|
||||
api,
|
||||
@@ -606,6 +610,7 @@ export const downloadFile = async (
|
||||
resolvedPath,
|
||||
onProgress,
|
||||
file,
|
||||
outPath === undefined,
|
||||
);
|
||||
};
|
||||
|
||||
@@ -618,11 +623,7 @@ export const downloadThumbnail = async (
|
||||
// As in `downloadFile`, with `thumb_` before the title.
|
||||
const resolvedPath =
|
||||
outPath ??
|
||||
"thumb_" +
|
||||
shortenKeepingExtension(
|
||||
sanitizeFileName(file.metadata.title, `file-${file.id}`),
|
||||
MAX_NAME_BYTES - "thumb_".length,
|
||||
);
|
||||
defaultFileName(file.metadata.title, file.id, { prefix: "thumb_" });
|
||||
const header = fromBase64(file.thumbnail.decryptionHeader);
|
||||
return fetchAndDecrypt(
|
||||
api,
|
||||
|
||||
+38
-5
@@ -6,6 +6,8 @@
|
||||
// directory the caller chose: never a path, never `..`, never hidden, never a
|
||||
// Windows device name. Such a name can still be too long for a file name, and
|
||||
// `shortenToBytes` or `shortenKeepingExtension` cuts one short.
|
||||
// `defaultFileName` does both for the name a file is saved under when the
|
||||
// caller gives no path.
|
||||
//
|
||||
// A path the user typed (`--out`, `outPath`) is not passed through here: the
|
||||
// caller is trusted, the server is not.
|
||||
@@ -61,10 +63,12 @@ export const shortenToBytes = (name: string, maxBytes: number): string => {
|
||||
// `name` cut short to fit in `maxBytes` bytes with `insert` put right before
|
||||
// its extension (`IMG_0001 (12).jpg`), each cut made as `shortenToBytes`
|
||||
// makes it. The part before the extension is cut first, so the extension is
|
||||
// kept whenever it fits. When the extension alone leaves no room for
|
||||
// `insert`, the extension is cut short too, and the part before it keeps only
|
||||
// the bytes that cut leaves over. `insert` is never cut. With no `insert`, a
|
||||
// name that fits comes back unchanged.
|
||||
// kept whenever it fits, but that part always keeps its first character:
|
||||
// when the extension leaves no room for `insert` and that character, the
|
||||
// extension is cut short too, and the part before it keeps the bytes that cut
|
||||
// leaves over. So a name that does not start with a dot never comes back
|
||||
// starting with one, which would hide it. `insert` is never cut. With no
|
||||
// `insert`, a name that fits comes back unchanged.
|
||||
export const shortenKeepingExtension = (
|
||||
name: string,
|
||||
maxBytes: number,
|
||||
@@ -72,7 +76,36 @@ export const shortenKeepingExtension = (
|
||||
): string => {
|
||||
const fullExt = extname(name);
|
||||
const stem = name.slice(0, name.length - fullExt.length);
|
||||
const ext = shortenToBytes(fullExt, maxBytes - Buffer.byteLength(insert));
|
||||
const first = Array.from(stem)[0] ?? "";
|
||||
const ext = shortenToBytes(
|
||||
fullExt,
|
||||
maxBytes - Buffer.byteLength(insert + first),
|
||||
);
|
||||
const room = maxBytes - Buffer.byteLength(insert + ext);
|
||||
return shortenToBytes(stem, room) + insert + ext;
|
||||
};
|
||||
|
||||
// The name a file is saved under in the working directory when the caller
|
||||
// gives no path (`quak get` without `--out`, `downloadFile` without
|
||||
// `outPath`): `prefix` (`thumb_` for a thumbnail), then the file's title with
|
||||
// unsafe characters replaced, or `file-<id>` when the title is empty, cut
|
||||
// short so that the whole fits in a file name, keeping the title's extension
|
||||
// whenever it fits. `ext` (".mov"), when given, takes the place of the
|
||||
// title's extension before the name is cut short; a live photo's image and
|
||||
// video are each named with their own.
|
||||
export const defaultFileName = (
|
||||
title: string,
|
||||
fileID: number,
|
||||
opts: { prefix?: string; ext?: string } = {},
|
||||
): string => {
|
||||
const { prefix = "", ext } = opts;
|
||||
const safe = sanitizeFileName(title, `file-${fileID}`);
|
||||
const name = ext === undefined ? safe : withExtension(safe, ext);
|
||||
return (
|
||||
prefix +
|
||||
shortenKeepingExtension(
|
||||
name,
|
||||
MAX_NAME_BYTES - Buffer.byteLength(prefix),
|
||||
)
|
||||
);
|
||||
};
|
||||
|
||||
+56
-10
@@ -18,6 +18,16 @@
|
||||
// the caller (the default reads stay silent and serve the last good copy). The
|
||||
// default methods and the background loop are unchanged.
|
||||
//
|
||||
// `freshCollections()` is the fresh read of the collection list alone (issue
|
||||
// #210): one collections request, no collection's files, and nothing stored,
|
||||
// for a caller such as `quak collections` that needs no more. A caller that
|
||||
// asks for each refresh it needs can also open the library with
|
||||
// `refreshOnOpen: false`, so that `open()` starts none.
|
||||
//
|
||||
// A refresh reports through `onProgress` each collection whose files it has
|
||||
// fetched, so a large account, whose first refresh fetches the files of every
|
||||
// collection, shows progress.
|
||||
//
|
||||
// A refresh stages all of its network work first and only mutates the store
|
||||
// once every fetch has succeeded. A refresh that fails partway therefore never
|
||||
// becomes visible to reads: the last good snapshot stays in place, and the
|
||||
@@ -161,18 +171,21 @@ export interface LibraryClient {
|
||||
// exactly one of "done" or "failed"; "failed" carries the error message and
|
||||
// an ML "done" reports how many payloads it stored and, in `mlDataErrors`,
|
||||
// each file whose ML data did not decrypt, gunzip or parse, with the reason.
|
||||
// The precache fills ("precacheThumbnails"/"precacheOriginals", #48) fire
|
||||
// "started"/"done" around each sweep that has work, "done" reporting the count
|
||||
// newly cached.
|
||||
// In between, a refresh fires "progress" after each collection whose files it
|
||||
// fetched: `fetched` is how many collections it has fetched the files of so
|
||||
// far, `total` how many it will. The precache fills
|
||||
// ("precacheThumbnails"/"precacheOriginals", #48) fire "started"/"done" around
|
||||
// each sweep that has work, "done" reporting the count newly cached.
|
||||
export interface RefreshEvent {
|
||||
operation:
|
||||
| "refresh"
|
||||
| "fetchMLData"
|
||||
| "precacheThumbnails"
|
||||
| "precacheOriginals";
|
||||
status: "started" | "done" | "failed";
|
||||
status: "started" | "progress" | "done" | "failed";
|
||||
error?: string;
|
||||
fetched?: number;
|
||||
total?: number;
|
||||
mlDataErrors?: { fileID: number; error: string }[];
|
||||
}
|
||||
|
||||
@@ -192,6 +205,10 @@ export interface LibraryOptions {
|
||||
// `Infinity` starts none: the library then refreshes only as it opens and
|
||||
// when `fresh()` or `backup()` asks.
|
||||
refreshIntervalSeconds?: number;
|
||||
// Whether `open()` starts a refresh. Defaults to true. With false, `open()`
|
||||
// returns at once onto what the cache holds, empty or not, and a refresh
|
||||
// starts only when `fresh()` or `backup()` asks or the timer fires.
|
||||
refreshOnOpen?: boolean;
|
||||
onProgress?: RefreshProgressCallback;
|
||||
// The bounded request pools (issue #45), shared by the ML-data fetch (the
|
||||
// metadata pool) and the content cache. Defaults to a fresh set at the
|
||||
@@ -382,7 +399,8 @@ export class Library {
|
||||
// then opens empty with the failure recorded in `status()`. With an
|
||||
// existing cache the first refresh runs in the background and `open()`
|
||||
// returns as soon as the cached data is ready — an unreachable server does
|
||||
// not block opening.
|
||||
// not block opening. With `refreshOnOpen: false` there is no first
|
||||
// refresh: `open()` returns onto the cache as it is.
|
||||
static async open(opts: LibraryOptions): Promise<Library> {
|
||||
const { userID } = opts.client.whoami();
|
||||
const cacheDirectory =
|
||||
@@ -485,7 +503,10 @@ export class Library {
|
||||
// refresh.
|
||||
precache?.start();
|
||||
|
||||
if (store.loadedFromDisk) {
|
||||
if (opts.refreshOnOpen === false) {
|
||||
// The caller asks for each refresh it needs.
|
||||
lib.scheduleNext();
|
||||
} else if (store.loadedFromDisk) {
|
||||
// An existing copy already answers reads; refresh in the background
|
||||
// and start the interval once that first cycle settles.
|
||||
void lib.runRefresh().then(() => lib.scheduleNext());
|
||||
@@ -589,6 +610,21 @@ export class Library {
|
||||
};
|
||||
}
|
||||
|
||||
// The fresh read of the collection list alone (issue #210): every live
|
||||
// collection that decrypts, in the server's order, from one collections
|
||||
// request. Unlike a refresh, it fetches no collection's files, which on a
|
||||
// large account take long. Nothing is stored: a refresh fetches a
|
||||
// collection's files only when the collection is new or has changed since
|
||||
// the stored copy, so a collection stored here without its files would
|
||||
// never have them fetched.
|
||||
async freshCollections(): Promise<Collection[]> {
|
||||
if (this.closed) throw new Error("the library is closed");
|
||||
const { collections } = await this.client.collectionsSince({
|
||||
sinceTime: 0,
|
||||
});
|
||||
return collections;
|
||||
}
|
||||
|
||||
// Back up every in-scope file to `opts.downloadDirectory`, or else the
|
||||
// library's, each original at its save path, with a durable failure
|
||||
// ledger (issue #51). Takes the lock in that directory first, unless the
|
||||
@@ -742,18 +778,28 @@ export class Library {
|
||||
// Stage per-collection file diffs. A collection's files are
|
||||
// re-enumerated only when its updationTime has advanced past the cached
|
||||
// copy; an unchanged album's file list cannot have changed. New
|
||||
// collections enumerate from the beginning of time.
|
||||
// collections enumerate from the beginning of time. Each collection
|
||||
// whose files are fetched is reported, with how many there are to
|
||||
// fetch.
|
||||
const changedCollections = page.collections.filter((collection) => {
|
||||
const known = this.store.getCollection(collection.id);
|
||||
return !known || collection.updationTime > known.updationTime;
|
||||
});
|
||||
const filePages: { collectionID: number; page: FilesPage }[] = [];
|
||||
for (const collection of page.collections) {
|
||||
for (const collection of changedCollections) {
|
||||
const known = this.store.getCollection(collection.id);
|
||||
if (known && collection.updationTime <= known.updationTime)
|
||||
continue;
|
||||
const filePage = await this.client.filesSince({
|
||||
collectionID: collection.id,
|
||||
collectionKey: collection.key,
|
||||
sinceTime: known ? known.updationTime : 0,
|
||||
});
|
||||
filePages.push({ collectionID: collection.id, page: filePage });
|
||||
this.emit({
|
||||
operation: "refresh",
|
||||
status: "progress",
|
||||
fetched: filePages.length,
|
||||
total: changedCollections.length,
|
||||
});
|
||||
}
|
||||
|
||||
// Network work done; commit to the store and persist only if something
|
||||
|
||||
+10
-11
@@ -1795,12 +1795,11 @@ describe("backup album folders", () => {
|
||||
expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 1 });
|
||||
});
|
||||
|
||||
// The extensions of these titles leave no room for the ID, so each is cut
|
||||
// short and the name before the ID keeps only the bytes that cut leaves
|
||||
// over. File 1's extension is 303 bytes and is cut to 251, which leaves
|
||||
// none. File 2's is 271 bytes and is cut to 250, since "€" takes three
|
||||
// bytes, which leaves one, for the "a". Neither extension is only letters
|
||||
// and digits, so both originals are saved with `.bin`, a name that fits.
|
||||
// The extensions of these titles leave no room for the ID and the "a"
|
||||
// before them, so each is cut short to 250 bytes, which leaves one, for
|
||||
// the "a". File 1's extension is 303 bytes and file 2's 271. Neither
|
||||
// extension is only letters and digits, so both originals are saved with
|
||||
// `.bin`, a name that fits.
|
||||
it("cuts the extension of a title short when it leaves no room for the ID", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf([
|
||||
@@ -1815,7 +1814,7 @@ describe("backup album folders", () => {
|
||||
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
const link = ` (1).${"L".repeat(250)}`;
|
||||
const link = `a (1).${"L".repeat(249)}`;
|
||||
const link2 = `a (2).${"€".repeat(83)}`;
|
||||
expect(Buffer.byteLength(link)).toBe(255);
|
||||
expect(Buffer.byteLength(link2)).toBe(255);
|
||||
@@ -1830,9 +1829,9 @@ describe("backup album folders", () => {
|
||||
|
||||
// Files 1 and 3 share a title, so each gets its ID before the
|
||||
// extension. File 2's title is too long, and its extension leaves no
|
||||
// room for the ID. Were the ID put at the end of file 2's title cut
|
||||
// short, its link would have file 1's name, and the link made second
|
||||
// would replace the first.
|
||||
// room for the ID and the "x" before it. Were the ID put at the end of
|
||||
// file 2's title cut short, its link would have file 1's name, and the
|
||||
// link made second would replace the first.
|
||||
it("keeps names unique when a title's extension leaves no room for the ID", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const shared = `x.${"Q".repeat(245)} (2)`;
|
||||
@@ -1852,8 +1851,8 @@ describe("backup album folders", () => {
|
||||
expect(result.failed).toBe(0);
|
||||
expect(tree(outDir)).toEqual([
|
||||
"Trip/",
|
||||
`Trip/ (2).${"Q".repeat(250)} -> ${linkTo("2.bin")}`,
|
||||
`Trip/x (1).${"Q".repeat(245)} (2) -> ${linkTo("1.bin")}`,
|
||||
`Trip/x (2).${"Q".repeat(249)} -> ${linkTo("2.bin")}`,
|
||||
`Trip/x (3).${"Q".repeat(245)} (2) -> ${linkTo("3.bin")}`,
|
||||
"Trip.json",
|
||||
]);
|
||||
|
||||
+248
-16
@@ -53,11 +53,21 @@ import {
|
||||
} from "../../src/cli-commands.js";
|
||||
import { run } from "../../src/cli-run.js";
|
||||
import { loadSession } from "../../src/cli-session.js";
|
||||
import type { Client, ClientSnapshot, LoginOptions } from "../../src/client.js";
|
||||
import {
|
||||
Client,
|
||||
type ClientSnapshot,
|
||||
type LoginOptions,
|
||||
} from "../../src/client.js";
|
||||
import { savePath, type ContentSource } from "../../src/library/content.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
import { encryptBlob, init, toBase64 } from "../../src/crypto/index.js";
|
||||
import { defaultCacheDirectory } from "../../src/library/index.js";
|
||||
import {
|
||||
buildAccount,
|
||||
buildMetaFetch,
|
||||
encryptSecretbox,
|
||||
encryptStreamBlob,
|
||||
} from "../ente-server.js";
|
||||
import { HEIC_WITH_EXIF } from "../exif-heic.js";
|
||||
import {
|
||||
asLivePhoto,
|
||||
@@ -171,6 +181,15 @@ const fakeClient = (
|
||||
return fake as unknown as Client;
|
||||
};
|
||||
|
||||
// What a read command prints on stderr while the refresh it waits for fetches
|
||||
// the files of each of the fake's two albums, on an empty cache.
|
||||
const FETCHED_BOTH_ALBUMS =
|
||||
"Fetched the files of 1 of 2 collections\n" +
|
||||
"Fetched the files of 2 of 2 collections\n";
|
||||
|
||||
// The same for an account with one album.
|
||||
const FETCHED_ONE_ALBUM = "Fetched the files of 1 of 1 collections\n";
|
||||
|
||||
// Collects everything written to it.
|
||||
class Output {
|
||||
text = "";
|
||||
@@ -510,7 +529,9 @@ describe("files", () => {
|
||||
|
||||
it("exits 1 for an unknown collection", async () => {
|
||||
expect(await filesCommand(context(), { collection: "9" })).toBe(1);
|
||||
expect(stderr.text).toBe("Collection 9 not found\n");
|
||||
expect(stderr.text).toBe(
|
||||
FETCHED_BOTH_ALBUMS + "Collection 9 not found\n",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -519,27 +540,27 @@ describe("get and get-thumb", () => {
|
||||
const out = join(root, "diagram.png");
|
||||
expect(await getCommand(context(), "200", { out })).toBe(0);
|
||||
expect(readFileSync(out)).toEqual(Buffer.alloc(7, 200));
|
||||
expect(stderr.text).toBe(`7 bytes -> ${out}\n`);
|
||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + `7 bytes -> ${out}\n`);
|
||||
});
|
||||
|
||||
it("get-thumb finds a file in any album without --collection", async () => {
|
||||
const out = join(root, "thumb.jpg");
|
||||
expect(await getThumbCommand(context(), "200", { out })).toBe(0);
|
||||
expect(readFileSync(out)).toEqual(Buffer.alloc(3, 200));
|
||||
expect(stderr.text).toBe(`3 bytes -> ${out}\n`);
|
||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + `3 bytes -> ${out}\n`);
|
||||
});
|
||||
|
||||
it("get exits 1 when no album has the file", async () => {
|
||||
const out = join(root, "x");
|
||||
expect(await getCommand(context(), "999", { out })).toBe(1);
|
||||
expect(stderr.text).toBe("File 999 not found\n");
|
||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + "File 999 not found\n");
|
||||
expect(existsSync(out)).toBe(false);
|
||||
});
|
||||
|
||||
it("get-thumb exits 1 when no album has the file", async () => {
|
||||
const out = join(root, "x");
|
||||
expect(await getThumbCommand(context(), "999", { out })).toBe(1);
|
||||
expect(stderr.text).toBe("File 999 not found\n");
|
||||
expect(stderr.text).toBe(FETCHED_BOTH_ALBUMS + "File 999 not found\n");
|
||||
expect(existsSync(out)).toBe(false);
|
||||
});
|
||||
});
|
||||
@@ -587,7 +608,7 @@ describe("names from an album name or a title over 255 bytes", () => {
|
||||
expect(Buffer.byteLength(name)).toBe(255);
|
||||
expect(code).toBe(0);
|
||||
expect(readdirSync(dir)).toEqual([name]);
|
||||
expect(stderr.text).toBe(`7 bytes -> ${name}\n`);
|
||||
expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `7 bytes -> ${name}\n`);
|
||||
});
|
||||
|
||||
it("get-thumb cuts the title short so that thumb_ and it fit in 255 bytes, and exits 0", async () => {
|
||||
@@ -599,7 +620,7 @@ describe("names from an album name or a title over 255 bytes", () => {
|
||||
expect(Buffer.byteLength(name)).toBe(255);
|
||||
expect(code).toBe(0);
|
||||
expect(readdirSync(dir)).toEqual([name]);
|
||||
expect(stderr.text).toBe(`3 bytes -> ${name}\n`);
|
||||
expect(stderr.text).toBe(FETCHED_ONE_ALBUM + `3 bytes -> ${name}\n`);
|
||||
});
|
||||
|
||||
it("backup-metadata cuts the album name short so that its folder's name fits in 255 bytes, and exits 0", async () => {
|
||||
@@ -641,9 +662,12 @@ describe("names from an album name or a title over 255 bytes", () => {
|
||||
// one live photo, 300, downloaded through the real download layer
|
||||
// (test/live-photo.ts).
|
||||
describe("a live photo", () => {
|
||||
const livePhotoClient = async (image = IMAGE): Promise<Client> => {
|
||||
const livePhotoClient = async (
|
||||
image = IMAGE,
|
||||
title = "IMG_0300.HEIC",
|
||||
): Promise<Client> => {
|
||||
const { file: live, body } = await asLivePhoto(
|
||||
file(300, 1, "IMG_0300.HEIC"),
|
||||
file(300, 1, title),
|
||||
livePhotoZip({ "image.heic": image, "video.mov": VIDEO }),
|
||||
livePhotoHash(image, VIDEO),
|
||||
);
|
||||
@@ -685,11 +709,50 @@ describe("a live photo", () => {
|
||||
Buffer.from(VIDEO),
|
||||
);
|
||||
expect(stderr.text).toBe(
|
||||
`${IMAGE.length} bytes -> IMG_0300.heic\n` +
|
||||
FETCHED_ONE_ALBUM +
|
||||
`${IMAGE.length} bytes -> IMG_0300.heic\n` +
|
||||
`${VIDEO.length} bytes -> IMG_0300.mov\n`,
|
||||
);
|
||||
});
|
||||
|
||||
// Run `get` without `--out` in a new directory for a live photo titled
|
||||
// `title`, and return its exit code and the names it wrote.
|
||||
const getInNewDirectory = async (
|
||||
title: string,
|
||||
): Promise<{ code: number; names: string[] }> => {
|
||||
const ctx = context(await livePhotoClient(IMAGE, title));
|
||||
const dir = mkdtempSync(join(root, "cwd-"));
|
||||
const previous = process.cwd();
|
||||
process.chdir(dir);
|
||||
try {
|
||||
const code = await getCommand(ctx, "300", {});
|
||||
return { code, names: readdirSync(dir).sort() };
|
||||
} finally {
|
||||
process.chdir(previous);
|
||||
}
|
||||
};
|
||||
const imageName = `${"t".repeat(250)}.heic`;
|
||||
const videoName = `${"t".repeat(251)}.mov`;
|
||||
|
||||
// The title fits, but ".heic" is one byte longer than ".jpg", so the
|
||||
// image's name is cut short once its extension is in it.
|
||||
it("get cuts its image's name short for a 255-byte title ending in .jpg, and exits 0", async () => {
|
||||
const title = `${"t".repeat(251)}.jpg`;
|
||||
|
||||
const result = await getInNewDirectory(title);
|
||||
|
||||
expect(Buffer.byteLength(title)).toBe(255);
|
||||
expect(Buffer.byteLength(imageName)).toBe(255);
|
||||
expect(Buffer.byteLength(videoName)).toBe(255);
|
||||
expect(result).toEqual({ code: 0, names: [imageName, videoName] });
|
||||
});
|
||||
|
||||
it("get cuts the names of its image and video short for a 300-byte title with no extension, and exits 0", async () => {
|
||||
const result = await getInNewDirectory("t".repeat(300));
|
||||
|
||||
expect(result).toEqual({ code: 0, names: [imageName, videoName] });
|
||||
});
|
||||
|
||||
it("get --out writes the image there and the video beside it", async () => {
|
||||
const out = join(root, "photo.jpg");
|
||||
const video = join(root, "photo.mov");
|
||||
@@ -701,7 +764,8 @@ describe("a live photo", () => {
|
||||
expect(readFileSync(out)).toEqual(Buffer.from(IMAGE));
|
||||
expect(readFileSync(video)).toEqual(Buffer.from(VIDEO));
|
||||
expect(stderr.text).toBe(
|
||||
`${IMAGE.length} bytes -> ${out}\n${VIDEO.length} bytes -> ${video}\n`,
|
||||
FETCHED_ONE_ALBUM +
|
||||
`${IMAGE.length} bytes -> ${out}\n${VIDEO.length} bytes -> ${video}\n`,
|
||||
);
|
||||
});
|
||||
|
||||
@@ -715,7 +779,8 @@ describe("a live photo", () => {
|
||||
expect(existsSync(out)).toBe(false);
|
||||
expect(existsSync(join(root, "photo.mov"))).toBe(false);
|
||||
expect(stderr.text).toBe(
|
||||
`File 300 is a live photo, and its video would also be written to ${out}\n`,
|
||||
FETCHED_ONE_ALBUM +
|
||||
`File 300 is a live photo, and its video would also be written to ${out}\n`,
|
||||
);
|
||||
});
|
||||
|
||||
@@ -1195,11 +1260,12 @@ describe("backup-metadata --exif", () => {
|
||||
});
|
||||
});
|
||||
|
||||
// Each test first runs `collections` so the cache holds the account as it was,
|
||||
// then changes the server under it.
|
||||
// Each test first runs `files` so the cache holds the account as it was, then
|
||||
// changes the server under it. (`collections` would not do: it reads the
|
||||
// collection list alone and stores nothing.)
|
||||
describe("backup-metadata and the thumbnail helpers refresh first", () => {
|
||||
beforeEach(async () => {
|
||||
expect(await collectionsCommand(context(), {})).toBe(0);
|
||||
expect(await filesCommand(context(), { collection: "1" })).toBe(0);
|
||||
stdout.text = "";
|
||||
stderr.text = "";
|
||||
});
|
||||
@@ -1384,3 +1450,169 @@ describe("an ID given on the command line", () => {
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
// An account with many albums (issue #210). On an empty cache `collections`
|
||||
// used to wait for a refresh that fetched and decrypted the files of every
|
||||
// album, one album after another, before it printed anything, so on such an
|
||||
// account it seemed to hang. The account here has 40 albums of one photo each,
|
||||
// encrypted as Ente's clients encrypt them, on the fake Ente server of
|
||||
// `test/ente-server.ts`, which also records the path of each request quak
|
||||
// sends. The session is the one logging in to that server saves.
|
||||
describe("an account with many albums", () => {
|
||||
const ALBUMS = 40;
|
||||
const PASSWORD = "many-albums";
|
||||
const paths: string[] = [];
|
||||
let server: typeof globalThis.fetch;
|
||||
let snapshot: ClientSnapshot;
|
||||
|
||||
// Album n is named `Album n` and holds one photo, n * 100, titled
|
||||
// `photo-n.jpg`.
|
||||
beforeAll(async () => {
|
||||
await sodium.ready;
|
||||
const account = await buildAccount(PASSWORD);
|
||||
const collections: Record<string, unknown>[] = [];
|
||||
const filesByCollection: Record<number, Record<string, unknown>[]> = {};
|
||||
for (let id = 1; id <= ALBUMS; id++) {
|
||||
const albumKey = sodium.crypto_secretbox_keygen();
|
||||
const albumKeyBox = encryptSecretbox(albumKey, account.masterKey);
|
||||
const name = encryptSecretbox(
|
||||
new TextEncoder().encode(`Album ${id}`),
|
||||
albumKey,
|
||||
);
|
||||
collections.push({
|
||||
id,
|
||||
owner: { id: USER_ID },
|
||||
encryptedKey: toBase64(albumKeyBox.ciphertext),
|
||||
keyDecryptionNonce: toBase64(albumKeyBox.nonce),
|
||||
encryptedName: toBase64(name.ciphertext),
|
||||
nameDecryptionNonce: toBase64(name.nonce),
|
||||
type: "album",
|
||||
updationTime: 1,
|
||||
});
|
||||
const photoKey =
|
||||
sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const photoKeyBox = encryptSecretbox(photoKey, albumKey);
|
||||
const metadata = encryptStreamBlob(
|
||||
new TextEncoder().encode(
|
||||
JSON.stringify({
|
||||
title: `photo-${id}.jpg`,
|
||||
fileType: 0,
|
||||
creationTime: 1000,
|
||||
modificationTime: 1000,
|
||||
}),
|
||||
),
|
||||
photoKey,
|
||||
);
|
||||
filesByCollection[id] = [
|
||||
{
|
||||
id: id * 100,
|
||||
collectionID: id,
|
||||
ownerID: USER_ID,
|
||||
encryptedKey: toBase64(photoKeyBox.ciphertext),
|
||||
keyDecryptionNonce: toBase64(photoKeyBox.nonce),
|
||||
metadata: {
|
||||
encryptedData: toBase64(metadata.ciphertext),
|
||||
decryptionHeader: toBase64(metadata.header),
|
||||
},
|
||||
// These tests fetch no original and no thumbnail.
|
||||
file: { decryptionHeader: "" },
|
||||
thumbnail: { decryptionHeader: "" },
|
||||
updationTime: 1,
|
||||
},
|
||||
];
|
||||
}
|
||||
const ente = buildMetaFetch({
|
||||
verifier: account.verifier,
|
||||
srpAttributes: account.srpAttributes,
|
||||
keyAttributes: account.keyAttributes,
|
||||
encryptedToken: account.encryptedToken,
|
||||
collections,
|
||||
filesByCollection,
|
||||
encryptedMLData: {},
|
||||
fileCiphertexts: {},
|
||||
});
|
||||
server = (async (input: RequestInfo | URL, init?: RequestInit) => {
|
||||
const url =
|
||||
typeof input === "string"
|
||||
? input
|
||||
: input instanceof URL
|
||||
? input.href
|
||||
: input.url;
|
||||
paths.push(new URL(url).pathname);
|
||||
return ente(input, init);
|
||||
}) as typeof globalThis.fetch;
|
||||
const client = await Client.login({
|
||||
email: "many@example.com",
|
||||
password: PASSWORD,
|
||||
apiOptions: { fetch: server },
|
||||
});
|
||||
snapshot = client.toJSON();
|
||||
});
|
||||
|
||||
beforeEach(() => {
|
||||
paths.length = 0;
|
||||
});
|
||||
|
||||
// A context whose saved session sends its requests to the fake server.
|
||||
const accountContext = (): CliContext => {
|
||||
const ctx = context();
|
||||
saveSession(ctx.sessionDir, snapshot);
|
||||
return {
|
||||
...ctx,
|
||||
loadSession: (path) => loadSession(path, { fetch: server }),
|
||||
};
|
||||
};
|
||||
|
||||
it("collections lists every album from one request, fetching no album's files", async () => {
|
||||
const ctx = accountContext();
|
||||
expect(await collectionsCommand(ctx, {})).toBe(0);
|
||||
|
||||
let albums = "";
|
||||
for (let id = 1; id <= ALBUMS; id++)
|
||||
albums += `${id}\talbum\tAlbum ${id}\n`;
|
||||
expect(stdout.text).toBe(albums);
|
||||
expect(stderr.text).toBe("");
|
||||
expect(paths).toEqual(["/collections/v2"]);
|
||||
|
||||
// It stored nothing that would make a refresh skip an album's files:
|
||||
// `files` after it still finds the album's photo.
|
||||
stdout.text = "";
|
||||
expect(await filesCommand(ctx, { collection: "7" })).toBe(0);
|
||||
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
||||
});
|
||||
|
||||
it("files counts on stderr each album whose files its refresh fetched", async () => {
|
||||
expect(await filesCommand(accountContext(), { collection: "7" })).toBe(
|
||||
0,
|
||||
);
|
||||
|
||||
expect(stdout.text).toBe("700\timage\tphoto-7.jpg\n");
|
||||
let progress = "";
|
||||
for (let n = 1; n <= ALBUMS; n++) {
|
||||
progress += `Fetched the files of ${n} of ${ALBUMS} collections\n`;
|
||||
}
|
||||
expect(stderr.text).toBe(progress);
|
||||
});
|
||||
|
||||
// With `--json` the refresh's progress is not printed, as `backup` and
|
||||
// the thumbnail helpers print none of theirs.
|
||||
it("files --json prints no progress", async () => {
|
||||
expect(
|
||||
await filesCommand(accountContext(), {
|
||||
collection: "7",
|
||||
json: true,
|
||||
}),
|
||||
).toBe(0);
|
||||
|
||||
expect(JSON.parse(stdout.text)).toEqual([
|
||||
{
|
||||
id: 700,
|
||||
title: "photo-7.jpg",
|
||||
fileType: "image",
|
||||
creationTime: 1000,
|
||||
collectionID: 7,
|
||||
},
|
||||
]);
|
||||
expect(stderr.text).toBe("");
|
||||
});
|
||||
});
|
||||
|
||||
+20
-11
@@ -3,10 +3,13 @@
|
||||
* issue #36, issue #52).
|
||||
*
|
||||
* The `collections`, `files`, `get`, and `get-thumb` commands must answer for
|
||||
* current server state, not the local cache, so each helper forces a
|
||||
* `Library.fresh()` round-trip before it reads. The stand-in library below
|
||||
* serves nothing until `fresh()` has been awaited, so a helper that read
|
||||
* without refreshing would come back empty and fail here.
|
||||
* current server state, not the local cache. `files`, `get` and `get-thumb`
|
||||
* force a `Library.fresh()` round-trip before they read; the stand-in library
|
||||
* below serves nothing until `fresh()` has been awaited, so a helper that read
|
||||
* without refreshing would come back empty and fail here. `collections` asks
|
||||
* the library for the collection list alone (`freshCollections()`, issue
|
||||
* #210), and must not wait for a refresh, which would fetch the files of every
|
||||
* collection.
|
||||
*
|
||||
* `collections` and `files` also list in the library's enumeration order
|
||||
* (`listCollections`/`listFiles`) — the order the pre-library CLI printed — not
|
||||
@@ -60,10 +63,13 @@ const file = (
|
||||
});
|
||||
|
||||
// A library that reveals its records only after `fresh()` has been awaited, and
|
||||
// serves them in the enumeration order it was given. `photos.byID` returns a
|
||||
// stand-in `Photo` carrying just the fileID the helper passes through.
|
||||
// serves them in the enumeration order it was given. `freshCollections()`
|
||||
// answers the collection list in that order without a refresh. `photos.byID`
|
||||
// returns a stand-in `Photo` carrying just the fileID the helper passes
|
||||
// through.
|
||||
class FakeLibrary implements FreshReadLibrary {
|
||||
freshCalls = 0;
|
||||
freshCollectionsCalls = 0;
|
||||
private refreshed = false;
|
||||
|
||||
constructor(
|
||||
@@ -77,12 +83,14 @@ class FakeLibrary implements FreshReadLibrary {
|
||||
return {};
|
||||
}
|
||||
|
||||
listCollections(): Collection[] {
|
||||
return this.refreshed ? this.collections : [];
|
||||
async freshCollections(): Promise<Collection[]> {
|
||||
this.freshCollectionsCalls++;
|
||||
return this.collections;
|
||||
}
|
||||
|
||||
getCollection(id: number): Collection | undefined {
|
||||
return this.listCollections().find((c) => c.id === id);
|
||||
if (!this.refreshed) return undefined;
|
||||
return this.collections.find((c) => c.id === id);
|
||||
}
|
||||
|
||||
listFiles(collectionID: number): EnteFile[] {
|
||||
@@ -106,7 +114,7 @@ class FakeLibrary implements FreshReadLibrary {
|
||||
}
|
||||
|
||||
describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
||||
it("freshCollections refreshes first, then lists in enumeration order", async () => {
|
||||
it("freshCollections reads the collection list alone, in enumeration order", async () => {
|
||||
// Enumeration order 2, 1, 3; a newest-first sort would be 3, 2, 1.
|
||||
const lib = new FakeLibrary(
|
||||
[collection(2, 200), collection(1, 300), collection(3, 100)],
|
||||
@@ -114,7 +122,8 @@ describe("CLI read helpers (issue #36 amendment, issue #52)", () => {
|
||||
);
|
||||
const rows = await freshCollections(lib);
|
||||
|
||||
expect(lib.freshCalls).toBe(1);
|
||||
expect(lib.freshCollectionsCalls).toBe(1);
|
||||
expect(lib.freshCalls).toBe(0);
|
||||
expect(rows.map((c) => c.id)).toEqual([2, 1, 3]);
|
||||
// The projection's newest-first order is a different sequence, so this
|
||||
// is not accidentally that order.
|
||||
|
||||
@@ -1923,6 +1923,53 @@ describe("downloadFile live photos", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
// A file system refuses a file name over 255 bytes. Each name is cut
|
||||
// short once its own extension is in it.
|
||||
const imageName = `${"t".repeat(250)}.heic`;
|
||||
const videoName = `${"t".repeat(251)}.mov`;
|
||||
|
||||
// The title fits, but ".heic" is one byte longer than ".jpg".
|
||||
it("cuts the image's name short for a 255-byte title ending in .jpg when no outPath is given", async () => {
|
||||
const title = `${"t".repeat(251)}.jpg`;
|
||||
const t = setup(livePhotoZip(), { ...livePhoto, title });
|
||||
|
||||
const result = await inDirectory(t.dir, () =>
|
||||
downloadFile(t.api, t.file),
|
||||
);
|
||||
|
||||
expect(Buffer.byteLength(title)).toBe(255);
|
||||
expect(Buffer.byteLength(imageName)).toBe(255);
|
||||
expect(Buffer.byteLength(videoName)).toBe(255);
|
||||
expect(result.path).toBe(imageName);
|
||||
expect(result.videoPath).toBe(videoName);
|
||||
expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]);
|
||||
});
|
||||
|
||||
it("cuts the names of the image and the video short for a 300-byte title with no extension when no outPath is given", async () => {
|
||||
const t = setup(livePhotoZip(), {
|
||||
...livePhoto,
|
||||
title: "t".repeat(300),
|
||||
});
|
||||
|
||||
const result = await inDirectory(t.dir, () =>
|
||||
downloadFile(t.api, t.file),
|
||||
);
|
||||
|
||||
expect(result.path).toBe(imageName);
|
||||
expect(result.videoPath).toBe(videoName);
|
||||
expect(readdirSync(t.dir).sort()).toEqual([imageName, videoName]);
|
||||
});
|
||||
|
||||
// The caller's path is used as is; only the extension is each part's own.
|
||||
it("keeps an outPath too long for a file name as given", async () => {
|
||||
const t = setup(livePhotoZip(), livePhoto);
|
||||
const outPath = join(t.dir, `${"t".repeat(251)}.jpg`);
|
||||
|
||||
await expect(downloadFile(t.api, t.file, outPath)).rejects.toThrow(
|
||||
/ENAMETOOLONG/,
|
||||
);
|
||||
});
|
||||
|
||||
it("gives each part its own extension from the ZIP, letters and digits only", async () => {
|
||||
const zip = livePhotoZip({ "image.JPG": IMAGE, "video.m-4v": VIDEO });
|
||||
const t = setup(zip, livePhoto);
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import {
|
||||
defaultFileName,
|
||||
safeExtension,
|
||||
sanitizeFileName,
|
||||
shortenKeepingExtension,
|
||||
@@ -141,9 +142,17 @@ describe("shortenKeepingExtension", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("cuts the extension short when it alone does not fit", () => {
|
||||
// A name cut down to its extension would start with a dot, and so be
|
||||
// hidden.
|
||||
it("keeps the first character before the extension, cutting the extension short to leave room for it", () => {
|
||||
expect(shortenKeepingExtension(`b.${"e".repeat(300)}`, 255)).toBe(
|
||||
`.${"e".repeat(254)}`,
|
||||
`b.${"e".repeat(253)}`,
|
||||
);
|
||||
expect(shortenKeepingExtension(`ab.${"e".repeat(300)}`, 255)).toBe(
|
||||
`a.${"e".repeat(253)}`,
|
||||
);
|
||||
expect(shortenKeepingExtension(`é.${"e".repeat(300)}`, 255)).toBe(
|
||||
`é.${"e".repeat(252)}`,
|
||||
);
|
||||
// "€" takes three bytes, so the extension is cut to 253 and the
|
||||
// part before it keeps the two bytes left over, for the "a".
|
||||
@@ -161,6 +170,50 @@ describe("shortenKeepingExtension", () => {
|
||||
).toBe(`${"t".repeat(247)} (7).jpg`);
|
||||
expect(
|
||||
shortenKeepingExtension(`b.${"e".repeat(300)}`, 255, " (7)"),
|
||||
).toBe(` (7).${"e".repeat(250)}`);
|
||||
).toBe(`b (7).${"e".repeat(249)}`);
|
||||
});
|
||||
});
|
||||
|
||||
describe("defaultFileName", () => {
|
||||
it("is the title when it is safe and fits", () => {
|
||||
expect(defaultFileName("IMG_0001.HEIC", 42)).toBe("IMG_0001.HEIC");
|
||||
});
|
||||
|
||||
it("replaces unsafe characters, and is file-<id> for an empty title", () => {
|
||||
expect(defaultFileName("../x.jpg", 42)).toBe("__x.jpg");
|
||||
expect(defaultFileName("", 42)).toBe("file-42");
|
||||
});
|
||||
|
||||
it("cuts a title over 255 bytes short, keeping its extension", () => {
|
||||
expect(defaultFileName(`${"t".repeat(296)}.jpg`, 42)).toBe(
|
||||
`${"t".repeat(251)}.jpg`,
|
||||
);
|
||||
expect(defaultFileName(`b.${"e".repeat(300)}`, 42)).toBe(
|
||||
`b.${"e".repeat(253)}`,
|
||||
);
|
||||
});
|
||||
|
||||
it("puts `prefix` before the title and leaves room for it", () => {
|
||||
expect(defaultFileName("beach.jpg", 42, { prefix: "thumb_" })).toBe(
|
||||
"thumb_beach.jpg",
|
||||
);
|
||||
expect(
|
||||
defaultFileName(`${"t".repeat(296)}.jpg`, 42, { prefix: "thumb_" }),
|
||||
).toBe(`thumb_${"t".repeat(245)}.jpg`);
|
||||
});
|
||||
|
||||
// A live photo's image and video are each named with their own
|
||||
// extension, and the name is cut short once that extension is in it.
|
||||
it("puts `ext` in place of the title's extension before cutting the name short", () => {
|
||||
expect(defaultFileName("IMG_0001.HEIC", 42, { ext: ".mov" })).toBe(
|
||||
"IMG_0001.mov",
|
||||
);
|
||||
expect(defaultFileName("", 42, { ext: ".mov" })).toBe("file-42.mov");
|
||||
expect(
|
||||
defaultFileName(`${"t".repeat(251)}.jpg`, 42, { ext: ".heic" }),
|
||||
).toBe(`${"t".repeat(250)}.heic`);
|
||||
expect(defaultFileName("t".repeat(300), 42, { ext: ".heic" })).toBe(
|
||||
`${"t".repeat(250)}.heic`,
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -14,6 +14,9 @@
|
||||
* three.
|
||||
* 3. A refresh that fails rejects the fresh read (currency was unavailable),
|
||||
* while the default reads stay silent and keep serving the last good copy.
|
||||
* 4. `freshCollections()` reads the collection list alone, with one request
|
||||
* and no collection's files, and stores nothing, so the next refresh still
|
||||
* fetches the files of every collection (issue #210).
|
||||
*
|
||||
* The client is the same metadata-only mock the background-refresh tests use:
|
||||
* no crypto, no network, scripted pages, and a record of each call. A long
|
||||
@@ -270,3 +273,70 @@ describe("Library.fresh", () => {
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("Library.freshCollections", () => {
|
||||
let dir: string;
|
||||
let cacheDirectory: string;
|
||||
|
||||
beforeEach(() => {
|
||||
dir = mkdtempSync(join(tmpdir(), "quak-fresh-collections-"));
|
||||
cacheDirectory = join(dir, "cache");
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
// A refresh fetches a collection's files only when the collection is new
|
||||
// or its updationTime has advanced past the stored copy's. Had the
|
||||
// collection list been stored, the next refresh would take every
|
||||
// collection for one it already holds the files of, and fetch none.
|
||||
it("asks for the collection list alone and stores nothing", async () => {
|
||||
const client = new MockClient();
|
||||
const page = {
|
||||
collections: [collection(2, 100), collection(1, 110)],
|
||||
deleted: [],
|
||||
cursor: 110,
|
||||
};
|
||||
client.collectionsQueue.push(page, page);
|
||||
client.filesFor(1, {
|
||||
files: [file(1001, 1, 90)],
|
||||
deleted: [],
|
||||
cursor: 90,
|
||||
});
|
||||
client.filesFor(2, {
|
||||
files: [file(2001, 2, 95)],
|
||||
deleted: [],
|
||||
cursor: 95,
|
||||
});
|
||||
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshOnOpen: false,
|
||||
refreshIntervalSeconds: SLOW_INTERVAL,
|
||||
});
|
||||
try {
|
||||
// One request, the whole list in the server's order, and no
|
||||
// collection's files.
|
||||
const collections = await lib.freshCollections();
|
||||
expect(collections.map((c) => c.id)).toEqual([2, 1]);
|
||||
expect(collections.map((c) => c.name)).toEqual([
|
||||
"album-2",
|
||||
"album-1",
|
||||
]);
|
||||
expect(client.collectionsSinceTimes).toEqual([0]);
|
||||
expect(client.filesCalls).toEqual([]);
|
||||
expect(lib.listCollections()).toEqual([]);
|
||||
|
||||
// The next refresh still asks from the beginning and fetches the
|
||||
// files of both collections.
|
||||
const reads = await lib.fresh();
|
||||
expect(client.collectionsSinceTimes).toEqual([0, 0]);
|
||||
expect(reads.photos.byID({ fileID: 1001 })?.fileID).toBe(1001);
|
||||
expect(reads.photos.byID({ fileID: 2001 })?.fileID).toBe(2001);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
@@ -26,6 +26,9 @@
|
||||
* opening.
|
||||
* 8. A save failure that leaves RAM ahead of disk keeps `status().lastError`
|
||||
* set and keeps retrying the write; a later empty refresh does not clear it.
|
||||
* 9. With `refreshOnOpen: false`, `open()` starts no refresh at all, and a
|
||||
* refresh reports through `onProgress` each collection whose files it
|
||||
* fetched.
|
||||
*
|
||||
* The client is a mock: no crypto, no network. It serves scripted pages and
|
||||
* records the `sinceTime` each call carried so cursor threading is provable.
|
||||
@@ -581,6 +584,107 @@ describe("Library.open and background refresh", () => {
|
||||
}
|
||||
});
|
||||
|
||||
// A caller that asks for each refresh it needs, as the CLI's read commands
|
||||
// do, opens without one: on an empty cache the refresh `open()` would
|
||||
// await fetches the files of every collection, which `quak collections`
|
||||
// never needs (issue #210).
|
||||
it("starts no refresh as it opens when refreshOnOpen is false", async () => {
|
||||
const client = new MockClient();
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 100)],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshOnOpen: false,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
});
|
||||
try {
|
||||
expect(client.collectionsSinceTimes).toEqual([]);
|
||||
expect(lib.listCollections()).toEqual([]);
|
||||
expect(lib.status().lastRefreshAt).toBeUndefined();
|
||||
|
||||
// The refresh comes when the caller asks for it.
|
||||
await lib.fresh();
|
||||
expect(client.collectionsSinceTimes).toEqual([0]);
|
||||
expect(lib.listCollections().map((c) => c.id)).toEqual([1]);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
// A refresh fetches the files of each collection that changed, one
|
||||
// collection after another, and stores nothing until it has them all. On
|
||||
// a large account that takes a while, so it reports each collection whose
|
||||
// files it has fetched, and how many it will fetch in all (issue #210).
|
||||
it("reports each collection whose files a refresh fetched", async () => {
|
||||
const events: RefreshEvent[] = [];
|
||||
const client = new MockClient();
|
||||
client.collectionsQueue.push({
|
||||
collections: [
|
||||
collection(1, 100),
|
||||
collection(2, 100),
|
||||
collection(3, 100),
|
||||
],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
onProgress: (e) => events.push(e),
|
||||
});
|
||||
try {
|
||||
expect(events).toEqual([
|
||||
{ operation: "refresh", status: "started" },
|
||||
{
|
||||
operation: "refresh",
|
||||
status: "progress",
|
||||
fetched: 1,
|
||||
total: 3,
|
||||
},
|
||||
{
|
||||
operation: "refresh",
|
||||
status: "progress",
|
||||
fetched: 2,
|
||||
total: 3,
|
||||
},
|
||||
{
|
||||
operation: "refresh",
|
||||
status: "progress",
|
||||
fetched: 3,
|
||||
total: 3,
|
||||
},
|
||||
{ operation: "refresh", status: "done" },
|
||||
]);
|
||||
|
||||
// Only collection 2 changed since, so the next refresh fetches
|
||||
// its files alone and counts one.
|
||||
events.length = 0;
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(2, 200)],
|
||||
deleted: [],
|
||||
cursor: 200,
|
||||
});
|
||||
await lib.fresh();
|
||||
expect(events.filter((e) => e.status === "progress")).toEqual([
|
||||
{
|
||||
operation: "refresh",
|
||||
status: "progress",
|
||||
fetched: 1,
|
||||
total: 1,
|
||||
},
|
||||
]);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("keeps a save failure visible until a save actually succeeds", async () => {
|
||||
const saveSpy = vi
|
||||
.spyOn(MetadataStore.prototype, "save")
|
||||
|
||||
Reference in New Issue
Block a user