Compare commits
2
Commits
4633150de9
...
375d8ae28a
| Author | SHA1 | Date | |
|---|---|---|---|
 sneak
|
375d8ae28a | ||
 clawbot
|
960a7ce8a6 |
No files matched your search
@@ -134,12 +134,12 @@ matters.
|
||||
|
||||
### Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
| ---- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `0` | The backup is complete: every original is at its save path, and no file failed. |
|
||||
| `1` | The run finished with files in `failures.json` or an album it could not decrypt, or it stopped on another error, printed as one line `quak: <message>`. The next run tries again. |
|
||||
| `2` | Another backup of the same directory is running. This run sent no request and changed nothing. |
|
||||
| `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. |
|
||||
| Code | Meaning |
|
||||
| ---- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `0` | The backup is complete: every original is at its save path, and no file failed. |
|
||||
| `1` | The run finished with files in `failures.json` or albums it could not decrypt or write (see "Backup layout"), or it stopped on another error, printed as one line `quak: <message>`. The next run tries again. |
|
||||
| `2` | Another backup of the same directory is running. This run sent no request and changed nothing. |
|
||||
| `3` | There is no usable session: none is saved, the saved one is corrupt, or the server no longer accepts it. Run `quak login` as the user the cron job runs as. |
|
||||
|
||||
`<message>` is the error's message, followed by the message of each error that
|
||||
caused it. Under Node, as `yarn quak` runs quak, `fetch` reports every failure
|
||||
@@ -697,8 +697,10 @@ into `failures.json` (see "Backup layout").
|
||||
|
||||
`backup-metadata` fetches ML data in requests of up to 200 files. When a request
|
||||
fails, the error is logged, each of its files is written with the reason in an
|
||||
`mlDataError` field instead of `mlData`, and the dump goes on. The exit code is
|
||||
non-zero if any ML data request failed.
|
||||
`mlDataError` field instead of `mlData`, and the dump goes on. A file whose ML
|
||||
data does not decrypt, gunzip or parse is logged and written the same way, with
|
||||
why in `mlDataError`. The exit code is non-zero if any file was written with
|
||||
`mlDataError`.
|
||||
|
||||
`helper fix-missing-thumbnails` regenerates thumbnails for JPEG images only,
|
||||
because the bundled decoder (`jpeg-js`) decodes only JPEG. A non-JPEG image
|
||||
@@ -763,8 +765,12 @@ A file's JSON holds Ente's ML data for it (its faces and its CLIP embedding) as
|
||||
for has no `mlData`. The backup waits for the library's ML data fetch to finish
|
||||
before it writes the JSON files. If that fetch fails, each file whose ML data is
|
||||
not in the cache gets the reason in `mlDataError` instead and counts as failed,
|
||||
and the next run fetches it again. The JSON files are rewritten on every run, so
|
||||
ML data that arrived since the last run appears.
|
||||
and the next run fetches it again. The same goes for a file with no ML data in
|
||||
the cache whose ML data the server sent but quak could not decrypt, gunzip or
|
||||
parse. Its `mlDataError` says why, and `failures.json` records it with an error
|
||||
starting `ML data: `. quak keeps that reason in memory only and caches nothing
|
||||
for the file, so the next run asks for its ML data again. The JSON files are
|
||||
rewritten on every run, so ML data that arrived since the last run appears.
|
||||
|
||||
A file's JSON holds, as `imageMetadata`, what `backup-metadata --exif` records
|
||||
from its original, or from a live photo's image: `format`, `width` and `height`
|
||||
@@ -827,8 +833,23 @@ after the file's title, both with unsafe characters replaced. When two
|
||||
collections would get the same name, or two symlinks in one collection the same
|
||||
name (ignoring case in both), each of them gets its ID added: two albums named
|
||||
`Trip` become `Trip (10)/` and `Trip (11)/`, and two files titled `IMG_0001.JPG`
|
||||
become `IMG_0001 (12345).JPG` and `IMG_0001 (12346).JPG`. IDs never change, so a
|
||||
name stays the same from run to run until such a clash appears or goes away.
|
||||
become `IMG_0001 (12345).JPG` and `IMG_0001 (12346).JPG`. A collection whose
|
||||
directory would have the name of another collection's JSON, again ignoring case,
|
||||
gets its ID added as well: beside an album `x`, whose JSON is `x.json`, an album
|
||||
named `x.json` gets the directory `x.json (12)/`. A name too long for a file
|
||||
name, which most file systems limit to 255 bytes, gets its ID added too. A name
|
||||
with its ID added is cut short between two characters in the part before the ID,
|
||||
just far enough that it fits. When a symlink's extension alone leaves no room
|
||||
for the ID, the extension is cut short and the part before the ID keeps only the
|
||||
bytes that cut leaves over. A collection's directory name also leaves room for
|
||||
the `.json` its JSON adds. IDs never change, so a name stays the same from run
|
||||
to run until such a clash appears or goes away.
|
||||
|
||||
A collection whose directory or JSON cannot be written, such as one whose
|
||||
directory name is taken by a file you put in `collections/`, does not stop the
|
||||
run. A run without `--json` prints `FAILED album <name>: <reason>` for it,
|
||||
`--json` and the `BackupResult` list its ID in `failedAlbums`, and the exit code
|
||||
is 1. The other collections and `failures.json` are still written.
|
||||
|
||||
A live photo, which Ente stores as one ZIP of its image and its video, is stored
|
||||
as those two files, which a photo viewer can open: each is
|
||||
@@ -963,6 +984,12 @@ resolves once that refresh (including its cache write), the ML data fetch and
|
||||
the precache fetches already running have all finished, so the cache directory
|
||||
can then be removed.
|
||||
|
||||
`lib.status().mlDataErrors` lists, as `{ fileID, error }`, each file whose ML
|
||||
data the last ML data fetch got but could not decrypt, gunzip or parse; the
|
||||
`fetchMLData` event with status `"done"` carries the same list. Such ML data is
|
||||
not cached, so the next ML data fetch asks for it again. The reasons are kept in
|
||||
memory only, never in the cache.
|
||||
|
||||
### Default reads vs. fresh reads
|
||||
|
||||
Default reads — `lib.albums`, `lib.photos`, `lib.timeline` — answer
|
||||
@@ -1122,9 +1149,10 @@ photos newest first). `lib.subscribe({ onChange })` delivers a `LibraryChange`
|
||||
`downloadDirectory` (falls back to the library's), `includeOriginals` (default
|
||||
`true`), `includeThumbnails` (default `false`), `onlyAlbumNames`, `verify`
|
||||
(default `false`), `onProgress`, and `lock`. See Backup layout above for the
|
||||
tree it writes. Whether the backup took the lock or its caller did, once the
|
||||
lock is lost the backup rejects with `lost the lock on <dir>: <reason>` before
|
||||
it starts another file.
|
||||
tree it writes, and for the albums `BackupResult` lists in
|
||||
`undecryptableAlbums` and `failedAlbums`. Whether the backup took the lock or
|
||||
its caller did, once the lock is lost the backup rejects with
|
||||
`lost the lock on <dir>: <reason>` before it starts another file.
|
||||
|
||||
### Request pools
|
||||
|
||||
|
||||
@@ -17,9 +17,7 @@ pre-1.0
|
||||
|
||||
# Next Step
|
||||
|
||||
Four bugs are open: https://git.eeqj.de/sneak/quak/issues/197,
|
||||
https://git.eeqj.de/sneak/quak/issues/198,
|
||||
https://git.eeqj.de/sneak/quak/issues/199 and
|
||||
Two bugs are open: https://git.eeqj.de/sneak/quak/issues/199 and
|
||||
https://git.eeqj.de/sneak/quak/issues/205. The check of quak against a live Ente
|
||||
account, https://git.eeqj.de/sneak/quak/issues/187, needs an account only sneak
|
||||
has, so it waits on him.
|
||||
@@ -29,6 +27,16 @@ declares one.
|
||||
|
||||
# Completed Steps
|
||||
|
||||
- 2026-10-16: ML data the server sends that does not decrypt, gunzip or parse is
|
||||
no longer taken for no ML data (issue 198). `fetchMLDataBatch` returns why for
|
||||
each such file beside the payloads. The library keeps those reasons from its
|
||||
last ML data fetch in memory, not in the cache, and gives them in
|
||||
`status().mlDataErrors` and in the `fetchMLData` event with status `"done"`.
|
||||
`quak backup` writes the reason as `mlDataError` in the file's JSON when no ML
|
||||
data is cached for it, and records the file in `failures.json` with an error
|
||||
starting `ML data: `; `backup-metadata` writes `mlDataError` too. Both exit 1.
|
||||
Tests serve ML data encrypted under a key that is not the file's.
|
||||
|
||||
- 2026-10-16: The `~/bin/quak` that `make install` builds retries a refused
|
||||
connection and a name that does not resolve, and the backup records either as
|
||||
transient (issue 206). Bun's `fetch` reports them with codes of its own,
|
||||
@@ -42,6 +50,15 @@ declares one.
|
||||
`unknown`. The shapes were found by running the binary against local
|
||||
listeners.
|
||||
|
||||
- 2026-10-15: An album whose folder or JSON `quak backup` cannot write no longer
|
||||
ends the run before the other albums and `failures.json` are written (issue
|
||||
197). The run prints `FAILED album <name>: <reason>`, lists the album's ID in
|
||||
`failedAlbums` in the result and in `--json`, goes on, and exits 1. An album
|
||||
whose folder would have the name of another album's JSON, as `x.json` beside
|
||||
`x` would, gets its ID added. A folder or link name over 255 bytes is cut
|
||||
short between two characters, by `shortenToBytes` in `src/filename.ts`, and
|
||||
gets the ID.
|
||||
|
||||
- 2026-10-15: A network failure is printed and recorded with its reason, not as
|
||||
just `fetch failed` (issue 196). `describeError` in `src/errors.ts` gives an
|
||||
error's message followed by the message of each error on its `cause` chain,
|
||||
|
||||
+110
-66
@@ -50,7 +50,9 @@
|
||||
// running attempt count, and the last-tried time, and the run continues.
|
||||
// A file the library could not decrypt is recorded there too, unless a copy of
|
||||
// it in another album decrypted; an album it could not decrypt is logged by ID
|
||||
// and listed in `result.undecryptableAlbums`, as its name is not known.
|
||||
// and listed in `result.undecryptableAlbums`, as its name is not known. An
|
||||
// album whose folder or JSON cannot be written is logged by name and
|
||||
// listed in `result.failedAlbums`, and the run goes on to the next album.
|
||||
// `result.failed` — and thus the CLI's exit code — stays non-zero while any
|
||||
// failure remains unresolved and clears once every one succeeds. Each run
|
||||
// reconciles the ledger against the files it attempted, so an entry for a file
|
||||
@@ -83,7 +85,12 @@ import {
|
||||
} from "./crypto/index.js";
|
||||
import { removeLeftoverTempFiles } from "./download/index.js";
|
||||
import { describeError } from "./errors.js";
|
||||
import { sanitizeFileName, withExtension } from "./filename.js";
|
||||
import {
|
||||
MAX_NAME_BYTES,
|
||||
sanitizeFileName,
|
||||
shortenToBytes,
|
||||
withExtension,
|
||||
} from "./filename.js";
|
||||
import {
|
||||
copyAtomic,
|
||||
placeOriginal,
|
||||
@@ -160,6 +167,9 @@ export interface BackupResult {
|
||||
// The IDs of the albums the library could not decrypt, whose files it
|
||||
// could not list; the CLI exits non-zero while there are any.
|
||||
undecryptableAlbums: number[];
|
||||
// The IDs of the albums whose folder or JSON this run could not write;
|
||||
// the CLI exits non-zero while there are any.
|
||||
failedAlbums: number[];
|
||||
}
|
||||
|
||||
// The slice of the library that backup drives. `Library` implements it; a test
|
||||
@@ -193,6 +203,9 @@ export interface BackupLibrary {
|
||||
// A file's cached ML data, as `lib.mldata.forFile()` returns it, or
|
||||
// undefined when none is cached.
|
||||
mlData(fileID: number): Promise<MLData | undefined>;
|
||||
// Why the ML data the last ML data fetch got for a file did not decrypt,
|
||||
// gunzip or parse, or undefined when it did or the fetch got none.
|
||||
mlDataError(fileID: number): unknown;
|
||||
}
|
||||
|
||||
type FailureClass = "transient" | "permanent" | "unknown";
|
||||
@@ -267,31 +280,47 @@ const rebuildSymlink = (linkPath: string, target: string): void => {
|
||||
// The on-disk names for the entries of one directory, in entry order. Each
|
||||
// name is used as is unless another entry would get the same name, ignoring
|
||||
// case (two names that differ only in case are one entry on a case-insensitive
|
||||
// file system); then every entry sharing it gets ` (<id>)`, before the
|
||||
// extension when `beforeExtension` is set. A name with an ID added can match
|
||||
// another entry's own name (`IMG (6).JPG`), so this repeats until no name is
|
||||
// shared. IDs are stable, so the names are too.
|
||||
// file system), or it is too long for a file name; then it gets ` (<id>)`,
|
||||
// right before the extension when `beforeExtension` is set. To fit, the part
|
||||
// before the ID is cut short as far as it must be. When the extension alone
|
||||
// leaves no room for the ID, the extension is cut short and the part before
|
||||
// the ID keeps only the bytes that cut leaves over. With `withJSON`, each
|
||||
// entry also has the JSON file `<name>.json` beside it, as an album does: a
|
||||
// name then leaves room for `.json`, and one that is another entry's JSON name
|
||||
// gets its ID added too.
|
||||
// Two names with different IDs added never match, but one can match another
|
||||
// entry's own name (`IMG (6).JPG`), so this repeats until no name is shared.
|
||||
// IDs are stable, so the names are too.
|
||||
const uniqueNames = (
|
||||
entries: { id: number; name: string }[],
|
||||
beforeExtension: boolean,
|
||||
opts: { beforeExtension?: boolean; withJSON?: boolean },
|
||||
): string[] => {
|
||||
const maxBytes = MAX_NAME_BYTES - (opts.withJSON ? ".json".length : 0);
|
||||
const withID = (id: number, name: string): string => {
|
||||
const ext = beforeExtension ? extname(name) : "";
|
||||
const stem = name.slice(0, name.length - ext.length);
|
||||
return `${stem} (${id})${ext}`;
|
||||
const tag = ` (${id})`;
|
||||
const fullExt = opts.beforeExtension ? extname(name) : "";
|
||||
const stem = name.slice(0, name.length - fullExt.length);
|
||||
const ext = shortenToBytes(fullExt, maxBytes - Buffer.byteLength(tag));
|
||||
const room = maxBytes - Buffer.byteLength(tag + ext);
|
||||
return shortenToBytes(stem, room) + tag + ext;
|
||||
};
|
||||
const names = entries.map((e) => e.name);
|
||||
const suffixed = new Set<number>();
|
||||
for (;;) {
|
||||
const counts = new Map<string, number>();
|
||||
for (const name of names) {
|
||||
const count = (name: string): void => {
|
||||
const key = name.toLowerCase();
|
||||
counts.set(key, (counts.get(key) ?? 0) + 1);
|
||||
};
|
||||
for (const name of names) {
|
||||
count(name);
|
||||
if (opts.withJSON) count(`${name}.json`);
|
||||
}
|
||||
let changed = false;
|
||||
for (const [i, { id, name }] of entries.entries()) {
|
||||
if (suffixed.has(i)) continue;
|
||||
if (counts.get(name.toLowerCase()) === 1) continue;
|
||||
const fits = Buffer.byteLength(name) <= maxBytes;
|
||||
if (fits && counts.get(name.toLowerCase()) === 1) continue;
|
||||
names[i] = withID(id, name);
|
||||
suffixed.add(i);
|
||||
changed = true;
|
||||
@@ -787,19 +816,18 @@ const runLockedBackup = async (
|
||||
// Phase 2: rebuild the derived views from the model. Sidecars first, for
|
||||
// every present original (this repairs stale ones), each with the file's
|
||||
// ML data once an ML data fetch has completed, and its image metadata.
|
||||
// When the fetch fails, a file with no cached ML data gets the reason
|
||||
// instead and is recorded as failed. The next run fetches its ML data
|
||||
// again because none is cached.
|
||||
// When the fetch got a file's ML data but could not decrypt, gunzip or
|
||||
// parse it, or when the fetch failed, a file with no cached ML data gets
|
||||
// the reason instead and is recorded as failed. The next run fetches its
|
||||
// ML data again because none is cached.
|
||||
if (includeOriginals) {
|
||||
let mlDataFailure: unknown;
|
||||
let mlDataError: string | undefined;
|
||||
try {
|
||||
log("Fetching ML data...");
|
||||
await lib.fetchMLData();
|
||||
} catch (err) {
|
||||
mlDataFailure = err;
|
||||
mlDataError = describeError(err);
|
||||
log(`FAILED ML data: ${mlDataError}`);
|
||||
log(`FAILED ML data: ${describeError(err)}`);
|
||||
}
|
||||
for (const file of distinct.values()) {
|
||||
if (lock.lost) throw lock.lost;
|
||||
@@ -816,13 +844,19 @@ const runLockedBackup = async (
|
||||
storedThisRun.has(file.id),
|
||||
);
|
||||
const mlData = await lib.mlData(file.id);
|
||||
if (mlData === undefined && mlDataError !== undefined) {
|
||||
writeSidecar(path, file, { mlDataError }, image);
|
||||
const failure = lib.mlDataError(file.id) ?? mlDataFailure;
|
||||
if (mlData === undefined && failure !== undefined) {
|
||||
writeSidecar(
|
||||
path,
|
||||
file,
|
||||
{ mlDataError: describeError(failure) },
|
||||
image,
|
||||
);
|
||||
recordFailure(
|
||||
file.id,
|
||||
file.metadata.title,
|
||||
collectionName.get(file.collectionID) ?? "",
|
||||
new Error("ML data", { cause: mlDataFailure }),
|
||||
new Error("ML data", { cause: failure }),
|
||||
);
|
||||
} else {
|
||||
writeSidecar(path, file, { mlData }, image);
|
||||
@@ -841,7 +875,7 @@ const runLockedBackup = async (
|
||||
id: c.id,
|
||||
name: sanitizeFileName(c.name, `collection-${c.id}`),
|
||||
})),
|
||||
false,
|
||||
{ withJSON: true },
|
||||
);
|
||||
const albumDirNames = new Map(
|
||||
allCollections.map((c, i) => [c.id, dirNames[i]!]),
|
||||
@@ -857,60 +891,69 @@ const runLockedBackup = async (
|
||||
log(`FAILED removing old album directories: ${describeError(err)}`);
|
||||
}
|
||||
|
||||
// An album whose folder or JSON cannot be written, such as one whose
|
||||
// folder name is taken by a file the user put there, is logged and listed
|
||||
// in the result, and the run goes on to the next album.
|
||||
const failedAlbums: number[] = [];
|
||||
for (const c of collections) {
|
||||
if (lock.lost) throw lock.lost;
|
||||
const colDirName = albumDirNames.get(c.id)!;
|
||||
const colDir = join(collectionsDir, colDirName);
|
||||
mkdirSync(colDir, { recursive: true });
|
||||
|
||||
// Every album links the one original, saved from the file's entry in
|
||||
// `distinct`.
|
||||
const files = filesByCollection.get(c.id) ?? [];
|
||||
const links = files.flatMap((f) =>
|
||||
linksFor(
|
||||
f,
|
||||
storedAtSavePath(downloadDirectory, distinct.get(f.id)!),
|
||||
),
|
||||
);
|
||||
const linkNames = uniqueNames(links, true);
|
||||
try {
|
||||
removeStaleLinks(colDir, new Set(linkNames), downloadDirectory);
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED removing old links in ${c.name}: ${describeError(err)}`,
|
||||
);
|
||||
}
|
||||
const colDirName = albumDirNames.get(c.id)!;
|
||||
const colDir = join(collectionsDir, colDirName);
|
||||
mkdirSync(colDir, { recursive: true });
|
||||
|
||||
const metaFiles = files.map((f) => ({
|
||||
id: f.id,
|
||||
metadata: f.metadata,
|
||||
}));
|
||||
for (const [i, link] of links.entries()) {
|
||||
if (!includeOriginals || link.target === undefined) continue;
|
||||
const linkName = linkNames[i]!;
|
||||
// Every album links the one original, saved from the file's entry
|
||||
// in `distinct`.
|
||||
const files = filesByCollection.get(c.id) ?? [];
|
||||
const links = files.flatMap((f) =>
|
||||
linksFor(
|
||||
f,
|
||||
storedAtSavePath(downloadDirectory, distinct.get(f.id)!),
|
||||
),
|
||||
);
|
||||
const linkNames = uniqueNames(links, { beforeExtension: true });
|
||||
try {
|
||||
rebuildSymlink(
|
||||
join(colDir, linkName),
|
||||
relative(colDir, link.target),
|
||||
);
|
||||
removeStaleLinks(colDir, new Set(linkNames), downloadDirectory);
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED symlink ${c.name}/${linkName}: ${describeError(err)}`,
|
||||
);
|
||||
recordFailure(
|
||||
link.file.id,
|
||||
link.file.metadata.title,
|
||||
c.name,
|
||||
err,
|
||||
`FAILED removing old links in ${c.name}: ${describeError(err)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
writeAlbumJSON(
|
||||
join(collectionsDir, `${colDirName}.json`),
|
||||
c,
|
||||
metaFiles,
|
||||
);
|
||||
const metaFiles = files.map((f) => ({
|
||||
id: f.id,
|
||||
metadata: f.metadata,
|
||||
}));
|
||||
for (const [i, link] of links.entries()) {
|
||||
if (!includeOriginals || link.target === undefined) continue;
|
||||
const linkName = linkNames[i]!;
|
||||
try {
|
||||
rebuildSymlink(
|
||||
join(colDir, linkName),
|
||||
relative(colDir, link.target),
|
||||
);
|
||||
} catch (err) {
|
||||
log(
|
||||
`FAILED symlink ${c.name}/${linkName}: ${describeError(err)}`,
|
||||
);
|
||||
recordFailure(
|
||||
link.file.id,
|
||||
link.file.metadata.title,
|
||||
c.name,
|
||||
err,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
writeAlbumJSON(
|
||||
join(collectionsDir, `${colDirName}.json`),
|
||||
c,
|
||||
metaFiles,
|
||||
);
|
||||
} catch (err) {
|
||||
log(`FAILED album ${c.name}: ${describeError(err)}`);
|
||||
failedAlbums.push(c.id);
|
||||
}
|
||||
}
|
||||
|
||||
// Reconcile the ledger against what this run actually attempted: an entry
|
||||
@@ -936,6 +979,7 @@ const runLockedBackup = async (
|
||||
failed: ledger.size,
|
||||
errors,
|
||||
undecryptableAlbums,
|
||||
failedAlbums,
|
||||
};
|
||||
};
|
||||
|
||||
|
||||
+5
-3
@@ -394,11 +394,11 @@ export const backupMetadataCommand = async (
|
||||
// Refresh first so the dump holds current account state, not what the
|
||||
// cache last held; a failed refresh throws.
|
||||
await lib.fresh();
|
||||
const { failedMLBatches } = await runMetadataBackup(lib, client, dir, {
|
||||
const { mlDataErrors } = await runMetadataBackup(lib, client, dir, {
|
||||
exif: opts.exif || opts.all,
|
||||
onProgress: (msg) => ctx.stderr.write(msg + "\n"),
|
||||
});
|
||||
return failedMLBatches > 0 ? 1 : 0;
|
||||
return mlDataErrors > 0 ? 1 : 0;
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
@@ -470,7 +470,9 @@ export const backupCommand = async (
|
||||
}
|
||||
}
|
||||
|
||||
return result.failed > 0 || result.undecryptableAlbums.length > 0
|
||||
return result.failed > 0 ||
|
||||
result.undecryptableAlbums.length > 0 ||
|
||||
result.failedAlbums.length > 0
|
||||
? 1
|
||||
: 0;
|
||||
} finally {
|
||||
|
||||
+8
-4
@@ -373,13 +373,17 @@ export class Client {
|
||||
}
|
||||
|
||||
// Fetch machine-learning data (face detections + CLIP embeddings) for up
|
||||
// to a batch of files, each decrypted with its own key. One request; the
|
||||
// library batches at `MLDATA_BATCH_SIZE` and schedules each batch through
|
||||
// its metadata request pool.
|
||||
// to a batch of files, each decrypted with its own key, and for each file
|
||||
// whose ML data did not decrypt, gunzip or parse, why not. One request;
|
||||
// the library batches at `MLDATA_BATCH_SIZE` and schedules each batch
|
||||
// through its metadata request pool.
|
||||
async fetchMLData(args: {
|
||||
fileIDs: number[];
|
||||
fileKeys: Map<number, Uint8Array>;
|
||||
}): Promise<Map<number, MLData>> {
|
||||
}): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}> {
|
||||
this.assertLoggedIn();
|
||||
return fetchMLDataBatch(this.api, args.fileIDs, args.fileKeys);
|
||||
}
|
||||
|
||||
+19
-1
@@ -4,7 +4,8 @@
|
||||
// hands us, and quak does not trust the server. Any name taken from them and
|
||||
// used on disk goes through here, so it can only ever name one file inside the
|
||||
// directory the caller chose: never a path, never `..`, never hidden, never a
|
||||
// Windows device name.
|
||||
// Windows device name. Such a name can still be too long for a file name, and
|
||||
// `shortenToBytes` cuts one short.
|
||||
//
|
||||
// A path the user typed (`--out`, `outPath`) is not passed through here: the
|
||||
// caller is trusted, the server is not.
|
||||
@@ -39,3 +40,20 @@ export const safeExtension = (title: string): string => {
|
||||
// `name` with its extension, if it has one, replaced by `ext` (".mov").
|
||||
export const withExtension = (name: string, ext: string): string =>
|
||||
name.slice(0, name.length - extname(name).length) + ext;
|
||||
|
||||
// The longest file name, in bytes, that ext4 and most other file systems allow.
|
||||
export const MAX_NAME_BYTES = 255;
|
||||
|
||||
// `name` cut short to at most `maxBytes` bytes of UTF-8, the encoding a file
|
||||
// name is written to disk in, between two characters, never inside one. A name
|
||||
// that fits comes back unchanged.
|
||||
export const shortenToBytes = (name: string, maxBytes: number): string => {
|
||||
let bytes = 0;
|
||||
let end = 0;
|
||||
for (const character of name) {
|
||||
bytes += Buffer.byteLength(character);
|
||||
if (bytes > maxBytes) break;
|
||||
end += character.length;
|
||||
}
|
||||
return name.slice(0, end);
|
||||
};
|
||||
+44
-9
@@ -89,6 +89,7 @@ export {
|
||||
} from "./content.js";
|
||||
export { type MLDataAPI, type SimilarResult } from "./mlsearch.js";
|
||||
import type { CollectionsPage, FilesPage } from "../client.js";
|
||||
import { describeError } from "../errors.js";
|
||||
import { MLDATA_BATCH_SIZE, type MLData } from "../mldata-fetch.js";
|
||||
import type { Collection, EnteFile } from "../model/types.js";
|
||||
import { runBackup, type BackupOptions, type BackupResult } from "../backup.js";
|
||||
@@ -138,12 +139,16 @@ export interface LibraryClient {
|
||||
sinceTime: number;
|
||||
}): Promise<FilesPage>;
|
||||
// Fetch ML data (face detections + CLIP embeddings) for up to a batch of
|
||||
// files. Optional: a client without it simply disables ML fetching, leaving
|
||||
// the metadata refresh untouched.
|
||||
// files, and for each file whose ML data did not decrypt, gunzip or parse,
|
||||
// why not. Optional: a client without it simply disables ML fetching,
|
||||
// leaving the metadata refresh untouched.
|
||||
fetchMLData?(args: {
|
||||
fileIDs: number[];
|
||||
fileKeys: Map<number, Uint8Array>;
|
||||
}): Promise<Map<number, MLData>>;
|
||||
}): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}>;
|
||||
// The byte source for the on-disk content cache. Optional so a mock client
|
||||
// that only serves metadata still satisfies the interface; when absent (and
|
||||
// no explicit `contentSource` is passed to `open`) the content cache is
|
||||
@@ -154,9 +159,11 @@ export interface LibraryClient {
|
||||
// A progress event for one unit of background work. A metadata "refresh" or an
|
||||
// ML "fetchMLData" pass each fire "started" before their network work and then
|
||||
// exactly one of "done" or "failed"; "failed" carries the error message and
|
||||
// an ML "done" reports how many payloads it stored. The precache fills
|
||||
// ("precacheThumbnails"/"precacheOriginals", #48) fire "started"/"done" around
|
||||
// each sweep that has work, "done" reporting the count newly cached.
|
||||
// an ML "done" reports how many payloads it stored and, in `mlDataErrors`,
|
||||
// each file whose ML data did not decrypt, gunzip or parse, with the reason.
|
||||
// The precache fills ("precacheThumbnails"/"precacheOriginals", #48) fire
|
||||
// "started"/"done" around each sweep that has work, "done" reporting the count
|
||||
// newly cached.
|
||||
export interface RefreshEvent {
|
||||
operation:
|
||||
| "refresh"
|
||||
@@ -166,6 +173,7 @@ export interface RefreshEvent {
|
||||
status: "started" | "done" | "failed";
|
||||
error?: string;
|
||||
fetched?: number;
|
||||
mlDataErrors?: { fileID: number; error: string }[];
|
||||
}
|
||||
|
||||
export type RefreshProgressCallback = (event: RefreshEvent) => void;
|
||||
@@ -225,6 +233,10 @@ export interface LibraryStatus {
|
||||
lastMLFetchAt?: number;
|
||||
// The most recent ML fetch pass's error, set only while it failed.
|
||||
lastMLError?: string;
|
||||
// Each file whose ML data the most recent ML fetch pass got but could not
|
||||
// decrypt, gunzip or parse, with the reason. Such ML data is not cached,
|
||||
// so the next pass asks for it again.
|
||||
mlDataErrors: { fileID: number; error: string }[];
|
||||
// ML payloads stored on disk and CLIP embeddings in the index; undefined
|
||||
// when ML fetching is disabled.
|
||||
mlStored?: number;
|
||||
@@ -292,6 +304,10 @@ export class Library {
|
||||
private lastError?: string;
|
||||
private lastMLFetchAt?: number;
|
||||
private lastMLError?: string;
|
||||
// For each file whose ML data the most recent ML fetch pass got but could
|
||||
// not decrypt, gunzip or parse, why not, by fileID. Kept in RAM only,
|
||||
// never in the cache.
|
||||
private readonly mlDataErrors = new Map<number, unknown>();
|
||||
// The plain-record projection as of the last refresh, and the GUI change
|
||||
// subscribers. A refresh that alters the projection notifies each with the
|
||||
// delta; `lastRecords` is kept current every refresh so a subscriber that
|
||||
@@ -542,6 +558,7 @@ export class Library {
|
||||
lastError: this.lastError,
|
||||
lastMLFetchAt: this.lastMLFetchAt,
|
||||
lastMLError: this.lastMLError,
|
||||
mlDataErrors: this.describeMLDataErrors(),
|
||||
mlStored: ml?.stored,
|
||||
mlIndexed: ml?.indexed,
|
||||
originalsUsedBytes: originals?.usedBytes,
|
||||
@@ -610,6 +627,7 @@ export class Library {
|
||||
contentConcurrency: this.pools.content.concurrency,
|
||||
fetchMLData: () => this.fetchMLDataNow(),
|
||||
mlData: (fileID) => this.mldata.forFile({ fileID }),
|
||||
mlDataError: (fileID) => this.mlDataErrors.get(fileID),
|
||||
},
|
||||
{ ...opts, downloadDirectory },
|
||||
);
|
||||
@@ -836,8 +854,10 @@ export class Library {
|
||||
|
||||
// One ML fetch pass: fetch, decrypt and store the ML data for every file
|
||||
// the store knows about that is not cached (or whose `updationTime` has
|
||||
// advanced), through the metadata pool, and update the CLIP index. A
|
||||
// failure is reported through `status()` and `onProgress`, then thrown.
|
||||
// advanced), through the metadata pool, and update the CLIP index. For a
|
||||
// file whose ML data does not decrypt, gunzip or parse, it keeps the
|
||||
// reason in `mlDataErrors` instead. A failure is reported through
|
||||
// `status()` and `onProgress`, then thrown.
|
||||
private async runMLFetch(): Promise<void> {
|
||||
const mldata = this.mlStore;
|
||||
// Bind so the call keeps the client as its receiver when invoked
|
||||
@@ -847,6 +867,9 @@ export class Library {
|
||||
|
||||
const files = this.uniqueFiles();
|
||||
const needed = mldata.neededFor(files);
|
||||
// A file whose ML data did not decode in the pass before is still
|
||||
// among `needed`, so this pass gives its reason anew.
|
||||
this.mlDataErrors.clear();
|
||||
if (needed.length === 0) return;
|
||||
|
||||
this.emit({ operation: "fetchMLData", status: "started" });
|
||||
@@ -862,10 +885,13 @@ export class Library {
|
||||
for (let i = 0; i < needed.length; i += MLDATA_BATCH_SIZE) {
|
||||
if (this.closed) break;
|
||||
const batch = needed.slice(i, i + MLDATA_BATCH_SIZE);
|
||||
const payloads = await this.pools.metadata.run(
|
||||
const { payloads, errors } = await this.pools.metadata.run(
|
||||
() => fetchMLData({ fileIDs: batch, fileKeys }),
|
||||
{ priority: "background" },
|
||||
);
|
||||
for (const [id, error] of errors) {
|
||||
this.mlDataErrors.set(id, error);
|
||||
}
|
||||
stored += (await mldata.storeFetched(payloads, updation))
|
||||
.stored;
|
||||
}
|
||||
@@ -876,6 +902,7 @@ export class Library {
|
||||
operation: "fetchMLData",
|
||||
status: "done",
|
||||
fetched: stored,
|
||||
mlDataErrors: this.describeMLDataErrors(),
|
||||
});
|
||||
} catch (err) {
|
||||
const error = err instanceof Error ? err.message : String(err);
|
||||
@@ -885,6 +912,14 @@ export class Library {
|
||||
}
|
||||
}
|
||||
|
||||
// `mlDataErrors` as `status()` and the ML "done" event give it.
|
||||
private describeMLDataErrors(): { fileID: number; error: string }[] {
|
||||
return [...this.mlDataErrors].map(([fileID, err]) => ({
|
||||
fileID,
|
||||
error: describeError(err),
|
||||
}));
|
||||
}
|
||||
|
||||
// The distinct files the store holds, one entry per fileID (a file in
|
||||
// several collections shares its ML data), each carrying the key and the
|
||||
// newest `updationTime` seen across its memberships.
|
||||
|
||||
+14
-8
@@ -2,6 +2,7 @@ import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { join } from "node:path";
|
||||
import * as jpeg from "jpeg-js";
|
||||
import type { Client } from "./client.js";
|
||||
import { describeError } from "./errors.js";
|
||||
import { readExifTags } from "./exif.js";
|
||||
import type { Library, Photo } from "./library/index.js";
|
||||
import { sanitizeFileName } from "./filename.js";
|
||||
@@ -91,14 +92,15 @@ const extractExif = async (
|
||||
// of plain JSON: account, per-collection, and per-file records including the
|
||||
// private and public magic metadata and (by default) the ML data. Collections
|
||||
// and files are enumerated from the library's cache, which the caller refreshes
|
||||
// first. Returns how many ML data requests failed; their files are still
|
||||
// first. Returns how many files have no ML data because their ML data request
|
||||
// failed or their ML data did not decrypt, gunzip or parse; they are still
|
||||
// written, with `mlDataError` in place of `mlData`.
|
||||
export const runMetadataBackup = async (
|
||||
lib: Library,
|
||||
client: Client,
|
||||
outDir: string,
|
||||
opts?: MetadataBackupOptions,
|
||||
): Promise<{ failedMLBatches: number }> => {
|
||||
): Promise<{ mlDataErrors: number }> => {
|
||||
const log = opts?.onProgress ?? (() => {});
|
||||
const wantExif = opts?.exif ?? false;
|
||||
|
||||
@@ -164,24 +166,28 @@ export const runMetadataBackup = async (
|
||||
}
|
||||
|
||||
// One failed request (retries exhausted) must not end the dump: its files
|
||||
// get the reason in `mlDataError` and the other batches go on.
|
||||
// get the reason in `mlDataError` and the other batches go on. So does a
|
||||
// file whose ML data does not decrypt, gunzip or parse.
|
||||
log("Fetching ML data (face detections, CLIP embeddings)...");
|
||||
const mlDataMap = new Map<number, MLData>();
|
||||
const mlDataErrors = new Map<number, string>();
|
||||
let failedMLBatches = 0;
|
||||
const fileIDs = [...fileKeys.keys()];
|
||||
for (let i = 0; i < fileIDs.length; i += MLDATA_BATCH_SIZE) {
|
||||
const batch = fileIDs.slice(i, i + MLDATA_BATCH_SIZE);
|
||||
try {
|
||||
const result = await fetchMLDataBatch(
|
||||
const { payloads, errors } = await fetchMLDataBatch(
|
||||
client.getApiClient(),
|
||||
batch,
|
||||
fileKeys,
|
||||
);
|
||||
for (const [id, payload] of result) mlDataMap.set(id, payload);
|
||||
for (const [id, payload] of payloads) mlDataMap.set(id, payload);
|
||||
for (const [id, err] of errors) {
|
||||
const reason = describeError(err);
|
||||
log(`ML data of file ${id} could not be read: ${reason}`);
|
||||
mlDataErrors.set(id, reason);
|
||||
}
|
||||
} catch (err) {
|
||||
const reason = err instanceof Error ? err.message : String(err);
|
||||
failedMLBatches++;
|
||||
log(
|
||||
`ML data request for ${batch.length} file(s) failed: ${reason}`,
|
||||
);
|
||||
@@ -229,5 +235,5 @@ export const runMetadataBackup = async (
|
||||
}
|
||||
|
||||
log("Metadata backup complete.");
|
||||
return { failedMLBatches };
|
||||
return { mlDataErrors: mlDataErrors.size };
|
||||
};
|
||||
+25
-25
@@ -28,43 +28,43 @@ interface RawRemoteFileData {
|
||||
updatedAt?: number;
|
||||
}
|
||||
|
||||
// Decrypt one entry with its file key and gunzip the JSON payload. Returns
|
||||
// undefined when the key is unknown or the entry does not decrypt/parse, so one
|
||||
// corrupt file never fails a whole batch.
|
||||
const decodeEntry = (
|
||||
entry: RawRemoteFileData,
|
||||
key: Uint8Array | undefined,
|
||||
): MLData | undefined => {
|
||||
if (!key) return undefined;
|
||||
try {
|
||||
const decrypted = decryptBlob(
|
||||
fromBase64(entry.encryptedData),
|
||||
fromBase64(entry.decryptionHeader),
|
||||
key,
|
||||
);
|
||||
const json = gunzipSync(Buffer.from(decrypted)).toString("utf-8");
|
||||
return JSON.parse(json) as MLData;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
// Decrypt one entry with its file key and gunzip the JSON payload. Throws why
|
||||
// when the entry does not decrypt, gunzip or parse.
|
||||
const decodeEntry = (entry: RawRemoteFileData, key: Uint8Array): MLData => {
|
||||
const decrypted = decryptBlob(
|
||||
fromBase64(entry.encryptedData),
|
||||
fromBase64(entry.decryptionHeader),
|
||||
key,
|
||||
);
|
||||
const json = gunzipSync(Buffer.from(decrypted)).toString("utf-8");
|
||||
return JSON.parse(json) as MLData;
|
||||
};
|
||||
|
||||
// Fetch ML data for up to `MLDATA_BATCH_SIZE` ids in a single request. This is
|
||||
// the unit the request pools schedule; callers with more ids split them into
|
||||
// batches and submit each batch to the pool.
|
||||
// batches and submit each batch to the pool. Returns the payload of each file
|
||||
// whose entry decodes and, in `errors`, why not for each file whose entry does
|
||||
// not, so one corrupt file never fails a whole batch. An entry for a file
|
||||
// whose key is not known, which quak did not ask for, is skipped.
|
||||
export const fetchMLDataBatch = async (
|
||||
api: ApiClient,
|
||||
fileIDs: number[],
|
||||
fileKeys: Map<number, Uint8Array>,
|
||||
): Promise<Map<number, MLData>> => {
|
||||
): Promise<{ payloads: Map<number, MLData>; errors: Map<number, unknown> }> => {
|
||||
const { data } = await api.postJSON<{ data: RawRemoteFileData[] }>(
|
||||
"/files/data/fetch",
|
||||
{ type: "mldata", fileIDs },
|
||||
);
|
||||
const result = new Map<number, MLData>();
|
||||
const payloads = new Map<number, MLData>();
|
||||
const errors = new Map<number, unknown>();
|
||||
for (const entry of data ?? []) {
|
||||
const payload = decodeEntry(entry, fileKeys.get(entry.fileID));
|
||||
if (payload) result.set(entry.fileID, payload);
|
||||
const key = fileKeys.get(entry.fileID);
|
||||
if (!key) continue;
|
||||
try {
|
||||
payloads.set(entry.fileID, decodeEntry(entry, key));
|
||||
} catch (err) {
|
||||
errors.set(entry.fileID, err);
|
||||
}
|
||||
}
|
||||
return result;
|
||||
return { payloads, errors };
|
||||
};
|
||||
@@ -274,17 +274,17 @@ afterAll(() => {
|
||||
|
||||
// Run `quak backup <dir>` as `bin/quak.ts` runs it, sending its requests
|
||||
// through `fetch`, and return its exit code and everything it printed on
|
||||
// stderr. Its requests are retried as those of `quak backup` are, except that
|
||||
// a retry does not wait. A download is given up after `downloadTimeoutMs` with
|
||||
// no bytes received, 60 seconds unless given. Its cache is beside `dir`, so
|
||||
// each run of one directory finds the cache the run before it left, as cron's
|
||||
// runs find the user's.
|
||||
// stderr and on stdout. Its requests are retried as those of `quak backup`
|
||||
// are, except that a retry does not wait. A download is given up after
|
||||
// `downloadTimeoutMs` with no bytes received, 60 seconds unless given. Its
|
||||
// cache is beside `dir`, so each run of one directory finds the cache the run
|
||||
// before it left, as cron's runs find the user's.
|
||||
const backup = async (
|
||||
dir: string,
|
||||
opts: { verify?: boolean } = {},
|
||||
opts: { verify?: boolean; json?: boolean } = {},
|
||||
fetch = server,
|
||||
downloadTimeoutMs?: number,
|
||||
): Promise<{ code: number; stderr: string }> => {
|
||||
): Promise<{ code: number; stderr: string; stdout: string }> => {
|
||||
let printed = "";
|
||||
const stderr = new Writable({
|
||||
write(chunk: Buffer, _encoding, done) {
|
||||
@@ -292,8 +292,10 @@ const backup = async (
|
||||
done();
|
||||
},
|
||||
});
|
||||
let output = "";
|
||||
const stdout = new Writable({
|
||||
write(_chunk, _encoding, done) {
|
||||
write(chunk: Buffer, _encoding, done) {
|
||||
output += chunk.toString();
|
||||
done();
|
||||
},
|
||||
});
|
||||
@@ -319,7 +321,7 @@ const backup = async (
|
||||
const code = await new Promise<number>((resolve) => {
|
||||
void run(backupCommand(ctx, dir, opts), stdout, stderr, resolve);
|
||||
});
|
||||
return { code, stderr: printed };
|
||||
return { code, stderr: printed, stdout: output };
|
||||
};
|
||||
|
||||
// The ID of the file whose original a request downloads, or undefined for any
|
||||
@@ -630,6 +632,40 @@ describe("quak backup", () => {
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
// The user has put a file in collections/ with the name of the Trips
|
||||
// folder, so quak cannot make that folder. It says so on one line, writes
|
||||
// the two albums after it, and exits 1, and --json lists the album's ID
|
||||
// in `failedAlbums`. No file failed, so there is no failures.json.
|
||||
it("exits 1 when an album's folder cannot be made, writes the other albums, and lists it in --json", async () => {
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
mkdirSync(join(dir, "collections"));
|
||||
writeFileSync(join(dir, "collections", "Trips"), "mine");
|
||||
|
||||
const first = await backup(dir);
|
||||
const { code, stdout } = await backup(dir, { json: true });
|
||||
|
||||
expect(first.code).toBe(1);
|
||||
expect(first.stderr).toMatch(/^FAILED album Trips: EEXIST: /m);
|
||||
expect(code).toBe(1);
|
||||
expect(JSON.parse(stdout)).toMatchObject({
|
||||
failed: 0,
|
||||
undecryptableAlbums: [],
|
||||
failedAlbums: [10],
|
||||
});
|
||||
const collections = join(dir, "collections");
|
||||
expect(readJSON(join(collections, "Family.json")).id).toBe(20);
|
||||
expect(readdirSync(join(collections, "Family")).sort()).toEqual([
|
||||
"IMG_0002.heic",
|
||||
"IMG_0002.mov",
|
||||
"beach.jpg",
|
||||
]);
|
||||
expect(readJSON(join(collections, "Friend's Wedding.json")).id).toBe(
|
||||
30,
|
||||
);
|
||||
expect(existsSync(join(collections, "Trips.json"))).toBe(false);
|
||||
expect(existsSync(join(dir, "failures.json"))).toBe(false);
|
||||
});
|
||||
|
||||
// The server no longer accepts the saved session's token, as after
|
||||
// `quak logout` elsewhere, and answers every request with 401. The run
|
||||
// stops at the refresh it starts with, says to log in again, and exits 3.
|
||||
@@ -841,6 +877,56 @@ describe("quak backup", () => {
|
||||
]);
|
||||
});
|
||||
|
||||
// The server sends beach.jpg's ML data encrypted under a key that is not
|
||||
// the file's, so quak cannot decrypt it. The file's JSON has the reason
|
||||
// in place of the ML data, and the file goes into `failures.json` with
|
||||
// it, so the exit code is 1. The other two files have no ML data, which
|
||||
// is no failure.
|
||||
it("records a file whose ML data does not decrypt in its JSON and in failures.json, and exits 1", async () => {
|
||||
const otherKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const mlData = encryptStreamBlob(
|
||||
gzipSync(JSON.stringify(ML_DATA)),
|
||||
otherKey,
|
||||
);
|
||||
const fetch: typeof globalThis.fetch = async (input, init) => {
|
||||
if (new URL(String(input)).pathname !== "/files/data/fetch") {
|
||||
return server(input, init);
|
||||
}
|
||||
const data = [
|
||||
{
|
||||
fileID: BEACH.id,
|
||||
encryptedData: toBase64(mlData.ciphertext),
|
||||
decryptionHeader: toBase64(mlData.header),
|
||||
},
|
||||
];
|
||||
return new Response(JSON.stringify({ data }), {
|
||||
headers: { "content-type": "application/json" },
|
||||
});
|
||||
};
|
||||
const dir = mkdtempSync(join(root, "backup-"));
|
||||
|
||||
const { code, stderr } = await backup(dir, {}, fetch);
|
||||
|
||||
expect(code).toBe(1);
|
||||
const reason = "secretstream chunk authentication failed";
|
||||
const beachJSON = readJSON(join(dir, DAY, "2024-03-01.100.json"));
|
||||
expect(beachJSON).not.toHaveProperty("mlData");
|
||||
expect(beachJSON.mlDataError).toBe(reason);
|
||||
const failures = readJSON(join(dir, "failures.json"));
|
||||
expect(Object.keys(failures.files)).toEqual(["100"]);
|
||||
expect(failures.files["100"]).toMatchObject({
|
||||
fileID: 100,
|
||||
title: "beach.jpg",
|
||||
error: `ML data: ${reason}`,
|
||||
});
|
||||
expect(stderr).toContain(`beach.jpg (id 100): ML data: ${reason}\n`);
|
||||
for (const fileID of [200, 300]) {
|
||||
const json = readJSON(join(dir, DAY, `2024-03-01.${fileID}.json`));
|
||||
expect(json).not.toHaveProperty("mlData");
|
||||
expect(json).not.toHaveProperty("mlDataError");
|
||||
}
|
||||
});
|
||||
|
||||
// Something changed beach.jpg on disk after it was backed up. A run with
|
||||
// --verify hashes every stored original, finds that one no longer matches
|
||||
// the hash Ente recorded, says so, and downloads it again; it put back the
|
||||
|
||||
+257
-7
@@ -938,13 +938,14 @@ const ML_PAYLOAD = { face: { faces: [] }, clip: { embedding: [0.5, 0.25] } };
|
||||
|
||||
class MLClient extends MockClient {
|
||||
mlError?: string;
|
||||
async fetchMLData(args: {
|
||||
fileIDs: number[];
|
||||
}): Promise<Map<number, MLData>> {
|
||||
async fetchMLData(args: { fileIDs: number[] }): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}> {
|
||||
if (this.mlError) throw new Error(this.mlError);
|
||||
const result = new Map<number, MLData>();
|
||||
if (args.fileIDs.includes(100)) result.set(100, ML_PAYLOAD);
|
||||
return result;
|
||||
const payloads = new Map<number, MLData>();
|
||||
if (args.fileIDs.includes(100)) payloads.set(100, ML_PAYLOAD);
|
||||
return { payloads, errors: new Map() };
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1498,13 +1499,15 @@ describe("backup album folders", () => {
|
||||
files: EnteFile[];
|
||||
}
|
||||
|
||||
const libraryOf = (albums: Album[]): BackupLibrary => ({
|
||||
// The original of the file `notFound`, if given, always fails to download.
|
||||
const libraryOf = (albums: Album[], notFound?: number): BackupLibrary => ({
|
||||
refresh: async () => {},
|
||||
listCollections: () => albums.map((a) => a.collection),
|
||||
listFiles: (id) =>
|
||||
albums.find((a) => a.collection.id === id)?.files ?? [],
|
||||
undecryptable: () => [],
|
||||
original: async (fileID) => {
|
||||
if (fileID === notFound) throw new Error("HTTP 404");
|
||||
const path = join(root, `source-${fileID}`);
|
||||
writeFileSync(path, `original ${fileID}`);
|
||||
return { path };
|
||||
@@ -1515,6 +1518,7 @@ describe("backup album folders", () => {
|
||||
contentConcurrency: 5,
|
||||
fetchMLData: async () => {},
|
||||
mlData: async () => undefined,
|
||||
mlDataError: () => undefined,
|
||||
whoami: () => ({ email: "backup@example.com", userID: USER_ID }),
|
||||
});
|
||||
|
||||
@@ -1600,6 +1604,252 @@ describe("backup album folders", () => {
|
||||
expect(albumID(outDir, "Trip (11) (12).json")).toBe(12);
|
||||
});
|
||||
|
||||
// The next three tests end with the album "ok" (14), whose one file
|
||||
// always fails to download, so each run has a failure to record in
|
||||
// failures.json after the album folders are written.
|
||||
const ok: Album = {
|
||||
collection: collection(14, "ok"),
|
||||
files: [file(5, 14, "e.jpg")],
|
||||
};
|
||||
|
||||
// The folder of an album named "x.json" would be the JSON file of the
|
||||
// album "x", and once ended every run there. Two runs, as on two nights,
|
||||
// write the same tree, and the failed file's attempt count goes up.
|
||||
it("adds the ID to an album whose folder would have another album's JSON name", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf(
|
||||
[
|
||||
{
|
||||
collection: collection(10, "x"),
|
||||
files: [file(1, 10, "a.jpg")],
|
||||
},
|
||||
{
|
||||
collection: collection(11, "x.json"),
|
||||
files: [file(2, 11, "b.jpg")],
|
||||
},
|
||||
{
|
||||
collection: collection(12, "Trip"),
|
||||
files: [file(3, 12, "c.jpg")],
|
||||
},
|
||||
{
|
||||
collection: collection(13, "TRIP.json"),
|
||||
files: [file(4, 13, "d.jpg")],
|
||||
},
|
||||
ok,
|
||||
],
|
||||
5,
|
||||
);
|
||||
|
||||
await runBackup(lib, { downloadDirectory: outDir });
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
expect(result.failedAlbums).toEqual([]);
|
||||
expect(tree(outDir)).toEqual([
|
||||
"TRIP.json (13)/",
|
||||
`TRIP.json (13)/d.jpg -> ${linkTo("4.jpg")}`,
|
||||
"TRIP.json (13).json",
|
||||
"Trip/",
|
||||
`Trip/c.jpg -> ${linkTo("3.jpg")}`,
|
||||
"Trip.json",
|
||||
"ok/",
|
||||
"ok.json",
|
||||
"x/",
|
||||
`x/a.jpg -> ${linkTo("1.jpg")}`,
|
||||
"x.json",
|
||||
"x.json (11)/",
|
||||
`x.json (11)/b.jpg -> ${linkTo("2.jpg")}`,
|
||||
"x.json (11).json",
|
||||
]);
|
||||
expect(albumID(outDir, "x.json")).toBe(10);
|
||||
expect(albumID(outDir, "x.json (11).json")).toBe(11);
|
||||
expect(albumID(outDir, "TRIP.json (13).json")).toBe(13);
|
||||
expect(albumID(outDir, "ok.json")).toBe(14);
|
||||
expect(result.failed).toBe(1);
|
||||
expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 2 });
|
||||
});
|
||||
|
||||
// A name over 255 bytes is refused as a file name, and an album folder's
|
||||
// name leaves room for `.json` in its JSON's. "é" takes two bytes, and a
|
||||
// name is cut between two characters.
|
||||
it("cuts an album name and a title too long for a file name short and adds the ID", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf(
|
||||
[
|
||||
{
|
||||
collection: collection(10, "L".repeat(300)),
|
||||
files: [file(1, 10, `${"é".repeat(150)}.jpg`)],
|
||||
},
|
||||
ok,
|
||||
],
|
||||
5,
|
||||
);
|
||||
|
||||
await runBackup(lib, { downloadDirectory: outDir });
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
const folder = `${"L".repeat(245)} (10)`;
|
||||
const link = `${"é".repeat(123)} (1).jpg`;
|
||||
expect(Buffer.byteLength(`${folder}.json`)).toBe(255);
|
||||
expect(Buffer.byteLength(link)).toBe(254);
|
||||
expect(result.failedAlbums).toEqual([]);
|
||||
expect(tree(outDir)).toEqual([
|
||||
`${folder}/`,
|
||||
`${folder}/${link} -> ${linkTo("1.jpg")}`,
|
||||
`${folder}.json`,
|
||||
"ok/",
|
||||
"ok.json",
|
||||
]);
|
||||
expect(albumID(outDir, `${folder}.json`)).toBe(10);
|
||||
expect(albumID(outDir, "ok.json")).toBe(14);
|
||||
expect(result.failed).toBe(1);
|
||||
expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 2 });
|
||||
});
|
||||
|
||||
// The user has put a file in collections/ with the name of the album
|
||||
// Trip's folder, so the folder cannot be made.
|
||||
it("goes on to the next album when an album's folder cannot be made, and lists it as failed", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf(
|
||||
[
|
||||
{
|
||||
collection: collection(10, "Trip"),
|
||||
files: [file(1, 10, "a.jpg")],
|
||||
},
|
||||
ok,
|
||||
],
|
||||
5,
|
||||
);
|
||||
mkdirSync(join(outDir, "collections"), { recursive: true });
|
||||
writeFileSync(join(outDir, "collections", "Trip"), "mine");
|
||||
const logged: string[] = [];
|
||||
|
||||
const result = await runBackup(lib, {
|
||||
downloadDirectory: outDir,
|
||||
onProgress: (message) => logged.push(message),
|
||||
});
|
||||
|
||||
expect(result.failedAlbums).toEqual([10]);
|
||||
expect(logged).toContainEqual(
|
||||
expect.stringMatching(/^FAILED album Trip: EEXIST: /),
|
||||
);
|
||||
expect(tree(outDir)).toEqual(["Trip", "ok/", "ok.json"]);
|
||||
expect(albumID(outDir, "ok.json")).toBe(14);
|
||||
expect(existsSync(saved(outDir, "1.jpg"))).toBe(true);
|
||||
expect(result.failed).toBe(1);
|
||||
expect(readLedger(outDir).files["5"]).toMatchObject({ attempts: 1 });
|
||||
});
|
||||
|
||||
// The extensions of these titles leave no room for the ID, so each is cut
|
||||
// short and the name before the ID keeps only the bytes that cut leaves
|
||||
// over. File 1's extension is 303 bytes and is cut to 251, which leaves
|
||||
// none. File 2's is 271 bytes and is cut to 250, since "€" takes three
|
||||
// bytes, which leaves one, for the "a". Neither extension is only letters
|
||||
// and digits, so both originals are saved with `.bin`, a name that fits.
|
||||
it("cuts the extension of a title short when it leaves no room for the ID", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf([
|
||||
{
|
||||
collection: collection(10, "Trip"),
|
||||
files: [
|
||||
file(1, 10, `a.${"L".repeat(300)}-x`),
|
||||
file(2, 10, `a.${"€".repeat(90)}`),
|
||||
],
|
||||
},
|
||||
]);
|
||||
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
const link = ` (1).${"L".repeat(250)}`;
|
||||
const link2 = `a (2).${"€".repeat(83)}`;
|
||||
expect(Buffer.byteLength(link)).toBe(255);
|
||||
expect(Buffer.byteLength(link2)).toBe(255);
|
||||
expect(result.failed).toBe(0);
|
||||
expect(tree(outDir)).toEqual([
|
||||
"Trip/",
|
||||
`Trip/${link} -> ${linkTo("1.bin")}`,
|
||||
`Trip/${link2} -> ${linkTo("2.bin")}`,
|
||||
"Trip.json",
|
||||
]);
|
||||
});
|
||||
|
||||
// Files 1 and 3 share a title, so each gets its ID before the
|
||||
// extension. File 2's title is too long, and its extension leaves no
|
||||
// room for the ID. Were the ID put at the end of file 2's title cut
|
||||
// short, its link would have file 1's name, and the link made second
|
||||
// would replace the first.
|
||||
it("keeps names unique when a title's extension leaves no room for the ID", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const shared = `x.${"Q".repeat(245)} (2)`;
|
||||
const lib = libraryOf([
|
||||
{
|
||||
collection: collection(10, "Trip"),
|
||||
files: [
|
||||
file(1, 10, shared),
|
||||
file(2, 10, `x (1).${"Q".repeat(250)}-`),
|
||||
file(3, 10, shared),
|
||||
],
|
||||
},
|
||||
]);
|
||||
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
expect(result.failed).toBe(0);
|
||||
expect(tree(outDir)).toEqual([
|
||||
"Trip/",
|
||||
`Trip/ (2).${"Q".repeat(250)} -> ${linkTo("2.bin")}`,
|
||||
`Trip/x (1).${"Q".repeat(245)} (2) -> ${linkTo("1.bin")}`,
|
||||
`Trip/x (3).${"Q".repeat(245)} (2) -> ${linkTo("3.bin")}`,
|
||||
"Trip.json",
|
||||
]);
|
||||
});
|
||||
|
||||
// A title of 255 bytes is the longest file name allowed, and an album
|
||||
// name of 250 bytes the longest whose JSON name fits; both are kept as
|
||||
// they are. A name one byte longer gets the ID and is cut short. "é"
|
||||
// takes two bytes, and in both names the room left ends inside one, so
|
||||
// the cut falls before it, one byte short of the room.
|
||||
it("keeps a name at the byte limit and adds the ID to one a byte over it", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const album = "A".repeat(250);
|
||||
const title = `${"t".repeat(251)}.jpg`;
|
||||
const longAlbum = `${"é".repeat(125)}B`;
|
||||
const longTitle = `${"é".repeat(126)}.jpg`;
|
||||
const lib = libraryOf([
|
||||
{
|
||||
collection: collection(10, album),
|
||||
files: [file(1, 10, title), file(2, 10, longTitle)],
|
||||
},
|
||||
{
|
||||
collection: collection(11, longAlbum),
|
||||
files: [file(3, 11, "c.jpg")],
|
||||
},
|
||||
]);
|
||||
|
||||
const result = await runBackup(lib, { downloadDirectory: outDir });
|
||||
|
||||
const folder = `${"é".repeat(122)} (11)`;
|
||||
const link = `${"é".repeat(123)} (2).jpg`;
|
||||
expect(Buffer.byteLength(`${album}.json`)).toBe(255);
|
||||
expect(Buffer.byteLength(title)).toBe(255);
|
||||
expect(Buffer.byteLength(longAlbum)).toBe(251);
|
||||
expect(Buffer.byteLength(longTitle)).toBe(256);
|
||||
expect(Buffer.byteLength(`${folder}.json`)).toBe(254);
|
||||
expect(Buffer.byteLength(link)).toBe(254);
|
||||
expect(result.failed).toBe(0);
|
||||
expect(result.failedAlbums).toEqual([]);
|
||||
expect(tree(outDir)).toEqual([
|
||||
`${album}/`,
|
||||
`${album}/${title} -> ${linkTo("1.jpg")}`,
|
||||
`${album}/${link} -> ${linkTo("2.jpg")}`,
|
||||
`${album}.json`,
|
||||
`${folder}/`,
|
||||
`${folder}/c.jpg -> ${linkTo("3.jpg")}`,
|
||||
`${folder}.json`,
|
||||
]);
|
||||
expect(albumID(outDir, `${album}.json`)).toBe(10);
|
||||
expect(albumID(outDir, `${folder}.json`)).toBe(11);
|
||||
});
|
||||
|
||||
it("changes nothing on a second run over an unchanged account", async () => {
|
||||
const outDir = join(root, "backup");
|
||||
const lib = libraryOf([
|
||||
|
||||
@@ -489,15 +489,15 @@ describe("quak backup-metadata", () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe("quak backup-metadata when an ML data request fails", () => {
|
||||
// Run the CLI command against the mock and return its exit code, stderr
|
||||
// and output directory.
|
||||
const runCommand = async (failMLDataFor?: number) => {
|
||||
describe("quak backup-metadata when a file's ML data cannot be read", () => {
|
||||
// Run the CLI command against `fetch`, the mock unless given, and return
|
||||
// its exit code, stderr and output directory.
|
||||
const runCommand = async (fetch = buildMetaFetch(mock)) => {
|
||||
const client = await Client.login({
|
||||
email: TEST_EMAIL,
|
||||
password: TEST_PASSWORD,
|
||||
apiOptions: {
|
||||
fetch: buildMetaFetch(mock, failMLDataFor),
|
||||
fetch,
|
||||
retry: { sleep: async () => {} },
|
||||
},
|
||||
});
|
||||
@@ -518,7 +518,9 @@ describe("quak backup-metadata when an ML data request fails", () => {
|
||||
};
|
||||
|
||||
it("writes every file, marks the failed batch's files, and exits 1", async () => {
|
||||
const { code, stderr, outDir } = await runCommand(200);
|
||||
const { code, stderr, outDir } = await runCommand(
|
||||
buildMetaFetch(mock, 200),
|
||||
);
|
||||
|
||||
expect(code).toBe(1);
|
||||
expect(stderr).toContain("ML data request for 1 file(s) failed");
|
||||
@@ -543,6 +545,49 @@ describe("quak backup-metadata when an ML data request fails", () => {
|
||||
expect(failed.mlDataError).toContain("500");
|
||||
});
|
||||
|
||||
// The server sends ML data for diagram.png encrypted under a key that is
|
||||
// not the file's, so quak cannot decrypt it.
|
||||
it("writes the reason for ML data that does not decrypt, and exits 1", async () => {
|
||||
const otherKey = sodium.crypto_secretstream_xchacha20poly1305_keygen();
|
||||
const { header, ciphertext } = encryptBlob(
|
||||
new Uint8Array(gzipSync(Buffer.from("{}"))),
|
||||
otherKey,
|
||||
);
|
||||
const { code, stderr, outDir } = await runCommand(
|
||||
buildMetaFetch({
|
||||
...mock,
|
||||
encryptedMLData: {
|
||||
...mock.encryptedMLData,
|
||||
200: {
|
||||
encryptedData: toBase64(ciphertext),
|
||||
decryptionHeader: toBase64(header),
|
||||
},
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
expect(code).toBe(1);
|
||||
const reason = "secretstream chunk authentication failed";
|
||||
expect(stderr).toContain(
|
||||
`ML data of file 200 could not be read: ${reason}\n`,
|
||||
);
|
||||
const failed = JSON.parse(
|
||||
readFileSync(
|
||||
join(outDir, "collections", "20-__Work", "200.json"),
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
expect(failed.mlData).toBeUndefined();
|
||||
expect(failed.mlDataError).toBe(reason);
|
||||
const ok = JSON.parse(
|
||||
readFileSync(
|
||||
join(outDir, "collections", "10-Vacation", "100.json"),
|
||||
"utf-8",
|
||||
),
|
||||
);
|
||||
expect(ok.mlData.clip.embedding).toEqual([0.5, 0.6, 0.7]);
|
||||
});
|
||||
|
||||
it("exits 0 when every ML data request succeeds", async () => {
|
||||
const { code } = await runCommand();
|
||||
expect(code).toBe(0);
|
||||
|
||||
@@ -14,7 +14,11 @@
|
||||
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
import { safeExtension, sanitizeFileName } from "../../src/filename.js";
|
||||
import {
|
||||
safeExtension,
|
||||
sanitizeFileName,
|
||||
shortenToBytes,
|
||||
} from "../../src/filename.js";
|
||||
|
||||
const FALLBACK = "file-42";
|
||||
|
||||
@@ -91,3 +95,26 @@ describe("safeExtension", () => {
|
||||
expect(safeExtension("x.jpg\0")).toBe(".bin");
|
||||
});
|
||||
});
|
||||
|
||||
describe("shortenToBytes", () => {
|
||||
it("returns a name that fits unchanged", () => {
|
||||
expect(shortenToBytes("beach", 5)).toBe("beach");
|
||||
expect(shortenToBytes("café", 5)).toBe("café");
|
||||
expect(shortenToBytes("", 0)).toBe("");
|
||||
});
|
||||
|
||||
it("cuts a name to the number of bytes given", () => {
|
||||
expect(shortenToBytes("L".repeat(300), 255)).toBe("L".repeat(255));
|
||||
});
|
||||
|
||||
// In UTF-8 "é" takes two bytes and an emoji four, which a JavaScript
|
||||
// string holds as two halves.
|
||||
it("cuts between two characters, never inside one", () => {
|
||||
expect(shortenToBytes("ééé", 5)).toBe("éé");
|
||||
expect(shortenToBytes("a\u{1F600}b", 4)).toBe("a");
|
||||
expect(shortenToBytes("a\u{1F600}b", 5)).toBe("a\u{1F600}");
|
||||
expect(Buffer.byteLength(shortenToBytes("é".repeat(200), 255))).toBe(
|
||||
254,
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -12,9 +12,10 @@
|
||||
*
|
||||
* 2. `Library` wiring: after each refresh the library fetches ML data through
|
||||
* the metadata pool for every known file not yet cached, is incremental on
|
||||
* later refreshes, and refetches a file whose `updationTime` advanced.
|
||||
* Opening a cache directory written by another account starts empty,
|
||||
* its ML data included (issue #104).
|
||||
* later refreshes, and refetches a file whose `updationTime` advanced. A
|
||||
* file whose ML data did not decode is reported with the reason, and
|
||||
* asked for again. Opening a cache directory written by another account
|
||||
* starts empty, its ML data included (issue #104).
|
||||
*
|
||||
* Embedding values are chosen to be exactly representable as float32 so the
|
||||
* round-trip through `clip.f32` compares equal.
|
||||
@@ -26,7 +27,7 @@ import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { MLDataStore } from "../../src/library/mldata.js";
|
||||
import { Library } from "../../src/library/index.js";
|
||||
import { Library, type RefreshEvent } from "../../src/library/index.js";
|
||||
import type { CollectionsPage, FilesPage } from "../../src/client.js";
|
||||
import type { MLData } from "../../src/mldata-fetch.js";
|
||||
import type { Collection, EnteFile } from "../../src/model/types.js";
|
||||
@@ -256,12 +257,14 @@ const file = (
|
||||
|
||||
// A mock client that serves scripted collection/file pages and per-file ML
|
||||
// payloads, recording every ML fetch request so incremental behaviour is
|
||||
// provable.
|
||||
// provable. A file in `mlErrorByFile` has ML data that did not decode, for
|
||||
// the reason given there.
|
||||
class MLMockClient {
|
||||
userID = USER_ID;
|
||||
collectionsQueue: CollectionsPage[] = [];
|
||||
filesByCollection = new Map<number, FilesPage[]>();
|
||||
mlByFile = new Map<number, MLData>();
|
||||
mlErrorByFile = new Map<number, Error>();
|
||||
mlFetchCalls: number[][] = [];
|
||||
|
||||
whoami(): { email: string; userID: number } {
|
||||
@@ -298,14 +301,20 @@ class MLMockClient {
|
||||
async fetchMLData(args: {
|
||||
fileIDs: number[];
|
||||
fileKeys: Map<number, Uint8Array>;
|
||||
}): Promise<Map<number, MLData>> {
|
||||
}): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}> {
|
||||
this.mlFetchCalls.push([...args.fileIDs]);
|
||||
const result = new Map<number, MLData>();
|
||||
const payloads = new Map<number, MLData>();
|
||||
const errors = new Map<number, unknown>();
|
||||
for (const id of args.fileIDs) {
|
||||
const p = this.mlByFile.get(id);
|
||||
if (p) result.set(id, p);
|
||||
if (p) payloads.set(id, p);
|
||||
const error = this.mlErrorByFile.get(id);
|
||||
if (error) errors.set(id, error);
|
||||
}
|
||||
return result;
|
||||
return { payloads, errors };
|
||||
}
|
||||
|
||||
filesFor(collectionID: number, ...pages: FilesPage[]): void {
|
||||
@@ -449,6 +458,74 @@ describe("Library ML-data fetch on refresh", () => {
|
||||
}
|
||||
});
|
||||
|
||||
it("reports a file whose ML data did not decode through status() and onProgress, and asks for it again", async () => {
|
||||
const client = new MLMockClient();
|
||||
client.collectionsQueue.push({
|
||||
collections: [collection(1, 100)],
|
||||
deleted: [],
|
||||
cursor: 100,
|
||||
});
|
||||
client.filesFor(1, {
|
||||
files: [file(1001, 1, 90), file(1002, 1, 95)],
|
||||
deleted: [],
|
||||
cursor: 95,
|
||||
});
|
||||
client.mlByFile.set(1001, payload([0.5, 0.25, 0.75]));
|
||||
client.mlErrorByFile.set(
|
||||
1002,
|
||||
new Error("secretstream chunk authentication failed"),
|
||||
);
|
||||
const events: RefreshEvent[] = [];
|
||||
|
||||
const lib = await Library.open({
|
||||
client,
|
||||
cacheDirectory,
|
||||
refreshIntervalSeconds: Infinity,
|
||||
onProgress: (event) => events.push(event),
|
||||
});
|
||||
try {
|
||||
await vi.waitFor(
|
||||
() => expect(lib.status().lastMLFetchAt).toBeGreaterThan(0),
|
||||
{ timeout: 2000, interval: 5 },
|
||||
);
|
||||
const reasons = [
|
||||
{
|
||||
fileID: 1002,
|
||||
error: "secretstream chunk authentication failed",
|
||||
},
|
||||
];
|
||||
expect(lib.status().mlDataErrors).toEqual(reasons);
|
||||
expect(lib.status().mlStored).toBe(1);
|
||||
expect(events.filter((e) => e.operation === "fetchMLData")).toEqual(
|
||||
[
|
||||
{ operation: "fetchMLData", status: "started" },
|
||||
{
|
||||
operation: "fetchMLData",
|
||||
status: "done",
|
||||
fetched: 1,
|
||||
mlDataErrors: reasons,
|
||||
},
|
||||
],
|
||||
);
|
||||
|
||||
// Nothing was cached for 1002, so the pass a refresh starts asks
|
||||
// for it again, and once its ML data decodes the reason is gone.
|
||||
client.mlErrorByFile.clear();
|
||||
client.mlByFile.set(1002, payload([1, -2, 0.5]));
|
||||
await lib.fresh();
|
||||
await vi.waitFor(
|
||||
() => {
|
||||
expect(lib.status().mlStored).toBe(2);
|
||||
expect(lib.status().mlDataErrors).toEqual([]);
|
||||
},
|
||||
{ timeout: 2000, interval: 5 },
|
||||
);
|
||||
expect(client.mlFetchCalls.at(-1)).toEqual([1002]);
|
||||
} finally {
|
||||
await lib.close();
|
||||
}
|
||||
});
|
||||
|
||||
it("close() resolves only after a running ML data fetch has stored its payloads", async () => {
|
||||
const client = new MLMockClient();
|
||||
client.collectionsQueue.push({
|
||||
|
||||
Reference in new issue
Block a user