ML data that does not decrypt is recorded with its reason (closes #198)
check / check (push) Canceled after 0s

`fetchMLDataBatch` dropped an entry that did not decrypt, gunzip or parse,
so the file looked like one with no ML data and the run exited 0. It now
returns the reason beside the payloads, also for ML data that is not a
JSON object. The library gives the reasons from its last ML data fetch
that succeeded in `status().mlDataErrors` and the `fetchMLData` "done"
event; a fetch that fails leaves them as they were. `quak backup` writes
the reason as `mlDataError` and records the file in `failures.json`;
`backup-metadata` writes `mlDataError` and exits 1.

Judgement call: a file whose ML data is cached keeps it in its JSON when a
newer copy does not decode, as when the fetch fails.

Model: opus-5-5
This commit was merged in pull request #208.
This commit is contained in:
2026-10-16 07:23:16 +02:00
parent 8583863ec0
commit 7ec71bf7ec
13 changed files with 558 additions and 87 deletions
+19 -9
View File
@@ -208,6 +208,11 @@ export interface BackupLibrary {
// A file's cached ML data, as `lib.mldata.forFile()` returns it, or
// undefined when none is cached.
mlData(fileID: number): Promise<MLData | undefined>;
// Why the ML data the last ML data fetch that succeeded got for a file did
// not decrypt, gunzip or parse, or undefined when it did or the fetch got
// none. A fetch still running does not change it. A fetch that fails
// leaves it as it was.
mlDataError(fileID: number): unknown;
}
type FailureClass = "transient" | "permanent" | "unknown";
@@ -791,19 +796,18 @@ const runLockedBackup = async (
// Phase 2: rebuild the derived views from the model. Sidecars first, for
// every present original (this repairs stale ones), each with the file's
// ML data once an ML data fetch has completed, and its image metadata.
// When the fetch fails, a file with no cached ML data gets the reason
// instead and is recorded as failed. The next run fetches its ML data
// again because none is cached.
// When the fetch got a file's ML data but could not decrypt, gunzip or
// parse it, or when the fetch failed, a file with no cached ML data gets
// the reason instead and is recorded as failed. The next run fetches its
// ML data again because none is cached.
if (includeOriginals) {
let mlDataFailure: unknown;
let mlDataError: string | undefined;
try {
log("Fetching ML data...");
await lib.fetchMLData();
} catch (err) {
mlDataFailure = err;
mlDataError = describeError(err);
log(`FAILED ML data: ${mlDataError}`);
log(`FAILED ML data: ${describeError(err)}`);
}
for (const file of distinct.values()) {
if (lock.lost) throw lock.lost;
@@ -820,13 +824,19 @@ const runLockedBackup = async (
storedThisRun.has(file.id),
);
const mlData = await lib.mlData(file.id);
if (mlData === undefined && mlDataError !== undefined) {
writeSidecar(path, file, { mlDataError }, image);
const failure = lib.mlDataError(file.id) ?? mlDataFailure;
if (mlData === undefined && failure !== undefined) {
writeSidecar(
path,
file,
{ mlDataError: describeError(failure) },
image,
);
recordFailure(
file.id,
file.metadata.title,
collectionName.get(file.collectionID) ?? "",
new Error("ML data", { cause: mlDataFailure }),
new Error("ML data", { cause: failure }),
);
} else {
writeSidecar(path, file, { mlData }, image);