ML data that does not decrypt is recorded with its reason (closes #198)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
`fetchMLDataBatch` dropped an entry that did not decrypt, gunzip or parse, so the file looked like one with no ML data and the run exited 0. It now returns the reason beside the payloads, also for ML data that is not a JSON object. The library gives the reasons from its last ML data fetch that succeeded in `status().mlDataErrors` and the `fetchMLData` "done" event; a fetch that fails leaves them as they were. `quak backup` writes the reason as `mlDataError` and records the file in `failures.json`; `backup-metadata` writes `mlDataError` and exits 1. Judgement call: a file whose ML data is cached keeps it in its JSON when a newer copy does not decode, as when the fetch fails. Model: opus-5-5
This commit was merged in pull request #208.
This commit is contained in:
+8
-4
@@ -373,13 +373,17 @@ export class Client {
|
||||
}
|
||||
|
||||
// Fetch machine-learning data (face detections + CLIP embeddings) for up
|
||||
// to a batch of files, each decrypted with its own key. One request; the
|
||||
// library batches at `MLDATA_BATCH_SIZE` and schedules each batch through
|
||||
// its metadata request pool.
|
||||
// to a batch of files, each decrypted with its own key, and for each file
|
||||
// whose ML data did not decrypt, gunzip or parse, why not. One request;
|
||||
// the library batches at `MLDATA_BATCH_SIZE` and schedules each batch
|
||||
// through its metadata request pool.
|
||||
async fetchMLData(args: {
|
||||
fileIDs: number[];
|
||||
fileKeys: Map<number, Uint8Array>;
|
||||
}): Promise<Map<number, MLData>> {
|
||||
}): Promise<{
|
||||
payloads: Map<number, MLData>;
|
||||
errors: Map<number, unknown>;
|
||||
}> {
|
||||
this.assertLoggedIn();
|
||||
return fetchMLDataBatch(this.api, args.fileIDs, args.fileKeys);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user