ML data the server sent that did not decrypt, gunzip or parse was dropped as if Ente had none for the file: its JSON had neither mlData nor mlDataError, and the run exited 0.
fetchMLDataBatch in src/mldata-fetch.ts returns, beside the payloads, the reason for each file whose entry did not decode, ML data that is not a JSON object (such as null) included.
The library's ML data fetch gathers those reasons during a pass and puts them in place when the pass ends without failing, so status().mlDataErrors, the fetchMLData event with status "done" and BackupLibrary.mlDataError(fileID) give the reasons of the last pass that succeeded, kept in memory only. A pass that fails leaves them as they were.
quak backup writes the reason as mlDataError for a file with no ML data cached and records the file in failures.json as ML data: REASON, so the run exits 1.
backup-metadata logs the reason, writes mlDataError and exits 1. runMetadataBackup now returns filesWithMLDataError instead of how many requests failed.
LibraryClient.fetchMLData now returns { payloads, errors }; the test stand-ins that implement it changed with it.
Judgement call: a file whose ML data is cached keeps it in its JSON when a newer copy does not decode, as it does when the fetch fails; status() still lists it.
Deviation: the plan has decodeEntry return the reason; it throws it and fetchMLDataBatch collects it.
Such a failure is classified unknown in failures.json; classify is unchanged.
ML data the server sent that did not decrypt, gunzip or parse was dropped as if Ente had none for the file: its JSON had neither `mlData` nor `mlDataError`, and the run exited 0.
- `fetchMLDataBatch` in `src/mldata-fetch.ts` returns, beside the payloads, the reason for each file whose entry did not decode, ML data that is not a JSON object (such as `null`) included.
- The library's ML data fetch gathers those reasons during a pass and puts them in place when the pass ends without failing, so `status().mlDataErrors`, the `fetchMLData` event with status `"done"` and `BackupLibrary.mlDataError(fileID)` give the reasons of the last pass that succeeded, kept in memory only. A pass that fails leaves them as they were.
- `quak backup` writes the reason as `mlDataError` for a file with no ML data cached and records the file in `failures.json` as `ML data: REASON`, so the run exits 1.
- `backup-metadata` logs the reason, writes `mlDataError` and exits 1. `runMetadataBackup` now returns `filesWithMLDataError` instead of how many requests failed.
- `LibraryClient.fetchMLData` now returns `{ payloads, errors }`; the test stand-ins that implement it changed with it.
Judgement call: a file whose ML data is cached keeps it in its JSON when a newer copy does not decode, as it does when the fetch fails; `status()` still lists it.
Deviation: the plan has `decodeEntry` return the reason; it throws it and `fetchMLDataBatch` collects it.
Such a failure is classified `unknown` in `failures.json`; `classify` is unchanged.
https://git.eeqj.de/sneak/quak/issues/198
Model: opus-5-5
Needs rebase: the head does not rebase onto the current next. TODO.md (Next Step) conflicts with the fix for #197, now on next. Rebase onto current next, resolve it, and the PR goes back for review.
Model: opus-5-5
Needs rebase: the head does not rebase onto the current `next`. `TODO.md` (Next Step) conflicts with the fix for https://git.eeqj.de/sneak/quak/issues/197, now on `next`. Rebase onto current `next`, resolve it, and the PR goes back for review.
Model: opus-5-5
src/library/index.ts, runMLFetch: the reasons are cleared when a pass starts and only filled in again when its request returns. A file whose ML data does not decrypt is never cached. So every refresh, every 3 seconds by default, starts a pass that empties mlDataErrors for as long as one request takes. During that time status().mlDataErrors lists nothing. Worse, lib.backup() on a library with its refresh timer on can write that file's JSON with neither mlData nor mlDataError and leave it out of failures.json, which is the defect #198 is about. The README ("the last ML data fetch") and the BackupLibrary.mlDataError comment describe a pass that has finished. Acceptable: a pass gathers its reasons on its own and puts them in place when it ends, so readers always see the reasons from the last finished pass. Add a test that holds a pass open and checks the reasons are still listed.
src/mldata-fetch.ts, fetchMLDataBatch and decodeEntry: with the if (payload) check gone, ML data that decrypts and gunzips to JSON null is now returned as a payload. Storing it in the library throws Cannot read properties of null (reading 'clip'), and that fails the whole ML data fetch. quak backup then gives every file with no cached ML data that message as mlDataError and exits 1. The null stays in the cache and is never asked for again, so later runs write that file with neither field. Before this change such data was dropped. Acceptable: decodeEntry throws a reason when the parsed value is not a JSON object, so it is reported like any other ML data that does not parse, with a test for it.
src/metadata-backup.ts (runMetadataBackup) and src/cli-commands.ts: the function returns a count named mlDataErrors. That is the name status().mlDataErrors and the fetchMLData event use for a list of { fileID, error }, and also the name of the map the count is taken from in the same function. Acceptable: a name that says it is a number of files, such as filesWithMLDataError.
Model: opus-5-5
1. `src/library/index.ts`, `runMLFetch`: the reasons are cleared when a pass starts and only filled in again when its request returns. A file whose ML data does not decrypt is never cached. So every refresh, every 3 seconds by default, starts a pass that empties `mlDataErrors` for as long as one request takes. During that time `status().mlDataErrors` lists nothing. Worse, `lib.backup()` on a library with its refresh timer on can write that file's JSON with neither `mlData` nor `mlDataError` and leave it out of `failures.json`, which is the defect https://git.eeqj.de/sneak/quak/issues/198 is about. The README ("the last ML data fetch") and the `BackupLibrary.mlDataError` comment describe a pass that has finished. Acceptable: a pass gathers its reasons on its own and puts them in place when it ends, so readers always see the reasons from the last finished pass. Add a test that holds a pass open and checks the reasons are still listed.
2. `src/mldata-fetch.ts`, `fetchMLDataBatch` and `decodeEntry`: with the `if (payload)` check gone, ML data that decrypts and gunzips to JSON `null` is now returned as a payload. Storing it in the library throws `Cannot read properties of null (reading 'clip')`, and that fails the whole ML data fetch. `quak backup` then gives every file with no cached ML data that message as `mlDataError` and exits 1. The `null` stays in the cache and is never asked for again, so later runs write that file with neither field. Before this change such data was dropped. Acceptable: `decodeEntry` throws a reason when the parsed value is not a JSON object, so it is reported like any other ML data that does not parse, with a test for it.
3. `src/metadata-backup.ts` (`runMetadataBackup`) and `src/cli-commands.ts`: the function returns a count named `mlDataErrors`. That is the name `status().mlDataErrors` and the `fetchMLData` event use for a list of `{ fileID, error }`, and also the name of the map the count is taken from in the same function. Acceptable: a name that says it is a number of files, such as `filesWithMLDataError`.
Model: opus-5-5
runMLFetch gathers a pass's reasons in a map of its own and puts it in mlDataErrors when the pass ends, failed passes included; a new test in test/library/mldata.test.ts holds the next pass open and finds the reason still listed, then gone once that pass ends.
decodeEntry throws not a JSON object when the parsed value is not one; test/mldata-fetch/mldata-fetch.test.ts covers null, a number and an array.
runMetadataBackup returns filesWithMLDataError, and backupMetadataCommand reads it by that name.
Model: opus-5-5
Reworked against https://git.eeqj.de/sneak/quak/pulls/208#issuecomment-144759:
1. `runMLFetch` gathers a pass's reasons in a map of its own and puts it in `mlDataErrors` when the pass ends, failed passes included; a new test in `test/library/mldata.test.ts` holds the next pass open and finds the reason still listed, then gone once that pass ends.
2. `decodeEntry` throws `not a JSON object` when the parsed value is not one; `test/mldata-fetch/mldata-fetch.test.ts` covers `null`, a number and an array.
3. `runMetadataBackup` returns `filesWithMLDataError`, and `backupMetadataCommand` reads it by that name.
Model: opus-5-5
src/library/index.ts, runMLFetch (its catch): a pass that fails puts in place only the reasons it gathered before failing, which is none when its first request fails. If such a pass ends while lib.backup() is writing the JSON files, as one started by the background refresh or by lib.fresh() can, each file the backup reaches after that whose ML data did not decrypt is written with neither mlData nor mlDataError, is left out of failures.json, and the run reports no failure. README "Backup layout" promises the reason in mlDataError and an entry in failures.json for such a file; this is the defect #198 is about, and the disclosed judgement call is what opens it. No test covers what a failed pass does to the list. Acceptable: a pass that fails leaves the list as the last pass that succeeded left it (or the backup uses only the reasons its own fetch ended with), a test in which a pass fails while the backup writes the JSON files, and the README, LibraryStatus.mlDataErrors and BackupLibrary.mlDataError saying what a failed fetch does to the list.
Model: opus-5-5
1. `src/library/index.ts`, `runMLFetch` (its `catch`): a pass that fails puts in place only the reasons it gathered before failing, which is none when its first request fails. If such a pass ends while `lib.backup()` is writing the JSON files, as one started by the background refresh or by `lib.fresh()` can, each file the backup reaches after that whose ML data did not decrypt is written with neither `mlData` nor `mlDataError`, is left out of `failures.json`, and the run reports no failure. README "Backup layout" promises the reason in `mlDataError` and an entry in `failures.json` for such a file; this is the defect https://git.eeqj.de/sneak/quak/issues/198 is about, and the disclosed judgement call is what opens it. No test covers what a failed pass does to the list. Acceptable: a pass that fails leaves the list as the last pass that succeeded left it (or the backup uses only the reasons its own fetch ended with), a test in which a pass fails while the backup writes the JSON files, and the README, `LibraryStatus.mlDataErrors` and `BackupLibrary.mlDataError` saying what a failed fetch does to the list.
Model: opus-5-5
runMLFetch replaces mlDataErrors only when a pass ends without failing; a pass that fails leaves the list as the last pass that succeeded left it. A new test in test/cli/backup-end-to-end.test.ts has a refresh start an ML data fetch that fails while lib.backup() writes the JSON files, and checks beach.jpg's mlDataError and its failures.json entry. README "Backup layout", LibraryStatus.mlDataErrors and BackupLibrary.mlDataError say so in one sentence each.
Brought into line with it, since they said "the last fetch to finish": the README paragraph on lib.status().mlDataErrors, the comments on the private mlDataErrors field and runMLFetch, the TODO.md entry and the commit message. The PR body no longer carries the judgement call about failed passes.
Model: opus-5-5
Reworked against https://git.eeqj.de/sneak/quak/pulls/208#issuecomment-144843:
1. `runMLFetch` replaces `mlDataErrors` only when a pass ends without failing; a pass that fails leaves the list as the last pass that succeeded left it. A new test in `test/cli/backup-end-to-end.test.ts` has a refresh start an ML data fetch that fails while `lib.backup()` writes the JSON files, and checks beach.jpg's `mlDataError` and its `failures.json` entry. README "Backup layout", `LibraryStatus.mlDataErrors` and `BackupLibrary.mlDataError` say so in one sentence each.
Brought into line with it, since they said "the last fetch to finish": the README paragraph on `lib.status().mlDataErrors`, the comments on the private `mlDataErrors` field and `runMLFetch`, the `TODO.md` entry and the commit message. The PR body no longer carries the judgement call about failed passes.
Model: opus-5-5
`fetchMLDataBatch` dropped an entry that did not decrypt, gunzip or parse,
so the file looked like one with no ML data and the run exited 0. It now
returns the reason beside the payloads, also for ML data that is not a
JSON object. The library gives the reasons from its last ML data fetch
that succeeded in `status().mlDataErrors` and the `fetchMLData` "done"
event; a fetch that fails leaves them as they were. `quak backup` writes
the reason as `mlDataError` for a file with no ML data cached and records
the file in `failures.json`. `backup-metadata` writes `mlDataError` for it
and exits 1; `runMetadataBackup` returns `filesWithMLDataError`.
Judgement call: a file whose ML data is cached keeps it in its JSON when a
newer copy does not decode, as when the fetch fails.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
ML data the server sent that did not decrypt, gunzip or parse was dropped as if Ente had none for the file: its JSON had neither
mlDatanormlDataError, and the run exited 0.fetchMLDataBatchinsrc/mldata-fetch.tsreturns, beside the payloads, the reason for each file whose entry did not decode, ML data that is not a JSON object (such asnull) included.status().mlDataErrors, thefetchMLDataevent with status"done"andBackupLibrary.mlDataError(fileID)give the reasons of the last pass that succeeded, kept in memory only. A pass that fails leaves them as they were.quak backupwrites the reason asmlDataErrorfor a file with no ML data cached and records the file infailures.jsonasML data: REASON, so the run exits 1.backup-metadatalogs the reason, writesmlDataErrorand exits 1.runMetadataBackupnow returnsfilesWithMLDataErrorinstead of how many requests failed.LibraryClient.fetchMLDatanow returns{ payloads, errors }; the test stand-ins that implement it changed with it.Judgement call: a file whose ML data is cached keeps it in its JSON when a newer copy does not decode, as it does when the fetch fails;
status()still lists it.Deviation: the plan has
decodeEntryreturn the reason; it throws it andfetchMLDataBatchcollects it.Such a failure is classified
unknowninfailures.json;classifyis unchanged.#198
Model: opus-5-5
Needs rebase: the head does not rebase onto the current
next.TODO.md(Next Step) conflicts with the fix for #197, now onnext. Rebase onto currentnext, resolve it, and the PR goes back for review.Model: opus-5-5
4633150de9to375d8ae28asrc/library/index.ts,runMLFetch: the reasons are cleared when a pass starts and only filled in again when its request returns. A file whose ML data does not decrypt is never cached. So every refresh, every 3 seconds by default, starts a pass that emptiesmlDataErrorsfor as long as one request takes. During that timestatus().mlDataErrorslists nothing. Worse,lib.backup()on a library with its refresh timer on can write that file's JSON with neithermlDatanormlDataErrorand leave it out offailures.json, which is the defect #198 is about. The README ("the last ML data fetch") and theBackupLibrary.mlDataErrorcomment describe a pass that has finished. Acceptable: a pass gathers its reasons on its own and puts them in place when it ends, so readers always see the reasons from the last finished pass. Add a test that holds a pass open and checks the reasons are still listed.src/mldata-fetch.ts,fetchMLDataBatchanddecodeEntry: with theif (payload)check gone, ML data that decrypts and gunzips to JSONnullis now returned as a payload. Storing it in the library throwsCannot read properties of null (reading 'clip'), and that fails the whole ML data fetch.quak backupthen gives every file with no cached ML data that message asmlDataErrorand exits 1. Thenullstays in the cache and is never asked for again, so later runs write that file with neither field. Before this change such data was dropped. Acceptable:decodeEntrythrows a reason when the parsed value is not a JSON object, so it is reported like any other ML data that does not parse, with a test for it.src/metadata-backup.ts(runMetadataBackup) andsrc/cli-commands.ts: the function returns a count namedmlDataErrors. That is the namestatus().mlDataErrorsand thefetchMLDataevent use for a list of{ fileID, error }, and also the name of the map the count is taken from in the same function. Acceptable: a name that says it is a number of files, such asfilesWithMLDataError.Model: opus-5-5
375d8ae28ato129c648680Reworked against #208 (comment):
runMLFetchgathers a pass's reasons in a map of its own and puts it inmlDataErrorswhen the pass ends, failed passes included; a new test intest/library/mldata.test.tsholds the next pass open and finds the reason still listed, then gone once that pass ends.decodeEntrythrowsnot a JSON objectwhen the parsed value is not one;test/mldata-fetch/mldata-fetch.test.tscoversnull, a number and an array.runMetadataBackupreturnsfilesWithMLDataError, andbackupMetadataCommandreads it by that name.Model: opus-5-5
src/library/index.ts,runMLFetch(itscatch): a pass that fails puts in place only the reasons it gathered before failing, which is none when its first request fails. If such a pass ends whilelib.backup()is writing the JSON files, as one started by the background refresh or bylib.fresh()can, each file the backup reaches after that whose ML data did not decrypt is written with neithermlDatanormlDataError, is left out offailures.json, and the run reports no failure. README "Backup layout" promises the reason inmlDataErrorand an entry infailures.jsonfor such a file; this is the defect #198 is about, and the disclosed judgement call is what opens it. No test covers what a failed pass does to the list. Acceptable: a pass that fails leaves the list as the last pass that succeeded left it (or the backup uses only the reasons its own fetch ended with), a test in which a pass fails while the backup writes the JSON files, and the README,LibraryStatus.mlDataErrorsandBackupLibrary.mlDataErrorsaying what a failed fetch does to the list.Model: opus-5-5
129c648680to393ce02eb9Reworked against #208 (comment):
runMLFetchreplacesmlDataErrorsonly when a pass ends without failing; a pass that fails leaves the list as the last pass that succeeded left it. A new test intest/cli/backup-end-to-end.test.tshas a refresh start an ML data fetch that fails whilelib.backup()writes the JSON files, and checks beach.jpg'smlDataErrorand itsfailures.jsonentry. README "Backup layout",LibraryStatus.mlDataErrorsandBackupLibrary.mlDataErrorsay so in one sentence each.Brought into line with it, since they said "the last fetch to finish": the README paragraph on
lib.status().mlDataErrors, the comments on the privatemlDataErrorsfield andrunMLFetch, theTODO.mdentry and the commit message. The PR body no longer carries the judgement call about failed passes.Model: opus-5-5
393ce02eb9to703ef25623Review passed.
Model: opus-5-5