ML data that does not decrypt is treated as absent, and the reason is lost #198

New Issue

2026-10-14T09:29:22+02:00

clawbot commented

2026-10-14 09:29:22 +02:00

Severity: low

Where: decodeEntry in src/mldata-fetch.ts returns undefined when an entry does not decrypt, gunzip or parse, and fetchMLDataBatch drops it.

Problem: the file's JSON in the backup then has neither mlData nor mlDataError, the run exits 0, and nothing says the server sent ML data that quak could not read. It looks exactly like a file Ente has no ML data for. Since nothing is cached for it, every later run asks the server for it again and drops it again. The README's "Backup layout" says a file whose ML data fetch fails gets the reason in mlDataError; a payload that fails to decrypt is a failure of the same kind with no reason recorded anywhere.

Reproduced: a backup through the fake Ente server of test/ente-server.ts whose ML data for its one file is encrypted under a key that is not the file's: exit 0, the file's JSON has no mlData and no mlDataError, the summary says Failed: 0.

Done when: an entry that does not decrypt or parse is reported with its reason, at least as mlDataError in the file's JSON and through the library's status() and progress events, so the backup can count it as backup-metadata counts a failed request; a test covers a payload under the wrong key.

Model: fable-5-1

Severity: low **Where:** `decodeEntry` in `src/mldata-fetch.ts` returns `undefined` when an entry does not decrypt, gunzip or parse, and `fetchMLDataBatch` drops it. **Problem:** the file's JSON in the backup then has neither `mlData` nor `mlDataError`, the run exits 0, and nothing says the server sent ML data that quak could not read. It looks exactly like a file Ente has no ML data for. Since nothing is cached for it, every later run asks the server for it again and drops it again. The README's "Backup layout" says a file whose ML data fetch fails gets the reason in `mlDataError`; a payload that fails to decrypt is a failure of the same kind with no reason recorded anywhere. **Reproduced:** a backup through the fake Ente server of `test/ente-server.ts` whose ML data for its one file is encrypted under a key that is not the file's: exit 0, the file's JSON has no `mlData` and no `mlDataError`, the summary says `Failed: 0`. **Done when:** an entry that does not decrypt or parse is reported with its reason, at least as `mlDataError` in the file's JSON and through the library's `status()` and progress events, so the backup can count it as `backup-metadata` counts a failed request; a test covers a payload under the wrong key. Model: fable-5-1

clawbot added this to the 1.0.0 milestone 2026-10-14 09:29:22 +02:00

clawbot referenced this issue

2026-10-14 09:32:15 +02:00

1.0: `v1.0.0-rc2` is a release candidate #201

clawbot commented

2026-10-16 01:01:27 +02:00

Plan.

decodeEntry in src/mldata-fetch.ts returns why an entry did not decrypt, gunzip or parse instead of nothing, and fetchMLDataBatch returns those reasons by file beside the payloads.
The library's ML data fetch (in src/library/, where it calls fetchMLDataBatch) keeps each such file's reason from its last fetch in memory, not in the cache, so the next run asks the server again. It reports them through status() and an onProgress event, and BackupLibrary in src/backup.ts gets a read for one file's reason beside mlData(fileID).
src/backup.ts writes the reason as mlDataError in the file's JSON and records the file in failures.json, exit 1, as it does now when the fetch request fails.
backup-metadata (src/metadata-backup.ts) uses the same function, so it writes mlDataError for such a file and exits 1 too.
README "Backup layout" and the API reference say it.
Tests: ML data under a key that is not the file's, for quak backup and for backup-metadata.

This unit waits until the other changes to src/backup.ts have landed on next.

Model: opus-5-5

Plan. - `decodeEntry` in `src/mldata-fetch.ts` returns why an entry did not decrypt, gunzip or parse instead of nothing, and `fetchMLDataBatch` returns those reasons by file beside the payloads. - The library's ML data fetch (in `src/library/`, where it calls `fetchMLDataBatch`) keeps each such file's reason from its last fetch in memory, not in the cache, so the next run asks the server again. It reports them through `status()` and an `onProgress` event, and `BackupLibrary` in `src/backup.ts` gets a read for one file's reason beside `mlData(fileID)`. - `src/backup.ts` writes the reason as `mlDataError` in the file's JSON and records the file in `failures.json`, exit 1, as it does now when the fetch request fails. - `backup-metadata` (`src/metadata-backup.ts`) uses the same function, so it writes `mlDataError` for such a file and exits 1 too. - README "Backup layout" and the API reference say it. - Tests: ML data under a key that is not the file's, for `quak backup` and for `backup-metadata`. This unit waits until the other changes to `src/backup.ts` have landed on `next`. Model: opus-5-5

clawbot referenced this issue

2026-10-16 01:07:27 +02:00

README says that `make install` needs nix (closes #200) #202

clawbot referenced this issue

2026-10-16 02:49:37 +02:00

An album whose folder cannot be made no longer stops the backup (closes #197) #204

clawbot referenced this issue

2026-10-16 02:52:04 +02:00

A network failure is printed and recorded with its reason (closes #196) #203

clawbot referenced this issue

2026-10-16 03:00:11 +02:00

A network failure is printed and recorded with its reason (closes #196) #203

clawbot referenced this issue

2026-10-16 03:07:07 +02:00

An album whose folder cannot be made no longer stops the backup (closes #197) #204

clawbot referenced this issue from a commit

2026-10-16 05:11:06 +02:00

ML data that does not decrypt is recorded with its reason (closes #198)

clawbot referenced a pull request that will close this issue

2026-10-16 05:17:05 +02:00

ML data that does not decrypt is recorded with its reason (closes #198) #208

clawbot commented

2026-10-16 05:17:11 +02:00

Built in #208. ML data that does not decrypt, gunzip or parse now gives the file mlDataError and an entry in failures.json reading ML data: REASON, and backup-metadata writes the reason too; both exit 1. The library lists the reasons in status().mlDataErrors and in the fetchMLData "done" event, and keeps them in memory only.

Model: opus-5-5

Built in https://git.eeqj.de/sneak/quak/pulls/208. ML data that does not decrypt, gunzip or parse now gives the file `mlDataError` and an entry in `failures.json` reading `ML data: REASON`, and `backup-metadata` writes the reason too; both exit 1. The library lists the reasons in `status().mlDataErrors` and in the `fetchMLData` "done" event, and keeps them in memory only. Model: opus-5-5

clawbot referenced this issue from a commit

2026-10-16 05:45:06 +02:00

ML data that does not decrypt is recorded with its reason (closes #198)

clawbot referenced this issue

2026-10-16 06:05:46 +02:00

ML data that does not decrypt is recorded with its reason (closes #198) #208

clawbot referenced this issue from a commit

2026-10-16 06:19:52 +02:00

ML data that does not decrypt is recorded with its reason (closes #198)

clawbot referenced this issue

2026-10-16 06:33:10 +02:00

ML data that does not decrypt is recorded with its reason (closes #198) #208