backup-metadata: keep going when an ML data request fails (closes #101) #114

Merged
clawbot merged 1 commits from issue-101-mldata-failure into next2 2026-09-23 05:49:38 +02:00
Collaborator

Fixes #101.

backup-metadata used to fetch ML data for the whole account in one call, so one request that failed after its retries ended the run with the per-file JSON unwritten. Now each request of up to 200 files is tried on its own. A failed one is logged, each of its files is written with the reason in mlDataError instead of mlData, and the dump goes on. runMetadataBackup returns how many requests failed, and the command exits 1 when that is more than zero, after the whole dump is written.

Things the diff does not make obvious:

  • fetchMLData in src/mldata-fetch.ts had no other caller, so it is removed; the loop now lives in runMetadataBackup and calls fetchMLDataBatch per batch.
  • The test file mocks MLDATA_BATCH_SIZE to 1 for all its tests, so the two files of the mock account take two requests and one can fail alone. The new tests run the real backupMetadataCommand, so they check the exit code as well as the files.
  • The README's CLI section now states the new field and exit code.

Model: opus-5-5

Fixes https://git.eeqj.de/sneak/quak/issues/101. `backup-metadata` used to fetch ML data for the whole account in one call, so one request that failed after its retries ended the run with the per-file JSON unwritten. Now each request of up to 200 files is tried on its own. A failed one is logged, each of its files is written with the reason in `mlDataError` instead of `mlData`, and the dump goes on. `runMetadataBackup` returns how many requests failed, and the command exits 1 when that is more than zero, after the whole dump is written. Things the diff does not make obvious: - `fetchMLData` in `src/mldata-fetch.ts` had no other caller, so it is removed; the loop now lives in `runMetadataBackup` and calls `fetchMLDataBatch` per batch. - The test file mocks `MLDATA_BATCH_SIZE` to 1 for all its tests, so the two files of the mock account take two requests and one can fail alone. The new tests run the real `backupMetadataCommand`, so they check the exit code as well as the files. - The README's CLI section now states the new field and exit code. Model: opus-5-5
clawbot self-assigned this 2026-09-23 05:40:02 +02:00
clawbot added 1 commit 2026-09-23 05:40:02 +02:00
Each ML data request of up to 200 files is now tried on its own. A request
that still fails after its retries is logged, its files are written with the
reason in `mlDataError`, and the command exits 1 once the dump is complete.
`fetchMLData`, used only here, is removed in favour of a per-batch loop over
`fetchMLDataBatch`.

Model: opus-5-5
clawbot added the needs-review label 2026-09-23 05:40:06 +02:00
Author
Collaborator

PASS on f051f9093bd8fb36ce4fc2f89af0453e8d6af8be rebased onto next2 d05b53d5609e29ea6d88ce8531da7d9a83452f1b.

Model: opus-5-5

PASS on `f051f9093bd8fb36ce4fc2f89af0453e8d6af8be` rebased onto `next2` `d05b53d5609e29ea6d88ce8531da7d9a83452f1b`. Model: opus-5-5
clawbot merged commit bf3b20df2f into next2 2026-09-23 05:49:38 +02:00
clawbot deleted branch issue-101-mldata-failure 2026-09-23 05:49:38 +02:00
Sign in to join this conversation.