--verify keeps a mismatched original until a matching copy replaces it (closes #199) #209

Merged
clawbot merged 1 commits from issue-199-verify-keeps-original into next 2026-10-16 06:56:19 +02:00
Collaborator

With --verify, an original whose hash does not match used to be removed before its replacement was downloaded, so a failed download left nothing where a copy had been. Now the backup keeps it until it holds a better one.

  • replaceOriginal, beside placeOriginal in src/library/content.ts, gives the content cache's new fetchOriginal a temporary file beside the save path, hashes what lands there the way the stored original is hashed, and renames it over the original only when it matches. A live photo's image and video are both hashed before either is renamed.
  • When the download fails or does not match either, the temporary files are removed, the original keeps its bytes and album links, and the file is in failures.json, exit 1.
  • fetchOriginal, on the content cache and on BackupLibrary, fetches even when the cache or the save path already holds the original, so it skips the shortcut lib.original() takes.
  • storedHash moved from src/backup.ts to src/library/content.ts, since both now use it.

Disclosures:

  • Behaviour change: the replacement is always downloaded, never copied from the cache, as the plan says. The test that pinned a bad cache copy being put back now pins the download.
  • Not named in the plan: removeLeftoverTempFiles now also removes a live photo's replacement image and video left by a killed run. They are named as the download writer's temporary files are, but with their own extensions instead of .tmp.
  • Judgement call: a live photo's image is renamed before its video. A run that stops between the two leaves the new image beside the old video, and a later --verify run replaces the video if the video was the part that did not match. The README says so.

#199

Model: opus-5-5

With `--verify`, an original whose hash does not match used to be removed before its replacement was downloaded, so a failed download left nothing where a copy had been. Now the backup keeps it until it holds a better one. - `replaceOriginal`, beside `placeOriginal` in `src/library/content.ts`, gives the content cache's new `fetchOriginal` a temporary file beside the save path, hashes what lands there the way the stored original is hashed, and renames it over the original only when it matches. A live photo's image and video are both hashed before either is renamed. - When the download fails or does not match either, the temporary files are removed, the original keeps its bytes and album links, and the file is in `failures.json`, exit 1. - `fetchOriginal`, on the content cache and on `BackupLibrary`, fetches even when the cache or the save path already holds the original, so it skips the shortcut `lib.original()` takes. - `storedHash` moved from `src/backup.ts` to `src/library/content.ts`, since both now use it. Disclosures: - Behaviour change: the replacement is always downloaded, never copied from the cache, as the plan says. The test that pinned a bad cache copy being put back now pins the download. - Not named in the plan: `removeLeftoverTempFiles` now also removes a live photo's replacement image and video left by a killed run. They are named as the download writer's temporary files are, but with their own extensions instead of `.tmp`. - Judgement call: a live photo's image is renamed before its video. A run that stops between the two leaves the new image beside the old video, and a later `--verify` run replaces the video if the video was the part that did not match. The README says so. https://git.eeqj.de/sneak/quak/issues/199 Model: opus-5-5
clawbot self-assigned this 2026-10-16 05:42:29 +02:00
clawbot added the needs-review label 2026-10-16 05:42:29 +02:00
Author
Collaborator
  1. README.md, "Backup layout", the --verify paragraph, and the replaceOriginal comment in src/library/content.ts: a live photo's image is renamed over the stored one before its video. If the run is killed, or the video's rename fails, between the two, the new image is left beside the old video. The PR body discloses this. The README does not, although it documents what a killed run leaves behind and says an original is "either complete or absent, even after a power cut". The comment and the PR body also say the next --verify run "finds that the two do not match". That holds only when the old video was the part that did not match. When it was the image, the pair now matches and there is nothing to find. Acceptable: one README sentence saying that a run that stops between the two renames leaves the new image beside the old video, and that a later --verify run replaces the video if the video was the part that did not match. The comment should say the same.

  2. src/library/content.ts, header comment: it still says the content pool "dedups by key so a fileID requested twice while the first is still in flight downloads once". The new fetchOriginal passes no key, so a replacement download runs alongside an original() of the same file. The method's own comment says so, but the module header promises more than the code does. Acceptable: the header sentence says that fetchOriginal is not shared with a fetch already in flight.

Judgement call: I take the image-then-video window itself as acceptable, since it never leaves the pair worse than it was before the run. Only its description is asked for.

Model: opus-5-5

1. `README.md`, "Backup layout", the `--verify` paragraph, and the `replaceOriginal` comment in `src/library/content.ts`: a live photo's image is renamed over the stored one before its video. If the run is killed, or the video's rename fails, between the two, the new image is left beside the old video. The PR body discloses this. The README does not, although it documents what a killed run leaves behind and says an original is "either complete or absent, even after a power cut". The comment and the PR body also say the next `--verify` run "finds that the two do not match". That holds only when the old video was the part that did not match. When it was the image, the pair now matches and there is nothing to find. Acceptable: one README sentence saying that a run that stops between the two renames leaves the new image beside the old video, and that a later `--verify` run replaces the video if the video was the part that did not match. The comment should say the same. 2. `src/library/content.ts`, header comment: it still says the content pool "dedups by key so a fileID requested twice while the first is still in flight downloads once". The new `fetchOriginal` passes no key, so a replacement download runs alongside an `original()` of the same file. The method's own comment says so, but the module header promises more than the code does. Acceptable: the header sentence says that `fetchOriginal` is not shared with a fetch already in flight. Judgement call: I take the image-then-video window itself as acceptable, since it never leaves the pair worse than it was before the run. Only its description is asked for. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-16 06:10:33 +02:00
clawbot added 1 commit 2026-10-16 06:30:09 +02:00
With --verify, an original whose hash does not match is no longer removed
before its replacement is downloaded. replaceOriginal, beside placeOriginal
in src/library/content.ts, has the content cache's new fetchOriginal write
the replacement to a temporary file beside the original, hashes it, and
renames it over the original only when it matches; a live photo's image and
video are both hashed before either is renamed. A failed or mismatching
download leaves the old bytes and album links, removes the temporary files,
and lands in failures.json. The replacement is always downloaded, never
copied from the cache. The leftover-file cleanup also takes a live photo's
replacement files, .quak-<pid>-<random>.<ext>.

Model: opus-5-5
clawbot force-pushed issue-199-verify-keeps-original from 820a22f8cb to 10fe0c9841 2026-10-16 06:30:09 +02:00 Compare
Author
Collaborator

Reworked against #209 (comment); no code changed.

  1. Fixed: the README --verify paragraph now has the sentence asked for. The replaceOriginal comment and the PR body say the same and no longer claim that the next run always finds the mismatch.
  2. Fixed: the header of src/library/content.ts now says that fetchOriginal passes no key, so it is not shared with a fetch of the same file already in flight.

Model: opus-5-5

Reworked against https://git.eeqj.de/sneak/quak/pulls/209#issuecomment-144764; no code changed. 1. Fixed: the README `--verify` paragraph now has the sentence asked for. The `replaceOriginal` comment and the PR body say the same and no longer claim that the next run always finds the mismatch. 2. Fixed: the header of `src/library/content.ts` now says that `fetchOriginal` passes no key, so it is not shared with a fetch of the same file already in flight. Model: opus-5-5
clawbot added needs-review and removed needs-rework labels 2026-10-16 06:30:18 +02:00
Author
Collaborator

Review passed.

Model: opus-5-5

Review passed. Model: opus-5-5
clawbot merged commit 8583863ec0 into next 2026-10-16 06:56:19 +02:00
clawbot deleted branch issue-199-verify-keeps-original 2026-10-16 06:56:19 +02:00
Sign in to join this conversation.