the reason the library exists: checksummed import off the card #5

New Issue

2026-08-30T13:14:54+02:00

clawbot commented

2026-08-30 13:14:54 +02:00

README § why: "I wanted to copy images off my memory cards for processing." The library has Hash() but no import. Provide a copy-off-card operation: copy object groups to a destination, verify by SHA256 after write, optionally skip files already present at the destination (by hash), never delete from the card. Exact call shape follows sneak's API pick.

Done: importing a fixture store to a temp dir round-trips every byte (hash-verified), duplicate re-import is a no-op when skip-existing is on, and a corrupted destination write is detected and reported, not silently kept.

README § why: "I wanted to copy images off my memory cards for processing." The library has `Hash()` but no import. Provide a copy-off-card operation: copy object groups to a destination, verify by SHA256 after write, optionally skip files already present at the destination (by hash), never delete from the card. Exact call shape follows sneak's API pick. Done: importing a fixture store to a temp dir round-trips every byte (hash-verified), duplicate re-import is a no-op when skip-existing is on, and a corrupted destination write is detected and reported, not silently kept.

clawbot referenced this issue

2026-09-25 11:10:13 +02:00

API cleanup: fs.FS-based store access, no *[]DCFStore, contexts on walks #4

clawbot commented

2026-10-03 14:38:24 +02:00

State, 2026-10-03: waits on sneak's answer to the question on #4 (where the import puts files, and how a program gets a card); planned after #3, since it copies file groups.

Model: opus-5-5

State, 2026-10-03: waits on sneak's answer to the question on https://git.eeqj.de/sneak/dcf/issues/4 (where the import puts files, and how a program gets a card); planned after https://git.eeqj.de/sneak/dcf/issues/3, since it copies file groups. Model: opus-5-5

clawbot referenced this issue

2026-11-02 22:22:21 +01:00

discovery is extension-grepping, not DCF: parse the CP-3461 structure #2

clawbot commented

2026-11-03 20:54:44 +01:00

Plan, built to the ruling at #4 (comment): one folder per day, and a different file already at the path is reported as a conflict and left alone. The 2026-10-03 note above saying this waits on sneak is out of date.

The call is result, err := store.Import(ctx, dest), in a new file pkg/dcf/import.go. ImportResult holds Copied, Skipped and Conflicts, each a list of destination paths.

What is copied: every member of every file group, and the images and videos under PRIVATE/M4ROOT. Files in Store.NonConforming are not copied.
Where: dest/YYYY-MM-DD/<file name>, from the file's modification time, in a new field Object.ModTime. Every member of a file group goes to the day of its primary member, so a group never splits across two folders.
Each file: hash the source. If a file is already at the path, the same hash means skipped, and anything else means conflict. Otherwise create the path with O_CREATE|O_EXCL, copy, sync, close, hash the written file, and compare. On a mismatch or failure, remove it and report the error. Set the copy's modification time to the source's.
One file that fails does not stop the others. The returned error joins the failures with errors.Join, each naming its file, and the result still lists what was done.
A cancelled context stops the import at once, inside a file too, and removes the partial copy. The error matches context.Canceled.
The card is only ever read through fs.FS, which has no call that can delete.

Judgement calls, built unless sneak says otherwise:

No options argument. A file with the same hash is always skipped, because fewest options wins and re-copying it would gain nothing.
The day comes from the modification time, not Exif data, because reading Exif would need a parser for every file type.
The copy is written at its final name, so nothing can ever overwrite a file. If the process dies mid-copy, the partial file stays, and the next import reports it as a conflict.

Test on a fstest.MapFS card: every byte round-trips; a second import changes nothing and reports all skipped; a different file at the path is a conflict and stays unchanged; a source whose second read returns other bytes gives an error and keeps no file; one unreadable file does not stop the others; a group whose members straddle midnight stays together; a clip under PRIVATE/M4ROOT is copied; cancellation.

Model: opus-5-5

Plan, built to the ruling at https://git.eeqj.de/sneak/dcf/issues/4#issuecomment-145367: one folder per day, and a different file already at the path is reported as a conflict and left alone. The 2026-10-03 note above saying this waits on sneak is out of date. The call is `result, err := store.Import(ctx, dest)`, in a new file `pkg/dcf/import.go`. `ImportResult` holds `Copied`, `Skipped` and `Conflicts`, each a list of destination paths. - What is copied: every member of every file group, and the images and videos under `PRIVATE/M4ROOT`. Files in `Store.NonConforming` are not copied. - Where: `dest/YYYY-MM-DD/<file name>`, from the file's modification time, in a new field `Object.ModTime`. Every member of a file group goes to the day of its primary member, so a group never splits across two folders. - Each file: hash the source. If a file is already at the path, the same hash means skipped, and anything else means conflict. Otherwise create the path with `O_CREATE|O_EXCL`, copy, sync, close, hash the written file, and compare. On a mismatch or failure, remove it and report the error. Set the copy's modification time to the source's. - One file that fails does not stop the others. The returned error joins the failures with `errors.Join`, each naming its file, and the result still lists what was done. - A cancelled context stops the import at once, inside a file too, and removes the partial copy. The error matches `context.Canceled`. - The card is only ever read through `fs.FS`, which has no call that can delete. Judgement calls, built unless sneak says otherwise: - No options argument. A file with the same hash is always skipped, because fewest options wins and re-copying it would gain nothing. - The day comes from the modification time, not Exif data, because reading Exif would need a parser for every file type. - The copy is written at its final name, so nothing can ever overwrite a file. If the process dies mid-copy, the partial file stays, and the next import reports it as a conflict. Test on a `fstest.MapFS` card: every byte round-trips; a second import changes nothing and reports all skipped; a different file at the path is a conflict and stays unchanged; a source whose second read returns other bytes gives an error and keeps no file; one unreadable file does not stop the others; a group whose members straddle midnight stays together; a clip under `PRIVATE/M4ROOT` is copied; cancellation. Model: opus-5-5

clawbot referenced this issue from a commit

2026-11-03 21:07:59 +01:00

copy files off a card, verified by SHA256 (closes #5)

clawbot referenced a pull request that will close this issue

2026-11-03 21:08:05 +01:00

copy files off a card, verified by SHA256 (closes #5) #14

clawbot commented

2026-11-03 21:08:13 +01:00