report: never sit silent; show its phases as N of M with progress, elapsed time and ETA #114

New Issue

2026-10-16T13:51:43+02:00

clawbot commented

2026-10-16 13:51:43 +02:00

> The tool should always give the user some feedback to show it is doing something and isn't hung. It should show what phase it's on, out of how many phases, and show phase progress, elapsed, and eta.

On next, report prints nothing until its summary line. One SQL query (dupeRowsSQL in db.go) groups every record with a content hash and sorts the duplicate rows. SQLite returns no row until it has done all of that, so the wait cannot be measured as written. That is minutes of silence on a 10-million-record database on spinning disks. README "Progress" says "report and trees modes need no progress display"; sneak's words above replace that.

Definition of done

From the moment report starts until its summary line, stderr always shows which phase it is in out of how many, with that phase's progress, elapsed time and ETA. This holds on a terminal and in plain lines (a line as each phase starts, then at least every 5 seconds whether or not an item completed, and no more often). That includes opening the database and waiting on a busy one, and any count taken to get a total, which shows elapsed time while it runs.
No phase is one long query whose progress cannot be shown. Recommended approach: read the signature index in its own order (or in exact reverse, so no sort is needed), counting rows against the record count loadDupeRows already takes. Form groups in Go as consecutive rows with an equal size, head, tail and content. Hold only the groups of the current size, sort those by first path, and write them before the next size. If the implementer finds a plainer way that meets the rule above, record the reading on the PR.
stdout stays exactly as it is: same rows, same order (groups by size descending, then by first path, paths ascending), same summary line. The count and the rows still come from one read transaction.
Memory: at most the duplicate paths of one size at a time, stated in README "report mode"; no longer every row in SQLite's sort.
It uses the same display and phase numbering as scan (#112), which lands first, without a second display.
README "Progress" and "report mode" updated.
Tests: the plain lines name each phase with its number and count; stdout unchanged for every existing report test; a case with several groups of one size in an order different from their first paths. Keep each test well under a second (#109).
TODO.md Completed Steps entry in the same commit; script/cibuild green on a fresh clone.

Notes for the implementer

Package main (the repo's only package). Queries stay in db.go, grouping and writing in report.go, the display in progress.go.
Small and plain; no new flags, no new dependencies, no schema change.

Model: opus-5-5

sneak, on https://git.eeqj.de/sneak/sfdupes/issues/56#issuecomment-145000: > The tool should always give the user some feedback to show it is doing something and isn't hung. It should show what phase it's on, out of how many phases, and show phase progress, elapsed, and eta. On `next`, `report` prints nothing until its summary line. One SQL query (`dupeRowsSQL` in `db.go`) groups every record with a content hash and sorts the duplicate rows. SQLite returns no row until it has done all of that, so the wait cannot be measured as written. That is minutes of silence on a 10-million-record database on spinning disks. README "Progress" says "`report` and `trees` modes need no progress display"; sneak's words above replace that. ## Definition of done - From the moment `report` starts until its summary line, stderr always shows which phase it is in out of how many, with that phase's progress, elapsed time and ETA. This holds on a terminal and in plain lines (a line as each phase starts, then at least every 5 seconds whether or not an item completed, and no more often). That includes opening the database and waiting on a busy one, and any count taken to get a total, which shows elapsed time while it runs. - No phase is one long query whose progress cannot be shown. Recommended approach: read the signature index in its own order (or in exact reverse, so no sort is needed), counting rows against the record count `loadDupeRows` already takes. Form groups in Go as consecutive rows with an equal size, head, tail and content. Hold only the groups of the current size, sort those by first path, and write them before the next size. If the implementer finds a plainer way that meets the rule above, record the reading on the PR. - stdout stays exactly as it is: same rows, same order (groups by size descending, then by first path, paths ascending), same summary line. The count and the rows still come from one read transaction. - Memory: at most the duplicate paths of one size at a time, stated in README "`report` mode"; no longer every row in SQLite's sort. - It uses the same display and phase numbering as `scan` (https://git.eeqj.de/sneak/sfdupes/issues/112), which lands first, without a second display. - README "Progress" and "`report` mode" updated. - Tests: the plain lines name each phase with its number and count; stdout unchanged for every existing report test; a case with several groups of one size in an order different from their first paths. Keep each test well under a second (https://git.eeqj.de/sneak/sfdupes/issues/109). - `TODO.md` Completed Steps entry in the same commit; `script/cibuild` green on a fresh clone. ## Notes for the implementer - Package `main` (the repo's only package). Queries stay in `db.go`, grouping and writing in `report.go`, the display in `progress.go`. - Small and plain; no new flags, no new dependencies, no schema change. Model: opus-5-5

clawbot added this to the 1.0.0 milestone 2026-10-16 13:51:43 +02:00

clawbot referenced this issue from a commit

2026-10-16 21:13:21 +02:00

report: show its phases as N of M, with progress, elapsed time and ETA (closes #114)

clawbot referenced a pull request that will close this issue

2026-10-16 21:15:05 +02:00

report: show its phases as N of M, with progress, elapsed time and ETA (closes #114) #122

clawbot commented

2026-10-16 21:15:10 +02:00

Built in #122: report runs in two phases, [1/2] open and [2/2] read, on the display scan and trees use, and reads the records in signature order from the index without a sort. The PR body says how the groups are formed and gives the judgement call.

Model: opus-5-5

Built in https://git.eeqj.de/sneak/sfdupes/pulls/122: `report` runs in two phases, `[1/2] open` and `[2/2] read`, on the display `scan` and `trees` use, and reads the records in signature order from the index without a sort. The PR body says how the groups are formed and gives the judgement call. Model: opus-5-5

clawbot closed this issue

2026-10-16 21:44:56 +02:00

clawbot referenced this issue from a commit

2026-10-16 21:44:57 +02:00

report: show its phases as N of M, with progress, elapsed time and ETA (closes #114)

clawbot referenced this issue

2026-10-16 21:45:18 +02:00

next into main #110

Sign in to join this conversation.