sneak/sfdupes
1
0
Fork 0
Code Issues 3 Pull Requests 1 Actions 12 Packages Projects Releases Wiki Activity

scan: show each phase as N of M, with progress, elapsed time and ETA, also while it waits #112

New Issue
Closed
opened 2026-10-16 13:50:50 +02:00 by clawbot · 1 comment
clawbot commented 2026-10-16 13:50:50 +02:00
Collaborator
Copy Link
Copy Source

sneak, on #56 (comment):

> The tool should always give the user some feedback to show it is doing something and isn't hung. It should show what phase it's on, out of how many phases, and show phase progress, elapsed, and eta.
>
> The scan mode walk doesn't tell you how many phases besides walk there are.

This issue covers scan; report and trees get their own issues.

What is wrong on next

  • scan shows five displays in turn (load, walk, hash, update, content), and none says which phase it is out of how many.
  • load and walk show a count, a rate and elapsed time but no ETA, and the content phase shows none while SQLite looks for the files to read. README "Progress" asks for exactly that; sneak's words above replace it.
  • When stderr is not a terminal (cron), a plain line is printed only when an item completes. A phase waiting on something (a query that has not returned its first row, one slow file, the 10-second busy timeout) prints nothing for as long as it waits.

Definition of done

  • Every display names its phase and its place, from the moment the phase starts, on a terminal and in plain lines: for example [1/5] load, [2/5] walk, [3/5] hash, [4/5] update, [5/5] content. The exact layout is the implementer's. One list in the code defines the phases, so the numbers cannot drift. The README counts five phases (load included) wherever it counts them.
  • Every phase shows its progress, elapsed time and ETA:
    • load: total from a count of the records taken first. While that count runs, the display already shows the phase and its elapsed time.
    • walk: estimated total = the records under the operands from the last scan, known once load is done, shown as an estimate (for example ~), with percent and ETA from it. If the walk passes the estimate, the ETA reads unknown, not a wrong or negative time. With no records under the operands (the first scan), the display says the ETA is unknown.
    • hash and update: as now, plus the phase number.
    • content: while SQLite searches for the files to read, the display shows the phase and elapsed time and says it is searching the database; then the checks and reads with totals and ETA, as now.
  • A phase that is waiting keeps showing it is alive. On a terminal the elapsed time keeps moving, for bars as well as spinners. In plain mode a line is printed at least every 5 seconds whether or not an item completed, and no more often than that.
  • README "Progress", "scan mode" and "Daily scan from cron" updated to match.
  • Tests: plain lines carry the phase number and count; plain mode prints during a wait with no completed items; the walk estimate comes from the records under the operands, and passing it gives an unknown ETA. Keep each test well under a second (no long sleeps; make the interval or clock settable in tests), since make test already runs close to its limit (#109).
  • TODO.md Completed Steps entry in the same commit; script/cibuild green on a fresh clone.

Notes for the implementer

  • Package main (the repo's only package). The display belongs in progress.go, whose doc covers rendering a phase's progress; the phase list and the totals belong to the scan in scan.go; the record count query goes with the other queries in db.go.
  • Printing a plain line on a timer means a second goroutine touches the display. Guard the shared state with one mutex and keep it obvious; workers must never block on the display.
  • Keep the change small and plain: no new flags, no new dependencies.

Model: opus-5-5

sneak, on https://git.eeqj.de/sneak/sfdupes/issues/56#issuecomment-145000: > The tool should always give the user some feedback to show it is doing something and isn't hung. It should show what phase it's on, out of how many phases, and show phase progress, elapsed, and eta. > > The scan mode walk doesn't tell you how many phases besides walk there are. This issue covers `scan`; `report` and `trees` get their own issues. ## What is wrong on `next` - `scan` shows five displays in turn (`load`, `walk`, `hash`, `update`, `content`), and none says which phase it is out of how many. - `load` and `walk` show a count, a rate and elapsed time but no ETA, and the content phase shows none while SQLite looks for the files to read. README "Progress" asks for exactly that; sneak's words above replace it. - When stderr is not a terminal (cron), a plain line is printed only when an item completes. A phase waiting on something (a query that has not returned its first row, one slow file, the 10-second busy timeout) prints nothing for as long as it waits. ## Definition of done - Every display names its phase and its place, from the moment the phase starts, on a terminal and in plain lines: for example `[1/5] load`, `[2/5] walk`, `[3/5] hash`, `[4/5] update`, `[5/5] content`. The exact layout is the implementer's. One list in the code defines the phases, so the numbers cannot drift. The README counts five phases (load included) wherever it counts them. - Every phase shows its progress, elapsed time and ETA: - `load`: total from a count of the records taken first. While that count runs, the display already shows the phase and its elapsed time. - `walk`: estimated total = the records under the operands from the last scan, known once `load` is done, shown as an estimate (for example `~`), with percent and ETA from it. If the walk passes the estimate, the ETA reads unknown, not a wrong or negative time. With no records under the operands (the first scan), the display says the ETA is unknown. - `hash` and `update`: as now, plus the phase number. - `content`: while SQLite searches for the files to read, the display shows the phase and elapsed time and says it is searching the database; then the checks and reads with totals and ETA, as now. - A phase that is waiting keeps showing it is alive. On a terminal the elapsed time keeps moving, for bars as well as spinners. In plain mode a line is printed at least every 5 seconds whether or not an item completed, and no more often than that. - README "Progress", "`scan` mode" and "Daily scan from cron" updated to match. - Tests: plain lines carry the phase number and count; plain mode prints during a wait with no completed items; the walk estimate comes from the records under the operands, and passing it gives an unknown ETA. Keep each test well under a second (no long sleeps; make the interval or clock settable in tests), since `make test` already runs close to its limit (https://git.eeqj.de/sneak/sfdupes/issues/109). - `TODO.md` Completed Steps entry in the same commit; `script/cibuild` green on a fresh clone. ## Notes for the implementer - Package `main` (the repo's only package). The display belongs in `progress.go`, whose doc covers rendering a phase's progress; the phase list and the totals belong to the scan in `scan.go`; the record count query goes with the other queries in `db.go`. - Printing a plain line on a timer means a second goroutine touches the display. Guard the shared state with one mutex and keep it obvious; workers must never block on the display. - Keep the change small and plain: no new flags, no new dependencies. Model: opus-5-5
clawbot added this to the 1.0.0 milestone 2026-10-16 13:50:50 +02:00
clawbot commented 2026-10-16 15:50:34 +02:00
Author
Collaborator
Copy Link
Copy Source

Built in #120. Every scan display now names its phase as [n/5] from its first line, with progress, rate, elapsed time and ETA, and keeps showing that it is alive while it waits. The PR body has the details and two judgement calls.

Model: opus-5-5

Built in https://git.eeqj.de/sneak/sfdupes/pulls/120. Every `scan` display now names its phase as `[n/5]` from its first line, with progress, rate, elapsed time and ETA, and keeps showing that it is alive while it waits. The PR body has the details and two judgement calls. Model: opus-5-5
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
critical
Extremely important: data loss, a security hole, production down, or basic use impossible
 merge-ready
Review passed; ready to merge
 needs-checks
CI checks failing or not yet green
 needs-rebase
Branch conflicts with main; rebase required
 needs-review
PR awaiting independent adversarial review
 needs-rework
Review found blocking problems; author must rework
No labels
Milestone
No items
No Milestone 1.0.0
Projects
Clear projects
No projects
Assignees
Clear assignees
clawbot sneak (Jeffrey Paul)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: sneak/sfdupes#112