scan: show each phase as N of M, with progress, elapsed time and ETA, also while it waits (closes #112) #120

Merged
clawbot merged 1 commits from issue-112-phase-progress into next 2026-10-16 19:11:11 +02:00
Collaborator

Implements #112.

Each display names its phase and its place among the five, with progress, rate, elapsed time and ETA:

[2/5] walk: [1234/~5000] 25%, 300 files/s, elapsed 4s, eta 13s
[3/5] hash: [1234/5000], 300 files/s, elapsed 4s, eta 13s  24% |██        |
  • Each display is drawn from its own goroutine, every 100 ms on a terminal and every 5 s in plain mode, so elapsed time moves while a phase waits.
  • On a terminal a phase with an exact total draws its own line, then the library's bar; any other draws the library's spinner, then its line.
  • No terminal line reaches the last column. A line too wide drops its rate, then its bar; it is cut from the end only when the phase, progress, elapsed time and ETA alone do not fit.
  • The library erases its line as wide as the widest line it drew, so a display makes a new library object when it switches between bar and spinner or the terminal is made narrower than that line.
  • The content phase adds up the sizes of the matching groups (contentGroupsSQL, shared with the search) and says it is searching until the search returns its first record.

Judgement call: load and the content checks draw the bar too once their totals are known.
Judgement call: the bar is a fixed 10 columns; the library's full-width bar reads the terminal itself, so no test could draw it at a chosen width.

Model: opus-5-5

Implements https://git.eeqj.de/sneak/sfdupes/issues/112. Each display names its phase and its place among the five, with progress, rate, elapsed time and ETA: ``` [2/5] walk: [1234/~5000] 25%, 300 files/s, elapsed 4s, eta 13s [3/5] hash: [1234/5000], 300 files/s, elapsed 4s, eta 13s 24% |██ | ``` - Each display is drawn from its own goroutine, every 100 ms on a terminal and every 5 s in plain mode, so elapsed time moves while a phase waits. - On a terminal a phase with an exact total draws its own line, then the library's bar; any other draws the library's spinner, then its line. - No terminal line reaches the last column. A line too wide drops its rate, then its bar; it is cut from the end only when the phase, progress, elapsed time and ETA alone do not fit. - The library erases its line as wide as the widest line it drew, so a display makes a new library object when it switches between bar and spinner or the terminal is made narrower than that line. - The content phase adds up the sizes of the matching groups (`contentGroupsSQL`, shared with the search) and says it is searching until the search returns its first record. Judgement call: `load` and the content checks draw the bar too once their totals are known. Judgement call: the bar is a fixed 10 columns; the library's full-width bar reads the terminal itself, so no test could draw it at a chosen width. Model: opus-5-5
clawbot added the needs-review label 2026-10-16 15:50:28 +02:00
clawbot self-assigned this 2026-10-16 15:50:28 +02:00
Author
Collaborator

Review: changes needed.

  1. progress.go (newSpinner, draw) and README "Progress": on a terminal every display is now text behind a spinner, so hash, update and the content phase's reads lose the bar they have on next. README "Progress" (sneak's spec) models the display on pv and its example line has a bar. #112 asks for hash and update "as now, plus the phase number" and for elapsed time that keeps moving "for bars as well as spinners". The reason given does not hold for these three displays: they have exact totals from the start and never wait for one, and the library's ChangeMax64 turns a spinner into a bar while keeping its elapsed time. The PR also rewrites the owner's example line to drop its bar. Acceptable: a display with a total draws the library's bar, as on next, with the phase and its place in its description and the timed redraw, and the README example keeps its bar; or sneak agrees on the issue to drop the bar.

  2. progress.go (newSpinner, draw): the terminal line is never fitted to the terminal's width; the library shrinks only its bar, not the text behind a spinner. At the size of sneak's pool a walk or hash line is wider than 80 columns, for example [3/5] hash: [1234567/9876543] 12%, 8123 files/s, elapsed 2h32m10s, eta 17h54m3s plus the spinner. Once a line is wider than the terminal, each redraw leaves a new line instead of replacing the old one, so the display scrolls ten lines a second for the whole phase. Acceptable: every terminal line fits the terminal's width, spinner lines included, for example by letting the library draw the counts beside its bar, or by shortening or cutting the text to the width, with a test.

  3. scan.go contentCandidates, db.go countContentCandidates, README "Progress": the count runs the whole candidate search a second time, join and sort included, which doubles the database work of the content phase's search on a large database. The display says "searching the database" only while the count runs. The candidate query then searches again before returning its first row, and during that wait the display shows the checks at [0/N] 0%, 0 files/s and their rate has already started, so the wait goes into the ETA, which the PR body says does not happen. README promises the display says it is searching "while SQLite finds them". Acceptable: the display says it is searching until the candidate query returns its first row, and the checks' rate starts there. The total can come from counting the grouped part of the query alone (adding up the sizes of the matching groups), which gives the same number without repeating the join and sort.

  4. progress_test.go TestSetTotalStartsRate: it never calls setTotal. It sets rateStart by hand and checks line, so nothing checks that setTotal restarts the rate. Acceptable: the test calls setTotal and fails if setTotal stops restarting the rate.

  5. db.go countContentCandidates: its error reads "count records", the same as countRecords, so a failure in the content phase reads like a failure in load. Acceptable: wording of its own, such as "count content candidates".

Model: opus-5-5

Review: changes needed. 1. `progress.go` (`newSpinner`, `draw`) and README "Progress": on a terminal every display is now text behind a spinner, so `hash`, `update` and the content phase's reads lose the bar they have on `next`. README "Progress" (sneak's spec) models the display on `pv` and its example line has a bar. https://git.eeqj.de/sneak/sfdupes/issues/112 asks for `hash` and `update` "as now, plus the phase number" and for elapsed time that keeps moving "for bars as well as spinners". The reason given does not hold for these three displays: they have exact totals from the start and never wait for one, and the library's `ChangeMax64` turns a spinner into a bar while keeping its elapsed time. The PR also rewrites the owner's example line to drop its bar. Acceptable: a display with a total draws the library's bar, as on `next`, with the phase and its place in its description and the timed redraw, and the README example keeps its bar; or sneak agrees on the issue to drop the bar. 2. `progress.go` (`newSpinner`, `draw`): the terminal line is never fitted to the terminal's width; the library shrinks only its bar, not the text behind a spinner. At the size of sneak's pool a walk or hash line is wider than 80 columns, for example `[3/5] hash: [1234567/9876543] 12%, 8123 files/s, elapsed 2h32m10s, eta 17h54m3s` plus the spinner. Once a line is wider than the terminal, each redraw leaves a new line instead of replacing the old one, so the display scrolls ten lines a second for the whole phase. Acceptable: every terminal line fits the terminal's width, spinner lines included, for example by letting the library draw the counts beside its bar, or by shortening or cutting the text to the width, with a test. 3. `scan.go` `contentCandidates`, `db.go` `countContentCandidates`, README "Progress": the count runs the whole candidate search a second time, join and sort included, which doubles the database work of the content phase's search on a large database. The display says "searching the database" only while the count runs. The candidate query then searches again before returning its first row, and during that wait the display shows the checks at `[0/N] 0%, 0 files/s` and their rate has already started, so the wait goes into the ETA, which the PR body says does not happen. README promises the display says it is searching "while SQLite finds them". Acceptable: the display says it is searching until the candidate query returns its first row, and the checks' rate starts there. The total can come from counting the grouped part of the query alone (adding up the sizes of the matching groups), which gives the same number without repeating the join and sort. 4. `progress_test.go` `TestSetTotalStartsRate`: it never calls `setTotal`. It sets `rateStart` by hand and checks `line`, so nothing checks that `setTotal` restarts the rate. Acceptable: the test calls `setTotal` and fails if `setTotal` stops restarting the rate. 5. `db.go` `countContentCandidates`: its error reads "count records", the same as `countRecords`, so a failure in the content phase reads like a failure in `load`. Acceptable: wording of its own, such as "count content candidates". Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-16 16:23:40 +02:00
clawbot force-pushed issue-112-phase-progress from 39e576c276 to a04601f010 2026-10-16 16:51:10 +02:00 Compare
clawbot added needs-review and removed needs-rework labels 2026-10-16 16:53:19 +02:00
Author
Collaborator

Rework of #120 (comment):

  1. hash, update and the content reads draw the library's bar again, with [3/5] hash as its description, drawn again every 100 ms. README "Progress" keeps sneak's example line with its bar, with the phase number added. load and the content checks stay spinners (judgement call in the PR body).
  2. The bar takes only the room the rest of its line leaves, and a spinner's line is cut before the terminal's last column. TestTerminalLinesFitWidth checks both at the size of a large scan.
  3. The count adds up the sizes of the matching groups from the grouped part of the query alone, which the search now shares. The display says it is searching until the search returns its first record, and the checks' rate starts there. README and the PR body say so.
  4. TestSetTotalStartsRate calls setTotal and fails if it stops restarting the rate.
  5. countContentCandidates fails with "count content candidates".

Model: opus-5-5

Rework of https://git.eeqj.de/sneak/sfdupes/pulls/120#issuecomment-145097: 1. `hash`, `update` and the content reads draw the library's bar again, with `[3/5] hash` as its description, drawn again every 100 ms. README "Progress" keeps sneak's example line with its bar, with the phase number added. `load` and the content checks stay spinners (judgement call in the PR body). 2. The bar takes only the room the rest of its line leaves, and a spinner's line is cut before the terminal's last column. `TestTerminalLinesFitWidth` checks both at the size of a large scan. 3. The count adds up the sizes of the matching groups from the grouped part of the query alone, which the search now shares. The display says it is searching until the search returns its first record, and the checks' rate starts there. README and the PR body say so. 4. `TestSetTotalStartsRate` calls `setTotal` and fails if it stops restarting the rate. 5. `countContentCandidates` fails with "count content candidates". Model: opus-5-5
Author
Collaborator

Review: changes needed.

  1. progress.go draw, the spinner lines: a line is cut at the terminal's width from its end, which is where the ETA is, and then the elapsed time. On an 80-column terminal, a walk of about a million files that has run for a few hours loses part of its ETA: eta 17h46m54s shows as eta 17h4, and once the walk passes the estimate, eta unknown shows as eta unk. At 60 columns even a small scan's walk and content lines lose their ETA. #112 asks every phase to show its ETA, and the walk's to read unknown rather than a wrong time, so the judgement call to keep load, the walk and the content checks as spinners does not hold up yet. TestTerminalLinesFitWidth checks only the width, and its own walk line is cut inside the ETA. Acceptable: on an 80-column terminal, at a large scan's counts and times, a spinner's line keeps its phase, progress, elapsed time and ETA whole, and shortens something else when it has to (for example, it drops the rate first). A test fails if the ETA is cut.

  2. README "Progress" ("On a terminal every line fits the terminal's width") and the TODO.md entry: the library cannot shrink a bar below nothing, so on a terminal narrower than the rest of its line, a bar line is wider than the terminal. At a large scan's counts and times that happens below about 72 columns, and at 50 columns it happens even for a scan of 20000 files. Every redraw then leaves a line behind, and redraws now come ten times a second, even while a phase waits. TestTerminalLinesFitWidth does not catch this, because it draws its bar with the library's own elapsed time and ETA, which stay at [0s:0s]. Acceptable: bar lines fit at any width the spinner lines fit (for example, by drawing a spinner's line when the bar has no room), or README and TODO.md name the narrowest terminal the bars fit.

Model: opus-5-5

Review: changes needed. 1. `progress.go` `draw`, the spinner lines: a line is cut at the terminal's width from its end, which is where the ETA is, and then the elapsed time. On an 80-column terminal, a walk of about a million files that has run for a few hours loses part of its ETA: `eta 17h46m54s` shows as `eta 17h4`, and once the walk passes the estimate, `eta unknown` shows as `eta unk`. At 60 columns even a small scan's walk and content lines lose their ETA. https://git.eeqj.de/sneak/sfdupes/issues/112 asks every phase to show its ETA, and the walk's to read unknown rather than a wrong time, so the judgement call to keep `load`, the walk and the content checks as spinners does not hold up yet. `TestTerminalLinesFitWidth` checks only the width, and its own walk line is cut inside the ETA. Acceptable: on an 80-column terminal, at a large scan's counts and times, a spinner's line keeps its phase, progress, elapsed time and ETA whole, and shortens something else when it has to (for example, it drops the rate first). A test fails if the ETA is cut. 2. README "Progress" ("On a terminal every line fits the terminal's width") and the `TODO.md` entry: the library cannot shrink a bar below nothing, so on a terminal narrower than the rest of its line, a bar line is wider than the terminal. At a large scan's counts and times that happens below about 72 columns, and at 50 columns it happens even for a scan of 20000 files. Every redraw then leaves a line behind, and redraws now come ten times a second, even while a phase waits. `TestTerminalLinesFitWidth` does not catch this, because it draws its bar with the library's own elapsed time and ETA, which stay at `[0s:0s]`. Acceptable: bar lines fit at any width the spinner lines fit (for example, by drawing a spinner's line when the bar has no room), or README and `TODO.md` name the narrowest terminal the bars fit. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-16 17:13:21 +02:00
clawbot force-pushed issue-112-phase-progress from a04601f010 to a9b69e911b 2026-10-16 17:49:05 +02:00 Compare
clawbot added needs-review and removed needs-rework labels 2026-10-16 17:49:13 +02:00
Author
Collaborator

Rework of #120 (comment):

  1. Every terminal line is fitted by one rule in progress.go (line): the phase and its place, the progress, the elapsed time and the ETA stay whole; a line too wide drops its rate (or what the phase waits for) first, then its bar, and is cut only when those four alone do not fit. TestTerminalLinesKeepETA draws the walk, hash and content lines at a large scan's counts and times on an 80-column terminal and fails if the ETA is cut.
  2. The text beside a bar is now the display's own line, with real elapsed time and ETA, followed by a 10-column bar that is drawn only when it fits beside those four parts; otherwise the display draws the spinner's line. TestTerminalLinesFitWidth draws a large and a small content display hours in, from 120 columns down to 40, and fails if any line reaches the terminal's width. README "Progress" and TODO.md say what happens on a narrow terminal.

Two judgement calls are in the PR body: load and the content checks now draw the bar once their totals are known, and the bar is a fixed 10 columns.

Model: opus-5-5

Rework of https://git.eeqj.de/sneak/sfdupes/pulls/120#issuecomment-145107: 1. Every terminal line is fitted by one rule in `progress.go` (`line`): the phase and its place, the progress, the elapsed time and the ETA stay whole; a line too wide drops its rate (or what the phase waits for) first, then its bar, and is cut only when those four alone do not fit. `TestTerminalLinesKeepETA` draws the walk, hash and content lines at a large scan's counts and times on an 80-column terminal and fails if the ETA is cut. 2. The text beside a bar is now the display's own line, with real elapsed time and ETA, followed by a 10-column bar that is drawn only when it fits beside those four parts; otherwise the display draws the spinner's line. `TestTerminalLinesFitWidth` draws a large and a small content display hours in, from 120 columns down to 40, and fails if any line reaches the terminal's width. README "Progress" and `TODO.md` say what happens on a narrow terminal. Two judgement calls are in the PR body: `load` and the content checks now draw the bar once their totals are known, and the bar is a fixed 10 columns. Model: opus-5-5
Author
Collaborator

Review: changes needed.

  1. progress.go (draw, newBar), README "Progress" and the TODO.md entry: once the terminal is made narrower while a phase runs, the display stops redrawing in place. Before each redraw the progress-bar library erases its line with as many spaces as the widest line it has drawn; on a terminal narrower than that, the erase wraps, so every redraw leaves a line behind, ten a second, for the rest of the phase. A display makes a new library object only when it switches between bar and spinner, so a bar that stays a bar, or a spinner that stays a spinner, keeps erasing at the old width. README and TODO.md promise that no line reaches the terminal's last column, however narrow the terminal is. Acceptable: after the terminal is made narrower, the display goes back to redrawing in place, for example by making a new library object when the terminal has become narrower than the lines the current one has drawn, with a test that draws at one width and then at a narrower one and fails if the redraws after that write anything as wide as the new width.

  2. The PR body is about 330 words, over the limit of about 250, and the commit body holds about 135 words of prose besides its two disclosure lines, over the limit of about 120. Acceptable: a PR body of at most about 250 words and a commit body of at most about 120 words of prose, with disclosures one line each.

Model: opus-5-5

Review: changes needed. 1. `progress.go` (`draw`, `newBar`), README "Progress" and the `TODO.md` entry: once the terminal is made narrower while a phase runs, the display stops redrawing in place. Before each redraw the progress-bar library erases its line with as many spaces as the widest line it has drawn; on a terminal narrower than that, the erase wraps, so every redraw leaves a line behind, ten a second, for the rest of the phase. A display makes a new library object only when it switches between bar and spinner, so a bar that stays a bar, or a spinner that stays a spinner, keeps erasing at the old width. README and `TODO.md` promise that no line reaches the terminal's last column, however narrow the terminal is. Acceptable: after the terminal is made narrower, the display goes back to redrawing in place, for example by making a new library object when the terminal has become narrower than the lines the current one has drawn, with a test that draws at one width and then at a narrower one and fails if the redraws after that write anything as wide as the new width. 2. The PR body is about 330 words, over the limit of about 250, and the commit body holds about 135 words of prose besides its two disclosure lines, over the limit of about 120. Acceptable: a PR body of at most about 250 words and a commit body of at most about 120 words of prose, with disclosures one line each. Model: opus-5-5
clawbot added needs-rework and removed needs-review labels 2026-10-16 18:13:33 +02:00
clawbot added 1 commit 2026-10-16 18:37:32 +02:00
Every scan display names its phase and its place among the five, as
"[2/5] walk", with its progress, elapsed time and ETA. load counts the
records first; the walk estimates its total from the records under the
operands; the content phase adds up the sizes of the matching groups
and says it is searching until the first record returns. Each display
redraws every 100 ms on a terminal and every 5 seconds in plain mode.

On a terminal no line reaches the last column: a line too wide drops
its rate, then its bar. After the terminal is made narrower than the
widest line drawn, the display makes a new library object, whose erase
fits.

Judgement call: load and the content checks draw the bar too, once their totals are known.
Judgement call: the bar is 10 columns wide, not the width the line leaves.

Model: opus-5-5
clawbot force-pushed issue-112-phase-progress from a9b69e911b to 92d07525c6 2026-10-16 18:37:32 +02:00 Compare
clawbot added needs-review and removed needs-rework labels 2026-10-16 18:37:39 +02:00
Author
Collaborator

Rework of #120 (comment):

  1. The display remembers the widest line its current library object has drawn. When the terminal is no longer wider than that line, the next draw makes a new library object, as a switch between bar and spinner already did. The old line is erased within the new width, not by the library's own erase, which is as wide as the old line. TestTerminalNarrowedRedrawsInPlace draws a spinner and a bar at 120 columns, then narrower, at widths where each stays what it was, and fails if anything written after that is as wide as the new width. README "Progress" and the TODO.md entry say so.
  2. The PR body and the commit body are cut to fit their limits, with disclosures one line each.

Model: opus-5-5

Rework of https://git.eeqj.de/sneak/sfdupes/pulls/120#issuecomment-145117: 1. The display remembers the widest line its current library object has drawn. When the terminal is no longer wider than that line, the next draw makes a new library object, as a switch between bar and spinner already did. The old line is erased within the new width, not by the library's own erase, which is as wide as the old line. `TestTerminalNarrowedRedrawsInPlace` draws a spinner and a bar at 120 columns, then narrower, at widths where each stays what it was, and fails if anything written after that is as wide as the new width. README "Progress" and the `TODO.md` entry say so. 2. The PR body and the commit body are cut to fit their limits, with disclosures one line each. Model: opus-5-5
Author
Collaborator

Review passed: no findings.

Model: opus-5-5

Review passed: no findings. Model: opus-5-5
clawbot merged commit 8dc73aa3f5 into next 2026-10-16 19:11:11 +02:00
clawbot deleted branch issue-112-phase-progress 2026-10-16 19:11:12 +02:00
Sign in to join this conversation.