trees: show its phases as N of M, with progress, elapsed time and ETA (closes #113)
check / check (push) Canceled after 0s

trees now runs in four phases, open, read, group and write, each under
the progress display scan uses, until the summary line. open covers a
wait on a busy database. read counts the records and reads them in one
read-only transaction, so the two agree during a scan; group counts the
directories, and write the groups. The rows reach stdout through a
writer that clears the display and holds its lock while writing, so no
row shares a line with the display on a shared terminal. Each rate
names what its phase counts.

Judgement call: opening the database is a phase of its own, so there are four phases, not the three the issue sketches.

Model: opus-5-5
This commit is contained in:
2026-10-16 17:58:04 +00:00
committed by sneak
parent 8dc73aa3f5
commit 510b545ae5
11 changed files with 566 additions and 99 deletions
+37 -17
View File
@@ -695,6 +695,12 @@ Processing:
- Order groups by total tree size descending, tie-broken by `first` path
ascending. Output must be fully deterministic for a given input.
`trees` runs in four phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `read` counts the records, then reads
them in path order, building the hierarchy and computing the digests as it goes;
`group` groups the directories by digest and picks the maximal groups; `write`
writes the groups to stdout.
#### Trees output format
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
@@ -715,13 +721,16 @@ units.
### Progress
Use the progress-bar library for all scan progress; rendering in the style of
`pv` is the model. All progress goes to stderr.
Use the progress-bar library for all progress of `scan` and `trees`; rendering
in the style of `pv` is the model. All progress goes to stderr.
A scan must always show that it is working and not hung. Each phase gets its own
display from the moment it starts, naming the phase and its place among the five
(`[1/5] load`, `[2/5] walk`, `[3/5] hash`, `[4/5] update`, `[5/5] content`) and
showing its progress, its elapsed time and its estimated time remaining (ETA):
`scan` and `trees` must always show that they are working and not hung. Each
phase gets its own display from the moment it starts, naming the phase and its
place among the command's phases and showing its progress, its elapsed time and
its estimated time remaining (ETA).
A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
`[4/5] update`, `[5/5] content`):
- `load` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
@@ -738,12 +747,23 @@ showing its progress, its elapsed time and its estimated time remaining (ETA):
and ETA, start there. The files to read then get a display of their own, with
their count as its total.
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
`[4/4] write`, see "`trees` mode"):
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `read` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
- `group` has the directories `read` found as its total, and `write` the
duplicate-tree groups.
Each display is one line holding:
- the phase and its place among the phases
- a `[m/n] x%` display (items processed / total items, percent); before a phase
has a total, the items processed, or what the phase is waiting for
- current rate (items/s)
- current rate (items/s), naming the items the phase counts: files, records,
directories or groups
- elapsed time
- estimated time remaining, or `unknown`
@@ -756,13 +776,13 @@ Example shape (exact layout is flexible, content is not):
Additional requirements:
- On a terminal, a phase with an exact total (every phase but `walk`, once it
has counted what it will do) is drawn as its line followed by the progress-bar
library's bar, 10 columns wide, which shows the percent in place of the line's
own. Any other phase, and a phase whose bar has no room, is drawn as the
library's spinner followed by its line. Each display is drawn again several
times a second, so its elapsed time keeps moving while a phase waits for its
next item.
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`
and `trees`' `open`, once it has counted what it will do) is drawn as its line
followed by the progress-bar library's bar, 10 columns wide, which shows the
percent in place of the line's own. Any other phase, and a phase whose bar has
no room, is drawn as the library's spinner followed by its line. Each display
is drawn again several times a second, so its elapsed time keeps moving while
a phase waits for its next item.
- On a terminal no line reaches the terminal's last column, however narrow the
terminal is. The phase and its place, the progress, the elapsed time and the
ETA are kept whole: a line too wide for the terminal leaves out its rate first
@@ -781,11 +801,11 @@ Additional requirements:
goroutine counts the items, and each display draws them from a goroutine of
its own.
- A warning printed during a phase always lands on a line of its own, never
inside the progress display.
inside the progress display. So does every line written to stdout when stdout
is the same terminal, as in a plain `sfdupes trees`.
- A phase that stops short of its total, as an interrupted one does, is left
showing the count it stopped at rather than filled up.
- `report` and `trees` modes need no progress display, only their stderr
summaries.
- `report` mode needs no progress display, only its stderr summary.
### Error handling and exit codes