trees: show its phases as N of M, with progress, elapsed time and ETA (closes #113)
check / check (push) Canceled after 0s

trees now runs in four phases, open, read, group and write, each under
the progress display scan uses, from the line naming the database to
the summary line. open covers a wait on a busy database. read counts
the records first, as scan's load does, then reads them against that
count; group counts the directories, and write the groups. The display
gains a unit so that each rate names what its phase counts; every scan
phase passes files and prints as before. The tests that pinned trees'
stderr now allow the phase lines between the database line and the
summary.

Judgement call: opening the database is a phase of its own, so there are four phases, not the three the issue sketches.
Judgement call: the count and the read are separate queries, as in scan's load, so during a scan the read can end slightly past or short of its total.

Model: opus-5-5
This commit is contained in:
2026-10-16 17:23:52 +00:00
parent 8dc73aa3f5
commit f838410bd0
9 changed files with 324 additions and 82 deletions
+35 -16
View File
@@ -695,6 +695,12 @@ Processing:
- Order groups by total tree size descending, tie-broken by `first` path
ascending. Output must be fully deterministic for a given input.
`trees` runs in four phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `read` counts the records, then reads
them in path order, building the hierarchy and computing the digests as it goes;
`group` groups the directories by digest and picks the maximal groups; `write`
writes the groups to stdout.
#### Trees output format
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
@@ -715,13 +721,16 @@ units.
### Progress
Use the progress-bar library for all scan progress; rendering in the style of
`pv` is the model. All progress goes to stderr.
Use the progress-bar library for all progress of `scan` and `trees`; rendering
in the style of `pv` is the model. All progress goes to stderr.
A scan must always show that it is working and not hung. Each phase gets its own
display from the moment it starts, naming the phase and its place among the five
(`[1/5] load`, `[2/5] walk`, `[3/5] hash`, `[4/5] update`, `[5/5] content`) and
showing its progress, its elapsed time and its estimated time remaining (ETA):
`scan` and `trees` must always show that they are working and not hung. Each
phase gets its own display from the moment it starts, naming the phase and its
place among the command's phases and showing its progress, its elapsed time and
its estimated time remaining (ETA).
A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
`[4/5] update`, `[5/5] content`):
- `load` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
@@ -738,12 +747,23 @@ showing its progress, its elapsed time and its estimated time remaining (ETA):
and ETA, start there. The files to read then get a display of their own, with
their count as its total.
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
`[4/4] write`, see "`trees` mode"):
- `open` has no total. While it opens the database, waiting on a busy one
included, the display says it is opening the database.
- `read` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
- `group` has the directories `read` found as its total, and `write` the
duplicate-tree groups.
Each display is one line holding:
- the phase and its place among the phases
- a `[m/n] x%` display (items processed / total items, percent); before a phase
has a total, the items processed, or what the phase is waiting for
- current rate (items/s)
- current rate (items/s), naming the items the phase counts: files, records,
directories or groups
- elapsed time
- estimated time remaining, or `unknown`
@@ -756,13 +776,13 @@ Example shape (exact layout is flexible, content is not):
Additional requirements:
- On a terminal, a phase with an exact total (every phase but `walk`, once it
has counted what it will do) is drawn as its line followed by the progress-bar
library's bar, 10 columns wide, which shows the percent in place of the line's
own. Any other phase, and a phase whose bar has no room, is drawn as the
library's spinner followed by its line. Each display is drawn again several
times a second, so its elapsed time keeps moving while a phase waits for its
next item.
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`
and `trees`' `open`, once it has counted what it will do) is drawn as its line
followed by the progress-bar library's bar, 10 columns wide, which shows the
percent in place of the line's own. Any other phase, and a phase whose bar has
no room, is drawn as the library's spinner followed by its line. Each display
is drawn again several times a second, so its elapsed time keeps moving while
a phase waits for its next item.
- On a terminal no line reaches the terminal's last column, however narrow the
terminal is. The phase and its place, the progress, the elapsed time and the
ETA are kept whole: a line too wide for the terminal leaves out its rate first
@@ -784,8 +804,7 @@ Additional requirements:
inside the progress display.
- A phase that stops short of its total, as an interrupted one does, is left
showing the count it stopped at rather than filled up.
- `report` and `trees` modes need no progress display, only their stderr
summaries.
- `report` mode needs no progress display, only its stderr summary.
### Error handling and exit codes