trees: show its phases as N of M, with progress, elapsed time and ETA (closes #113)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
trees now runs in four phases, open, read, group and write, each under the progress display scan uses, from the line naming the database to the summary line. open covers a wait on a busy database. read counts the records first, as scan's load does, then reads them against that count; group counts the directories, and write the groups. The display gains a unit so that each rate names what its phase counts; every scan phase passes files and prints as before. The tests that pinned trees' stderr now allow the phase lines between the database line and the summary. Judgement call: opening the database is a phase of its own, so there are four phases, not the three the issue sketches. Judgement call: the count and the read are separate queries, as in scan's load, so during a scan the read can end slightly past or short of its total. Model: opus-5-5
This commit is contained in:
@@ -695,6 +695,12 @@ Processing:
|
||||
- Order groups by total tree size descending, tie-broken by `first` path
|
||||
ascending. Output must be fully deterministic for a given input.
|
||||
|
||||
`trees` runs in four phases, each with its own progress display on stderr (see
|
||||
"Progress"): `open` opens the database; `read` counts the records, then reads
|
||||
them in path order, building the hierarchy and computing the digests as it goes;
|
||||
`group` groups the directories by digest and picks the maximal groups; `write`
|
||||
writes the groups to stdout.
|
||||
|
||||
#### Trees output format
|
||||
|
||||
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
|
||||
@@ -715,13 +721,16 @@ units.
|
||||
|
||||
### Progress
|
||||
|
||||
Use the progress-bar library for all scan progress; rendering in the style of
|
||||
`pv` is the model. All progress goes to stderr.
|
||||
Use the progress-bar library for all progress of `scan` and `trees`; rendering
|
||||
in the style of `pv` is the model. All progress goes to stderr.
|
||||
|
||||
A scan must always show that it is working and not hung. Each phase gets its own
|
||||
display from the moment it starts, naming the phase and its place among the five
|
||||
(`[1/5] load`, `[2/5] walk`, `[3/5] hash`, `[4/5] update`, `[5/5] content`) and
|
||||
showing its progress, its elapsed time and its estimated time remaining (ETA):
|
||||
`scan` and `trees` must always show that they are working and not hung. Each
|
||||
phase gets its own display from the moment it starts, naming the phase and its
|
||||
place among the command's phases and showing its progress, its elapsed time and
|
||||
its estimated time remaining (ETA).
|
||||
|
||||
A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
|
||||
`[4/5] update`, `[5/5] content`):
|
||||
|
||||
- `load` counts the records first, then reads them against that count. While the
|
||||
count runs, the display says it is counting the records.
|
||||
@@ -738,12 +747,23 @@ showing its progress, its elapsed time and its estimated time remaining (ETA):
|
||||
and ETA, start there. The files to read then get a display of their own, with
|
||||
their count as its total.
|
||||
|
||||
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
|
||||
`[4/4] write`, see "`trees` mode"):
|
||||
|
||||
- `open` has no total. While it opens the database, waiting on a busy one
|
||||
included, the display says it is opening the database.
|
||||
- `read` counts the records first, then reads them against that count. While the
|
||||
count runs, the display says it is counting the records.
|
||||
- `group` has the directories `read` found as its total, and `write` the
|
||||
duplicate-tree groups.
|
||||
|
||||
Each display is one line holding:
|
||||
|
||||
- the phase and its place among the phases
|
||||
- a `[m/n] x%` display (items processed / total items, percent); before a phase
|
||||
has a total, the items processed, or what the phase is waiting for
|
||||
- current rate (items/s)
|
||||
- current rate (items/s), naming the items the phase counts: files, records,
|
||||
directories or groups
|
||||
- elapsed time
|
||||
- estimated time remaining, or `unknown`
|
||||
|
||||
@@ -756,13 +776,13 @@ Example shape (exact layout is flexible, content is not):
|
||||
|
||||
Additional requirements:
|
||||
|
||||
- On a terminal, a phase with an exact total (every phase but `walk`, once it
|
||||
has counted what it will do) is drawn as its line followed by the progress-bar
|
||||
library's bar, 10 columns wide, which shows the percent in place of the line's
|
||||
own. Any other phase, and a phase whose bar has no room, is drawn as the
|
||||
library's spinner followed by its line. Each display is drawn again several
|
||||
times a second, so its elapsed time keeps moving while a phase waits for its
|
||||
next item.
|
||||
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`
|
||||
and `trees`' `open`, once it has counted what it will do) is drawn as its line
|
||||
followed by the progress-bar library's bar, 10 columns wide, which shows the
|
||||
percent in place of the line's own. Any other phase, and a phase whose bar has
|
||||
no room, is drawn as the library's spinner followed by its line. Each display
|
||||
is drawn again several times a second, so its elapsed time keeps moving while
|
||||
a phase waits for its next item.
|
||||
- On a terminal no line reaches the terminal's last column, however narrow the
|
||||
terminal is. The phase and its place, the progress, the elapsed time and the
|
||||
ETA are kept whole: a line too wide for the terminal leaves out its rate first
|
||||
@@ -784,8 +804,7 @@ Additional requirements:
|
||||
inside the progress display.
|
||||
- A phase that stops short of its total, as an interrupted one does, is left
|
||||
showing the count it stopped at rather than filled up.
|
||||
- `report` and `trees` modes need no progress display, only their stderr
|
||||
summaries.
|
||||
- `report` mode needs no progress display, only its stderr summary.
|
||||
|
||||
### Error handling and exit codes
|
||||
|
||||
|
||||
Reference in New Issue
Block a user