trees: show its phases as N of M, with progress, elapsed time and ETA (closes #113)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
trees now runs in four phases, open, read, group and write, each under the progress display scan uses, until the summary line. open covers a wait on a busy database. read counts the records and reads them in one read-only transaction, so the two agree during a scan; group counts the directories, and write the groups. The rows reach stdout through a writer that clears the display and holds its lock while writing, so no row shares a line with the display on a shared terminal. Each rate names what its phase counts. Judgement call: opening the database is a phase of its own, so there are four phases, not the three the issue sketches. Model: opus-5-5
This commit is contained in:
@@ -695,6 +695,12 @@ Processing:
|
||||
- Order groups by total tree size descending, tie-broken by `first` path
|
||||
ascending. Output must be fully deterministic for a given input.
|
||||
|
||||
`trees` runs in four phases, each with its own progress display on stderr (see
|
||||
"Progress"): `open` opens the database; `read` counts the records, then reads
|
||||
them in path order, building the hierarchy and computing the digests as it goes;
|
||||
`group` groups the directories by digest and picks the maximal groups; `write`
|
||||
writes the groups to stdout.
|
||||
|
||||
#### Trees output format
|
||||
|
||||
TSV on stdout: a header line, then one row per duplicate tree (N-1 rows for a
|
||||
@@ -715,13 +721,16 @@ units.
|
||||
|
||||
### Progress
|
||||
|
||||
Use the progress-bar library for all scan progress; rendering in the style of
|
||||
`pv` is the model. All progress goes to stderr.
|
||||
Use the progress-bar library for all progress of `scan` and `trees`; rendering
|
||||
in the style of `pv` is the model. All progress goes to stderr.
|
||||
|
||||
A scan must always show that it is working and not hung. Each phase gets its own
|
||||
display from the moment it starts, naming the phase and its place among the five
|
||||
(`[1/5] load`, `[2/5] walk`, `[3/5] hash`, `[4/5] update`, `[5/5] content`) and
|
||||
showing its progress, its elapsed time and its estimated time remaining (ETA):
|
||||
`scan` and `trees` must always show that they are working and not hung. Each
|
||||
phase gets its own display from the moment it starts, naming the phase and its
|
||||
place among the command's phases and showing its progress, its elapsed time and
|
||||
its estimated time remaining (ETA).
|
||||
|
||||
A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
|
||||
`[4/5] update`, `[5/5] content`):
|
||||
|
||||
- `load` counts the records first, then reads them against that count. While the
|
||||
count runs, the display says it is counting the records.
|
||||
@@ -738,12 +747,23 @@ showing its progress, its elapsed time and its estimated time remaining (ETA):
|
||||
and ETA, start there. The files to read then get a display of their own, with
|
||||
their count as its total.
|
||||
|
||||
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
|
||||
`[4/4] write`, see "`trees` mode"):
|
||||
|
||||
- `open` has no total. While it opens the database, waiting on a busy one
|
||||
included, the display says it is opening the database.
|
||||
- `read` counts the records first, then reads them against that count. While the
|
||||
count runs, the display says it is counting the records.
|
||||
- `group` has the directories `read` found as its total, and `write` the
|
||||
duplicate-tree groups.
|
||||
|
||||
Each display is one line holding:
|
||||
|
||||
- the phase and its place among the phases
|
||||
- a `[m/n] x%` display (items processed / total items, percent); before a phase
|
||||
has a total, the items processed, or what the phase is waiting for
|
||||
- current rate (items/s)
|
||||
- current rate (items/s), naming the items the phase counts: files, records,
|
||||
directories or groups
|
||||
- elapsed time
|
||||
- estimated time remaining, or `unknown`
|
||||
|
||||
@@ -756,13 +776,13 @@ Example shape (exact layout is flexible, content is not):
|
||||
|
||||
Additional requirements:
|
||||
|
||||
- On a terminal, a phase with an exact total (every phase but `walk`, once it
|
||||
has counted what it will do) is drawn as its line followed by the progress-bar
|
||||
library's bar, 10 columns wide, which shows the percent in place of the line's
|
||||
own. Any other phase, and a phase whose bar has no room, is drawn as the
|
||||
library's spinner followed by its line. Each display is drawn again several
|
||||
times a second, so its elapsed time keeps moving while a phase waits for its
|
||||
next item.
|
||||
- On a terminal, a phase with an exact total (every phase but `scan`'s `walk`
|
||||
and `trees`' `open`, once it has counted what it will do) is drawn as its line
|
||||
followed by the progress-bar library's bar, 10 columns wide, which shows the
|
||||
percent in place of the line's own. Any other phase, and a phase whose bar has
|
||||
no room, is drawn as the library's spinner followed by its line. Each display
|
||||
is drawn again several times a second, so its elapsed time keeps moving while
|
||||
a phase waits for its next item.
|
||||
- On a terminal no line reaches the terminal's last column, however narrow the
|
||||
terminal is. The phase and its place, the progress, the elapsed time and the
|
||||
ETA are kept whole: a line too wide for the terminal leaves out its rate first
|
||||
@@ -781,11 +801,11 @@ Additional requirements:
|
||||
goroutine counts the items, and each display draws them from a goroutine of
|
||||
its own.
|
||||
- A warning printed during a phase always lands on a line of its own, never
|
||||
inside the progress display.
|
||||
inside the progress display. So does every line written to stdout when stdout
|
||||
is the same terminal, as in a plain `sfdupes trees`.
|
||||
- A phase that stops short of its total, as an interrupted one does, is left
|
||||
showing the count it stopped at rather than filled up.
|
||||
- `report` and `trees` modes need no progress display, only their stderr
|
||||
summaries.
|
||||
- `report` mode needs no progress display, only its stderr summary.
|
||||
|
||||
### Error handling and exit codes
|
||||
|
||||
|
||||
Reference in New Issue
Block a user