Show progress while loading the record index
On a database with tens of millions of records, indexing the existing rows before the walk takes real single-core time with no output, which is indistinguishable from a hang. Give the load its own spinner, and render every phase display the moment the phase starts instead of waiting for its first completed item.
This commit is contained in:
@@ -395,9 +395,11 @@ all dupe rows) in human units.
|
||||
Use the progress-bar library for all scan progress; rendering in the
|
||||
style of `pv` is the model. All progress goes to stderr.
|
||||
|
||||
Each phase gets its own display. The walk has no known total while
|
||||
running: show a live file count, rate, and elapsed time
|
||||
(spinner-style, no percentage or ETA). The hash and update phases
|
||||
Each phase gets its own display, rendered the moment the phase
|
||||
starts — a scan must never look hung. Loading the existing-record
|
||||
index (`load`) and the walk have no known totals while running: show
|
||||
a live count, rate, and elapsed time (spinner-style, no percentage or
|
||||
ETA). The hash and update phases
|
||||
have exact totals — only files that actually need hashing appear in
|
||||
the hash total, so its ETA is meaningful. Required elements for the
|
||||
bars with known totals:
|
||||
|
||||
@@ -65,6 +65,9 @@ func newProgress(label string, total int64) *progress {
|
||||
progressbar.OptionSetItsString("files"),
|
||||
progressbar.OptionSetElapsedTime(true),
|
||||
progressbar.OptionThrottle(barThrottle),
|
||||
// Render at zero immediately: a phase must be visible the
|
||||
// moment it starts, even before its first item completes.
|
||||
progressbar.OptionSetRenderBlankState(true),
|
||||
}
|
||||
if total >= 0 {
|
||||
opts = append(opts,
|
||||
|
||||
7
scan.go
7
scan.go
@@ -186,10 +186,17 @@ func syncScan(db *sql.DB, roots []string, workers int,
|
||||
// can be recognized as a possible duplicate of a tree scanned
|
||||
// separately into the same database.
|
||||
func (s *scanState) loadIndex(roots []string) error {
|
||||
// Indexing tens of millions of records takes real time; without a
|
||||
// display the scan looks hung before the walk begins.
|
||||
prog := newProgress("load", -1)
|
||||
defer prog.finish()
|
||||
|
||||
s.existing = make(map[string]fileMeta)
|
||||
|
||||
return loadFileMeta(s.db,
|
||||
func(path string, size, mtime int64, hashed bool) {
|
||||
prog.increment()
|
||||
|
||||
if underAnyRoot(path, roots) {
|
||||
s.existing[path] = fileMeta{
|
||||
size: size, mtime: mtime, hashed: hashed,
|
||||
|
||||
Reference in New Issue
Block a user