scan prints nothing for long stretches and looks frozen, then reports over 1,000 files a second #125

New Issue

2026-10-16T22:39:35+02:00

clawbot commented

2026-10-16 22:39:35 +02:00

sneak, 2026-10-16, in chat: sfdupes scan often prints nothing for a long while and appears frozen, yet when the display does update it reports more than 1,000 files a second. That should not happen.

Done when: while any phase runs, the display redraws at a steady interval (at most about a second apart) whatever the walk, hashing or database writes are doing, so it never looks frozen while work goes on; the rate shown covers the time since the last redraw, not a burst; if the work itself is stuck (waiting on a lock, a slow disk or the database), the display says what it waits for; a test drives a phase whose producer stalls and checks the display keeps redrawing. First check whether the progress changes already on next (issues 112-114) cure it, and say on this issue which build shows it.

Model: opus-5-5

sneak, 2026-10-16, in chat: `sfdupes scan` often prints nothing for a long while and appears frozen, yet when the display does update it reports more than 1,000 files a second. That should not happen. **Done when:** while any phase runs, the display redraws at a steady interval (at most about a second apart) whatever the walk, hashing or database writes are doing, so it never looks frozen while work goes on; the rate shown covers the time since the last redraw, not a burst; if the work itself is stuck (waiting on a lock, a slow disk or the database), the display says what it waits for; a test drives a phase whose producer stalls and checks the display keeps redrawing. First check whether the progress changes already on `next` (issues 112-114) cure it, and say on this issue which build shows it. Model: opus-5-5

clawbot commented

2026-10-16 22:40:51 +02:00

Which build shows it: v0.0.2 (what @latest installs today) and next alike, in part.

On v0.0.2, a phase with a total (hash, update, the content reads) redraws on a terminal only when a file finishes, and off a terminal a line is printed only when a file finishes. A stretch with no finished file, such as one large file being read or a slow disk, prints nothing. When something finishes, the rate shown is the phase's average since it started, so after a stall it can still read over 1,000 files/s.
next (#112) already redraws every 100 ms on a terminal and prints a line every 5 seconds off one, whatever the work is doing, so it no longer looks frozen. But its rate is still the phase's average, and a stalled phase does not say what it is waiting for, apart from the database waits.

Plan for the implementer (one PR on next, package main):

Rate. The rate shown is the items finished over the last 5 seconds: since the last line off a terminal, and over the last 5 seconds on one. A stall shows the rate falling to 0, and a burst after one does not read as a steady rate. The ETA keeps coming from the phase's average rate, so it does not swing with each burst.
Stall. When no item has finished for 10 seconds, the line says so and what the phase is doing, for example no file finished for 42s, reading files. Each phase gives one short phrase where it is defined: listing directories, reading files, writing to the database, reading the database, searching the database, writing the report. The phrase sits with the kept parts of the line, so it is not dropped on a narrow terminal before the rate.
Tests with a settable clock and interval, under a second each: a producer that stalls, with the display going on redrawing, the rate reaching 0 and the stall line appearing; a burst after a stall, with the rate covering only the last 5 seconds.
README "Progress" says what the rate covers and when the stall line appears.

Model: opus-5-5

Which build shows it: `v0.0.2` (what `@latest` installs today) and `next` alike, in part. - On `v0.0.2`, a phase with a total (`hash`, `update`, the content reads) redraws on a terminal only when a file finishes, and off a terminal a line is printed only when a file finishes. A stretch with no finished file, such as one large file being read or a slow disk, prints nothing. When something finishes, the rate shown is the phase's average since it started, so after a stall it can still read over 1,000 files/s. - `next` (https://git.eeqj.de/sneak/sfdupes/issues/112) already redraws every 100 ms on a terminal and prints a line every 5 seconds off one, whatever the work is doing, so it no longer looks frozen. But its rate is still the phase's average, and a stalled phase does not say what it is waiting for, apart from the database waits. Plan for the implementer (one PR on `next`, package `main`): - **Rate.** The rate shown is the items finished over the last 5 seconds: since the last line off a terminal, and over the last 5 seconds on one. A stall shows the rate falling to 0, and a burst after one does not read as a steady rate. The ETA keeps coming from the phase's average rate, so it does not swing with each burst. - **Stall.** When no item has finished for 10 seconds, the line says so and what the phase is doing, for example `no file finished for 42s, reading files`. Each phase gives one short phrase where it is defined: `listing directories`, `reading files`, `writing to the database`, `reading the database`, `searching the database`, `writing the report`. The phrase sits with the kept parts of the line, so it is not dropped on a narrow terminal before the rate. - **Tests** with a settable clock and interval, under a second each: a producer that stalls, with the display going on redrawing, the rate reaching 0 and the stall line appearing; a burst after a stall, with the rate covering only the last 5 seconds. - README "Progress" says what the rate covers and when the stall line appears. Model: opus-5-5

clawbot added this to the 1.0.0 milestone 2026-10-16 22:40:52 +02:00

clawbot referenced this issue from a commit

2026-10-16 23:00:40 +02:00

Show the progress rate over the last 5 seconds, and what a phase is doing after 10 seconds with no item done (closes #125)

clawbot referenced this issue from a commit

2026-10-16 23:00:57 +02:00

Show the progress rate over the last 5 seconds, and what a phase is doing after 10 seconds with no item done (closes #125)

clawbot referenced this issue from a commit

2026-10-16 23:01:06 +02:00

Show the progress rate over the last 5 seconds, and what a phase is doing after 10 seconds with no item done (closes #125)

clawbot referenced a pull request that will close this issue

2026-10-16 23:03:26 +02:00

Show the progress rate over the last 5 seconds, and where a phase waits after 10 seconds with no item done (closes #125) #126

clawbot commented

2026-10-16 23:03:37 +02:00

Built as planned above in #126. The PR body names its two judgement calls, and notes that on an 80-column terminal the words about nothing finishing rarely fit and are left out.

Model: opus-5-5

Built as planned above in https://git.eeqj.de/sneak/sfdupes/pulls/126. The PR body names its two judgement calls, and notes that on an 80-column terminal the words about nothing finishing rarely fit and are left out. Model: opus-5-5

clawbot referenced this issue

2026-10-16 23:19:13 +02:00

Show the progress rate over the last 5 seconds, and where a phase waits after 10 seconds with no item done (closes #125) #126

clawbot referenced this issue from a commit

2026-10-16 23:41:01 +02:00

Show the progress rate over the last 5 seconds, and what a phase is doing after 10 seconds with no item done (closes #125)

clawbot referenced this issue

2026-10-16 23:58:44 +02:00

Show the progress rate over the last 5 seconds, and where a phase waits after 10 seconds with no item done (closes #125) #126

clawbot referenced this issue from a commit

2026-10-17 00:13:33 +02:00

Show the progress rate over the last 5 seconds, and where a phase waits after 10 seconds with no item done (closes #125)