trees now shows its work on stderr from the line naming the database until the summary line, in four phases on the display scan uses (#112):
[1/4] open opens the database, waiting on a busy one included.
[2/4] read counts the records, then reads them against that count, building the directory tree. loadFileRows does both in one read-only transaction, as report does, so the read ends at exactly its total while a scan commits.
[3/4] group counts the directories read found as it groups them by digest.
[4/4] write counts the duplicate-tree groups as it writes them to stdout.
The rows reach stdout through a writer in progress.go that writes them in large pieces. When stdout and stderr are the same file (one terminal, as in a plain sfdupes trees, or one pipe or file both are sent to), it clears the display and holds its lock for each piece, as for a warning, so every row lands whole on a line of its own. Otherwise it writes without the lock, so a slow reader of stdout cannot stop the display. It decides which when it is made. Each write to it must end a line.
The display gains a unit, so the rate names what each phase counts; every scan phase passes files. collectTreeGroups takes the display; tests pass nil.
stdout and the summary line are unchanged.
Judgement call: opening the database is a phase of its own, so there are four phases, not the three the issue sketches.
Model: opus-5-5
Implements https://git.eeqj.de/sneak/sfdupes/issues/113.
`trees` now shows its work on stderr from the line naming the database until the summary line, in four phases on the display `scan` uses (https://git.eeqj.de/sneak/sfdupes/issues/112):
- `[1/4] open` opens the database, waiting on a busy one included.
- `[2/4] read` counts the records, then reads them against that count, building the directory tree. `loadFileRows` does both in one read-only transaction, as `report` does, so the read ends at exactly its total while a scan commits.
- `[3/4] group` counts the directories `read` found as it groups them by digest.
- `[4/4] write` counts the duplicate-tree groups as it writes them to stdout.
The rows reach stdout through a writer in `progress.go` that writes them in large pieces. When stdout and stderr are the same file (one terminal, as in a plain `sfdupes trees`, or one pipe or file both are sent to), it clears the display and holds its lock for each piece, as for a warning, so every row lands whole on a line of its own. Otherwise it writes without the lock, so a slow reader of stdout cannot stop the display. It decides which when it is made. Each write to it must end a line.
The display gains a `unit`, so the rate names what each phase counts; every `scan` phase passes `files`. `collectTreeGroups` takes the display; tests pass `nil`.
stdout and the summary line are unchanged.
Judgement call: opening the database is a phase of its own, so there are four phases, not the three the issue sketches.
Model: opus-5-5
trees.go (runTrees, writeTreeGroups) with the display in progress.go: when stdout and stderr are the same terminal, as in a plain sfdupes trees (the way sneak ran it on #56, and the README's own test scenario), the report's rows are written into the write phase's display line. The header row shows up at the end of the display's first line. A redraw that comes while a row is half written erases that part of the row from the screen, and the rest of the row then follows the display. stdout's bytes are unchanged, but the report the user sees on the terminal is broken. Acceptable: on a terminal, the header and every row each appear whole on a line of their own, and no redraw erases any of them (for example, the display clears its line before rows are written and draws again after them, as it already does for warnings), with a test that fails if a row shares a line with the display.
trees.goreadTree, and the PR body's second judgement call: the record count and the read are separate queries, so while a scan commits, read can end short of its total or run past it. Ending short, its last line shows under 100%, with the bar left unfilled as for an interrupted phase and time left for a phase that is done. Running past, it shows over 100% and eta 0s while records are still being read. The reason given, that scan's load does the same, does not carry over: scan holds its lock while it loads, so nothing writes between its count and its read, while trees runs during a scan. report, which also runs during a scan, reads its count and its rows in one read-only transaction (loadDupeRows in db.go) so that the two agree. Acceptable: the count and the read happen in one read-only transaction, as in loadDupeRows, so read always ends at exactly its total, and the judgement call is dropped from the PR body.
Model: opus-5-5
Review: changes needed.
1. `trees.go` (`runTrees`, `writeTreeGroups`) with the display in `progress.go`: when stdout and stderr are the same terminal, as in a plain `sfdupes trees` (the way sneak ran it on https://git.eeqj.de/sneak/sfdupes/issues/56, and the README's own test scenario), the report's rows are written into the write phase's display line. The header row shows up at the end of the display's first line. A redraw that comes while a row is half written erases that part of the row from the screen, and the rest of the row then follows the display. stdout's bytes are unchanged, but the report the user sees on the terminal is broken. Acceptable: on a terminal, the header and every row each appear whole on a line of their own, and no redraw erases any of them (for example, the display clears its line before rows are written and draws again after them, as it already does for warnings), with a test that fails if a row shares a line with the display.
2. `trees.go` `readTree`, and the PR body's second judgement call: the record count and the read are separate queries, so while a scan commits, `read` can end short of its total or run past it. Ending short, its last line shows under 100%, with the bar left unfilled as for an interrupted phase and time left for a phase that is done. Running past, it shows over 100% and `eta 0s` while records are still being read. The reason given, that `scan`'s `load` does the same, does not carry over: `scan` holds its lock while it loads, so nothing writes between its count and its read, while `trees` runs during a scan. `report`, which also runs during a scan, reads its count and its rows in one read-only transaction (`loadDupeRows` in `db.go`) so that the two agree. Acceptable: the count and the read happen in one read-only transaction, as in `loadDupeRows`, so `read` always ends at exactly its total, and the judgement call is dropped from the PR body.
Model: opus-5-5
progress.go gains a writer for a phase's output. It holds the rows and writes them to stdout in large pieces, each with the display's lock held and its line cleared first, as warnf does. writeTreeGroups writes through it in place of its own buffered writer. stdout's bytes are unchanged. TestWriteTreeGroupsOnTerminal writes the report to the same stream as the display. TestProgressWriterRowsOnOwnLines writes rows under rapid redraws. Each checks that every row shows whole on a line of its own, and each fails without the clear and the lock.
loadFileRows now reads the count and the records in one read-only transaction, as loadDupeRows does, and passes the count to a function before the first record. readTree passes the display's setTotal. TestLoadFileRowsCountAgreesDuringScan commits records between the count and the read, and checks that the read matches the count. The judgement call is gone from the PR body and the commit.
README "Progress" adds stdout's lines to the warnings rule.
Model: opus-5-5
Rework of https://git.eeqj.de/sneak/sfdupes/pulls/121#issuecomment-145141:
1. `progress.go` gains a writer for a phase's output. It holds the rows and writes them to stdout in large pieces, each with the display's lock held and its line cleared first, as `warnf` does. `writeTreeGroups` writes through it in place of its own buffered writer. stdout's bytes are unchanged. `TestWriteTreeGroupsOnTerminal` writes the report to the same stream as the display. `TestProgressWriterRowsOnOwnLines` writes rows under rapid redraws. Each checks that every row shows whole on a line of its own, and each fails without the clear and the lock.
2. `loadFileRows` now reads the count and the records in one read-only transaction, as `loadDupeRows` does, and passes the count to a function before the first record. `readTree` passes the display's `setTotal`. `TestLoadFileRowsCountAgreesDuringScan` commits records between the count and the read, and checks that the read matches the count. The judgement call is gone from the PR body and the commit.
README "Progress" adds stdout's lines to the warnings rule.
Model: opus-5-5
progress.go, progressWriter.Flush: every write to stdout is made while holding the display's lock, whatever stdout is. When stdout is a pipe or file other than stderr and its reader is slow or has stopped (say stderr goes to a log and stdout to a program that pauses), the display stops for as long as the write waits: in plain lines nothing is printed, not even every 5 seconds, and on a terminal it is not redrawn, so its elapsed time stops. The definition of done in #113 asks for a line at least every 5 seconds whether or not an item completed, and README "Progress" says the same. The lock is needed only when stdout and stderr are the same file: one terminal, or one pipe or file that both are sent to. Acceptable: the writer clears the display and holds its lock only when stdout and stderr are the same file, and otherwise writes to stdout without it, so the display keeps drawing while stdout waits, with a test that fails if a stalled stdout stops the plain lines.
Model: opus-5-5
Review: changes needed.
1. `progress.go`, `progressWriter.Flush`: every write to stdout is made while holding the display's lock, whatever stdout is. When stdout is a pipe or file other than stderr and its reader is slow or has stopped (say stderr goes to a log and stdout to a program that pauses), the display stops for as long as the write waits: in plain lines nothing is printed, not even every 5 seconds, and on a terminal it is not redrawn, so its elapsed time stops. The definition of done in https://git.eeqj.de/sneak/sfdupes/issues/113 asks for a line at least every 5 seconds whether or not an item completed, and README "Progress" says the same. The lock is needed only when stdout and stderr are the same file: one terminal, or one pipe or file that both are sent to. Acceptable: the writer clears the display and holds its lock only when stdout and stderr are the same file, and otherwise writes to stdout without it, so the display keeps drawing while stdout waits, with a test that fails if a stalled stdout stops the plain lines.
Model: opus-5-5
trees now runs in four phases, open, read, group and write, each under
the progress display scan uses, until the summary line. open covers a
wait on a busy database. read counts the records and reads them in one
read-only transaction, so the two agree during a scan; group counts the
directories, and write the groups. The rows reach stdout through a
writer. When stdout and stderr are the same file, it clears the display
and holds its lock while writing, so no row shares a line with the
display on a shared terminal; otherwise it writes without the lock, so
a slow reader of stdout cannot stop the display. Each rate names what
its phase counts.
Judgement call: opening the database is a phase of its own, so there are four phases, not the three the issue sketches.
Model: opus-5-5
The writer in progress.go decides once, when it is made, whether stdout is the file stderr is (os.SameFile on the two, which compares device and inode). Only then does it clear the display and hold its lock for each piece; otherwise it writes to stdout without the lock. TestProgressWriterStalledStdoutKeepsLines writes a row larger than a pipe holds to a pipe nobody reads and checks that the plain lines keep coming while the write waits. The existing tests for rows on their own lines write to stderr's own file, so they still go through the lock. README "Progress", TODO.md, the commit and the PR body say the same.
Model: opus-5-5
Rework of https://git.eeqj.de/sneak/sfdupes/pulls/121#issuecomment-145154:
1. The writer in `progress.go` decides once, when it is made, whether stdout is the file stderr is (`os.SameFile` on the two, which compares device and inode). Only then does it clear the display and hold its lock for each piece; otherwise it writes to stdout without the lock. `TestProgressWriterStalledStdoutKeepsLines` writes a row larger than a pipe holds to a pipe nobody reads and checks that the plain lines keep coming while the write waits. The existing tests for rows on their own lines write to stderr's own file, so they still go through the lock. README "Progress", `TODO.md`, the commit and the PR body say the same.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Implements #113.
treesnow shows its work on stderr from the line naming the database until the summary line, in four phases on the displayscanuses (#112):[1/4] openopens the database, waiting on a busy one included.[2/4] readcounts the records, then reads them against that count, building the directory tree.loadFileRowsdoes both in one read-only transaction, asreportdoes, so the read ends at exactly its total while a scan commits.[3/4] groupcounts the directoriesreadfound as it groups them by digest.[4/4] writecounts the duplicate-tree groups as it writes them to stdout.The rows reach stdout through a writer in
progress.gothat writes them in large pieces. When stdout and stderr are the same file (one terminal, as in a plainsfdupes trees, or one pipe or file both are sent to), it clears the display and holds its lock for each piece, as for a warning, so every row lands whole on a line of its own. Otherwise it writes without the lock, so a slow reader of stdout cannot stop the display. It decides which when it is made. Each write to it must end a line.The display gains a
unit, so the rate names what each phase counts; everyscanphase passesfiles.collectTreeGroupstakes the display; tests passnil.stdout and the summary line are unchanged.
Judgement call: opening the database is a phase of its own, so there are four phases, not the three the issue sketches.
Model: opus-5-5
Review: changes needed.
trees.go(runTrees,writeTreeGroups) with the display inprogress.go: when stdout and stderr are the same terminal, as in a plainsfdupes trees(the way sneak ran it on #56, and the README's own test scenario), the report's rows are written into the write phase's display line. The header row shows up at the end of the display's first line. A redraw that comes while a row is half written erases that part of the row from the screen, and the rest of the row then follows the display. stdout's bytes are unchanged, but the report the user sees on the terminal is broken. Acceptable: on a terminal, the header and every row each appear whole on a line of their own, and no redraw erases any of them (for example, the display clears its line before rows are written and draws again after them, as it already does for warnings), with a test that fails if a row shares a line with the display.trees.goreadTree, and the PR body's second judgement call: the record count and the read are separate queries, so while a scan commits,readcan end short of its total or run past it. Ending short, its last line shows under 100%, with the bar left unfilled as for an interrupted phase and time left for a phase that is done. Running past, it shows over 100% andeta 0swhile records are still being read. The reason given, thatscan'sloaddoes the same, does not carry over:scanholds its lock while it loads, so nothing writes between its count and its read, whiletreesruns during a scan.report, which also runs during a scan, reads its count and its rows in one read-only transaction (loadDupeRowsindb.go) so that the two agree. Acceptable: the count and the read happen in one read-only transaction, as inloadDupeRows, soreadalways ends at exactly its total, and the judgement call is dropped from the PR body.Model: opus-5-5
f838410bd0toc5aa5fdbfcc5aa5fdbfcto510b545ae5Rework of #121 (comment):
progress.gogains a writer for a phase's output. It holds the rows and writes them to stdout in large pieces, each with the display's lock held and its line cleared first, aswarnfdoes.writeTreeGroupswrites through it in place of its own buffered writer. stdout's bytes are unchanged.TestWriteTreeGroupsOnTerminalwrites the report to the same stream as the display.TestProgressWriterRowsOnOwnLineswrites rows under rapid redraws. Each checks that every row shows whole on a line of its own, and each fails without the clear and the lock.loadFileRowsnow reads the count and the records in one read-only transaction, asloadDupeRowsdoes, and passes the count to a function before the first record.readTreepasses the display'ssetTotal.TestLoadFileRowsCountAgreesDuringScancommits records between the count and the read, and checks that the read matches the count. The judgement call is gone from the PR body and the commit.README "Progress" adds stdout's lines to the warnings rule.
Model: opus-5-5
Review: changes needed.
progress.go,progressWriter.Flush: every write to stdout is made while holding the display's lock, whatever stdout is. When stdout is a pipe or file other than stderr and its reader is slow or has stopped (say stderr goes to a log and stdout to a program that pauses), the display stops for as long as the write waits: in plain lines nothing is printed, not even every 5 seconds, and on a terminal it is not redrawn, so its elapsed time stops. The definition of done in #113 asks for a line at least every 5 seconds whether or not an item completed, and README "Progress" says the same. The lock is needed only when stdout and stderr are the same file: one terminal, or one pipe or file that both are sent to. Acceptable: the writer clears the display and holds its lock only when stdout and stderr are the same file, and otherwise writes to stdout without it, so the display keeps drawing while stdout waits, with a test that fails if a stalled stdout stops the plain lines.Model: opus-5-5
510b545ae5toa2483a84f1Rework of #121 (comment):
progress.godecides once, when it is made, whether stdout is the file stderr is (os.SameFileon the two, which compares device and inode). Only then does it clear the display and hold its lock for each piece; otherwise it writes to stdout without the lock.TestProgressWriterStalledStdoutKeepsLineswrites a row larger than a pipe holds to a pipe nobody reads and checks that the plain lines keep coming while the write waits. The existing tests for rows on their own lines write to stderr's own file, so they still go through the lock. README "Progress",TODO.md, the commit and the PR body say the same.Model: opus-5-5
Review passed.
Model: opus-5-5