info counted the records and added up their sizes with a query over files. SQLite answers it only by reading every record, which takes minutes on a large database. scan, report and trees ran the same count for the totals of their progress displays.
What changed:
A new totals table holds one row: the number of records and the sum of their sizes. Three triggers on files keep it current: after insert, after delete, and after an update of size. Each runs in the transaction of the change that fires it.
Every count reads that row instead of files.
scan writes files with the same statements. Its upsert of an existing record fires the size trigger, not the insert trigger.
scan sets temp_store(MEMORY). Each trigger needs a statement journal, which costs far less in memory than in a temporary file.
New tests: a query-plan check fails if a count reads files or its index. Another sets the row to numbers files does not hold and checks that each count reports them. A first scan, a rescan that changes sizes and one that deletes records must each leave the row equal to COUNT(*) and SUM(size) over files.
Disclosures:
TestRunFatalAfterOpenClosesDatabase now expects "no such table: totals", the error of each subcommand's first query.
The triggers make each write to files slower: a first scan of many small files takes about two fifths longer than on next; on large files, where reading them takes most of the time, the difference is small.
Model: opus-5-5
Fixes https://git.eeqj.de/sneak/sfdupes/issues/146.
`info` counted the records and added up their sizes with a query over `files`. SQLite answers it only by reading every record, which takes minutes on a large database. `scan`, `report` and `trees` ran the same count for the totals of their progress displays.
What changed:
- A new `totals` table holds one row: the number of records and the sum of their sizes. Three triggers on `files` keep it current: after insert, after delete, and after an update of `size`. Each runs in the transaction of the change that fires it.
- Every count reads that row instead of `files`.
- `scan` writes `files` with the same statements. Its upsert of an existing record fires the `size` trigger, not the insert trigger.
- `scan` sets `temp_store(MEMORY)`. Each trigger needs a statement journal, which costs far less in memory than in a temporary file.
New tests: a query-plan check fails if a count reads `files` or its index. Another sets the row to numbers `files` does not hold and checks that each count reports them. A first scan, a rescan that changes sizes and one that deletes records must each leave the row equal to `COUNT(*)` and `SUM(size)` over `files`.
Disclosures:
- `TestRunFatalAfterOpenClosesDatabase` now expects "no such table: totals", the error of each subcommand's first query.
- The triggers make each write to `files` slower: a first scan of many small files takes about two fifths longer than on `next`; on large files, where reading them takes most of the time, the difference is small.
Model: opus-5-5
Before review, one fix: the new TODO.md entry ends "A database made before this has no totals table and must be removed and rescanned". sfdupes is pre-1.0 with no installed base, so its documents say nothing about databases made by earlier builds. Remove that sentence; nothing else changes.
Model: opus-5-5
Before review, one fix: the new `TODO.md` entry ends "A database made before this has no `totals` table and must be removed and rescanned". sfdupes is pre-1.0 with no installed base, so its documents say nothing about databases made by earlier builds. Remove that sentence; nothing else changes.
Model: opus-5-5
Removed the sentence about databases made before this change from the TODO.md entry, and the matching paragraph from the commit message; the code comments and README had none.
Model: opus-5-5
Removed the sentence about databases made before this change from the `TODO.md` entry, and the matching paragraph from the commit message; the code comments and README had none.
Model: opus-5-5
db_test.go line 594, TestCountsReadOnlyTotals: it checks the query plans of the two SQL constants only. If loadFileRows, loadSignatureRows or countRecords goes back to SELECT COUNT(*) FROM files, this test still passes. Only TestRunFatalAfterOpenClosesDatabase fails, through its expected error text, and its message is about something else. Acceptable: a test that fails whichever count goes back to reading files. For example, it sets the totals row to numbers files does not hold, then checks that scan's load, the read phases of report and trees, and info each report those numbers.
README.md line 711: the touched sentence "report runs in two phases, each with its own progress display on stderr (see "Progress"): open opens the database; read gets the number of records from totals, then reads the records in signature order, largest size first." runs to about 36 words and chains the phases with semicolons. Acceptable: the lead-in, then one bullet or one short sentence per phase.
README.md line 792: the touched sentence "trees runs in four phases, each with its own progress display on stderr (see "Progress"): open opens the database; read gets the number of records from totals, ... write writes the groups to stdout." is about 60 words and chains four phases with semicolons. Acceptable: as in 2.
README.md line 846: the touched sentence "info runs in two phases, each with its own progress display on stderr (see "Progress"): open opens the database; count reads the schema version and the row of totals ... while a scan is committing." is about 45 words. Acceptable: as in 2.
Model: opus-5-5
1. `db_test.go` line 594, `TestCountsReadOnlyTotals`: it checks the query plans of the two SQL constants only. If `loadFileRows`, `loadSignatureRows` or `countRecords` goes back to `SELECT COUNT(*) FROM files`, this test still passes. Only `TestRunFatalAfterOpenClosesDatabase` fails, through its expected error text, and its message is about something else. Acceptable: a test that fails whichever count goes back to reading `files`. For example, it sets the `totals` row to numbers `files` does not hold, then checks that scan's `load`, the `read` phases of `report` and `trees`, and `info` each report those numbers.
2. `README.md` line 711: the touched sentence "`report` runs in two phases, each with its own progress display on stderr (see "Progress"): `open` opens the database; `read` gets the number of records from `totals`, then reads the records in signature order, largest size first." runs to about 36 words and chains the phases with semicolons. Acceptable: the lead-in, then one bullet or one short sentence per phase.
3. `README.md` line 792: the touched sentence "`trees` runs in four phases, each with its own progress display on stderr (see "Progress"): `open` opens the database; `read` gets the number of records from `totals`, ... `write` writes the groups to stdout." is about 60 words and chains four phases with semicolons. Acceptable: as in 2.
4. `README.md` line 846: the touched sentence "`info` runs in two phases, each with its own progress display on stderr (see "Progress"): `open` opens the database; `count` reads the schema version and the row of `totals` ... while a scan is committing." is about 45 words. Acceptable: as in 2.
Model: opus-5-5
New test TestCountsComeFromTotals in db_test.go sets the row of totals to 7 records of 9000 bytes over an empty files, then checks that countRecords, loadFileRows, loadSignatureRows and readDatabaseCounts each report those numbers.
report: a lead-in, then one bullet each for open and read; the next sentence now names signature order instead of starting with "That".
trees: a lead-in, then one bullet per phase.
info: a lead-in, then one bullet per phase, the count bullet split into two sentences; the sentence about when info writes its lines is now its own paragraph.
Model: opus-5-5
Rework for the review https://git.eeqj.de/sneak/sfdupes/pulls/148#issuecomment-159332:
1. New test `TestCountsComeFromTotals` in `db_test.go` sets the row of `totals` to 7 records of 9000 bytes over an empty `files`, then checks that `countRecords`, `loadFileRows`, `loadSignatureRows` and `readDatabaseCounts` each report those numbers.
2. `report`: a lead-in, then one bullet each for `open` and `read`; the next sentence now names signature order instead of starting with "That".
3. `trees`: a lead-in, then one bullet per phase.
4. `info`: a lead-in, then one bullet per phase, the `count` bullet split into two sentences; the sentence about when `info` writes its lines is now its own paragraph.
Model: opus-5-5
db.go lines 89 to 108 and 474: the three triggers run once for every record scan inserts, updates or deletes, and they make SQLite's writes to files several times slower. A first scan of a tree of many small files takes nearly twice as long as on next. Neither the PR nor README.md says so. Acceptable: scan's write speed brought back near next's, or the remaining cost stated in the PR body and in README's totals paragraph so the owner can weigh it. Adding temp_store(MEMORY) to scanParams recovers about half of the loss; keeping totals current with one update per committed batch, instead of one per record, is the other way.
report.go line 48 and trees.go line 43: the doc comments of runReport and runTrees still say the read phase counts the records. It now reads the number from totals, as the matching comments in info.go and scan.go and the README now say. Acceptable: the same wording in these two comments.
Model: opus-5-5
1. `db.go` lines 89 to 108 and 474: the three triggers run once for every record `scan` inserts, updates or deletes, and they make SQLite's writes to `files` several times slower. A first scan of a tree of many small files takes nearly twice as long as on `next`. Neither the PR nor `README.md` says so. Acceptable: `scan`'s write speed brought back near `next`'s, or the remaining cost stated in the PR body and in README's `totals` paragraph so the owner can weigh it. Adding `temp_store(MEMORY)` to `scanParams` recovers about half of the loss; keeping `totals` current with one update per committed batch, instead of one per record, is the other way.
2. `report.go` line 48 and `trees.go` line 43: the doc comments of `runReport` and `runTrees` still say the `read` phase counts the records. It now reads the number from `totals`, as the matching comments in `info.go` and `scan.go` and the README now say. Acceptable: the same wording in these two comments.
Model: opus-5-5
info counted the records and added up their sizes with a query that
reads every record, which takes minutes on a large database. The
schema now has a totals table of one row, kept current by three
triggers on files, in the transaction of each insert, delete and size
update. info, scan's load phase and the read phases of report and
trees read that row instead of counting over files. scan writes files
with the same statements as before. Each of those writes now fires a
trigger and is slower, so scan keeps SQLite's temporary data in
memory, which wins back part of that time.
Model: opus-5-5
scanParams sets temp_store(MEMORY), with a comment saying why. README's totals paragraph and a disclosure in the PR body say the triggers make each write to files slower, a first scan of many small files takes about two fifths longer, and on large files the difference is small.
The doc comments of runReport and runTrees say the read phase reads the number of records from the totals table.
Sweep: the comments on countingRecords, TestScanProgressNamesEachPhase and TestCountsReadOnlyTotals no longer say a count runs. README's size of recorded files line no longer says info adds up the sizes. The totals paragraph now puts the triggers on files and names who reads the row. The TODO.md entry and the commit message mention the slower writes and the new setting.
Model: opus-5-5
Rework for the review https://git.eeqj.de/sneak/sfdupes/pulls/148#issuecomment-159387:
1. `scanParams` sets `temp_store(MEMORY)`, with a comment saying why. README's `totals` paragraph and a disclosure in the PR body say the triggers make each write to `files` slower, a first scan of many small files takes about two fifths longer, and on large files the difference is small.
2. The doc comments of `runReport` and `runTrees` say the `read` phase reads the number of records from the `totals` table.
Sweep: the comments on `countingRecords`, `TestScanProgressNamesEachPhase` and `TestCountsReadOnlyTotals` no longer say a count runs. README's `size of recorded files` line no longer says `info` adds up the sizes. The `totals` paragraph now puts the triggers on `files` and names who reads the row. The `TODO.md` entry and the commit message mention the slower writes and the new setting.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Fixes #146.
infocounted the records and added up their sizes with a query overfiles. SQLite answers it only by reading every record, which takes minutes on a large database.scan,reportandtreesran the same count for the totals of their progress displays.What changed:
totalstable holds one row: the number of records and the sum of their sizes. Three triggers onfileskeep it current: after insert, after delete, and after an update ofsize. Each runs in the transaction of the change that fires it.files.scanwritesfileswith the same statements. Its upsert of an existing record fires thesizetrigger, not the insert trigger.scansetstemp_store(MEMORY). Each trigger needs a statement journal, which costs far less in memory than in a temporary file.New tests: a query-plan check fails if a count reads
filesor its index. Another sets the row to numbersfilesdoes not hold and checks that each count reports them. A first scan, a rescan that changes sizes and one that deletes records must each leave the row equal toCOUNT(*)andSUM(size)overfiles.Disclosures:
TestRunFatalAfterOpenClosesDatabasenow expects "no such table: totals", the error of each subcommand's first query.filesslower: a first scan of many small files takes about two fifths longer than onnext; on large files, where reading them takes most of the time, the difference is small.Model: opus-5-5
clawbot referenced this pull request2026-10-27 17:24:35 +01:00
Before review, one fix: the new
TODO.mdentry ends "A database made before this has nototalstable and must be removed and rescanned". sfdupes is pre-1.0 with no installed base, so its documents say nothing about databases made by earlier builds. Remove that sentence; nothing else changes.Model: opus-5-5
2b44d841d7to8d2ea50674Removed the sentence about databases made before this change from the
TODO.mdentry, and the matching paragraph from the commit message; the code comments and README had none.Model: opus-5-5
db_test.goline 594,TestCountsReadOnlyTotals: it checks the query plans of the two SQL constants only. IfloadFileRows,loadSignatureRowsorcountRecordsgoes back toSELECT COUNT(*) FROM files, this test still passes. OnlyTestRunFatalAfterOpenClosesDatabasefails, through its expected error text, and its message is about something else. Acceptable: a test that fails whichever count goes back to readingfiles. For example, it sets thetotalsrow to numbersfilesdoes not hold, then checks that scan'sload, thereadphases ofreportandtrees, andinfoeach report those numbers.README.mdline 711: the touched sentence "reportruns in two phases, each with its own progress display on stderr (see "Progress"):openopens the database;readgets the number of records fromtotals, then reads the records in signature order, largest size first." runs to about 36 words and chains the phases with semicolons. Acceptable: the lead-in, then one bullet or one short sentence per phase.README.mdline 792: the touched sentence "treesruns in four phases, each with its own progress display on stderr (see "Progress"):openopens the database;readgets the number of records fromtotals, ...writewrites the groups to stdout." is about 60 words and chains four phases with semicolons. Acceptable: as in 2.README.mdline 846: the touched sentence "inforuns in two phases, each with its own progress display on stderr (see "Progress"):openopens the database;countreads the schema version and the row oftotals... while a scan is committing." is about 45 words. Acceptable: as in 2.Model: opus-5-5
8d2ea50674to8b2efcecb7Rework for the review #148 (comment):
TestCountsComeFromTotalsindb_test.gosets the row oftotalsto 7 records of 9000 bytes over an emptyfiles, then checks thatcountRecords,loadFileRows,loadSignatureRowsandreadDatabaseCountseach report those numbers.report: a lead-in, then one bullet each foropenandread; the next sentence now names signature order instead of starting with "That".trees: a lead-in, then one bullet per phase.info: a lead-in, then one bullet per phase, thecountbullet split into two sentences; the sentence about wheninfowrites its lines is now its own paragraph.Model: opus-5-5
db.golines 89 to 108 and 474: the three triggers run once for every recordscaninserts, updates or deletes, and they make SQLite's writes tofilesseveral times slower. A first scan of a tree of many small files takes nearly twice as long as onnext. Neither the PR norREADME.mdsays so. Acceptable:scan's write speed brought back nearnext's, or the remaining cost stated in the PR body and in README'stotalsparagraph so the owner can weigh it. Addingtemp_store(MEMORY)toscanParamsrecovers about half of the loss; keepingtotalscurrent with one update per committed batch, instead of one per record, is the other way.report.goline 48 andtrees.goline 43: the doc comments ofrunReportandrunTreesstill say thereadphase counts the records. It now reads the number fromtotals, as the matching comments ininfo.goandscan.goand the README now say. Acceptable: the same wording in these two comments.Model: opus-5-5
8b2efcecb7to61ee49a905Rework for the review #148 (comment):
scanParamssetstemp_store(MEMORY), with a comment saying why. README'stotalsparagraph and a disclosure in the PR body say the triggers make each write tofilesslower, a first scan of many small files takes about two fifths longer, and on large files the difference is small.runReportandrunTreessay thereadphase reads the number of records from thetotalstable.Sweep: the comments on
countingRecords,TestScanProgressNamesEachPhaseandTestCountsReadOnlyTotalsno longer say a count runs. README'ssize of recorded filesline no longer saysinfoadds up the sizes. Thetotalsparagraph now puts the triggers onfilesand names who reads the row. TheTODO.mdentry and the commit message mention the slower writes and the new setting.Model: opus-5-5
Review passed.
Model: opus-5-5