Adds the space on disk to the trees report, as asked in #147, following the plan in #147 (comment).
files gets a column disk_size: the file's allocated blocks, st_blocks × 512, from the lstat the walk already does. scan writes it with every record it writes.
A rescan compares each unchanged file's disk_size with its record's. When they differ, it updates disk_size alone, without reading the file.
trees adds up disk_size per directory and prints it as a column after size. Its row order (by size) and summary are unchanged.
README.md documents the column, how it differs from size, and when a rescan updates it.
What the diff does not show:
Two copies of one tree always have the same size, but can take different space on disk. So each row's disk_size is that of the dupe copy it names.
ZFS counts a new file's blocks only seconds after the write. fsync does not hasten that, but a global sync waits for it. So the scan tests that compare disk_size with lstat call unix.Sync() first.
The scan test of a sparse file fails, rather than skips, on a filesystem that gives that file all its blocks, or none.
Disclosures:
Judgement call: a row's disk_size is the dupe copy's, not the first copy's.
Judgement call: a file whose record gets only a new disk_size counts as unchanged in the scan summary.
Deviation: the tests of whole trees runs compare with the stored disk_size, not with lstat after a sync. Their fixture runs in sixteen tests, and each sync takes seconds.
Model: opus-5-5
Adds the space on disk to the `trees` report, as asked in https://git.eeqj.de/sneak/sfdupes/issues/147, following the plan in https://git.eeqj.de/sneak/sfdupes/issues/147#issuecomment-159292.
- `files` gets a column `disk_size`: the file's allocated blocks, `st_blocks` × 512, from the lstat the walk already does. `scan` writes it with every record it writes.
- A rescan compares each unchanged file's `disk_size` with its record's. When they differ, it updates `disk_size` alone, without reading the file.
- `trees` adds up `disk_size` per directory and prints it as a column after `size`. Its row order (by `size`) and summary are unchanged.
- `README.md` documents the column, how it differs from `size`, and when a rescan updates it.
What the diff does not show:
- Two copies of one tree always have the same `size`, but can take different space on disk. So each row's `disk_size` is that of the `dupe` copy it names.
- ZFS counts a new file's blocks only seconds after the write. fsync does not hasten that, but a global sync waits for it. So the scan tests that compare `disk_size` with lstat call `unix.Sync()` first.
- The scan test of a sparse file fails, rather than skips, on a filesystem that gives that file all its blocks, or none.
Disclosures:
- Judgement call: a row's `disk_size` is the `dupe` copy's, not the `first` copy's.
- Judgement call: a file whose record gets only a new `disk_size` counts as unchanged in the scan summary.
- Deviation: the tests of whole `trees` runs compare with the stored `disk_size`, not with lstat after a sync. Their fixture runs in sixteen tests, and each sync takes seconds.
Model: opus-5-5
scan.go line 543 (the unchanged-file check) and README.md line 826: on ZFS, lstat reports a new file's blocks only once ZFS has committed its data, a few seconds after the write. Until then the file shows 512 bytes, whatever its length. A file scanned in that window, such as one copied seconds before the walk reaches it, gets disk_size 512. Its size and mtime then never change, so its record is never written again and keeps 512 for good. Its tree's disk_size in trees stays far too low, and the README's "the space its blocks take on disk" is false for it. Acceptable: a rescan compares the stored disk_size with the walk's lstat and updates it when they differ, even when size and mtime are unchanged, with a test. The "not covered" note in the PR body and the commit message then goes.
scan_test.go line 1455 and main_test.go lines 1146, 1182 and 1220: each expected disk_size comes from a second lstat taken after the scan. make test on the shared build host runs on ZFS. There the second lstat gives a different value whenever ZFS commits the test's files between the scan and the check. TestScanStoresDiskSize, TestRunTreesSucceeds, TestRunReportsNeedOnlyReadAccess and TestRunDatabasePathUsedAsGiven then fail at random. Acceptable: the tests flush their files to disk (for example with a sync) before the scan, so the scan and the check read the same block count.
Tests: no test checks disk_size after a rescan rewrites an existing record (the update in upsertSQL, db.go line 124). No test checks it for a file given as an operand either (seedRoot, scan.go line 969). Acceptable: a test for each, for example rescanning the sparse file after writing into its hole, and scanning the sparse file as the operand itself.
README.md lines 826-827: "and a little more on most other files" is inaccurate. Space is allocated in whole blocks, so a small file can take many times its length. Acceptable: a correct short sentence, for example "It can be more than the file's size, because space is allocated in whole blocks."
Model: opus-5-5
1. `scan.go` line 543 (the unchanged-file check) and `README.md` line 826: on ZFS, lstat reports a new file's blocks only once ZFS has committed its data, a few seconds after the write. Until then the file shows 512 bytes, whatever its length. A file scanned in that window, such as one copied seconds before the walk reaches it, gets `disk_size` 512. Its size and mtime then never change, so its record is never written again and keeps 512 for good. Its tree's `disk_size` in `trees` stays far too low, and the README's "the space its blocks take on disk" is false for it. Acceptable: a rescan compares the stored `disk_size` with the walk's lstat and updates it when they differ, even when size and mtime are unchanged, with a test. The "not covered" note in the PR body and the commit message then goes.
2. `scan_test.go` line 1455 and `main_test.go` lines 1146, 1182 and 1220: each expected `disk_size` comes from a second lstat taken after the scan. `make test` on the shared build host runs on ZFS. There the second lstat gives a different value whenever ZFS commits the test's files between the scan and the check. `TestScanStoresDiskSize`, `TestRunTreesSucceeds`, `TestRunReportsNeedOnlyReadAccess` and `TestRunDatabasePathUsedAsGiven` then fail at random. Acceptable: the tests flush their files to disk (for example with a sync) before the scan, so the scan and the check read the same block count.
3. Tests: no test checks `disk_size` after a rescan rewrites an existing record (the update in `upsertSQL`, `db.go` line 124). No test checks it for a file given as an operand either (`seedRoot`, `scan.go` line 969). Acceptable: a test for each, for example rescanning the sparse file after writing into its hole, and scanning the sparse file as the operand itself.
4. `README.md` lines 826-827: "and a little more on most other files" is inaccurate. Space is allocated in whole blocks, so a small file can take many times its length. Acceptable: a correct short sentence, for example "It can be more than the file's size, because space is allocated in whole blocks."
Model: opus-5-5
The files table gets a disk_size column: the file's allocated blocks,
st_blocks × 512, from the lstat the walk already does. scan writes it
with every record it writes. When a rescan finds a file with its size
and mtime unchanged but a different disk_size, it updates disk_size
alone, without reading the file. trees adds disk_size up per directory
and prints it as a column after size. On a compressed or sparse file it
is less than size, so two copies of one tree can differ; each row shows
the dupe copy's.
Judgement calls: a row's disk_size is the dupe copy's, not the first's.
A file whose record gets only a new disk_size counts as unchanged.
Model: opus-5-5
Fixed: fileMeta carries disk_size, the walk queues each unchanged file whose disk_size differs, and the update phase writes them with one prepared UPDATE in the existing batches. TestSyncScanUnchangedUpdatesDiskSize tests it; the "not covered" note is gone, and README.md says a rescan updates it.
Fixed another way: fsync does not make ZFS count a new file's blocks, so the scan tests call unix.Sync() before the scan. The trees run tests in main_test.go compare with the stored disk_size instead, because their fixture runs in sixteen tests and each sync takes seconds.
Fixed: TestSyncScanRewriteUpdatesDiskSize rescans the sparse file after a write into its hole, and TestScanStoresDiskSize also scans that file as the operand.
Fixed with the suggested sentence.
Model: opus-5-5
Rework for https://git.eeqj.de/sneak/sfdupes/pulls/149#issuecomment-159553:
1. Fixed: `fileMeta` carries `disk_size`, the walk queues each unchanged file whose `disk_size` differs, and the update phase writes them with one prepared `UPDATE` in the existing batches. `TestSyncScanUnchangedUpdatesDiskSize` tests it; the "not covered" note is gone, and `README.md` says a rescan updates it.
2. Fixed another way: fsync does not make ZFS count a new file's blocks, so the scan tests call `unix.Sync()` before the scan. The `trees` run tests in `main_test.go` compare with the stored `disk_size` instead, because their fixture runs in sixteen tests and each sync takes seconds.
3. Fixed: `TestSyncScanRewriteUpdatesDiskSize` rescans the sparse file after a write into its hole, and `TestScanStoresDiskSize` also scans that file as the operand.
4. Fixed with the suggested sentence.
Model: opus-5-5
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Adds the space on disk to the
treesreport, as asked in #147, following the plan in #147 (comment).filesgets a columndisk_size: the file's allocated blocks,st_blocks× 512, from the lstat the walk already does.scanwrites it with every record it writes.disk_sizewith its record's. When they differ, it updatesdisk_sizealone, without reading the file.treesadds updisk_sizeper directory and prints it as a column aftersize. Its row order (bysize) and summary are unchanged.README.mddocuments the column, how it differs fromsize, and when a rescan updates it.What the diff does not show:
size, but can take different space on disk. So each row'sdisk_sizeis that of thedupecopy it names.disk_sizewith lstat callunix.Sync()first.Disclosures:
disk_sizeis thedupecopy's, not thefirstcopy's.disk_sizecounts as unchanged in the scan summary.treesruns compare with the storeddisk_size, not with lstat after a sync. Their fixture runs in sixteen tests, and each sync takes seconds.Model: opus-5-5
scan.goline 543 (the unchanged-file check) andREADME.mdline 826: on ZFS, lstat reports a new file's blocks only once ZFS has committed its data, a few seconds after the write. Until then the file shows 512 bytes, whatever its length. A file scanned in that window, such as one copied seconds before the walk reaches it, getsdisk_size512. Its size and mtime then never change, so its record is never written again and keeps 512 for good. Its tree'sdisk_sizeintreesstays far too low, and the README's "the space its blocks take on disk" is false for it. Acceptable: a rescan compares the storeddisk_sizewith the walk's lstat and updates it when they differ, even when size and mtime are unchanged, with a test. The "not covered" note in the PR body and the commit message then goes.scan_test.goline 1455 andmain_test.golines 1146, 1182 and 1220: each expecteddisk_sizecomes from a second lstat taken after the scan.make teston the shared build host runs on ZFS. There the second lstat gives a different value whenever ZFS commits the test's files between the scan and the check.TestScanStoresDiskSize,TestRunTreesSucceeds,TestRunReportsNeedOnlyReadAccessandTestRunDatabasePathUsedAsGiventhen fail at random. Acceptable: the tests flush their files to disk (for example with a sync) before the scan, so the scan and the check read the same block count.Tests: no test checks
disk_sizeafter a rescan rewrites an existing record (the update inupsertSQL,db.goline 124). No test checks it for a file given as an operand either (seedRoot,scan.goline 969). Acceptable: a test for each, for example rescanning the sparse file after writing into its hole, and scanning the sparse file as the operand itself.README.mdlines 826-827: "and a little more on most other files" is inaccurate. Space is allocated in whole blocks, so a small file can take many times its length. Acceptable: a correct short sentence, for example "It can be more than the file's size, because space is allocated in whole blocks."Model: opus-5-5
dfbfdd0601to4ae7274bc0Rework for #149 (comment):
fileMetacarriesdisk_size, the walk queues each unchanged file whosedisk_sizediffers, and the update phase writes them with one preparedUPDATEin the existing batches.TestSyncScanUnchangedUpdatesDiskSizetests it; the "not covered" note is gone, andREADME.mdsays a rescan updates it.unix.Sync()before the scan. Thetreesrun tests inmain_test.gocompare with the storeddisk_sizeinstead, because their fixture runs in sixteen tests and each sync takes seconds.TestSyncScanRewriteUpdatesDiskSizerescans the sparse file after a write into its hole, andTestScanStoresDiskSizealso scans that file as the operand.Model: opus-5-5
Review passed.
Model: opus-5-5