next holds the fixes from your trial beyond main, a faster info, the space on disk in trees, path arguments resolved to their real paths, and the new dwim subcommand.
sfdupes dwim scans / with the default excludes, then prints the duplicate trees largest first; its last line says sfdupes trees prints the report again (your order in #154).
The duplicate check you specified in #132: the SHA-256 of a file's first and last 64 KiB, plus that of its whole content under 100 MiB or of a sample at each gigabyte above, so a large file is never read in full. Files of unique size are hashed too.
scan resolves every path argument to its real path, symlinks followed, before it starts (your ruling in #144). A symlink argument is scanned at its target; one whose target is missing is a fatal error.
An argument at or inside a .zfs directory, judged by its real path, is never scanned (your ruling in #142).
info answers at once, from a totals table kept by triggers (#146). Each write to the records is slower for it: a first scan of many small files takes about two fifths longer.
trees prints disk_size, the space one copy's files take on disk (#147).
README.md and TODO.md are rewritten for a newcomer (#140).
The disk size tests no longer fail at random on ZFS.
Model: opus-5-5
`next` holds the fixes from your trial beyond `main`, a faster `info`, the space on disk in `trees`, path arguments resolved to their real paths, and the new `dwim` subcommand.
- `sfdupes dwim` scans `/` with the default excludes, then prints the duplicate trees largest first; its last line says `sfdupes trees` prints the report again (your order in https://git.eeqj.de/sneak/sfdupes/issues/154).
- The duplicate check you specified in https://git.eeqj.de/sneak/sfdupes/issues/132: the SHA-256 of a file's first and last 64 KiB, plus that of its whole content under 100 MiB or of a sample at each gigabyte above, so a large file is never read in full. Files of unique size are hashed too.
- Each hash is stored as its raw 32 bytes (https://git.eeqj.de/sneak/sfdupes/issues/134).
- The progress display fixes from https://git.eeqj.de/sneak/sfdupes/issues/133.
- `scan` resolves every path argument to its real path, symlinks followed, before it starts (your ruling in https://git.eeqj.de/sneak/sfdupes/issues/144). A symlink argument is scanned at its target; one whose target is missing is a fatal error.
- An argument at or inside a `.zfs` directory, judged by its real path, is never scanned (your ruling in https://git.eeqj.de/sneak/sfdupes/issues/142).
- `info` answers at once, from a `totals` table kept by triggers (https://git.eeqj.de/sneak/sfdupes/issues/146). Each write to the records is slower for it: a first scan of many small files takes about two fifths longer.
- `trees` prints `disk_size`, the space one copy's files take on disk (https://git.eeqj.de/sneak/sfdupes/issues/147).
- `README.md` and `TODO.md` are rewritten for a newcomer (https://git.eeqj.de/sneak/sfdupes/pulls/140).
- The disk size tests no longer fail at random on ZFS.
Model: opus-5-5
Every file the walk records now gets three hashes: head and tail, of
its first and last 64 KiB, and content. The content hash covers the
whole file under 100 MiB, and 1 MiB at each gigabyte from 100 MiB up,
so a large file is never read in full. One 100 MiB constant replaces
the 10 MiB and 50 MiB thresholds. The size census, the records stored
without hashes and the content phase are gone, so a rescan reads only
new and changed files. report and trees no longer special-case a
missing content hash. README "Duplicate detection" and "Rationale" are
rewritten.
Judgement call: info drops its two hash-count lines, which now always
equal the record count.
Model: opus-5-5
The head, tail and content columns of the files table are BLOB NOT
NULL and hold the raw 32-byte SHA-256, half the size of the hex text,
in the table and in the files_signature index. The hash phase, the
records, report and trees carry each hash as a [sha256.Size]byte and
compare it with ==; nothing is hex-encoded. Reading a record whose
hash column is not 32 bytes is an error naming its path.
In the tests, testHash is a stand-in hash: a string's bytes padded
with zeros, so fixtures keep the sort order their strings give.
Judgement call: a hash of the wrong length is refused on reading,
not copied silently.
Model: opus-5-5
clawbot
changed title from next into main: the duplicate check from the trial to next into main: the duplicate check and hash storage from the trial2026-10-26 03:16:29 +01:00
A path removed while the scan runs gets no warning and no skipped
count, in the walk or the hash phase. If the first of several hard
links is gone, the next is read. Other errors are still warned about.
On a terminal every display is the library's spinner, then the line. A
phase with an exact total ends the padded line with its percent and a
bar at a fixed column, drawn by sfdupes, as the library shows no
spinner beside its own bar. A too-wide line loses its rate first.
The time remaining is labelled remain, not eta.
Judgement call: an operand removed mid-scan is skipped silently too.
Deviation: README's stalled-line example keeps its words on 100
columns, not 80.
Model: opus-5-5
clawbot
changed title from next into main: the duplicate check and hash storage from the trial to next into main: the duplicate check, hash storage and display fixes from the trial2026-10-26 06:33:34 +01:00
An operand inside a directory named .zfs was walked, because the
operand check looked only at the operand's own name. The check now
also looks at every directory above the operand, both as given and
with its symlinks resolved. Such an operand gets a warning, counts as
skipped and is not walked, whether given alone or under another
operand. The check runs before overlapping operands are pruned, so an
operand under another one still gets its warning.
The test for an operand named .zfs now writes its record beneath .zfs
straight into the database, since no scan stores one there any more.
Judgement call: a path component named .zfs counts whatever it is, a
symlink included.
Model: opus-5-5
clawbot
changed title from next into main: the duplicate check, hash storage and display fixes from the trial to next into main: the duplicate check, hash storage, display and .zfs fixes from the trial2026-10-26 15:16:31 +01:00
info counted the records and added up their sizes with a query that
reads every record, which takes minutes on a large database. The
schema now has a totals table of one row, kept current by three
triggers on files, in the transaction of each insert, delete and size
update. info, scan's load phase and the read phases of report and
trees read that row instead of counting over files. scan writes files
with the same statements as before. Each of those writes now fires a
trigger and is slower, so scan keeps SQLite's temporary data in
memory, which wins back part of that time.
Model: opus-5-5
The files table gets a disk_size column: the file's allocated blocks,
st_blocks × 512, from the lstat the walk already does. scan writes it
with every record it writes. When a rescan finds a file with its size
and mtime unchanged but a different disk_size, it updates disk_size
alone, without reading the file. trees adds disk_size up per directory
and prints it as a column after size. On a compressed or sparse file it
is less than size, so two copies of one tree can differ.
Judgement calls: each row's disk_size is the dupe copy's, not the
first's. A file whose record gets only a new disk_size counts as
unchanged.
Model: opus-5-5
Rewrites `README.md` from scratch for #135, so that a newcomer can follow each sentence on one reading. It leads with what the program is and how to use it, then the reference, then development. Where the old text and the code disagreed, it states what the code does; #140 lists each point. Test comments that cited moved text cite the new headings.
Review scope: full reviews through the 16th, then only text changed since.
Dropped: the permission to add `github.com/spf13/viper`.
Model: fable-5-1 (rewrite); opus-5-5 (review, landing message)
Rewrites `TODO.md` from scratch for #135, by the rules applied to `README.md`. Every completed step keeps its date, its issue or branch and its facts, in short sentences, with lists as bullets. An entry superseded later says so. Entries record history, so they were not corrected against the code.
Known stale: the 2026-10-17 release-rule entry.
Changed: one product name in the 2026-10-15 entry is replaced by a description.
Model: fable-5-1 (rewrite); opus-5-5 (review, landing message)
The three disk size tests in `scan_test.go` failed at random on ZFS, which failed the build on `next`. ZFS counts a write's blocks up to seconds later, sometimes through several sizes, and `unix.Sync` does not wait for that. The rewrite test now waits for the size to change, then rescans until the size is the same on both sides of a rescan, and compares exactly, all within 30 seconds. The other two tests fail only on a `disk_size` outside the reads around their scan.
Limit: those two can pass on a wrong 512 when ZFS has not yet counted the blocks.
Model: opus-5-5
`README.md` and a comment in `scan.go` said ZFS counts a new file's blocks only seconds after the write. The count can land in under a second, so both now say up to seconds after the write, as the comment on `diskSizeOf` already does.
Model: opus-5-5
Since the last state, next gained the rewritten README.md and TODO.md (#140, #141) and the fix for tests that failed at random (#150).
Disclosures:
Judgement call: after 16 full reviews I froze the README, and the later reviews read only what had changed. A false sentence found later gets its own small PR, as #153 did.
Limit: two of the disk size tests can pass on a wrong size of 512 while ZFS has not yet counted a file's blocks.
Model: opus-5-5
No known bug is open, and nothing is left that does not wait on you.
- https://git.eeqj.de/sneak/sfdupes/issues/144 waits on your answer about resolving every path before the scan starts.
- https://git.eeqj.de/sneak/sfdupes/issues/56 waits on your trial at real scale.
- https://git.eeqj.de/sneak/sfdupes/issues/98 stays open until you call it finished.
Since the last state, `next` gained the rewritten `README.md` and `TODO.md` (https://git.eeqj.de/sneak/sfdupes/pulls/140, https://git.eeqj.de/sneak/sfdupes/pulls/141) and the fix for tests that failed at random (https://git.eeqj.de/sneak/sfdupes/issues/150).
Disclosures:
- Judgement call: after 16 full reviews I froze the README, and the later reviews read only what had changed. A false sentence found later gets its own small PR, as https://git.eeqj.de/sneak/sfdupes/pulls/153 did.
- Limit: two of the disk size tests can pass on a wrong size of 512 while ZFS has not yet counted a file's blocks.
Model: opus-5-5
scan now resolves each operand to its real path, every symlink followed, before the walk, as sneak ruled. Files are recorded and warnings printed under that path. With /srv/link a symlink to a directory outside /srv, scan /srv /srv/link/sub now walks sub on its own and keeps its records, where before it dropped sub and deleted them. A symlink operand is scanned at its target. A dangling symlink is a fatal error, exit 1. A .. after a symlink counts from the symlink's target, as realpath does. The .zfs rule applies to the real path. The walk still follows no symlink it finds.
Judgement call: records an earlier scan stored under a path through a symlink are not migrated; there is no installed base.
Model: opus-5-5
clawbot
changed title from next into main: the duplicate check, hash storage, display and .zfs fixes from the trial to next into main: the trial's duplicate check, hash storage, display and .zfs fixes, and real-path arguments2026-11-06 19:58:31 +01:00
sfdupes dwim, as sneak ordered: with no arguments it runs what scan / runs with no flags, default excludes on, then what trees runs, printing the duplicate trees largest first on stdout. Its last line, on stderr, says that sfdupes trees prints the report again. It needs no setup beyond what scan creates. SIGINT or SIGTERM during the scan stops it as it stops scan, exit 1, with no report; during the report a signal ends it at once, as it ends trees. A positional argument is a usage error, exit 2.
Model: opus-5-5
clawbot
changed title from next into main: the trial's duplicate check, hash storage, display and .zfs fixes, and real-path arguments to next into main: the trial's fixes, real-path arguments, and sfdupes dwim2026-11-06 20:28:56 +01:00
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
nextholds the fixes from your trial beyondmain, a fasterinfo, the space on disk intrees, path arguments resolved to their real paths, and the newdwimsubcommand.sfdupes dwimscans/with the default excludes, then prints the duplicate trees largest first; its last line sayssfdupes treesprints the report again (your order in #154).scanresolves every path argument to its real path, symlinks followed, before it starts (your ruling in #144). A symlink argument is scanned at its target; one whose target is missing is a fatal error..zfsdirectory, judged by its real path, is never scanned (your ruling in #142).infoanswers at once, from atotalstable kept by triggers (#146). Each write to the records is slower for it: a first scan of many small files takes about two fifths longer.treesprintsdisk_size, the space one copy's files take on disk (#147).README.mdandTODO.mdare rewritten for a newcomer (#140).Model: opus-5-5
next into main: the duplicate check from the trialto next into main: the duplicate check and hash storage from the trialnext into main: the duplicate check and hash storage from the trialto next into main: the duplicate check, hash storage and display fixes from the trialnext into main: the duplicate check, hash storage and display fixes from the trialto next into main: the duplicate check, hash storage, display and .zfs fixes from the trialNo known bug is open, and nothing is left that does not wait on you.
Since the last state,
nextgained the rewrittenREADME.mdandTODO.md(#140, #141) and the fix for tests that failed at random (#150).Disclosures:
Model: opus-5-5
next into main: the duplicate check, hash storage, display and .zfs fixes from the trialto next into main: the trial's duplicate check, hash storage, display and .zfs fixes, and real-path argumentsnext into main: the trial's duplicate check, hash storage, display and .zfs fixes, and real-path argumentsto next into main: the trial's fixes, real-path arguments, and sfdupes dwim