Compare commits

..
1 Commits
Author SHA1 Message Date
sneak e210fcde97 Say in --help where the database is and what each report prints (closes #177)
check / check (push) Canceled after 0s
The help of sfdupes and of scan names the database, its default path
and SFDUPES_DATABASE. The help of report and trees lists the columns,
the order of the rows and the four escapes. The help of info lists its
lines. The help of dwim says that it scans /, should run as root and
keeps the scan in the database.

Model: opus-5-5
2026-11-11 03:57:43 +00:00
6 changed files with 33 additions and 99 deletions

No files matched your search

+8 -7
View File
@@ -208,8 +208,8 @@ makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
of reads, at the price of a comparison that is not exact. of reads, at the price of a comparison that is not exact.
The hashes live in a database that is kept between runs. A rescan reads only new The hashes live in a database that is kept between runs. A rescan reads only new
files, and files whose size or modification time (mtime) differs from their files, and files whose size changed or whose modification time (mtime) is newer
record. All analysis then works from the database alone. than recorded. All analysis then works from the database alone.
The goal is whole duplicated trees, not only single files. A download extracted The goal is whole duplicated trees, not only single files. A download extracted
twice, or a project copied once too often, is such a tree. It can be removed as twice, or a project copied once too often, is such a tree. It can be removed as
@@ -412,16 +412,17 @@ operands:
- A file not yet in the database is hashed and inserted. - A file not yet in the database is hashed and inserted.
- A file already in the database is not read again when its size and mtime match - A file already in the database is not read again when its size and mtime match
its record. This is what makes a daily rescan cheap. its record. The mtime matches when it is not newer than the recorded one. This
is what makes a daily rescan cheap.
- When such a file's size on disk differs from its recorded `disk_size`, `scan` - When such a file's size on disk differs from its recorded `disk_size`, `scan`
updates `disk_size` alone. It still does not read the file, and counts it as updates `disk_size` alone. It still does not read the file, and counts it as
unchanged. ZFS, for example, counts a new file's blocks up to seconds after unchanged. ZFS, for example, counts a new file's blocks up to seconds after
the write. A block is the unit in which a filesystem allocates disk space. A the write. A block is the unit in which a filesystem allocates disk space. A
file scanned within those seconds gets its real `disk_size` at the next scan. file scanned within those seconds gets its real `disk_size` at the next scan.
- `scan` hashes a file again when its size or its mtime differs from its record. - `scan` hashes a file again when its size differs or its mtime is newer. The
An older mtime counts as much as a newer one. A file put back by `cp -p`, mtime is compared to the nanosecond, as finely as the filesystem records it. A
`rsync -a` or a restore from backup often has an older mtime. The mtime is same-size rewrite counts as a change whenever the filesystem gives it a later
compared to the nanosecond, as finely as the filesystem records it. mtime, even within the same second.
- The scan deletes each record under a scanned operand whose file it could not - The scan deletes each record under a scanned operand whose file it could not
verify. That covers deleted files, and paths the scan could not stat or read verify. That covers deleted files, and paths the scan could not stat or read
this run. To stat a file is to read its details, such as its size, without this run. To stat a file is to read its details, such as its size, without
-3
View File
@@ -34,9 +34,6 @@ are known.
order of the rows and the escapes. The help of `info` lists its lines. The order of the rows and the escapes. The help of `info` lists its lines. The
help of `dwim` says that it scans `/`, should run as root and keeps the scan help of `dwim` says that it scans `/`, should run as root and keeps the scan
in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177) in the database (2026-11-11, https://git.eeqj.de/sneak/sfdupes/issues/177)
- The usage line of `scan` no longer ends in `[flags]`. It is the line
`README.md` shows under "Subcommands" (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/169)
- `report` and `trees` leave out empty files. `report` writes no group of size - `report` and `trees` leave out empty files. `report` writes no group of size
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still 0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
counts toward a tree's digest (2026-11-11, counts toward a tree's digest (2026-11-11,
+13 -16
View File
@@ -33,22 +33,22 @@ const (
// databaseHelp is the paragraph of the root and scan help that names // databaseHelp is the paragraph of the root and scan help that names
// the database (README "Location"). // the database (README "Location").
const databaseHelp = "scan and dwim store what they find in a database " + const databaseHelp = "scan stores what it finds in a database file. " +
"file. report, trees\nand info read it. The file is $" + databaseEnv + "report, trees, info and dwim\nread it. The file is $" + databaseEnv +
" when that is set and\nnot empty, and otherwise " + " when that is set and not empty,\nand otherwise " +
defaultDatabasePath + ", where creating it\nneeds root.\n" defaultDatabasePath + ", where creating it needs root.\n"
// escapesHelp is the paragraph of the report and trees help on how // escapesHelp is the paragraph of the report and trees help on how
// paths are escaped (README "Report output format"). // paths are escaped (README "Report output format").
const escapesHelp = `Four characters in a path are escaped, so that each row stays one const escapesHelp = `Four characters in a path are escaped, so that each
line: a backslash as \\, a tab as \t, a newline as \n and a carriage row stays one line: a backslash as \\, a tab as \t, a newline as \n
return as \r. and a carriage return as \r.
` `
// reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line // reportHelp, treesHelp, infoHelp and dwimHelp follow the one-line
// summary in the help of their subcommands. // summary in the help of their subcommands.
const reportHelp = `The report is tab-separated text on stdout: a header line, then one const reportHelp = `The report is tab-separated text on stdout: a header
row per duplicate file. The columns are: line, then one row per duplicate file. The columns are:
first the path that sorts first, byte by byte, in its group of copies first the path that sorts first, byte by byte, in its group of copies
dupe another copy of that file dupe another copy of that file
@@ -58,8 +58,8 @@ Rows are ordered by size, largest first, then by path.
` + escapesHelp ` + escapesHelp
const treesHelp = `The report is tab-separated text on stdout: a header line, then one const treesHelp = `The report is tab-separated text on stdout: a header
row per duplicate directory tree. The columns are: line, then one row per duplicate directory tree. The columns are:
first the path that sorts first, byte by byte, in its group of copies first the path that sorts first, byte by byte, in its group of copies
dupe another copy of that tree dupe another copy of that tree
@@ -279,11 +279,8 @@ func newScanCommand() *cobra.Command {
short := "Walk trees and synchronize the scan database" short := "Walk trees and synchronize the scan database"
scanCmd := &cobra.Command{ scanCmd := &cobra.Command{
Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...", Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
// The Use line already names every flag, so cobra must not Short: short,
// append its generic "[flags]" to it.
DisableFlagsInUseLine: true,
Short: short,
Long: short + ".\n\n" + databaseHelp + "\n" + Long: short + ".\n\n" + databaseHelp + "\n" +
defaultExcludesHelp(runtime.GOOS), defaultExcludesHelp(runtime.GOOS),
Args: cobra.MinimumNArgs(1), Args: cobra.MinimumNArgs(1),
+1 -31
View File
@@ -717,42 +717,12 @@ func TestRunHelp(t *testing.T) {
} }
} }
func TestRunScanUsageLine(t *testing.T) {
t.Parallel()
// README §Subcommands shows this usage line for scan, and the help
// and the usage error both print it.
const want = "\n sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...\n"
cases := map[string][]string{
"help": {cmdScan, "-h"},
"missing paths": {cmdScan},
}
for name, args := range cases {
t.Run(name, func(t *testing.T) {
t.Parallel()
var stdout, stderr bytes.Buffer
run(args, &stdout, &stderr)
if !strings.Contains(stderr.String(), want) {
t.Errorf("run(%v) stderr = %q, want the usage line %q",
args, stderr.String(), want)
}
})
}
}
func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) { func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
t.Parallel() t.Parallel()
// README §Settings: the help names the database, and says what // README §Settings: the help names the database, and says what
// report, trees, info and dwim print. // report, trees, info and dwim print.
database := []string{ database := []string{databaseEnv, defaultDatabasePath, "needs root"}
"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
}
escapes := []string{`\\`, `\t`, `\n`, `\r`} escapes := []string{`\\`, `\t`, `\n`, `\r`}
// The subcommand whose help is printed; none for the root. // The subcommand whose help is printed; none for the root.
+11 -1
View File
@@ -545,7 +545,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
prog.increment() prog.increment()
old, ok := s.existing[ev.rec.path] old, ok := s.existing[ev.rec.path]
if !ok || old.size != ev.rec.size || !ev.rec.mtime.Equal(old.mtime) { if !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime) {
s.toHash = append(s.toHash, ev.rec) s.toHash = append(s.toHash, ev.rec)
continue continue
@@ -780,6 +780,16 @@ func (s *scanState) updatePhase(ctx context.Context) error {
return applyChanges(ctx, s.db, nil, deletes, prog) return applyChanges(ctx, s.db, nil, deletes, prog)
} }
// mtimeAfter reports whether mtime a is later than mtime b.
// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
func mtimeAfter(a, b time.Time) bool {
if a.Unix() != b.Unix() {
return a.Unix() > b.Unix()
}
return a.Nanosecond() > b.Nanosecond()
}
// underAnyRoot reports whether path is any of the roots or lies under // underAnyRoot reports whether path is any of the roots or lies under
// one of them. // one of them.
func underAnyRoot(path string, roots []string) bool { func underAnyRoot(path string, roots []string) bool {
-41
View File
@@ -1384,47 +1384,6 @@ func TestSyncScanMtimeBump(t *testing.T) {
} }
} }
// TestSyncScanMtimeMovedBack replaces a file with other content of the
// same size and an older mtime, as cp -p or a restore from backup does.
// The next scan must hash the file again.
func TestSyncScanMtimeMovedBack(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
recorded := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
err := os.Chtimes(a, recorded, recorded)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dir)
replaced := pattern(3, 500)
writeFile(t, dir, "a.bin", replaced)
older := recorded.Add(-time.Hour)
err = os.Chtimes(a, older, older)
if err != nil {
t.Fatal(err)
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
assertWholeFileHashed(t, db, a, replaced)
if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(older) {
t.Fatalf("mtime = %v, want %v", r.mtime, older)
}
}
// assertWholeFileHashed fails unless the record for path holds the // assertWholeFileHashed fails unless the record for path holds the
// whole-file hash of data as its head, tail, and content. // whole-file hash of data as its head, tail, and content.
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string, func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,