Leave removed copies out of report and trees (closes #173)

A deleted duplicate stayed in both reports until a scan covered it, and
a missing scan operand is a fatal error. report now runs lstat on each
path of a possible duplicate group and drops paths that are gone; trees
does the same for the directory at the top of each copy, never the
files inside. A group left with one copy is not reported. Only "does not
exist" and "not a directory" count as gone, so a permission error keeps
the copy for an unprivileged user. README says a stale report is fixed
by scanning again.

Model: opus-5-5

README.md

@@ -99,15 +99,13 @@ Each `-v` flag below makes a bind mount, which shows a host directory at a path
 inside the container. Bind-mount `/var/lib/sfdupes` to keep the database between
 runs. Mount the tree to scan at the same path inside the container as on the
 host. Read-only access is enough. The database then records the host's paths,
-and so do the reports. `report` and `trees` need that mount too. Without it,
-every recorded path counts as gone, and each report holds only its header line.
+and so do the reports.
 
 ```sh
 make docker
 docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
     sfdupes scan /srv
-docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
-    sfdupes report > dupes.tsv
+docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
 ```
 
 ### Daily scan from `cron`
@@ -213,9 +211,9 @@ makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
 of reads, at the price of a comparison that is not exact.
 
 The hashes live in a database that is kept between runs. A rescan reads only new
-files, and files whose size or modification time (mtime) differs from their
-record. All analysis then works from the database. The reports only check that
-each copy they name still exists.
+files, and files whose size changed or whose modification time (mtime) is newer
+than recorded. All analysis then works from the database. The reports only check
+that each copy they name still exists.
 
 The goal is whole duplicated trees, not only single files. A download extracted
 twice, or a project copied once too often, is such a tree. It can be removed as
@@ -417,16 +415,17 @@ operands:
 
 - A file not yet in the database is hashed and inserted.
 - A file already in the database is not read again when its size and mtime match
-  its record. This is what makes a daily rescan cheap.
+  its record. The mtime matches when it is not newer than the recorded one. This
+  is what makes a daily rescan cheap.
 - When such a file's size on disk differs from its recorded `disk_size`, `scan`
   updates `disk_size` alone. It still does not read the file, and counts it as
   unchanged. ZFS, for example, counts a new file's blocks up to seconds after
   the write. A block is the unit in which a filesystem allocates disk space. A
   file scanned within those seconds gets its real `disk_size` at the next scan.
-- `scan` hashes a file again when its size or its mtime differs from its record.
-  An older mtime counts as much as a newer one. A file put back by `cp -p`,
-  `rsync -a` or a restore from backup often has an older mtime. The mtime is
-  compared to the nanosecond, as finely as the filesystem records it.
+- `scan` hashes a file again when its size differs or its mtime is newer. The
+  mtime is compared to the nanosecond, as finely as the filesystem records it. A
+  same-size rewrite counts as a change whenever the filesystem gives it a later
+  mtime, even within the same second.
 - The scan deletes each record under a scanned operand whose file it could not
   verify. That covers deleted files, and paths the scan could not stat or read
   this run. To stat a file is to read its details, such as its size, without
@@ -739,6 +738,7 @@ when they hold the same relative paths, and each path holds a file with the same
 signature in both. It rebuilds the directory tree from the paths in the records,
 split on `/`. Like `report`, it never opens a scanned file. It checks only that
 the directory at the top of each copy still exists, as `report` checks a path.
+It looks at no file or directory inside a copy.
 
 Definitions:
 
@@ -765,8 +765,8 @@ Processing:
   directories that all have the same digest. The parents' group, or an
   ancestor's, already implies it. A group is always reported when it holds
   sibling directories, or when its members' parents have different digests.
-- A directory that is gone is dropped from its group. A group left with fewer
-  than two directories is not reported.
+- In each group that is not left out, a directory that is gone is left out. A
+  group that still has two or more directories is reported.
 - Within a group, paths are sorted in byte order. The first is `first`, and
   every other path is a `dupe`.
 - Groups are ordered by total tree size, largest first, then by `first` path.

TODO.md

@@ -34,13 +34,12 @@ are known.
   top of a tree. A group left with one copy is not reported. A missing `scan`
   operand is still a fatal error (2026-11-11,
   https://git.eeqj.de/sneak/sfdupes/issues/173)
-- The usage line of `scan` no longer ends in `[flags]`. It is the line
-  `README.md` shows under "Subcommands" (2026-11-11,
-  https://git.eeqj.de/sneak/sfdupes/issues/169)
+
 - `report` and `trees` leave out empty files. `report` writes no group of size
   0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
   counts toward a tree's digest (2026-11-11,
   https://git.eeqj.de/sneak/sfdupes/issues/178)
+
 - The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
   command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
   became `internal/sfdupes/cli.go`. Every other file kept its name. The install

internal/sfdupes/cli.go

@@ -215,13 +215,10 @@ func newScanCommand() *cobra.Command {
 	short := "Walk trees and synchronize the scan database"
 
 	scanCmd := &cobra.Command{
-		Use: cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
-		// The Use line already names every flag, so cobra must not
-		// append its generic "[flags]" to it.
-		DisableFlagsInUseLine: true,
-		Short:                 short,
-		Long:                  short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
-		Args:                  cobra.MinimumNArgs(1),
+		Use:   cmdScan + " [--workers N] [-x] [--no-default-excludes] PATH...",
+		Short: short,
+		Long:  short + ".\n\n" + defaultExcludesHelp(runtime.GOOS),
+		Args:  cobra.MinimumNArgs(1),
 		PreRunE: func(cmd *cobra.Command, _ []string) error {
 			return checkScanWorkers(cmd, scanWorkers)
 		},

internal/sfdupes/cli_test.go

@@ -714,34 +714,6 @@ func TestRunHelp(t *testing.T) {
 	}
 }
 
-func TestRunScanUsageLine(t *testing.T) {
-	t.Parallel()
-
-	// README §Subcommands shows this usage line for scan, and the help
-	// and the usage error both print it.
-	const want = "\n  sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...\n"
-
-	cases := map[string][]string{
-		"help":          {cmdScan, "-h"},
-		"missing paths": {cmdScan},
-	}
-
-	for name, args := range cases {
-		t.Run(name, func(t *testing.T) {
-			t.Parallel()
-
-			var stdout, stderr bytes.Buffer
-
-			run(args, &stdout, &stderr)
-
-			if !strings.Contains(stderr.String(), want) {
-				t.Errorf("run(%v) stderr = %q, want the usage line %q",
-					args, stderr.String(), want)
-			}
-		})
-	}
-}
-
 func TestRunVersion(t *testing.T) {
 	t.Parallel()
 

internal/sfdupes/report_test.go

@@ -491,9 +491,9 @@ func TestReadReportChecksOnlyPossibleDuplicates(t *testing.T) {
 
 	a, u, e := testHash("a"), testHash("u"), sha256.Sum256(nil)
 	recs := []scanRec{
-		{size: 9, head: a, tail: a, content: a, path: "/copy/1"},
-		{size: 9, head: a, tail: a, content: a, path: "/copy/2"},
-		{size: 9, head: a, tail: a, content: a, path: "/copy/3"},
+		{size: 9, head: a, tail: a, content: a, path: "/x/1"},
+		{size: 9, head: a, tail: a, content: a, path: "/x/2"},
+		{size: 9, head: a, tail: a, content: a, path: "/x/3"},
 		{size: 9, head: u, tail: u, content: u, path: "/lonely"},
 		{size: 0, head: e, tail: e, content: e, path: "/empty1"},
 		{size: 0, head: e, tail: e, content: e, path: "/empty2"},
@@ -521,81 +521,12 @@ func TestReadReportChecksOnlyPossibleDuplicates(t *testing.T) {
 
 	slices.Sort(checked)
 
-	want := []string{"/copy/1", "/copy/2", "/copy/3"}
+	want := []string{"/x/1", "/x/2", "/x/3"}
 	if !slices.Equal(checked, want) {
 		t.Errorf("paths checked on disk = %q, want %q", checked, want)
 	}
 }
 
-// TestReadReportDropsGroupLeftWithOnePath checks that a group of two
-// paths, one of them gone, is no duplicate group. If kept, it would
-// write no row, but the summary line would count it.
-func TestReadReportDropsGroupLeftWithOnePath(t *testing.T) {
-	t.Parallel()
-
-	a := testHash("a")
-	recs := []scanRec{
-		{size: 9, head: a, tail: a, content: a, path: "/a/kept"},
-		{size: 9, head: a, tail: a, content: a, path: "/a/gone"},
-	}
-
-	db := openTestDB(t)
-
-	err := applyChanges(t.Context(), db, recs, nil, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	exists := func(path string) bool { return path != "/a/gone" }
-
-	report, err := readReport(t.Context(), db, "", io.Discard, exists, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if report.dupeGroups != 0 {
-		t.Errorf("duplicate groups = %d, want 0", report.dupeGroups)
-	}
-}
-
-// TestReadReportKeepsGroupLeftWithTwoPaths checks that a group of three
-// paths, one of them gone, is a duplicate group of the other two.
-func TestReadReportKeepsGroupLeftWithTwoPaths(t *testing.T) {
-	t.Parallel()
-
-	a := testHash("a")
-	recs := []scanRec{
-		{size: 9, head: a, tail: a, content: a, path: "/b/kept1"},
-		{size: 9, head: a, tail: a, content: a, path: "/b/gone"},
-		{size: 9, head: a, tail: a, content: a, path: "/b/kept2"},
-	}
-
-	db := openTestDB(t)
-
-	err := applyChanges(t.Context(), db, recs, nil, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	exists := func(path string) bool { return path != "/b/gone" }
-
-	var out bytes.Buffer
-
-	report, err := readReport(t.Context(), db, "", &out, exists, nil)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	if report.dupeGroups != 1 {
-		t.Errorf("duplicate groups = %d, want 1", report.dupeGroups)
-	}
-
-	want := "first\tdupe\tsize\n/b/kept1\t/b/kept2\t9\n"
-	if got := out.String(); got != want {
-		t.Errorf("stdout = %q, want %q", got, want)
-	}
-}
-
 func TestDupeGroupsContentSeparates(t *testing.T) {
 	t.Parallel()
 

internal/sfdupes/scan.go

@@ -545,7 +545,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
 		prog.increment()
 
 		old, ok := s.existing[ev.rec.path]
-		if !ok || old.size != ev.rec.size || !ev.rec.mtime.Equal(old.mtime) {
+		if !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime) {
 			s.toHash = append(s.toHash, ev.rec)
 
 			continue
@@ -780,6 +780,16 @@ func (s *scanState) updatePhase(ctx context.Context) error {
 	return applyChanges(ctx, s.db, nil, deletes, prog)
 }
 
+// mtimeAfter reports whether mtime a is later than mtime b.
+// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
+func mtimeAfter(a, b time.Time) bool {
+	if a.Unix() != b.Unix() {
+		return a.Unix() > b.Unix()
+	}
+
+	return a.Nanosecond() > b.Nanosecond()
+}
+
 // underAnyRoot reports whether path is any of the roots or lies under
 // one of them.
 func underAnyRoot(path string, roots []string) bool {

internal/sfdupes/scan_test.go

@@ -1384,47 +1384,6 @@ func TestSyncScanMtimeBump(t *testing.T) {
 	}
 }
 
-// TestSyncScanMtimeMovedBack replaces a file with other content of the
-// same size and an older mtime, as cp -p or a restore from backup does.
-// The next scan must hash the file again.
-func TestSyncScanMtimeMovedBack(t *testing.T) {
-	t.Parallel()
-
-	dir := t.TempDir()
-	db := openTestDB(t)
-	a := writeFile(t, dir, "a.bin", pattern(1, 500))
-
-	recorded := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
-
-	err := os.Chtimes(a, recorded, recorded)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	syncTree(t, db, dir)
-
-	replaced := pattern(3, 500)
-	writeFile(t, dir, "a.bin", replaced)
-
-	older := recorded.Add(-time.Hour)
-
-	err = os.Chtimes(a, older, older)
-	if err != nil {
-		t.Fatal(err)
-	}
-
-	st := syncTree(t, db, dir)
-	if st != (scanStats{walked: 1, updated: 1}) {
-		t.Fatalf("rescan stats = %+v, want 1 updated", st)
-	}
-
-	assertWholeFileHashed(t, db, a, replaced)
-
-	if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(older) {
-		t.Fatalf("mtime = %v, want %v", r.mtime, older)
-	}
-}
-
 // assertWholeFileHashed fails unless the record for path holds the
 // whole-file hash of data as its head, tail, and content.
 func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,

internal/sfdupes/trees_test.go

@@ -503,16 +503,20 @@ func TestCollectTreeGroupsDifferingParents(t *testing.T) {
 // TestCollectTreeGroupsChecksOnlyReportedCopies checks that trees asks
 // whether a directory is on disk only for the members of the groups it
 // returns. It never asks about /a/proj/sub, which lies inside a copy
-// and whose group the group of /a/proj implies.
+// and whose group the group of /a/proj implies, nor about /a/empty,
+// which holds 0 bytes.
 func TestCollectTreeGroupsChecksOnlyReportedCopies(t *testing.T) {
 	t.Parallel()
 
-	f, g, u := testHash("f"), testHash("g"), testHash("u")
+	f, g, u, e := testHash("f"), testHash("g"), testHash("u"),
+		sha256.Sum256(nil)
 	recs := []scanRec{
 		{size: 10, head: f, tail: f, content: f, path: "/a/proj/f"},
 		{size: 20, head: g, tail: g, content: g, path: "/a/proj/sub/g"},
+		{size: 0, head: e, tail: e, content: e, path: "/a/empty/.gitkeep"},
 		{size: 10, head: f, tail: f, content: f, path: "/b/proj/f"},
 		{size: 20, head: g, tail: g, content: g, path: "/b/proj/sub/g"},
+		{size: 0, head: e, tail: e, content: e, path: "/b/empty/.gitkeep"},
 		// /a lacks this file, so /a and /b are no copies of each other.
 		{size: 30, head: u, tail: u, content: u, path: "/b/lonely"},
 	}
@@ -541,49 +545,3 @@ func TestCollectTreeGroupsChecksOnlyReportedCopies(t *testing.T) {
 			checked, want[0])
 	}
 }
-
-// TestCollectTreeGroupsDropsGroupLeftWithOneCopy checks that a group of
-// two copies, one of them gone, is not returned. If kept, it would
-// write no row, but the summary line would count it.
-func TestCollectTreeGroupsDropsGroupLeftWithOneCopy(t *testing.T) {
-	t.Parallel()
-
-	f := testHash("f")
-	recs := []scanRec{
-		{size: 10, head: f, tail: f, content: f, path: "/c/kept/f"},
-		{size: 10, head: f, tail: f, content: f, path: "/c/gone/f"},
-	}
-
-	super, dirs := treeOf(t, recs)
-
-	exists := func(path string) bool { return path != "/c/gone" }
-
-	got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
-	if len(got) != 0 {
-		t.Errorf("groups = %v, want none", got)
-	}
-}
-
-// TestCollectTreeGroupsKeepsGroupLeftWithTwoCopies checks that a group
-// of three copies, one of them gone, is returned with the other two.
-func TestCollectTreeGroupsKeepsGroupLeftWithTwoCopies(t *testing.T) {
-	t.Parallel()
-
-	f := testHash("f")
-	recs := []scanRec{
-		{size: 10, head: f, tail: f, content: f, path: "/d/kept1/f"},
-		{size: 10, head: f, tail: f, content: f, path: "/d/gone/f"},
-		{size: 10, head: f, tail: f, content: f, path: "/d/kept2/f"},
-	}
-
-	super, dirs := treeOf(t, recs)
-
-	exists := func(path string) bool { return path != "/d/gone" }
-
-	got := groupPaths(collectTreeGroups(dirs, super, exists, nil))
-
-	want := [][]string{{"/d/kept1", "/d/kept2"}}
-	if !slices.EqualFunc(got, want, slices.Equal) {
-		t.Errorf("groups = %v, want %v", got, want)
-	}
-}