Compare commits

...
Author SHA1 Message Date
sneak f9f309e94a Hash a file again when its mtime moved back, not only forward (closes #176)
check / check (push) Successful in 3m46s
scan treated a recorded file as unchanged when its size matched and its
mtime was not newer. A file replaced by other content of the same size
with an older mtime, as cp -p, rsync -a or a restore writes it, kept the
hash of the file it replaced. Any mtime difference now counts as a
change. README.md says so, and a new test covers the older mtime.

Model: opus-5-5
2026-11-11 02:54:08 +00:00
clawbot 6c529fcc6d Leave empty files out of report and trees (closes #178) (#183)
check / check (push) Canceled after 0s
2026-11-11 03:52:45 +01:00
sneak 202f083cc0 Merge branch 'main' into next
check / check (push) Canceled after 0s
2026-11-10 23:11:52 +01:00
clawbot 1017332d68 Move the command to cmd/sfdupes and the code to internal/sfdupes (closes #157)
check / check (push) Canceled after 0s
`go install sneak.berlin/go/sfdupes/cmd/sfdupes@...` found no package: the command was built from `package main` in the repository root. It now lives in `cmd/sfdupes/main.go`, whose one call is `sfdupes.Main()`, and every other Go file moved with `git mv` to `internal/sfdupes`, as `REPO_POLICIES.md` and the Go style guide require. The `Makefile` builds `./cmd/sfdupes` and sets `Version` by its new import path. README "Install" and `TODO.md` give the new paths; the fleet-written "flat layout" divergence is gone.

Rule suppressed: each test file carries `//nolint:testpackage`, since the tests call unexported functions throughout.

Model: opus-5-5
2026-11-10 21:15:53 +01:00
27 changed files with 248 additions and 129 deletions

No files matched your search

+2 -2
View File
@@ -4,7 +4,7 @@ export CGO_ENABLED = 0
BINARY := sfdupes BINARY := sfdupes
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo dev) VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
LDFLAGS := -X main.Version=$(VERSION) LDFLAGS := -X sneak.berlin/go/sfdupes/internal/sfdupes.Version=$(VERSION)
.PHONY: sfdupes build bootstrap setup test lint fmt fmt-check check docker hooks clean .PHONY: sfdupes build bootstrap setup test lint fmt fmt-check check docker hooks clean
@@ -14,7 +14,7 @@ LDFLAGS := -X main.Version=$(VERSION)
# Default target: build the binary. Phony so go build (which has its # Default target: build the binary. Phony so go build (which has its
# own build cache) always decides what to recompile. # own build cache) always decides what to recompile.
sfdupes: sfdupes:
go build -ldflags "$(LDFLAGS)" -o $(BINARY) go build -ldflags "$(LDFLAGS)" -o $(BINARY) ./cmd/sfdupes
build: sfdupes build: sfdupes
+26 -22
View File
@@ -57,25 +57,25 @@ shell shows both on the terminal, and `>` redirects stdout alone.
With Go installed, this builds and installs the newest tagged release: With Go installed, this builds and installs the newest tagged release:
```sh ```sh
go install sneak.berlin/go/sfdupes@latest go install sneak.berlin/go/sfdupes/cmd/sfdupes@latest
``` ```
`@main` installs the tip of the `main` branch instead: `@main` installs the tip of the `main` branch instead:
```sh ```sh
go install sneak.berlin/go/sfdupes@main go install sneak.berlin/go/sfdupes/cmd/sfdupes@main
``` ```
The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set. The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set.
`sfdupes --version` prints `sfdupes VERSION` and exits. For a binary from `sfdupes --version` prints `sfdupes VERSION` and exits. For a binary from
`go install`, `VERSION` is the version Go recorded in it. For `@latest` that is `go install`, `VERSION` is the version Go recorded in it. For `@latest` that is
a release tag such as `v0.0.3`. For `@main` it is the commit's tag when it has a release tag. For `@main` it is the commit's tag when it has one. Otherwise it
one. Otherwise it is a pseudo-version, Go's version string for an untagged is a pseudo-version, Go's version string for an untagged commit. That string
commit. That string ends in the commit's hash. A binary built from a clone ends in the commit's hash. A binary built from a clone carries the output of
carries the output of `git describe --tags --always --dirty` in that clone, such `git describe --tags --always --dirty` in that clone, such as `v0.0.3` on a
as `v0.0.3` on a tagged commit. `VERSION` is `dev` when no version is known. To tagged commit. `VERSION` is `dev` when no version is known. To see everything Go
see everything Go recorded in an installed binary: recorded in an installed binary:
```sh ```sh
go version -m "$(command -v sfdupes)" go version -m "$(command -v sfdupes)"
@@ -208,8 +208,8 @@ makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
of reads, at the price of a comparison that is not exact. of reads, at the price of a comparison that is not exact.
The hashes live in a database that is kept between runs. A rescan reads only new The hashes live in a database that is kept between runs. A rescan reads only new
files, and files whose size changed or whose modification time (mtime) is newer files, and files whose size or modification time (mtime) differs from their
than recorded. All analysis then works from the database alone. record. All analysis then works from the database alone.
The goal is whole duplicated trees, not only single files. A download extracted The goal is whole duplicated trees, not only single files. A download extracted
twice, or a project copied once too often, is such a tree. It can be removed as twice, or a project copied once too often, is such a tree. It can be removed as
@@ -410,17 +410,16 @@ operands:
- A file not yet in the database is hashed and inserted. - A file not yet in the database is hashed and inserted.
- A file already in the database is not read again when its size and mtime match - A file already in the database is not read again when its size and mtime match
its record. The mtime matches when it is not newer than the recorded one. This its record. This is what makes a daily rescan cheap.
is what makes a daily rescan cheap.
- When such a file's size on disk differs from its recorded `disk_size`, `scan` - When such a file's size on disk differs from its recorded `disk_size`, `scan`
updates `disk_size` alone. It still does not read the file, and counts it as updates `disk_size` alone. It still does not read the file, and counts it as
unchanged. ZFS, for example, counts a new file's blocks up to seconds after unchanged. ZFS, for example, counts a new file's blocks up to seconds after
the write. A block is the unit in which a filesystem allocates disk space. A the write. A block is the unit in which a filesystem allocates disk space. A
file scanned within those seconds gets its real `disk_size` at the next scan. file scanned within those seconds gets its real `disk_size` at the next scan.
- `scan` hashes a file again when its size differs or its mtime is newer. The - `scan` hashes a file again when its size or its mtime differs from its record.
mtime is compared to the nanosecond, as finely as the filesystem records it. A An older mtime counts as much as a newer one. A file put back by `cp -p`,
same-size rewrite counts as a change whenever the filesystem gives it a later `rsync -a` or a restore from backup often has an older mtime. The mtime is
mtime, even within the same second. compared to the nanosecond, as finely as the filesystem records it.
- The scan deletes each record under a scanned operand whose file it could not - The scan deletes each record under a scanned operand whose file it could not
verify. That covers deleted files, and paths the scan could not stat or read verify. That covers deleted files, and paths the scan could not stat or read
this run. To stat a file is to read its details, such as its size, without this run. To stat a file is to read its details, such as its size, without
@@ -665,6 +664,7 @@ Processing:
- Records are grouped by signature, the four values - Records are grouped by signature, the four values
`(size, head, tail, content)`. `(size, head, tail, content)`.
- Every group with two or more paths is a duplicate group. - Every group with two or more paths is a duplicate group.
- Empty files form no group, because removing one frees nothing.
- Within a group, paths are sorted in byte order. The first is `first`, and - Within a group, paths are sorted in byte order. The first is `first`, and
every other path is a `dupe`. every other path is a `dupe`.
- Groups are ordered by size, largest first, then by `first` path. The output is - Groups are ordered by size, largest first, then by `first` path. The output is
@@ -713,7 +713,7 @@ The summary line on stderr gives:
- the reclaimable bytes, the sum of `size` over all rows. - the reclaimable bytes, the sum of `size` over all rows.
``` ```
report: 14 records read, 5 duplicate groups, 8 dupe files, 10.0 KiB reclaimable report: 14 records read, 4 duplicate groups, 7 dupe files, 10.0 KiB reclaimable
``` ```
### `trees` mode ### `trees` mode
@@ -741,6 +741,9 @@ Processing:
- `trees` builds the directory tree and computes every directory's digest. It - `trees` builds the directory tree and computes every directory's digest. It
then groups the directories by digest. Every group with two or more then groups the directories by digest. Every group with two or more
directories is a duplicate-tree group. directories is a duplicate-tree group.
- `trees` leaves out a group whose trees hold 0 bytes, because removing one
frees nothing. An empty file still counts toward the digest of a tree that
holds other files.
- `trees` leaves out a group that a group of its members' parents already - `trees` leaves out a group that a group of its members' parents already
implies. A group is left out when its members' parents are distinct implies. A group is left out when its members' parents are distinct
directories that all have the same digest. The parents' group, or an directories that all have the same digest. The parents' group, or an
@@ -1382,7 +1385,7 @@ All of the following, run in this directory, must pass:
head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content head -c 2000 /dev/urandom > "$d/a/unique.bin" # same size, different content
printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates printf 'x' > "$d/tiny1"; printf 'x' > "$d/tiny2" # 1-byte duplicates
printf 'y' > "$d/tiny3" # 1-byte non-duplicate printf 'y' > "$d/tiny3" # 1-byte non-duplicate
: > "$d/empty1"; : > "$d/empty2" # empty duplicates : > "$d/empty1"; : > "$d/empty2" # empty files
# duplicate trees: t1 and t2 are identical; t3 differs by one filename # duplicate trees: t1 and t2 are identical; t3 differs by one filename
mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub" mkdir -p "$d/t1/sub" "$d/t2/sub" "$d/t3/sub"
head -c 3000 /dev/urandom > "$d/t1/f1" head -c 3000 /dev/urandom > "$d/t1/f1"
@@ -1407,8 +1410,8 @@ All of the following, run in this directory, must pass:
is the path that sorts first. is the path that sorts first.
- `t1/f1`, `t2/f1` and `t3/f1` form one group. - `t1/f1`, `t2/f1` and `t3/f1` form one group.
- `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group. - `t1/sub/f2`, `t2/sub/f2` and `t3/sub/f2renamed` form one group.
- `tiny1` and `tiny2` form a pair, and so do `empty1` and `empty2`. - `tiny1` and `tiny2` form a pair.
- `unique.bin` and `tiny3` appear nowhere. - `unique.bin`, `tiny3`, `empty1` and `empty2` appear nowhere.
- The groups come in order of size, largest first. - The groups come in order of size, largest first.
`trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`. `trees` prints exactly one row, `$d/t1` as `first` and `$d/t2` as `dupe`.
@@ -1421,8 +1424,9 @@ All of the following, run in this directory, must pass:
because its content changed. `copy.bin` and `copy2.bin` remain as one pair. because its content changed. `copy.bin` and `copy2.bin` remain as one pair.
`unique.bin` is gone from the database. `unique.bin` is gone from the database.
The test suite automates this scenario (see `scan_test.go`). It also checks The test suite automates this scenario (see
that `report` and `trees` never touch the scanned filesystem. `internal/sfdupes/scan_test.go`). It also checks that `report` and `trees`
never touch the scanned filesystem.
## TODO ## TODO
+53 -39
View File
@@ -29,6 +29,16 @@
Newest first. An entry ends with its date and its issue or branch, where those Newest first. An entry ends with its date and its issue or branch, where those
are known. are known.
- `report` and `trees` leave out empty files. `report` writes no group of size
0, and `trees` writes no group whose trees hold 0 bytes. An empty file still
counts toward a tree's digest (2026-11-11,
https://git.eeqj.de/sneak/sfdupes/issues/178)
- The Go code moved out of the repository root. `cmd/sfdupes/main.go` is the
command, and it calls `Main` in the package `internal/sfdupes`. `main.go`
became `internal/sfdupes/cli.go`. Every other file kept its name. The install
path is now `sneak.berlin/go/sfdupes/cmd/sfdupes` (2026-11-10,
https://git.eeqj.de/sneak/sfdupes/issues/157)
- New subcommand `dwim`. It runs what `sfdupes scan /` runs with no flags, then - New subcommand `dwim`. It runs what `sfdupes scan /` runs with no flags, then
what `sfdupes trees` runs. Its last line on stderr says that `sfdupes trees` what `sfdupes trees` runs. Its last line on stderr says that `sfdupes trees`
prints the report again (2026-11-06, prints the report again (2026-11-06,
@@ -40,18 +50,19 @@ are known.
walks `sub` on its own instead of dropping it and deleting its records walks `sub` on its own instead of dropping it and deleting its records
(2026-11-06, https://git.eeqj.de/sneak/sfdupes/issues/144) (2026-11-06, https://git.eeqj.de/sneak/sfdupes/issues/144)
- `README.md` and a comment in `scan.go` no longer say ZFS counts a new file's - `README.md` and a comment in `internal/sfdupes/scan.go` no longer say ZFS
blocks only seconds after the write. The count can land in under a second. counts a new file's blocks only seconds after the write. The count can land in
Both now say up to seconds after the write, as the comment on `diskSizeOf` in under a second. Both now say up to seconds after the write, as the comment on
`scan_test.go` does (2026-11-03, https://git.eeqj.de/sneak/sfdupes/issues/152) `diskSizeOf` in `internal/sfdupes/scan_test.go` does (2026-11-03,
https://git.eeqj.de/sneak/sfdupes/issues/152)
- The three disk size tests in `scan_test.go` no longer fail at random on ZFS. - The three disk size tests in `internal/sfdupes/scan_test.go` no longer fail at
`unix.Sync` can return before ZFS has counted a file's blocks, and one ZFS random on ZFS. `unix.Sync` can return before ZFS has counted a file's blocks,
commit can take a file through several sizes. The test that needs the size to and one ZFS commit can take a file through several sizes. The test that needs
change waits for it, for at most 30 seconds. Each test reads the size before the size to change waits for it, for at most 30 seconds. Each test reads the
and after its scan. The rewrite test rescans until the two reads match, then size before and after its scan. The rewrite test rescans until the two reads
compares exactly. The other two fail only on a `disk_size` outside the two match, then compares exactly. The other two fail only on a `disk_size` outside
reads (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/150) the two reads (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/150)
- `TODO.md` rewritten from scratch by the same rules as `README.md`, sentence by - `TODO.md` rewritten from scratch by the same rules as `README.md`, sentence by
sentence. Every completed step is kept, with its date and its issue or branch. sentence. Every completed step is kept, with its date and its issue or branch.
@@ -62,9 +73,13 @@ are known.
it. It leads with what the program is and how to use it. A reference section it. It leads with what the program is and how to use it. A reference section
covers every setting and every subcommand. Development and testing come last. covers every setting and every subcommand. Development and testing come last.
Every statement was checked against the code on `next`. Five comments in Every statement was checked against the code on `next`. Five comments in
`main_test.go` cite "Settings", where their rules now live. One comment each `internal/sfdupes/cli_test.go` cite "Settings", where their rules now live.
in `scan_test.go`, `trees_test.go` and `exclude_test.go` cites what the README One comment each in these files cites what the README now says:
now says (2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135) - `internal/sfdupes/scan_test.go`;
- `internal/sfdupes/trees_test.go`;
- `internal/sfdupes/exclude_test.go`.
(2026-11-02, https://git.eeqj.de/sneak/sfdupes/issues/135)
- Each record holds `disk_size`, the bytes the file takes on disk. That is its - Each record holds `disk_size`, the bytes the file takes on disk. That is its
allocated blocks, `st_blocks` × 512, from the `lstat` call the walk already allocated blocks, `st_blocks` × 512, from the `lstat` call the walk already
@@ -137,12 +152,12 @@ are known.
https://git.eeqj.de/sneak/sfdupes/issues/109) https://git.eeqj.de/sneak/sfdupes/issues/109)
- The release rule was set. Until 1.0, each tip of `main` that passes the build - The release rule was set. Until 1.0, each tip of `main` that passes the build
gets the next `v0.0.N` tag. `go install sneak.berlin/go/sfdupes@latest` then gets the next `v0.0.N` tag.
installs it. `go.mod` retracts `v0.0.1`, the early build without a database. `go install sneak.berlin/go/sfdupes/cmd/sfdupes@latest` then installs it.
When the linker set no version, `sfdupes --version` prints the version Go `go.mod` retracts `v0.0.1`, the early build without a database. When the
recorded in the binary. That is the tag, or the pseudo-version of a linker set no version, `sfdupes --version` prints the version Go recorded in
`go install` build. It prints `dev` only when neither is known. README the binary. That is the tag, or the pseudo-version of a `go install` build. It
"Install" documented: prints `dev` only when neither is known. README "Install" documented:
- the rule; - the rule;
- what `@main` and `v0.0.1` give; - what `@main` and `v0.0.1` give;
- how to check an installed binary. - how to check an installed binary.
@@ -393,8 +408,9 @@ are known.
`gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04, `gomodguard_v2`, so lint no longer prints a deprecation warning (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/26) https://git.eeqj.de/sneak/sfdupes/issues/26)
- A test fails when either `hashWorker` cancellation check in `scan.go` is - A test fails when either `hashWorker` cancellation check in
removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/83) `internal/sfdupes/scan.go` is removed (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/83)
- A database path holding `?`, `#` or `%` opens exactly the file it names - A database path holding `?`, `#` or `%` opens exactly the file it names
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55) (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/55)
@@ -402,15 +418,15 @@ are known.
- `scan` rejects `--workers` below 1 as a usage error instead of running - `scan` rejects `--workers` below 1 as a usage error instead of running
single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10) single-threaded (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/10)
- A test fails when either walk cancellation check in `scan.go` is removed - A test fails when either walk cancellation check in `internal/sfdupes/scan.go`
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81) is removed (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/81)
- A test checks that `scan` refuses a database with another schema version - A test checks that `scan` refuses a database with another schema version
(2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64) (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/64)
- Four inaccurate comments in `cancel_test.go` were corrected, and - Four inaccurate comments in `internal/sfdupes/cancel_test.go` were corrected,
`walkCancelInFlightDirs` was renamed to `walkCancelInFlightFiles` (2026-10-04, and `walkCancelInFlightDirs` was renamed to `walkCancelInFlightFiles`
https://git.eeqj.de/sneak/sfdupes/issues/33) (2026-10-04, https://git.eeqj.de/sneak/sfdupes/issues/33)
- The `-x` filesystem-boundary rules in `subdirJob` are tested (2026-10-04, - The `-x` filesystem-boundary rules in `subdirJob` are tested (2026-10-04,
https://git.eeqj.de/sneak/sfdupes/issues/17) https://git.eeqj.de/sneak/sfdupes/issues/17)
@@ -617,15 +633,15 @@ are known.
- `loadRecords`; - `loadRecords`;
- `resolveRoots`. - `resolveRoots`.
The single exit point is `run` in `main.go`. It maps a `fatalError`, The single exit point is `run` in `internal/sfdupes/cli.go`. It maps a
anything a subcommand returned, to exit 1, and cobra's own argument and flag `fatalError`, anything a subcommand returned, to exit 1, and cobra's own
errors to exit 2. So a runtime failure is never reported as a usage error, argument and flag errors to exit 2. So a runtime failure is never reported
nor printed with the usage text. `main_test.go` drives the CLI in-process as a usage error, nor printed with the usage text.
and asserts the exit codes from README "Error handling" and the `internal/sfdupes/cli_test.go` drives the CLI in-process and asserts the
stdout/stderr split. It also checks that a fatal error after the database is exit codes from README "Error handling" and the stdout/stderr split. It also
open leaves no `-wal` or `-shm` file behind, for `scan`, `report` or `trees` checks that a fatal error after the database is open leaves no `-wal` or
(2026-08-09, branch `db-close-on-fatal`, `-shm` file behind, for `scan`, `report` or `trees` (2026-08-09, branch
https://git.eeqj.de/sneak/sfdupes/issues/4) `db-close-on-fatal`, https://git.eeqj.de/sneak/sfdupes/issues/4)
- `golangci-lint` was updated to v2.12.2 with the canonical config, merged as - `golangci-lint` was updated to v2.12.2 with the canonical config, merged as
`38a01bd`. The pinned linter in the `Dockerfile` lint stage and `38a01bd`. The pinned linter in the `Dockerfile` lint stage and
@@ -758,6 +774,4 @@ are known.
# Accepted Divergences # Accepted Divergences
- Flat single-package layout, with the `.go` files in the repo root. That is None.
fine for a small single-binary tool per the Go styleguide, and the tracker
audit agrees.
+29
View File
@@ -0,0 +1,29 @@
// Command sfdupes quickly identifies candidate duplicate files across
// very large filesystems without reading every byte of every file.
// Files are considered duplicates when their sizes are equal and they
// agree on three SHA-256 hashes: of the first 64 KiB, of the last 64
// KiB, and of the content. The content hash covers the whole file under
// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large
// file is never read in full. scan maintains a persistent SQLite
// database of file signatures (SFDUPES_DATABASE, default
// /var/lib/sfdupes/db.sqlite) that the other subcommands read.
//
// Usage:
//
// sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
// sfdupes report > dupes.tsv
// sfdupes trees > dupetrees.tsv
// sfdupes info
// sfdupes dwim
// sfdupes --version
//
// See README.md for the complete specification.
package main
import (
"os"
"sneak.berlin/go/sfdupes/internal/sfdupes"
)
func main() { os.Exit(sfdupes.Main()) }
@@ -1,4 +1,4 @@
package main package sfdupes //nolint:testpackage // cancels the unexported runScan and hashWorker
import ( import (
"context" "context"
+7 -23
View File
@@ -1,24 +1,6 @@
// Command sfdupes quickly identifies candidate duplicate files across // Package sfdupes implements the sfdupes command and the database its
// very large filesystems without reading every byte of every file. // subcommands share.
// Files are considered duplicates when their sizes are equal and they package sfdupes
// agree on three SHA-256 hashes: of the first 64 KiB, of the last 64
// KiB, and of the content. The content hash covers the whole file under
// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large
// file is never read in full. scan maintains a persistent SQLite
// database of file signatures (SFDUPES_DATABASE, default
// /var/lib/sfdupes/db.sqlite) that the other subcommands read.
//
// Usage:
//
// sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
// sfdupes report > dupes.tsv
// sfdupes trees > dupetrees.tsv
// sfdupes info
// sfdupes dwim
// sfdupes --version
//
// See README.md for the complete specification.
package main
import ( import (
"context" "context"
@@ -86,12 +68,14 @@ func buildVersion(linkerVersion string, info *debug.BuildInfo) string {
return info.Main.Version return info.Main.Version
} }
func main() { // Main runs the command line in os.Args and returns the process exit
// code.
func Main() int {
// Once the reader of a stdout pipe has gone, as in "sfdupes report | // Once the reader of a stdout pipe has gone, as in "sfdupes report |
// head", the Go runtime ends the process with SIGPIPE on the next // head", the Go runtime ends the process with SIGPIPE on the next
// write instead of returning an error (README "Error handling"). // write instead of returning an error (README "Error handling").
// Registering for SIGPIPE with os/signal would change that. // Registering for SIGPIPE with os/signal would change that.
os.Exit(run(os.Args[1:], os.Stdout, os.Stderr)) return run(os.Args[1:], os.Stdout, os.Stderr)
} }
// run executes args against the command tree and returns the process // run executes args against the command tree and returns the process
@@ -1,4 +1,4 @@
package main package sfdupes //nolint:testpackage // drives the unexported run
import ( import (
"bytes" "bytes"
+1 -1
View File
@@ -1,4 +1,4 @@
package main package sfdupes
import ( import (
"context" "context"
+1 -1
View File
@@ -1,4 +1,4 @@
package main package sfdupes //nolint:testpackage // calls the unexported database openers
import ( import (
"context" "context"
+1 -1
View File
@@ -1,4 +1,4 @@
package main package sfdupes
import ( import (
"context" "context"
@@ -1,4 +1,4 @@
package main package sfdupes //nolint:testpackage // calls the unexported runDwim
import ( import (
"bytes" "bytes"
+1 -1
View File
@@ -1,4 +1,4 @@
package main package sfdupes
// This file holds two rules for the walk (README "Rules for the walk"): // This file holds two rules for the walk (README "Rules for the walk"):
// the default excludes, the paths a scan's walk does not enter below an // the default excludes, the paths a scan's walk does not enter below an
@@ -1,6 +1,6 @@
//go:build darwin //go:build darwin
package main package sfdupes
import ( import (
"slices" "slices"
@@ -1,6 +1,6 @@
//go:build !darwin //go:build !darwin
package main package sfdupes
import ( import (
"fmt" "fmt"
@@ -1,4 +1,4 @@
package main package sfdupes //nolint:testpackage // checks the unexported defaultExcludes
import ( import (
"bytes" "bytes"
+1 -1
View File
@@ -1,4 +1,4 @@
package main package sfdupes
import ( import (
"context" "context"
@@ -1,4 +1,4 @@
package main package sfdupes //nolint:testpackage // seeds the database with unexported scanRec
import ( import (
"fmt" "fmt"
+1 -1
View File
@@ -1,4 +1,4 @@
package main package sfdupes
import ( import (
"fmt" "fmt"
@@ -1,4 +1,4 @@
package main package sfdupes //nolint:testpackage // drives the unexported startProgress
import ( import (
"fmt" "fmt"
+5 -4
View File
@@ -1,4 +1,4 @@
package main package sfdupes
import ( import (
"context" "context"
@@ -195,10 +195,11 @@ func (w *reportWriter) add(r scanRec) error {
} }
// endGroup ends the paths read with w.sig: two or more are a duplicate // endGroup ends the paths read with w.sig: two or more are a duplicate
// group, held with its paths sorted until writeGroups writes it. // group, held with its paths sorted until writeGroups writes it. Empty
// files form no group, since removing one frees nothing.
func (w *reportWriter) endGroup() { func (w *reportWriter) endGroup() {
if len(w.paths) < minGroupSize { if len(w.paths) < minGroupSize || w.sig.size == 0 {
// A path with no duplicate is dropped, and its slice reused. // The paths are dropped, and their slice reused.
w.paths = w.paths[:0] w.paths = w.paths[:0]
return return
@@ -1,4 +1,4 @@
package main package sfdupes //nolint:testpackage // checks the unexported dupeGroup results
import ( import (
"bytes" "bytes"
@@ -434,6 +434,27 @@ func TestDupeGroups(t *testing.T) {
} }
} }
// TestDupeGroupsLeavesOutEmptyFiles checks that empty files form no
// duplicate group, while 1-byte files still do.
func TestDupeGroupsLeavesOutEmptyFiles(t *testing.T) {
t.Parallel()
e, x := sha256.Sum256(nil), testHash("x")
recs := []scanRec{
{size: 0, head: e, tail: e, content: e, path: "/a/.gitkeep"},
{size: 0, head: e, tail: e, content: e, path: "/b/.gitkeep"},
{size: 0, head: e, tail: e, content: e, path: "/c/__init__.py"},
{size: 1, head: x, tail: x, content: x, path: "/tiny1"},
{size: 1, head: x, tail: x, content: x, path: "/tiny2"},
}
groups := dupeGroupsOf(t, recs)
if len(groups) != 1 || groups[0].size != 1 ||
!slices.Equal(groups[0].paths, []string{"/tiny1", "/tiny2"}) {
t.Fatalf("groups = %+v, want only /tiny1 /tiny2 of size 1", groups)
}
}
func TestDupeGroupsContentSeparates(t *testing.T) { func TestDupeGroupsContentSeparates(t *testing.T) {
t.Parallel() t.Parallel()
+2 -12
View File
@@ -1,4 +1,4 @@
package main package sfdupes
import ( import (
"cmp" "cmp"
@@ -545,7 +545,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
prog.increment() prog.increment()
old, ok := s.existing[ev.rec.path] old, ok := s.existing[ev.rec.path]
if !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime) { if !ok || old.size != ev.rec.size || !ev.rec.mtime.Equal(old.mtime) {
s.toHash = append(s.toHash, ev.rec) s.toHash = append(s.toHash, ev.rec)
continue continue
@@ -780,16 +780,6 @@ func (s *scanState) updatePhase(ctx context.Context) error {
return applyChanges(ctx, s.db, nil, deletes, prog) return applyChanges(ctx, s.db, nil, deletes, prog)
} }
// mtimeAfter reports whether mtime a is later than mtime b.
// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
func mtimeAfter(a, b time.Time) bool {
if a.Unix() != b.Unix() {
return a.Unix() > b.Unix()
}
return a.Nanosecond() > b.Nanosecond()
}
// underAnyRoot reports whether path is any of the roots or lies under // underAnyRoot reports whether path is any of the roots or lies under
// one of them. // one of them.
func underAnyRoot(path string, roots []string) bool { func underAnyRoot(path string, roots []string) bool {
@@ -1,6 +1,6 @@
//go:build darwin //go:build darwin
package main package sfdupes
import "syscall" import "syscall"
@@ -1,6 +1,6 @@
//go:build !darwin //go:build !darwin
package main package sfdupes
import "syscall" import "syscall"
+48 -6
View File
@@ -1,4 +1,4 @@
package main package sfdupes //nolint:testpackage // drives the unexported syncTree
import ( import (
"bytes" "bytes"
@@ -1185,11 +1185,12 @@ func assertSmokeDupeGroups(t *testing.T, dir string, db *sql.DB) {
t.Helper() t.Helper()
groups := dupeGroups(t, db) groups := dupeGroups(t, db)
if len(groups) != 5 { if len(groups) != 4 {
t.Fatalf("len(groups) = %d, want 5", len(groups)) t.Fatalf("len(groups) = %d, want 4", len(groups))
} }
wantSizes := []int64{3000, 2000, 100, 1, 0} // The empty files empty1 and empty2 form no group.
wantSizes := []int64{3000, 2000, 100, 1}
for i, g := range groups { for i, g := range groups {
if g.size != wantSizes[i] { if g.size != wantSizes[i] {
t.Errorf("groups[%d].size = %d, want %d", t.Errorf("groups[%d].size = %d, want %d",
@@ -1383,6 +1384,47 @@ func TestSyncScanMtimeBump(t *testing.T) {
} }
} }
// TestSyncScanMtimeMovedBack replaces a file with other content of the
// same size and an older mtime, as cp -p or a restore from backup does.
// The next scan must hash the file again.
func TestSyncScanMtimeMovedBack(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
recorded := time.Date(2026, 1, 2, 3, 4, 5, 0, time.UTC)
err := os.Chtimes(a, recorded, recorded)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dir)
replaced := pattern(3, 500)
writeFile(t, dir, "a.bin", replaced)
older := recorded.Add(-time.Hour)
err = os.Chtimes(a, older, older)
if err != nil {
t.Fatal(err)
}
st := syncTree(t, db, dir)
if st != (scanStats{walked: 1, updated: 1}) {
t.Fatalf("rescan stats = %+v, want 1 updated", st)
}
assertWholeFileHashed(t, db, a, replaced)
if r := recordByPath(t, dbRecords(t, db), a); !r.mtime.Equal(older) {
t.Fatalf("mtime = %v, want %v", r.mtime, older)
}
}
// assertWholeFileHashed fails unless the record for path holds the // assertWholeFileHashed fails unless the record for path holds the
// whole-file hash of data as its head, tail, and content. // whole-file hash of data as its head, tail, and content.
func assertWholeFileHashed(t *testing.T, db *sql.DB, path string, func assertWholeFileHashed(t *testing.T, db *sql.DB, path string,
@@ -2367,8 +2409,8 @@ func TestRunScanPassesOverDatabase(t *testing.T) {
report := runStdout(t, cmdReport) report := runStdout(t, cmdReport)
if strings.Contains(report, "db.sqlite") || if strings.Contains(report, "db.sqlite") ||
strings.Contains(report, hardLink) || strings.Contains(report, hardLink) ||
!strings.Contains(report, filepath.Join(tree, "empty2")) { !strings.Contains(report, filepath.Join(tree, "b")) {
t.Errorf("report = %q, want the empty files and none of "+ t.Errorf("report = %q, want a and b and none of "+
"the database's files", report) "the database's files", report)
} }
}) })
+7 -3
View File
@@ -1,4 +1,4 @@
package main package sfdupes
import ( import (
"context" "context"
@@ -301,7 +301,9 @@ func fileEntry(name string, r scanRec) string {
// collectTreeGroups groups directories by digest, counting each one on // collectTreeGroups groups directories by digest, counting each one on
// prog as it is grouped, and returns every maximal group with two or // prog as it is grouped, and returns every maximal group with two or
// more members, each group's members sorted by path, groups ordered by // more members, each group's members sorted by path, groups ordered by
// tree size descending then by first path ascending. // tree size descending then by first path ascending. A group whose
// trees hold 0 bytes is left out, since removing one frees nothing. The
// groups it implies hold 0 bytes too, so leaving it out hides nothing.
func collectTreeGroups(allDirs []*treeNode, super *treeNode, func collectTreeGroups(allDirs []*treeNode, super *treeNode,
prog *progress, prog *progress,
) [][]*treeNode { ) [][]*treeNode {
@@ -315,7 +317,9 @@ func collectTreeGroups(allDirs []*treeNode, super *treeNode,
var dupes [][]*treeNode var dupes [][]*treeNode
for _, g := range groups { for _, g := range groups {
if len(g) < minGroupSize || suppressed(g, super) { // The members share a digest, so they share a total size.
if len(g) < minGroupSize || g[0].totalSize == 0 ||
suppressed(g, super) {
continue continue
} }
@@ -1,7 +1,8 @@
package main package sfdupes //nolint:testpackage // checks the unexported treeNode results
import ( import (
"bytes" "bytes"
"crypto/sha256"
"database/sql" "database/sql"
"os" "os"
"slices" "slices"
@@ -435,6 +436,35 @@ func TestCollectTreeGroupsSiblings(t *testing.T) {
} }
} }
// TestCollectTreeGroupsLeavesOutEmptyTrees checks that directories
// holding only empty files form no group, and that an empty file still
// counts toward the digest of a tree that holds other files.
func TestCollectTreeGroupsLeavesOutEmptyTrees(t *testing.T) {
t.Parallel()
h, tl, c, e := testHash("h"), testHash("t"), testHash("c"),
sha256.Sum256(nil)
recs := []scanRec{
{size: 0, head: e, tail: e, content: e, path: "/e/a/.gitkeep"},
{size: 0, head: e, tail: e, content: e, path: "/e/b/.gitkeep"},
{size: 10, head: h, tail: tl, content: c, path: "/p/x1/f"},
{size: 0, head: e, tail: e, content: e, path: "/p/x1/.gitkeep"},
{size: 10, head: h, tail: tl, content: c, path: "/p/x2/f"},
{size: 0, head: e, tail: e, content: e, path: "/p/x2/.gitkeep"},
// Without the empty file, /p/x3 is no copy of /p/x1.
{size: 10, head: h, tail: tl, content: c, path: "/p/x3/f"},
}
super, dirs := treeOf(t, recs)
got := groupPaths(collectTreeGroups(dirs, super, nil))
want := [][]string{{"/p/x1", "/p/x2"}}
if !slices.EqualFunc(got, want, slices.Equal) {
t.Fatalf("groups = %v, want %v", got, want)
}
}
func TestCollectTreeGroupsDifferingParents(t *testing.T) { func TestCollectTreeGroupsDifferingParents(t *testing.T) {
t.Parallel() t.Parallel()