Hash every file's head, tail and content in the hash phase (closes #132)
check / check (push) Canceled after 0s
check / check (push) Canceled after 0s
Every file the walk records now gets three hashes in the hash phase: head and tail, of its first and last 64 KiB, and content. The content hash covers the whole file under 100 MiB, and 1 MiB at each gigabyte from 100 MiB up, so a large file is never read in full. One 100 MiB constant replaces the 10 MiB and 50 MiB thresholds. The size census, the records stored without hashes and the content phase are gone, so a scan has four phases and a rescan reads only new and changed files. report and trees no longer special-case a missing content hash. README "Duplicate detection" and "Rationale" are rewritten. Judgement call: info drops its two hash-count lines, which would now always equal the record count. Model: opus-5-5
This commit is contained in:
@@ -1,14 +1,12 @@
|
||||
// Command sfdupes quickly identifies candidate duplicate files across
|
||||
// very large filesystems without reading every byte of every file.
|
||||
// Files are considered duplicates when their sizes are equal and they
|
||||
// agree on a short ladder of SHA-256 hashes. A file under 10 MiB is
|
||||
// hashed in full. A larger file is compared on the hashes of its first
|
||||
// and last 64 KiB, and only when those match another file's is its
|
||||
// content hash computed and compared: of the whole file when it is
|
||||
// under 50 MiB, or of gigabyte-spaced 1 MiB samples when it is 50 MiB
|
||||
// or larger. scan maintains a persistent SQLite database of file
|
||||
// signatures (SFDUPES_DATABASE, default /var/lib/sfdupes/db.sqlite)
|
||||
// that the other subcommands read.
|
||||
// agree on three SHA-256 hashes: of the first 64 KiB, of the last 64
|
||||
// KiB, and of the content. The content hash covers the whole file under
|
||||
// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large
|
||||
// file is never read in full. scan maintains a persistent SQLite
|
||||
// database of file signatures (SFDUPES_DATABASE, default
|
||||
// /var/lib/sfdupes/db.sqlite) that the other subcommands read.
|
||||
//
|
||||
// Usage:
|
||||
//
|
||||
@@ -238,7 +236,7 @@ func newScanCommand() *cobra.Command {
|
||||
}),
|
||||
}
|
||||
scanCmd.Flags().IntVar(&scanWorkers, "workers", runtime.NumCPU(),
|
||||
"concurrent workers for the walk, hash, and content phases")
|
||||
"concurrent workers for the walk and hash phases")
|
||||
scanCmd.Flags().BoolVarP(&scanOneFS, "one-file-system", "x", false,
|
||||
"do not cross filesystem boundaries")
|
||||
scanCmd.Flags().BoolVar(&scanNoDefaultExcludes, "no-default-excludes",
|
||||
|
||||
Reference in New Issue
Block a user