sneak/sfdupes
1
0
Fork 0
Code Issues 4 Pull Requests 3 Actions 7 Packages Projects Releases Wiki Activity

Hash every file's head, tail and content in the hash phase (closes #132)
check / check (push) Canceled after 0s
Details

Browse Source 
Every file the walk records now gets three hashes: head and tail, of
its first and last 64 KiB, and content. The content hash covers the
whole file under 100 MiB, and 1 MiB at each gigabyte from 100 MiB up,
so a large file is never read in full. One 100 MiB constant replaces
the 10 MiB and 50 MiB thresholds. The size census, the records stored
without hashes and the content phase are gone, so a rescan reads only
new and changed files. report and trees no longer special-case a
missing content hash. README "Duplicate detection" and "Rationale" are
rewritten.

Judgement call: info drops its two hash-count lines, which now always
equal the record count.

Model: opus-5-5
This commit was merged in pull request #136.
This commit is contained in:
clawbot
2026-10-26 02:02:48 +01:00
parent f4ddb58ae1
commit f7a746f912
16 changed files with 537 additions and 1774 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+201 -281
View File
@@ -3,22 +3,22 @@
## Description
`sfdupes` is an MIT-licensed Go CLI tool by [@sneak](https://sneak.berlin) that
quickly identifies _candidate_ duplicate files — and, ultimately, entire
duplicate directory trees — across very large filesystems without reading every
byte of every file. Files are considered duplicates when their sizes are equal
and they agree on a short ladder of hashes. A file under 10 MiB is hashed in
full and compared directly. A larger file is gated first on the SHA-256 of its
first 64 KiB and of its last 64 KiB, and only when its size and both of those
match another file's is it read for a content hash to compare — the SHA-256 of
the whole file when it is under 50 MiB, or of gigabyte-spaced 1 MiB samples when
it is 50 MiB or larger. Below 50 MiB the content hash is proof of identical
content; at or above 50 MiB it is a strong candidate signal rather than proof,
because the gaps between samples are never read. The intended use is finding
duplicate downloads and duplicated directory trees on multi-terabyte ZFS servers
where reading every byte of every file is prohibitively expensive. `scan`
maintains a persistent SQLite database of file signatures that survives between
runs, so it can be run from cron and the reports can be generated at any time
from the most recent scan.
quickly finds _candidate_ duplicate files, and ultimately whole duplicate
directory trees, on very large filesystems. It does this without reading every
byte of every file.
Two files are duplicates when they have the same size and the same three SHA-256
hashes. Two of the hashes cover the first and the last 64 KiB of the file. The
third, the content hash, covers the whole file when it is under 100 MiB, so a
match proves identical content. At 100 MiB and above it covers only 1 MiB at
each gigabyte, so such a file is never read in full. A match there is a strong
candidate rather than proof.
The intended use is finding duplicate downloads and duplicated directory trees
on multi-terabyte ZFS servers, where reading every byte of every file is
prohibitively expensive. `scan` keeps a persistent SQLite database of file
signatures between runs, so it can run from cron. The reports can be generated
at any time from the most recent scan.
This README is the complete and authoritative specification.
@@ -34,13 +34,13 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
`scan` walks one or more filesystem trees and maintains one database record per
regular file (path, size, mtime, head hash, tail hash, content hash). The
database persists between runs; a rescan only hashes files that are new or
changed, or that may have gained a duplicate since the last scan, and removes
records for files that no longer exist. `report` reads the database and prints
the file-level duplicates report. `trees` reads the same database and prints the
duplicate-tree report. `info` prints the database's details: its path, the sizes
of its files and the number of its records. A missing/invalid subcommand — or a
`scan` invocation with no `PATH` operand — prints a usage message and exits 2.
database persists between runs. A rescan hashes only the files that are new or
changed, and removes the records of files that no longer exist. `report` reads
the database and prints the file-level duplicates report. `trees` reads the same
database and prints the duplicate-tree report. `info` prints the database's
details: its path, the sizes of its files and the number of its records. A
missing/invalid subcommand — or a `scan` invocation with no `PATH` operand —
prints a usage message and exits 2.
The database defaults to `/var/lib/sfdupes/db.sqlite` and can be placed anywhere
by setting `SFDUPES_DATABASE`. The intended deployment is a daily `sfdupes scan`
@@ -122,14 +122,19 @@ As a file `/etc/cron.d/sfdupes`:
the first scan. To keep it elsewhere, set
`SFDUPES_DATABASE=/path/to/db.sqlite` before the command on the same line.
- `scan` writes nothing to stdout. Its stderr, appended here to
`/var/log/sfdupes.log`, holds a first line naming the database (see
"Database"), plain progress lines naming each phase and its place among the
five, as `[2/5] walk` (one as the phase starts, one every 5 seconds while it
runs, and one as it ends), a warning for each path it skips, a line for each
default exclude its walk does not enter and for each path it leaves out
because it is not on a persistent filesystem (see "Rules for the walk"), and
the summary line (see "Progress" and "`scan` mode"). The log grows with every
scan; rotate it like any other.
`/var/log/sfdupes.log`, holds these lines:
- a first line naming the database (see "Database");
- plain progress lines naming each phase and its place among the four, as
`[2/4] walk`: one as the phase starts, one every 5 seconds while it runs,
and one as it ends;
- a warning for each path it skips;
- a line for each default exclude its walk does not enter, and for each path
it leaves out because it is not on a persistent filesystem (see "Rules for
the walk");
- the summary line (see "Progress" and "`scan` mode").
The log grows with every scan; rotate it like any other.
- Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
reason among the last lines of the log; `tail` prints them, and cron mails
@@ -153,7 +158,7 @@ A row is a candidate, not proof:
- The reports read only the database, so they show the files as of the last
scan; a file may have changed or gone since.
- A file of 50 MiB or more is compared only on samples of its content (see
- A file of 100 MiB or more is compared only on samples of its content (see
"Duplicate detection").
- Paths that are hard links to one file are listed as duplicates, but they share
their data, so removing one frees nothing.
@@ -188,22 +193,23 @@ or a symlink, which `trees` does not see.
## Rationale
Duplicate finders that hash entire files do not scale to the target environment:
~10 million files and ~150 TB on possibly slow or busy disks (a ZFS pool under
resilver). sfdupes spends disk I/O only on files whose size at least one other
file shares, since a size-unique file cannot be a duplicate. Of those, a file
under 10 MiB is read in full; a larger one has its cheap end windows read first,
and is read for a content hash only when its size and both end windows match
another file's — the whole file below 50 MiB, but only gigabyte-spaced samples
at or above 50 MiB, so the largest files are never read in full. This keeps a
full-filesystem sweep tractable, and the signatures are kept in a persistent
database, so the expensive filesystem pass is incremental: a rescan re-hashes
only files whose recorded mtime or size changed, plus — for its content hash — a
file of 10 MiB or more whose size and end windows have come to match another
file's. All analysis happens offline from the database alone. The end goal is
not individual files but whole duplicated trees — duplicate extractions,
duplicate downloads, copied project trees — which an operator can consider
removing as a unit.
Duplicate finders that hash every file in full do not scale to the target
environment. It holds about 10 million files and 150 TB, on disks that may be
slow or busy, such as a ZFS pool under resilver.
sfdupes reads a file under 100 MiB in full, and a larger one only at its two
ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it
makes the comparison exact. A full read of every large file would take
unreasonably long. Sampling cuts a 150 GiB file to about 150 MiB of reads, at
the price of a comparison that is not exact.
The hashes are kept in a persistent database, so the expensive filesystem pass
is incremental. A rescan reads only the files that are new or whose size or
mtime changed. All analysis then works from the database alone.
The end goal is not individual files but whole duplicated trees, such as a
download extracted twice. An operator can consider removing such a tree as a
unit.
## Design
@@ -215,14 +221,10 @@ Goals, in order:
trees), so the operator can consider removing an entire subtree at once.
File-level duplicate detection is the foundation; tree-level detection is
built on top of it.
2. **Spend I/O in proportion to duplicate likelihood.** Only files whose size
at least one other file shares are read at all — a size-unique file cannot
be a duplicate. Those are compared by the ladder in "Duplicate detection"
below: a file under 10 MiB is hashed in full, while a larger file is gated
on cheap 64 KiB end windows first, and gets a content hash only when its
size and both end windows match another file's. That hash reads the whole
file below 50 MiB but only gigabyte-spaced 1 MiB samples at or above it, so
the very largest files are still never read in full. Scale target: tens of
2. **Bound the I/O per file.** A file is read only when it is new or has
changed. A file under 100 MiB is then read in full, and a larger one only at
its ends and at 1 MiB per gigabyte (see "Duplicate detection" below).
Therefore no file of 100 MiB or more is read in full. Scale target: tens of
millions of files, ~150 TB filesystem, possibly slow or busy disks (ZFS pool
under resilver). Holding one small record (path, size, mtime) per file in
memory during a scan is acceptable; holding every file's hashes is not (they
@@ -235,12 +237,10 @@ Goals, in order:
being read, so its memory grows with the number of directories and with the
size of the largest directory.
3. **Scan incrementally, analyze offline.** The expensive filesystem scan
maintains a persistent database; an unchanged file is never read again on a
rescan, except to compute its content hash once a file of 10 MiB or more
comes to match another on size and both end windows. All analysis (`report`,
`trees`) works from the database alone and must never touch the scanned
filesystem again. `scan` is designed to be cronned; the reports run at any
time against the last completed scan.
maintains a persistent database, and an unchanged file is never read again
on a rescan. All analysis (`report`, `trees`) works from the database alone
and must never touch the scanned filesystem again. `scan` is designed to be
cronned; the reports run at any time against the last completed scan.
4. **Clean stream separation.** Everything on stdout is machine-readable data.
All progress, warnings, summaries, and help and usage text go to stderr.
Never mix them.
@@ -272,7 +272,7 @@ Four subcommands, all implemented:
from the database records, compute a Merkle-style digest per directory, and
report maximal groups of identical trees.
4. `info` — the database's details: its path, the sizes of its files, its
schema version and the counts of its records.
schema version, the number of its records and their total size.
```
sfdupes scan [--workers N] [-x] [--no-default-excludes] PATH...
@@ -368,9 +368,7 @@ All four subcommands operate on a single SQLite database file:
- While it loads the existing records (`load`) and walks: no records. A
first scan's database holds none, however long its walk has run.
- During the hash phase: the first records, committed 10,000 at a time.
- In the update phase: the rest of the hashed records, the records for
size-unique files, and the deletions.
- In the content phase: content hashes, again committed 10,000 at a time.
- In the update phase: the rest of the hashed records, and the deletions.
- At the end, as it closes the database: the `-wal` and `-shm` files go
away, unless a report still has the database open (see above).
- `report`, `trees` and `info` open the database read-only and need only read
@@ -390,9 +388,9 @@ All four subcommands operate on a single SQLite database file:
size INTEGER NOT NULL, -- bytes, from lstat
mtime INTEGER NOT NULL, -- whole Unix seconds of the mtime, from lstat
mtime_nsec INTEGER NOT NULL, -- nanoseconds within that second, 0 to 999999999
head TEXT NOT NULL, -- lowercase-hex SHA-256; first 64 KiB, or whole file under 10 MiB
tail TEXT NOT NULL, -- lowercase-hex SHA-256; last 64 KiB, or whole file under 10 MiB
content TEXT NOT NULL -- lowercase-hex SHA-256, whole file or samples
head TEXT NOT NULL, -- lowercase-hex SHA-256 of the first 64 KiB
tail TEXT NOT NULL, -- lowercase-hex SHA-256 of the last 64 KiB
content TEXT NOT NULL -- lowercase-hex SHA-256; whole file under 100 MiB, else samples
) WITHOUT ROWID;
CREATE INDEX files_signature ON files (size, head, tail, content);
```
@@ -403,57 +401,46 @@ All four subcommands operate on a single SQLite database file:
nanoseconds past that second. Split this way they hold any mtime a
filesystem can record, one before 1678 or after 2262 included, which a
single 64-bit count of nanoseconds cannot. They are used only for change
detection and are not part of the duplicate key. For a file under 10 MiB
`head`, `tail`, and `content` all hold the whole-file hash (that range is
hashed in full, with no end windows); for a larger file `head` and `tail`
hold the first- and last-64 KiB hashes and `content` the whole-file or
sampled hash. All three are empty strings when the file has never been
hashed because its size was unique as of the last scan that covered it. For
a file of 10 MiB or more, `content` stays empty until the content phase of a
scan (see "`scan` mode" below) has read the file. A record with an empty
`content` is never part of a duplicate group, though it still defines the
file for tree reconstruction. The `files_signature` index lets `report` read
the records in signature order without sorting the table.
detection and are not part of the duplicate key. Every record holds the
three hashes that "Duplicate detection" below describes. The
`files_signature` index lets `report` read the records in signature order
without sorting the table.
### Duplicate detection
Two files are duplicates only when they agree on every rung of this ladder; a
mismatch at any rung means they are not duplicates. `scan` stores each file's
hashes, and `report` and `trees` group files by the whole signature — size,
`head`, `tail`, and `content` — so the grouping is exactly this ladder applied
across everything scanned into the database, even across separate scans.
Two files are duplicates when they have the same size and the same three hashes.
`scan` computes the hashes and stores them in the database. `report` and `trees`
group the files whose size and hashes all match, whichever scan recorded them.
1. **Size.** Files of different sizes are never compared. Only files whose size
at least one other file shares are hashed at all.
2. **Under 10 MiB: whole file.** A file smaller than 10 MiB is hashed in full
and compared directly, with no separate end-window step — small files are
cheap to read to the last byte, and doing so makes the comparison exact.
`head`, `tail`, and `content` all hold this whole-file SHA-256, so such a
file's signature is decided entirely by its size and its content.
3. **10 MiB and above: head and tail.** For a larger file, the SHA-256 of the
first 64 KiB (`head`) and of the last 64 KiB (`tail`) are a cheap gate that
eliminates most same-size pairs before any bulk reading: the content hash of
the next two rungs is computed only for a file whose size, `head`, and
`tail` match another file's, whether that file is scanned in the same run or
stored by an earlier scan. A stored file that first gains such a match in a
later scan gets its content hash then; until it has one, its `content` is
empty and it is not a duplicate. At 10 MiB and above the two windows never
overlap.
4. **10 MiB and above, content below 50 MiB.** The SHA-256 of the entire file.
Agreement here is proof of identical content (barring a SHA-256 collision).
5. **10 MiB and above, content 50 MiB and above.** A sampled SHA-256: the 1 MiB
window at each gigabyte-aligned offset (0, 1 GiB, 2 GiB, … while inside the
file, the final window truncated at end of file) is fed, in order, into one
hash. This is **deliberately probabilistic** — the gaps between samples are
never read, so two large files that agree on every sample are reported as
duplicates without being read in full. It is the price of never reading a
150 GB file end to end. Because size is already part of the signature, only
equal-size files reach this rung, so their sample boundaries always align.
`scan` gives every file it records three SHA-256 hashes:
`head`, `tail`, and `content` are one column each. A file below 10 MiB and one
at or above it never share a size, and neither do a file below 50 MiB and one at
or above it, so a stored value is never ambiguous between the whole-file,
end-window, and sampled forms.
- `head`, the hash of the file's first 64 KiB.
- `tail`, the hash of the file's last 64 KiB.
- `content`, the content hash described below.
In a file of 64 KiB or less, the first and the last 64 KiB are both the whole
file, so all three hashes are the hash of the whole file.
The content hash depends on the file's size:
- Under 100 MiB, it is the SHA-256 of the whole file. Agreement on it is proof
of identical content, barring a SHA-256 collision.
- At 100 MiB and above, it is the SHA-256 of samples, so the file is never read
in full. A sample is the 1 MiB at each gigabyte-aligned offset: 0, 1 GiB, 2
GiB and so on. The samples are fed into one hash in order. A sample that would
run past the end of the file stops there.
The sampled hash is **deliberately probabilistic**. Bytes outside the samples
and outside the first and last 64 KiB are never read. Two large files that
differ only in such bytes are reported as duplicates. That is the price of never
reading a 150 GB file end to end.
Only files of the same size are compared, so their samples always lie at the
same offsets. A file under 100 MiB and one of 100 MiB or more never share a
size, so a whole-file hash is never compared with a sampled one.
`scan` reads a file only to hash it, and only when the file is new or has
changed since its record was written (see "`scan` mode").
### `scan` mode
@@ -479,35 +466,22 @@ FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
a one-line warning naming the path and what it is, counts it as skipped, and
drops it from the scanned operands before reading the database. Another operand
beneath it is still scanned. The records stored beneath it are not deleted: they
are treated like any other record outside the scanned operands, including the
content-phase exception below. If it lies under another operand, they are under
that operand instead, and are treated like any other record there. This is not
an error: a scan whose every operand is dropped walks nothing and exits 0. An
operand that is not on a persistent filesystem (see "Rules for the walk") is
dropped the same way, except that the line `scan` prints for it is not a
warning, and it is not counted as skipped.
are treated like any other record outside the scanned operands. If it lies under
another operand, they are under that operand instead, and are treated like any
other record there. This is not an error: a scan whose every operand is dropped
walks nothing and exits 0. An operand that is not on a persistent filesystem
(see "Rules for the walk") is dropped the same way. The line `scan` prints for
it is not a warning, though, and it is not counted as skipped.
`scan` synchronizes the database with the filesystem state under the scanned
operands:
- Only a file whose size at least one other file shares is ever read: a
size-unique file cannot be a duplicate, so it is recorded without hashes
(`head`, `tail`, and `content` empty). The size census covers every file
walked this scan plus every database record outside the scanned operands, so a
possible duplicate of a separately scanned tree is still recognized.
- A file not yet in the database is inserted: hashed when its size is shared,
without hashes otherwise.
- A file not yet in the database is hashed and inserted.
- A file already in the database is **skipped without reading its contents**
when its lstat size equals the recorded size and its lstat mtime is not newer
than the recorded mtime. This is what makes a daily rescan cheap. Exception:
an unchanged file whose record lacks hashes is hashed — and its record updated
— once its size becomes shared, so hashing deferred by size-uniqueness happens
as soon as it could matter. Likewise, an unchanged file of 10 MiB or more
whose record has no `content` hash is read for one by the content phase below
once its size, `head`, and `tail` match another record's.
- A file whose mtime is newer than recorded, or whose size differs, is processed
as if new: re-hashed, or recorded without hashes, per the shared-size rule.
Change detection compares the mtime to the nanosecond, as finely as the
than the recorded mtime. This is what makes a daily rescan cheap.
- A file whose mtime is newer than recorded, or whose size differs, is hashed
again. Change detection compares the mtime to the nanosecond, as finely as the
filesystem records it, so a same-size rewrite counts as a change whenever the
filesystem gives it a later mtime than recorded, even within the same second.
- A database record whose path lies under one of the scanned operands but was
@@ -515,31 +489,26 @@ operands:
deleted files. It also removes records for paths that failed to stat or hash
this run: the database only ever contains signatures verified by the most
recent scan that covered them (a subsequent successful scan re-adds such
files). A failure in the content phase below removes nothing: the record is
left as it is.
files).
- Database records outside the scanned operands are untouched, so disjoint trees
can be scanned on different schedules into the same database. The one
exception is the content phase below: a stored file of 10 MiB or more without
a `content` hash is read for one, wherever it lies, once its size, `head`, and
`tail` match another record's. If that file is gone, has changed since its
record was written, or is not on a persistent filesystem, it is not read and
the record is left as it is.
- A record at or below a path the walk left out, because that path is a default
exclude, is not on a persistent filesystem, or, with `-x`, is a directory on
another filesystem than its operand, counts as outside the scanned operands:
it is not deleted, and its size joins the size census. A filesystem mounted
under an operand can therefore be scanned on its own schedule into the same
database. Records below an operand at or inside such a path are not kept this
way, since that operand's own walk verifies them.
can be scanned on different schedules into the same database.
- A record at or below a path the walk left out counts as outside the scanned
operands, so it is not deleted. The walk leaves out a path that:
- is a default exclude;
- is not on a persistent filesystem;
- with `-x`, is a directory on another filesystem than its operand.
`scan` runs **five sequential phases over the whole scan**. Parallelism lives
inside the walk, hash and content phases; batched database writes begin during
the hash phase:
A filesystem mounted under an operand can therefore be scanned on its own
schedule into the same database. Records below an operand at or inside such
a path are not kept this way, since that operand's own walk verifies them.
`scan` runs **four sequential phases over the whole scan**. Parallelism lives
inside the walk and hash phases, and batched database writes begin during the
hash phase:
1. **load** — read the existing records. Each record under the operands is kept
in memory, by path, with its size, its mtime and whether it has hashes, for
the walk to compare each file with; the size of every other record joins the
size census.
in memory, by path, with its size and its mtime, for the walk to compare
each file with.
2. **walk + stat** — enumerate the trees under all `PATH` operands concurrently
with the walk worker pool: every operand seeds the shared queue, and each
worker reads one directory at a time, handing discovered subdirectories back
@@ -547,49 +516,23 @@ the hash phase:
(while the directory's metadata is still hot). Sequential directory
enumeration is metadata-latency-bound and takes hours at tens of millions of
files; per-directory parallelism is what makes the walk tractable on large
or busy pools. The walk builds the size census and resolves unchanged
already-hashed files on the fly; every other file is carried to the hash
phase as a (path, size, mtime) record.
3. **hash** — with the census complete, each carried file's size decides its
fate. Size-unique files are never read: new or changed ones are recorded
without hashes in the update phase, unchanged unhashed ones simply keep
their records. Every file with a shared size is hashed by the worker pool as
described in "Duplicate detection" above: a file under 10 MiB in full, which
gives its `head`, `tail`, and `content` alike, and a larger file only in its
end windows, which give its `head` and `tail`; its content hash is left to
the content phase. Zero-length files have constant hashes and are never
opened. Files are hashed in **inode order** (minimizing seeks on spinning
disks), and paths that are hard links to the same inode are **read once**,
all sharing the one result — a hard-link backup farm costs one read per
inode, not per path. The phase total counts actual reads, so progress and
ETA are meaningful. Completed records are committed in batched transactions
**while hashing runs**, so a scan interrupted after hours keeps everything
hashed so far and the next scan resumes cheaply, skipping records already
written.
4. **update** — commit the final partial batch, the hash-less records for
size-unique new and changed files, and the deletions for records the scan
did not verify (vanished files, plus paths that failed to stat or hash).
5. **content** — find every record of 10 MiB or more without a `content` hash
whose size, `head`, and `tail` equal another record's, anywhere in the
database: records from this scan and records stored by earlier scans, inside
or outside the scanned operands. SQLite finds them, so only the records to
be read are kept in memory, never every file's hashes. Every record sharing
their size, `head`, and `tail`, including one that already has a `content`
hash, has its file checked with `lstat` first. A file that is gone, is no
longer a regular file, has changed (a different size, or an mtime newer than
recorded), or is not on a persistent filesystem (see "Rules for the walk")
keeps its record as it is and does not count as a match for the others. Any
other `lstat` error is warned about and counted as skipped, with the same
result. If such a record has no `content` hash, it stays out of duplicate
groups; if it has one, it is still reported until a scan covering its own
tree updates or removes it. The files that pass and have no `content` hash
are read only if at least two of those records pass, so a file whose only
matches are stale costs no read; a file that already has a `content` hash is
never read again. They are read by a worker pool as in the hash phase, in
inode order and once per inode, and their content hashes are committed in
batches. A failed read is warned about and counted as skipped; its record
keeps an empty `content`, so it is not a duplicate, and a later scan tries
again.
or busy pools. The walk compares each file with its record as it goes, and
an unchanged file needs nothing more. Every new or changed file is carried
to the hash phase as a (path, size, mtime) record.
3. **hash** — the worker pool hashes every carried file as described in
"Duplicate detection" above, which gives the file its `head`, `tail` and
`content`. Zero-length files have constant hashes and are never opened.
Files are hashed in **inode order**, which minimizes seeks on spinning
disks. Paths that are hard links to the same inode are **read once**, and
all share the one result. A hard-link backup farm therefore costs one read
per inode, not per path. The phase total counts actual reads, so progress
and ETA are meaningful. Completed records are committed in batched
transactions **while hashing runs**. A scan interrupted after hours
therefore keeps everything hashed so far, and the next scan resumes cheaply,
skipping records already written.
4. **update** — commit the final partial batch of hashed records. Then delete
the records the scan did not verify: those of vanished files, and of paths
that failed to stat or hash.
Rules for the walk:
@@ -612,10 +555,10 @@ Rules for the walk:
- Filesystem boundaries are crossed by default. With `-x` (long form
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend
into a directory on a different filesystem than its `PATH` operand; each
operand is bounded by its own filesystem. The records below such a directory,
unless they lie below an operand at or inside that directory, count as outside
the scanned operands, as described in "`scan` mode" above: they are not
deleted, and their sizes join the size census.
operand is bounded by its own filesystem. The records below such a directory
count as outside the scanned operands, as described in "`scan` mode" above:
they are not deleted. That does not hold below an operand at or inside that
directory.
- Never record anything that is not on a persistent filesystem. A filesystem is
not persistent when the kernel makes it up or keeps it in memory: one of type
`proc`, `sysfs`, `devtmpfs`, `devpts`, `tmpfs`, `ramfs`, `cgroup`, `cgroup2`,
@@ -632,21 +575,19 @@ Rules for the walk:
`scan` cannot list them, it fails (exit 1). The walk then never enters a
directory, and never records a file, on one of those devices, however it
reaches it: as an operand, below one, or through a bind mount, which keeps the
device ID of what it binds. The content phase never reads a file on one of
them either. No flag turns this off. For each operand, directory or file it
leaves out this way, `scan` prints a line on stderr naming it and the type of
its filesystem, as it prints a warning, so it never lands inside a progress
display:
device ID of what it binds. No flag turns this off. For each operand,
directory or file it leaves out this way, `scan` prints a line on stderr
naming it and the type of its filesystem. It prints that line as it prints a
warning, so the line never lands inside a progress display:
```
scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
```
That line is not a warning, and the path is not counted as skipped. Records
that an earlier scan left at or below it, unless they lie below an operand
at or inside it, count as outside the scanned operands, as described in
"`scan` mode" above: they are not deleted, and their sizes join the size
census.
That line is not a warning, and the path is not counted as skipped. An
earlier scan's records at or below it count as outside the scanned operands,
as described in "`scan` mode" above: they are not deleted. That does not
hold below an operand at or inside it.
- Below an operand, never descend into a default exclude: a path that holds the
kernel's view of the running system, or temporary files, so that
@@ -678,32 +619,30 @@ Rules for the walk:
That line is not a warning, and the directory is not counted as skipped. A
default exclude that is not on a persistent filesystem, as `/proc` is, gets
the line for its filesystem instead, and one that is itself an operand gets
no line, since it is walked as that operand. The records below a default
exclude the walk does not enter, unless they lie below an operand at or
inside it, count as outside the scanned operands, as described in "`scan`
mode" above: they are not deleted, and their sizes join the size census.
the line for its filesystem instead. One that is itself an operand gets no
line, since it is walked as that operand. When the walk does not enter a
default exclude, the records below it count as outside the scanned operands,
as described in "`scan` mode" above: they are not deleted. That does not
hold below an operand at or inside it.
- On any per-path error (permission denied, file vanished between passes,
unreadable): print a one-line warning to stderr, skip the path, and continue.
Per-file errors never abort the run; the final summary reports how many were
skipped. As specified above, a skipped path that has a database record from an
earlier scan loses that record, unless it failed only in the content phase, or
is an operand dropped before the database was read whose record counts as
outside the scanned operands; an unreadable directory subtree likewise loses
its records (accepted: the database mirrors what the latest scan could
actually verify).
earlier scan loses that record. An unreadable directory subtree likewise loses
its records. This is accepted: the database mirrors what the latest scan could
actually verify. The exception is an operand dropped before the database was
read, whose record counts as outside the scanned operands.
Concurrency: the walk phase (which also stats files), the hash phase, and the
content phase each use a worker pool of `--workers` workers (default
`runtime.NumCPU()`); the walk parallelizes across directories, hashing across
files. `--workers` must be at least 1: a smaller value is a usage error,
reported in one line on stderr with exit 2 before anything is scanned. All three
phases are seek-bound on spinning disks, so raising `--workers` well past the
core count can help on pools with many spindles. The main goroutine owns
partitioning, database writes, and the progress counts, which each progress
display draws from a goroutine of its own; progress display must never block the
workers.
Concurrency: the walk phase, which also stats files, and the hash phase each use
a worker pool of `--workers` workers (default `runtime.NumCPU()`). The walk
parallelizes across directories, and hashing across files. `--workers` must be
at least 1: a smaller value is a usage error, reported in one line on stderr
with exit 2 before anything is scanned. Both phases are seek-bound on spinning
disks, so raising `--workers` well past the core count can help on pools with
many spindles. The main goroutine owns the database writes and the progress
counts. Each progress display draws the counts from a goroutine of its own, and
must never block the workers.
`scan` writes nothing to stdout. The summary line on stderr reports the files
seen this run broken down by disposition, plus skips:
@@ -726,9 +665,7 @@ output whether or not the scanned filesystem is still mounted.
Processing:
- Records without a `content` hash (see "Database" above) are excluded: their
content is unknown, so they are never reported as duplicates.
- Group the remaining records by the key `(size, head, tail, content)`.
- Group the records by the key `(size, head, tail, content)`.
- Every group with two or more paths is a duplicate group.
- Within each group, sort paths lexicographically (byte order). The first path
is the group's `first`; every other path is a `dupe`.
@@ -782,9 +719,7 @@ records, split on `/`.
Definitions:
- A file's **signature** is `(size, head, tail, content)` — mtime is
informational and excluded. A record without a `content` hash has unknown
content: its signature is treated as unique to that file, so a tree containing
such a file never compares equal to any other tree.
informational and excluded.
- A directory's **digest** is a SHA-256 Merkle digest computed bottom-up:
serialize the directory's child entries — for a file child, its name and
signature; for a subdirectory child, its name and that subdirectory's digest —
@@ -858,8 +793,6 @@ database file size 2147483648 (2.0 GiB)
modified 2026-10-16T12:02:50Z
schema version 1
records 12345678
records with head and tail hashes 1234567
records with content hash 123456
size of recorded files 150000000000000 (136.4 TiB)
```
@@ -872,10 +805,8 @@ size of recorded files 150000000000000 (136.4 TiB)
database is in WAL mode, the newest changes may be in the `-wal` file alone.
- `schema version` is the database's `PRAGMA user_version`. A database with
another version than this build's is refused, as by `report`.
- `records` is the number of records. `records with head and tail hashes` and
`records with content hash` count the records that have those hashes (see
"Database"), and `size of recorded files` adds up the sizes of all records, in
bytes, then in human units.
- `records` is the number of records. `size of recorded files` adds up the sizes
of all records, in bytes, then in human units.
`info` runs in two phases, each with its own progress display on stderr (see
"Progress"): `open` opens the database; `count` reads the schema version and
@@ -895,8 +826,8 @@ not hung. Each phase gets its own display from the moment it starts, naming the
phase and its place among the command's phases and showing its progress, its
elapsed time and its estimated time remaining (ETA).
A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
`[4/5] update`, `[5/5] content`):
A scan has four phases (`[1/4] load`, `[2/4] walk`, `[3/4] hash`,
`[4/4] update`):
- `load` counts the records first, then reads them against that count. While the
count runs, the display says it is counting the records.
@@ -906,13 +837,6 @@ A scan has five phases (`[1/5] load`, `[2/5] walk`, `[3/5] hash`,
scan, the ETA is unknown.
- `hash` and `update` have exact totals. Only files that actually need hashing
appear in the hash total, so its ETA is meaningful.
- `content` first counts the records whose files it checks with `lstat`, which
SQLite does by adding up the number of records with each matching size,
`head`, and `tail`, without fetching the records. It then says it is searching
the database until SQLite returns the first record; the checks, and their rate
and ETA, start there. The files to read then get a display of their own, with
their count as its total, and so does the writing of the last of their records
to the database, with the count of those records as its total.
`trees` has four phases (`[1/4] open`, `[2/4] read`, `[3/4] group`,
`[4/4] write`, see "`trees` mode"):
@@ -958,23 +882,21 @@ Each display is one line holding:
Example shape (exact layout is flexible, content is not):
```
[3/5] hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
[2/5] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown
[3/5] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s
[3/4] hash: [12345/98765] 12% |████ | 92 files/s elapsed 2:32 eta 17:54
[2/4] walk: [1234567/~1200000] 103%, 5120 files/s, elapsed 4m1s, eta unknown
[3/4] hash: [12345/45678] 27%, stalled disk/db, elapsed 4m30s, eta 12m9s
```
When no item has completed for 10 seconds, as while one large file is read, a
disk is slow or the database is busy, the line says `stalled` and where the
phase waits in place of the rate, which would read 0: `disk` for files, `db` for
the database, `out` for stdout, and two of them, as `disk/db`, for a phase that
may wait on either. `walk` says `disk`. `hash` and the content phase's reads say
`disk/db`, as they read files and write their records to the database between
them, and so do the content phase's checks, which `lstat` each record's file as
the database search returns it. `load`, `update`, the content phase's last write
and the `read` phase of `trees` say `db`. `trees`' `write` says `out`, and its
`group`, which works in memory, says `cpu`. The `read` phase of `report` writes
the report while it reads the records, so it says `db/out`. A phase still
waiting for its total already says what it waits for, and says nothing more.
may wait on either. `walk` says `disk`. `hash` says `disk/db`, as it reads files
and writes their records to the database between them. `load`, `update` and the
`read` phase of `trees` say `db`. `trees`' `write` says `out`, and its `group`,
which works in memory, says `cpu`. The `read` phase of `report` writes the
report while it reads the records, so it says `db/out`. A phase still waiting
for its total already says what it waits for, and says nothing more.
Additional requirements:
@@ -990,17 +912,15 @@ Additional requirements:
terminal is. A line too wide for the terminal first loses its bar, which gives
way to the spinner, then its rate, the words of a stalled phase, or what the
phase is waiting for. The phase and its place, the progress, the elapsed time
and the ETA are kept whole. On an 80-column terminal, at counts like those of
the stalled line in the example above, a stalled line keeps its words whenever
the elapsed time and the ETA are both in minutes, except in the content
phase's checks and reads. Their lines are the widest, and at most times in
minutes they lose the words. Where the words and the ETA do not both fit, as
at millions of files and times in hours, the words give way. Only when those
four alone are too wide, as on a terminal narrower than about 70 columns hours
into a scan of millions of files, is the line cut short at the terminal's
edge, so the end of the ETA is lost first. A terminal made narrower while a
phase runs gets the same: the next draw erases the line within the new width
and fits the line to it, and the display goes on redrawing in place.
and the ETA are kept whole. On an 80-column terminal, at counts like those in
the example above, a stalled line keeps its words whenever the elapsed time
and the ETA are both in minutes. Where the words and the ETA do not both fit,
as at millions of files and times in hours, the words give way. Only when
those four alone are too wide, as on a terminal narrower than about 70 columns
hours into a scan of millions of files, is the line cut short at the
terminal's edge, so the end of the ETA is lost first. A terminal made narrower
while a phase runs gets the same: the next draw erases the line within the new
width and fits the line to it, and the display goes on redrawing in place.
- When stderr is not a terminal (a pipe, a file, `/dev/null`), do not emit ANSI
redraws: print the line the moment each phase starts, every 5 seconds while it
runs (whether or not an item has completed, and never more often), and once
@@ -1232,8 +1152,8 @@ Tracked in [TODO.md](TODO.md).
## Non-goals
- No byte-for-byte compare, and no deletion or linking of duplicates. Files that
match are compared by a SHA-256 of the whole file below 50 MiB, and only by
samples at 50 MiB and over. The reports are advisory; acting on them is the
match are compared by a SHA-256 of the whole file below 100 MiB, and only by
samples at 100 MiB and over. The reports are advisory; acting on them is the
user's job.
- No persistence beyond the SQLite database described above; no export/import
formats.