Author SHA1 Message Date
sneak a2fdb0b55d Rewrite README.md sentence by sentence for a newcomer
check / check (push) Canceled after 0s
The old README was one dense specification: long sentences, lists
chained with commas, exceptions ahead of the rules, and release
history in the user text. This rewrite leads with what the program is
and how to use it, then gives a reference for the settings, each
subcommand, the database, the output formats, the progress display
and the exit codes. Non-goals come before the reference; the
constraints, the entrypoints and the build come last. Every statement
was checked against the code on next and against the built binary on
small trees. The section names the code comments cite are kept. Five
comments in main_test.go cite Settings, and one each in scan_test.go,
trees_test.go and exclude_test.go cites what the README now says.
TODO.md records the step.

Model: fable-5-1
2026-10-26 16:15:49 +00:00
2 changed files with 140 additions and 164 deletions
+132 -157
View File
@@ -38,11 +38,9 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
`scan` walks the trees named by its operands, the paths given after `scan` on `scan` walks the trees named by its operands, the paths given after `scan` on
the command line. The walk is the listing of every directory under them. `scan` the command line. The walk is the listing of every directory under them. `scan`
keeps one database record per regular file. A regular file is an ordinary file, keeps one database record per regular file. A regular file is an ordinary file,
as opposed to a directory, a symlink or a device node. A symlink is a file that as opposed to a directory, a symlink or a device. `report` prints the file-level
holds the path of another file and stands in for it. A device node is a file duplicates. `trees` prints the duplicate directory trees. `info` prints the
that stands for a piece of hardware, such as a disk or a terminal. `report` database's path, size and record count.
prints the file-level duplicates. `trees` prints the duplicate directory trees.
`info` prints the database's path, size and record count.
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
another file. The first `scan` creates it. another file. The first `scan` creates it.
@@ -69,12 +67,11 @@ The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set.
`sfdupes --version` prints `sfdupes VERSION` and exits. For a binary from `sfdupes --version` prints `sfdupes VERSION` and exits. For a binary from
`go install`, `VERSION` is the version Go recorded in it. For `@latest` that is `go install`, `VERSION` is the version Go recorded in it. For `@latest` that is
a release tag such as `v0.0.3`. For `@main` it is the commit's tag when it has a release tag such as `v0.0.3`. For `@main` it is a pseudo-version, Go's version
one. Otherwise it is a pseudo-version, Go's version string for an untagged string for an untagged commit. That string ends in the commit's hash. A binary
commit. That string ends in the commit's hash. A binary built from a clone built from a clone carries the output of `git describe --tags --always --dirty`
carries the output of `git describe --tags --always --dirty` in that clone, such in that clone, such as `v0.0.3` on a tagged commit. `VERSION` is `dev` when no
as `v0.0.3` on a tagged commit. `VERSION` is `dev` when no version is known. To version is known. To see everything Go recorded in an installed binary:
see everything Go recorded in an installed binary:
```sh ```sh
go version -m "$(command -v sfdupes)" go version -m "$(command -v sfdupes)"
@@ -108,10 +105,9 @@ docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
### Daily scan from `cron` ### Daily scan from `cron`
Run `scan` as root, so that it can read every file. `scan` warns about a Run `scan` as root, so that it can read every file. A path that it cannot read
directory it cannot read, and about a new or changed file it cannot read. It is skipped with a warning and loses its database record. As the file
skips the path and deletes the records at or below it (see "Rules for the `/etc/cron.d/sfdupes`:
walk"). As the file `/etc/cron.d/sfdupes`:
``` ```
30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log 30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log
@@ -127,9 +123,8 @@ walk"). As the file `/etc/cron.d/sfdupes`:
- a warning for each path it skips; - a warning for each path it skips;
- a line for each system or temporary directory the walk leaves out (see - a line for each system or temporary directory the walk leaves out (see
"Rules for the walk"); "Rules for the walk");
- a line for each path left out because its filesystem is one the kernel, - a line for each path left out because its filesystem is one the kernel
the core of the operating system, makes up or keeps in memory, such as makes up or keeps in memory, such as `tmpfs`;
`tmpfs`;
- the summary line; - the summary line;
- a warning when a report still has the database open as the scan ends (see - a warning when a report still has the database open as the scan ends (see
"Concurrent reads"). "Concurrent reads").
@@ -158,8 +153,8 @@ to keep is your choice.
A row is a candidate, not proof: A row is a candidate, not proof:
- The reports show each file as it was when a scan last recorded it. A file may - The reports show the files as they were at the last scan. A file may have
have changed or gone since. changed or gone since.
- A file of 100 MiB or more is compared only on samples of its content (see - A file of 100 MiB or more is compared only on samples of its content (see
"Duplicate detection"). "Duplicate detection").
- Two paths that are hard links, two names for one file, are listed as - Two paths that are hard links, two names for one file, are listed as
@@ -206,9 +201,9 @@ two ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it
makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
of reads, at the price of a comparison that is not exact. of reads, at the price of a comparison that is not exact.
The hashes live in a database that is kept between runs. A rescan reads only new The hashes live in a database that is kept between runs. A rescan reads only the
files, and files whose size changed or whose modification time (mtime) is newer files that are new or whose size or modification time (mtime) changed. All
than recorded. All analysis then works from the database alone. analysis then works from the database alone.
The goal is whole duplicated trees, not only single files. A download extracted The goal is whole duplicated trees, not only single files. A download extracted
twice, or a project copied once too often, is such a tree. It can be removed as twice, or a project copied once too often, is such a tree. It can be removed as
@@ -229,9 +224,10 @@ In order:
its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read
in full. The target is a filesystem of about ten million files and 150 TB, on in full. The target is a filesystem of about ten million files and 150 TB, on
disks that may be slow or busy. During a scan, one small record per file is disks that may be slow or busy. During a scan, one small record per file is
held in memory. The hashes stay in the database. `report` holds the duplicate held in memory. It holds the path and the two values the walk compares, the
paths of one file size at a time. `trees` holds one entry per directory, plus size and the mtime. The hashes stay in the database. `report` holds the
the files of the directories it is reading. duplicate paths of one file size at a time. `trees` holds one entry per
directory, plus the files of the directories it is reading.
3. **Rescan cheaply, analyze from the database.** The scan keeps its database 3. **Rescan cheaply, analyze from the database.** The scan keeps its database
between runs, and an unchanged file is never read again. `report`, `trees` between runs, and an unchanged file is never read again. `report`, `trees`
and `info` work from the database alone and never touch the scanned and `info` work from the database alone and never touch the scanned
@@ -265,14 +261,11 @@ The reference below is also the program's specification.
## Non-goals ## Non-goals
- No byte-for-byte compare. - No byte-for-byte compare, and no deleting or linking of duplicates. The
- No deleting of duplicates. reports are advisory. Acting on them is your job.
- No replacing a copy with a hard link to another copy. The reports are
advisory. Acting on them is your job.
- The SQLite database is the only stored data. There are no export or import - The SQLite database is the only stored data. There are no export or import
formats. formats.
- `sfdupes` does not run in the background or watch the filesystem for changes. - No daemon or filesystem watcher. Scheduling rescans is `cron`'s job.
Scheduling rescans is `cron`'s job.
## Reference ## Reference
@@ -287,9 +280,9 @@ The reference below is also the program's specification.
hashes them. The default is the number of CPUs. `N` must be at least 1. A hashes them. The default is the number of CPUs. `N` must be at least 1. A
smaller value is a usage error, reported in one line on stderr with exit 2 smaller value is a usage error, reported in one line on stderr with exit 2
before anything is scanned. before anything is scanned.
- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of - `-x`, `--one-file-system` (`scan`) keeps the walk of each operand on that
directories on other filesystems, as the same flag does for `du` and `rsync`. operand's filesystem, as the same flag does for `du` and `rsync`. By default
By default the walk also enters directories on other filesystems. the walk also enters directories on other filesystems.
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes. - `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
Those are the system and temporary directories the walk otherwise leaves out, Those are the system and temporary directories the walk otherwise leaves out,
such as `/var/tmp` on Linux (see "Rules for the walk"). such as `/var/tmp` on Linux (see "Rules for the walk").
@@ -350,13 +343,12 @@ default. An operand may be a directory or a regular file.
the exit code is 2. the exit code is 2.
- An operand that does not exist is a fatal error. `scan` then stops before it - An operand that does not exist is a fatal error. `scan` then stops before it
walks anything, and exits 1. walks anything, and exits 1.
- Each operand is made absolute before the walk. A relative operand is joined to - Each operand is made absolute before the walk. A `.` or `..` in it is resolved
the working directory as `pwd` prints it, with its symlinks kept. A `.` or as text, and symlinks are not followed. Each record is stored under that
`..` in it is resolved as text, and symlinks are not followed. Each record is absolute path, so results do not depend on the working directory.
stored under that absolute path.
- All operands belong to one scan, and one walk covers them all at once (see - All operands belong to one scan, and one walk covers them all at once (see
"Concurrency"). "Concurrency").
- An operand that repeats another is dropped before the walk. - An operand that repeats another is dropped before the walk, without a line.
- An operand under another operand is dropped the same way, unless one of the - An operand under another operand is dropped the same way, unless one of the
cases below applies. No path is then walked twice. cases below applies. No path is then walked twice.
- An operand under another is kept, and walked on its own, when the inner - An operand under another is kept, and walked on its own, when the inner
@@ -367,21 +359,19 @@ default. An operand may be a directory or a regular file.
It is also kept when a directory between the two is on a filesystem that is It is also kept when a directory between the two is on a filesystem that is
not persistent. Such a filesystem is one the kernel makes up or keeps in not persistent. Such a filesystem is one the kernel makes up or keeps in
memory (see "Rules for the walk"). The outer operand's walk does not enter memory (see "Rules for the walk"). The outer operand's walk does not enter
such a directory, so the inner operand is walked on its own. such a directory, so the inner operand is walked on its own. For an inner
operand that is a regular file, the check starts at its directory.
- No other case keeps an operand under another. An operand reached through a - No other case keeps an operand under another. An operand reached through a
symlinked directory below the outer operand is dropped too. The outer symlinked directory below the outer operand is dropped too, without a line.
operand's walk does not follow the symlink, so no file under the inner operand The outer operand's walk does not follow the symlink, so no file under the
is scanned. The records that an earlier scan left under it are deleted. The inner operand is scanned. The records an earlier scan left under it are
`-x` case above also covers an operand reached that way. deleted.
- An operand that is neither a regular file nor a directory is not scanned. - An operand that is neither a regular file nor a directory is not scanned.
`scan` prints a warning naming the path and what it is. It counts the operand `scan` prints a warning naming the path and what it is. It counts the operand
as skipped and drops it. An operand beneath a symlink operand is scanned all as skipped and drops it. An operand beneath a dropped one is scanned all the
the same. The scan keeps the records beneath a dropped operand, unless another same. The scan keeps the records beneath a dropped operand, unless another
operand covers them. After a `cd` through a symlink, `scan .` names the operand covers them.
symlink and is skipped as a symlink operand. `scan sub` in that directory
records its files under the symlink's path. `report` can then list each as a
duplicate of the same file recorded under its real path.
- An operand at or inside a directory named `.zfs` is never scanned, whether - An operand at or inside a directory named `.zfs` is never scanned, whether
given alone or under another operand. The check looks at the path as given and given alone or under another operand. The check looks at the path as given and
at the path with its symlinks resolved. `scan` prints a warning naming the at the path with its symlinks resolved. `scan` prints a warning naming the
@@ -391,9 +381,6 @@ default. An operand may be a directory or a regular file.
- An operand on a filesystem that is not persistent is dropped too. `scan` - An operand on a filesystem that is not persistent is dropped too. `scan`
prints a line naming it and its filesystem type, and does not count it as prints a line naming it and its filesystem type, and does not count it as
skipped. skipped.
- `scan` prints these warnings, and the line for an operand that is not on a
persistent filesystem, even when the operand repeats another or lies under
another.
- A scan whose every operand is dropped walks nothing and exits 0. - A scan whose every operand is dropped walks nothing and exits 0.
#### What a scan does to the database #### What a scan does to the database
@@ -405,16 +392,15 @@ operands:
- A file already in the database is not read again when its size and mtime match - A file already in the database is not read again when its size and mtime match
its record. The mtime matches when it is not newer than the recorded one. This its record. The mtime matches when it is not newer than the recorded one. This
is what makes a daily rescan cheap. is what makes a daily rescan cheap.
- `scan` hashes a file again when its size differs or its mtime is newer. The - A file whose size differs, or whose mtime is newer, is hashed again. The mtime
mtime is compared to the nanosecond, as finely as the filesystem records it. A is compared to the nanosecond, as finely as the filesystem records it. A
same-size rewrite counts as a change whenever the filesystem gives it a later same-size rewrite counts as a change whenever the filesystem gives it a later
mtime, even within the same second. mtime, even within the same second.
- The scan deletes each record under a scanned operand whose file it could not - The scan deletes each record under a scanned operand whose file it could not
verify. That covers deleted files, and paths the scan could not stat or read verify. That covers deleted files, and paths the scan could not stat or read
this run. To stat a file is to read its details, such as its size, without this run. To stat a file is to read its details, such as its size, without
opening it. So after a complete scan, the database holds only what that scan opening it. So the database holds only what the latest scan over each path
could verify under its operands. A later successful scan adds such files could verify. A later successful scan adds such files again.
again.
- Records outside the scanned operands are left alone, so separate trees can be - Records outside the scanned operands are left alone, so separate trees can be
scanned on different schedules into one database. scanned on different schedules into one database.
- The scan keeps every record at or below a path that the walk leaves out for - The scan keeps every record at or below a path that the walk leaves out for
@@ -425,8 +411,9 @@ operands:
Such records count as outside the scanned operands. So with `-x`, a Such records count as outside the scanned operands. So with `-x`, a
filesystem mounted under an operand can be scanned on its own schedule into filesystem mounted under an operand can be scanned on its own schedule into
the same database. An operand may itself lie at or inside such a path, as the same database. An operand may lie at or inside a default exclude, or,
"Operands" describes. Its walk verifies the records below it, so they are with `-x`, at or inside a directory on another filesystem. Such an operand
is walked on its own. Its walk verifies the records below it, so they are
deleted when their files are gone. deleted when their files are gone.
#### Phases #### Phases
@@ -437,13 +424,12 @@ phases.
1. `load` reads the existing records. Each record under the operands is kept in 1. `load` reads the existing records. Each record under the operands is kept in
memory by path, with its size and mtime, for the walk to compare against. memory by path, with its size and mtime, for the walk to compare against.
2. `walk` lists the trees under all operands. Several workers share one queue of 2. `walk` lists the trees under all operands. Several workers share one queue of
directories to read. A worker is one thread of the program. A thread is one directories to read. A worker is one thread of the program, and "Concurrency"
of several parts of a program that run at the same time and share its memory. says how many there are. Each worker reads one directory at a time. It hands
"Concurrency" says how many workers there are. Each worker reads one the subdirectories it finds back to the queue, and it stats each regular file
directory at a time. It hands the subdirectories it finds back to the queue, as it finds it. The walk compares each file with its record. An unchanged
and it stats each regular file as it finds it. The walk compares each file file needs nothing more. Every new or changed file is passed to the hash
with its record. An unchanged file needs nothing more. Every new or changed phase.
file is passed to the hash phase.
3. `hash` reads every new or changed file and computes its three hashes. Files 3. `hash` reads every new or changed file and computes its three hashes. Files
are read in inode order. An inode is the filesystem's own record of a file. are read in inode order. An inode is the filesystem's own record of a file.
Files with nearby inode numbers usually lie near each other on disk. Reading Files with nearby inode numbers usually lie near each other on disk. Reading
@@ -462,7 +448,7 @@ walk practical on large or busy storage pools.
#### Rules for the walk #### Rules for the walk
- Only regular files are recorded. Every other kind of file, such as a symlink - Only regular files are recorded. Every other kind of file, such as a symlink
or a device node, is passed over. The walk follows no symlink it finds. or a device node, is passed over. Symlinks are never followed.
- The walk never enters a directory named `.zfs`. Such a directory holds the - The walk never enters a directory named `.zfs`. Such a directory holds the
filesystem's ZFS snapshots, which are its saved earlier states. Walking it filesystem's ZFS snapshots, which are its saved earlier states. Walking it
would list every file once per snapshot. `scan` never starts a walk at or would list every file once per snapshot. `scan` never starts a walk at or
@@ -480,14 +466,14 @@ walk practical on large or busy storage pools.
operand at the path given, even when that path runs through a symlinked operand at the path given, even when that path runs through a symlinked
directory. Such an operand can lead to the database files, and the walk directory. Such an operand can lead to the database files, and the walk
passes over them there as well. The walk prints no line for the database passes over them there as well. The walk prints no line for the database
files and does not count them as skipped. A record that an earlier scan left files and does not count them as skipped. A record an earlier scan left for
for one of them is deleted. one of them is deleted.
- By default the walk enters directories on other filesystems than its - By default the walk enters directories on other filesystems than its
operand's. With `-x`, it never enters a directory on a different filesystem operand's. With `-x`, it never enters a directory on a different filesystem
than its operand. `scan` prints no line for a directory that only `-x` leaves than its operand. `scan` prints no line for that directory and does not count
out, and does not count it as skipped. The records below it are kept, as it as skipped. The records below it are kept, as described under "What a scan
described under "What a scan does to the database". does to the database".
- The walk never records anything on a filesystem that is not persistent. A - The walk never records anything on a filesystem that is not persistent. A
filesystem is not persistent when the kernel makes it up or keeps it in filesystem is not persistent when the kernel makes it up or keeps it in
memory. These are the types, as `sfdupes scan --help` lists them on Linux and memory. These are the types, as `sfdupes scan --help` lists them on Linux and
@@ -523,10 +509,11 @@ walk practical on large or busy storage pools.
The path is not counted as skipped. The records below it are kept, as The path is not counted as skipped. The records below it are kept, as
described under "What a scan does to the database". described under "What a scan does to the database".
- Below an operand, the walk does not enter a default exclude. On Linux, the - Below an operand, the walk does not enter a default exclude. The default
default excludes hold temporary files and the kernel's view of the running excludes hold the kernel's view of the running system and temporary files, so
system. They depend on the system, and `sfdupes scan --help` lists those of that `sfdupes scan /` walks only the files worth comparing. They depend on the
the system it runs on. On Linux they are: system, and `sfdupes scan --help` lists those of the system it runs on. On
Linux they are:
``` ```
/proc, /sys, /dev, /run, /tmp, /var/tmp /proc, /sys, /dev, /run, /tmp, /var/tmp
@@ -555,27 +542,21 @@ walk practical on large or busy storage pools.
scan: skipped /var/tmp (default exclude) scan: skipped /var/tmp (default exclude)
``` ```
The directory is not counted as skipped. `scan` prints the filesystem line The directory is not counted as skipped. A default exclude on a filesystem
instead for a default exclude that is not on a persistent filesystem, such that is not persistent, such as `/proc`, gets the line for its filesystem
as `/proc`. One that is itself an operand gets no line, because it is walked instead. One that is itself an operand gets no line, because it is walked as
as that operand. The scan keeps the records below a default exclude that the that operand. The scan keeps the records below a default exclude that the
walk did not enter, as described under "What a scan does to the database". walk did not enter, as described under "What a scan does to the database".
- `scan` skips each of these, with a one-line warning on stderr: - A path that cannot be read, for lack of permission or any other error, gets a
- a directory it cannot read; one-line warning on stderr and is skipped. The scan continues, and the summary
- a path it cannot stat; line says how many paths were skipped. A skipped path that has a record from
- a new or changed file it cannot read. an earlier scan loses that record, and so does every path under an unreadable
directory. The database mirrors what the latest scan could verify.
An unchanged file is not opened, so it keeps its record even when it has
become unreadable. The scan continues, and the summary line says how many
paths were skipped. A skipped path that has a record from an earlier scan
loses that record, and so does every path under an unreadable directory.
- A file or directory that disappears while the scan runs is not an error. The - A file or directory that disappears while the scan runs is not an error. The
scan prints no line for it and does not count it as skipped. A path gone when scan prints no line for it and does not count it as skipped. A path gone when
the walk reaches it gets no record, and so does a file gone when the hash the walk reaches it gets no record, and so does a file gone when the hash
phase opens it. The records that an earlier scan left at or below it are phase opens it. The records an earlier scan left at or below it are deleted.
deleted.
#### Concurrency #### Concurrency
@@ -593,8 +574,8 @@ from a thread of its own and never blocks the workers.
- one line naming the database (see "Database"); - one line naming the database (see "Database");
- for each dropped operand, in operand order, the warning or the line described - for each dropped operand, in operand order, the warning or the line described
under "Operands". `scan` prints neither for an operand it drops only because under "Operands". An operand dropped as a repeat of another, or as lying under
it repeats another or lies under another; another, gets neither;
- the progress display of each phase (see "Progress"). `scan` prints each - the progress display of each phase (see "Progress"). `scan` prints each
warning for a skipped path as it happens. It prints a line the same way for warning for a skipped path as it happens. It prints a line the same way for
each path left out as a default exclude or as not on a persistent filesystem; each path left out as a default exclude or as not on a persistent filesystem;
@@ -623,12 +604,12 @@ signal `kill` sends by default:
- Nothing else is written or deleted. Records are deleted only after a complete - Nothing else is written or deleted. Records are deleted only after a complete
walk. So the records under paths that an interrupted walk never reached are walk. So the records under paths that an interrupted walk never reached are
kept. kept.
- The database is closed and the lock released. - The database is closed and the lock released, as on any other exit.
- The line `scan: interrupted after N files` goes to stderr. `N` is the number - The line `scan: interrupted after N files` goes to stderr. `N` is the number
of files the walk reached. The exit code is 1. of files the walk reached. The exit code is 1.
The next scan reads none of the unchanged files already recorded, and brings the The next scan reads none of the files already recorded and brings the database
database up to date as usual. up to date as usual.
After the first signal `scan` stops catching signals. A second one ends it at After the first signal `scan` stops catching signals. A second one ends it at
once, as any uncaught signal does. The records not yet written are lost, and the once, as any uncaught signal does. The records not yet written are lost, and the
@@ -687,8 +668,8 @@ Four bytes are written as escapes:
Every other byte is written unchanged, including bytes that are not valid UTF-8, Every other byte is written unchanged, including bytes that are not valid UTF-8,
the usual text encoding. Undoing the four escapes gives back the stored path. the usual text encoding. Undoing the four escapes gives back the stored path.
Grouping and ordering use the stored path. The warnings about skipped paths that Grouping and ordering use the stored path. The warnings that `scan` prints are
`scan` prints are escaped the same way, so each stays one line. escaped the same way, so each warning stays one line.
The summary line on stderr gives: The summary line on stderr gives:
@@ -726,11 +707,11 @@ Processing:
- `trees` builds the directory tree and computes every directory's digest. It - `trees` builds the directory tree and computes every directory's digest. It
then groups the directories by digest. Every group with two or more then groups the directories by digest. Every group with two or more
directories is a duplicate-tree group. directories is a duplicate-tree group.
- `trees` leaves out a group that a group of its members' parents already - Only the outermost duplicate trees are reported. A group is left out when its
implies. A group is left out when its members' parents are distinct members' parents are distinct directories that all have the same digest. The
directories that all have the same digest. The parents' group, or an parents' group, or an ancestor's, already implies it. A group holding sibling
ancestor's, already implies it. A group is always reported when it holds directories, or members whose parents have different digests, is always
sibling directories, or when its members' parents have different digests. reported.
- Within a group, paths are sorted in byte order. The first is `first`, and - Within a group, paths are sorted in byte order. The first is `first`, and
every other path is a `dupe`. every other path is a `dupe`.
- Groups are ordered by total tree size, largest first, then by `first` path. - Groups are ordered by total tree size, largest first, then by `first` path.
@@ -742,8 +723,8 @@ Two limitations are accepted:
They share storage, so removing one frees nothing. The scan uses the inode to They share storage, so removing one frees nothing. The scan uses the inode to
avoid reading a file twice, and does not store it. avoid reading a file twice, and does not store it.
- Only the regular files in the database define a tree. Empty directories are - Only the regular files in the database define a tree. Empty directories are
invisible. When the scan skips a file in one copy but not in the other, the invisible. A file skipped during the scan in one copy but not in the other
two trees differ. makes two otherwise identical trees differ.
`trees` has four phases, each with its own progress display on stderr (see `trees` has four phases, each with its own progress display on stderr (see
"Progress"). `open` opens the database. `read` counts the records, then reads "Progress"). `open` opens the database. `read` counts the records, then reads
@@ -854,21 +835,20 @@ the error for a missing database comes after it.
#### The scan lock #### The scan lock
Only one `scan` at a time can hold the lock. Only one `scan` runs against a database at a time.
- For its whole run, `scan` holds a lock on a lock file beside the database. The - For its whole run, `scan` holds a lock on a lock file beside the database. The
lock is taken with the `flock(2)` system call, a request a program makes to lock is taken with the `flock(2)` system call, and only one process can hold
the kernel, and only one process can hold it. The lock file's name is the it. The lock file's name is the database's path, with symlinks resolved, plus
database's path, with symlinks resolved, plus `.lock`. By default that is `.lock`. By default that is `/var/lib/sfdupes/db.sqlite.lock`. Every path to
`/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the same one database therefore names the same lock file.
lock file.
- `scan` takes the lock before it walks the filesystem or opens the database. - `scan` takes the lock before it walks the filesystem or opens the database.
Just before that, it creates the database file, empty, if it is missing. So Just before that, it creates the database file, empty, if it is missing. So
the lock file is named the same way before the first scan as after. A symlink the lock file is named the same way before the first scan as after. A symlink
at the database path is followed, and its target is created. at the database path is followed, and its target is created.
- A second `scan` that finds the lock held does not wait. It fails at once with - A second `scan` against the same database does not wait. It fails at once with
an error naming the lock file, and exits 1. It has walked nothing and has not an error naming the lock file, and exits 1. It has walked nothing and opened
opened the database. The running scan carries on. nothing. The running scan carries on.
- The lock file is created on first use and never deleted. Only its owner may - The lock file is created on first use and never deleted. Only its owner may
open it. A leftover file blocks nothing. The lock ends with the process that open it. A leftover file blocks nothing. The lock ends with the process that
holds it, however that process ends. Deleting the file while a scan runs would holds it, however that process ends. Deleting the file while a scan runs would
@@ -881,9 +861,8 @@ Only one `scan` at a time can hold the lock.
SQLite appends changes to a `-wal` file beside the database, and keeps a SQLite appends changes to a `-wal` file beside the database, and keeps a
lookup table for that file in a `-shm` file. Readers do not block the writer, lookup table for that file in a `-shm` file. Readers do not block the writer,
so running a report during a scan is safe. so running a report during a scan is safe.
- Every subcommand opens the database with a busy timeout of 10 seconds. That is - Every connection has a busy timeout of 10 seconds. That is how long it waits
how long it waits for a database another process has locked before it gives for a database another connection has locked before it gives up.
up.
- The filesystem is authoritative, and the database follows it. Hashed records - The filesystem is authoritative, and the database follows it. Hashed records
are written 10,000 at a time while the scan runs, so a report may see a scan's are written 10,000 at a time while the scan runs, so a report may see a scan's
changes partly applied. A scan that dies partway leaves a valid database changes partly applied. A scan that dies partway leaves a valid database
@@ -921,15 +900,13 @@ Only one `scan` at a time can hold the lock.
#### Schema #### Schema
The schema is the database's layout, its table and its index. The schema is the database's layout, its table and its index.
`PRAGMA user_version` holds the schema version, currently 1. An empty database `PRAGMA user_version` holds the schema version, currently 1. A database with any
is at version 0, and `scan` sets it up. Any other version is a fatal error. other version is a fatal error. `scan` creates the table and the index and sets
`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal the version in one transaction (see "`info` mode"). So the three land together
error too. `scan` creates the table and the index and sets the version in one or not at all. A first scan stopped while doing so leaves an empty database,
transaction (see "`info` mode"). So the three land together or not at all. A which the next scan sets up. A database at version 0 that already has a `files`
first scan stopped while doing so leaves an empty database, which the next scan table was not made by `sfdupes`. Every subcommand refuses it, with an error
sets up. A database at version 0 that already has a `files` table was not made telling you to remove the file and rescan.
by `sfdupes`. Every subcommand refuses it, with an error telling you to remove
the file and rescan.
```sql ```sql
CREATE TABLE files ( CREATE TABLE files (
@@ -959,9 +936,9 @@ CREATE INDEX files_signature ON files (size, head, tail, content);
### Progress ### Progress
Each phase of every subcommand gets a display of its own on stderr from the Every subcommand always shows that it is working. Each phase gets a display of
moment it starts. The display names the phase and shows its progress. The parts its own on stderr from the moment it starts. The display names the phase and
of its line are listed below. shows its progress. The parts of its line are listed below.
The phases, and what each display counts: The phases, and what each display counts:
@@ -1022,7 +999,7 @@ phase waits, in place of the rate. That happens while one large file is read, or
while a disk or the database is slow. Each phase names the places it may wait while a disk or the database is slow. Each phase names the places it may wait
in, and a phase that may wait in two places names both, as `disk/db`: in, and a phase that may wait in two places names both, as `disk/db`:
- `walk` says `disk`, because it reads directories and stats files. - `walk` says `disk`, for files.
- `hash` says `disk/db`, because it reads files and writes their records to the - `hash` says `disk/db`, because it reads files and writes their records to the
database between them. database between them.
- `load`, `update` and the `read` phase of `trees` say `db`. - `load`, `update` and the `read` phase of `trees` say `db`.
@@ -1050,18 +1027,19 @@ On a terminal:
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ | ⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
``` ```
- No line reaches the terminal's last column. A line too wide for the terminal - No line reaches the terminal's last column, however narrow the terminal is. A
first leaves out its rate, the words of a stalled phase, or what the phase is line too wide for the terminal first leaves out its rate, the words of a
waiting for. It leaves out its bar next. It is cut short at the edge only when stalled phase, or what the phase is waiting for. It leaves out its bar next.
the rest does not fit on its own. The rest is: It is cut short at the edge only when the rest does not fit on its own. The
rest is:
- the phase and its place; - the phase and its place;
- the progress, when the phase shows one; - the progress, when the phase shows one;
- the elapsed time; - the elapsed time;
- the time remaining. - the time remaining.
On a terminal 80 columns wide, no line is cut short, even hours into a scan On a terminal narrower than about 75 columns, a line is cut short only hours
of millions of files. A terminal made narrower during a phase is fitted at into a scan of millions of files. A terminal made narrower during a phase is
the next draw. fitted at the next draw.
- A warning printed during a phase lands on a line of its own, never inside the - A warning printed during a phase lands on a line of its own, never inside the
display. So does every line written to stdout when stdout and stderr are the display. So does every line written to stdout when stdout and stderr are the
@@ -1077,8 +1055,8 @@ Each phase prints its line:
- every 5 seconds while it runs, whether or not an item has completed; - every 5 seconds while it runs, whether or not an item has completed;
- once more as it ends. - once more as it ends.
When a phase stops short of its total, as an interrupted one does, it keeps A phase that stops short of its total, as an interrupted one does, keeps showing
showing the count it stopped at. the count it stopped at.
### Error handling and exit codes ### Error handling and exit codes
@@ -1087,14 +1065,12 @@ showing the count it stopped at.
error is one of these: error is one of these:
- an operand that does not exist, or whose details `scan` cannot read; - an operand that does not exist, or whose details `scan` cannot read;
- a list of mounted filesystems that `scan` cannot read; - a list of mounted filesystems that `scan` cannot read;
- another `scan` holding the lock (see "The scan lock"); - another `scan` already running against the same database;
- a database, its directory or its lock file that cannot be created or - a database that cannot be created or opened, or whose reads or writes
opened; fail, its directory and its lock file included;
- a read or write of the database that fails;
- a relative database path when the working directory cannot be read; - a relative database path when the working directory cannot be read;
- a database at a schema version other than 0 or 1; - a database with another schema version, or one at version 0 that already
- a database at version 0 read by `report`, `trees` or `info`; has a `files` table;
- a database at version 0 that already has a `files` table (see "Schema");
- a record with a hash that is not 32 bytes, read by `report` or `trees`; - a record with a hash that is not 32 bytes, read by `report` or `trees`;
- a missing database for `report`, `trees` or `info`; - a missing database for `report`, `trees` or `info`;
- a failed write to stdout. - a failed write to stdout.
@@ -1143,8 +1119,8 @@ the repository root itself, so it can be run from any directory.
- An installed `node` is used when its major version is 22. A `yarn` already - An installed `node` is used when its major version is 22. A `yarn` already
on `PATH` is then used as it is. When `yarn` is missing, `yarn` 1.22.22 is on `PATH` is then used as it is. When `yarn` is missing, `yarn` 1.22.22 is
activated through `corepack`, the `yarn` installer that ships with `node`. activated through `corepack`, the `yarn` installer that ships with `node`.
When there is no `corepack` either, the script installs `yarn` with `npm`, When there is no `corepack` either, `npm`, the package manager that ships
the package manager that ships with `node`. with `node`, installs it.
- Otherwise the script installs `node` 22.17.0 through `nvm`, a `node` - Otherwise the script installs `node` 22.17.0 through `nvm`, a `node`
version manager, and activates `yarn` 1.22.22 under that `node` through version manager, and activates `yarn` 1.22.22 under that `node` through
`corepack`. When `nvm` is missing, the script first downloads it and `corepack`. When `nvm` is missing, the script first downloads it and
@@ -1174,9 +1150,9 @@ the repository root itself, so it can be run from any directory.
- `script/lint` runs the linter, `golangci-lint`, which checks the Go sources - `script/lint` runs the linter, `golangci-lint`, which checks the Go sources
for common mistakes. It builds only the `lint` stage of the `Dockerfile` and for common mistakes. It builds only the `lint` stage of the `Dockerfile` and
writes no image. The stage starts from the `golangci/golangci-lint` image, writes no image. The stage starts from the `golangci/golangci-lint` image,
pinned by its digest, and runs these checks as build steps: pinned by its digest, and runs three build steps:
- the `gofmt` check, which fails when `gofmt`, Go's own formatter, would - the `gofmt` check, which fails on a Go source that `gofmt`, Go's own
change a Go source; formatter, would change;
- `golangci-lint config verify`, which checks the linter's settings file - `golangci-lint config verify`, which checks the linter's settings file
`.golangci.yml`; `.golangci.yml`;
- `golangci-lint run`, which lints the Go sources. - `golangci-lint run`, which lints the Go sources.
@@ -1210,8 +1186,8 @@ the repository root itself, so it can be run from any directory.
every check passed. The tests and the linter run twice, once in `script/check` every check passed. The tests and the linter run twice, once in `script/check`
and again as stages of the image build. and again as stages of the image build.
- `script/precommit` runs from the `git` pre-commit hook, a script that `git` - `script/precommit` runs from the `git` pre-commit hook, a script that `git`
runs before each commit. It runs `go mod tidy`, and fails the commit when that runs before each commit. It fails the commit when `go mod tidy` would change
changes `go.mod` or `go.sum`. Then it runs `script/check`. anything. Then it runs `script/check`.
- `script/install-precommit` installs `.git/hooks/pre-commit`, which runs - `script/install-precommit` installs `.git/hooks/pre-commit`, which runs
`script/precommit`. `script/precommit`.
@@ -1283,7 +1259,6 @@ All of the following, run in this directory, must pass:
data. data.
```sh ```sh
make build
d=$(mktemp -d /var/tmp/sfdupes.XXXXXX) d=$(mktemp -d /var/tmp/sfdupes.XXXXXX)
export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite" export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite"
mkdir -p "$d/a" "$d/b" mkdir -p "$d/a" "$d/b"
+8 -7
View File
@@ -773,8 +773,8 @@ func TestBuildVersion(t *testing.T) {
func TestRunVersionWriteFailureIsFatal(t *testing.T) { func TestRunVersionWriteFailureIsFatal(t *testing.T) {
t.Parallel() t.Parallel()
// README §Error handling: a failed write to stdout is a fatal error, // README §Error handling: a stdout write failure exits 1, reported
// exit 1, reported on stderr starting with "sfdupes:". // in one line on stderr.
var stderr bytes.Buffer var stderr bytes.Buffer
code := run([]string{"--version"}, failingWriter{}, &stderr) code := run([]string{"--version"}, failingWriter{}, &stderr)
@@ -1279,8 +1279,9 @@ func holdScanLock(t *testing.T, path string) {
} }
func TestRunSecondScanFails(t *testing.T) { func TestRunSecondScanFails(t *testing.T) {
// README §Database: a second scan that finds the lock held fails // README §Database: while one scan holds the lock, a second scan
// at once, naming the lock file, without opening the database. // fails at once, naming the lock file, without opening the
// database.
path := testDBPath(t) path := testDBPath(t)
t.Setenv(databaseEnv, path) t.Setenv(databaseEnv, path)
@@ -1339,7 +1340,7 @@ func assertSecondScanFails(t *testing.T, path string) {
} }
func TestRunSecondScanThroughSymlinkFails(t *testing.T) { func TestRunSecondScanThroughSymlinkFails(t *testing.T) {
// README §Database: a path through a symlink names the same lock // README §Database: every path to one database names the same lock
// file, so while a scan holds it through one path, a scan through // file, so while a scan holds it through one path, a scan through
// another fails, naming the lock file beside the database's real // another fails, naming the lock file beside the database's real
// path. // path.
@@ -1444,8 +1445,8 @@ func TestRunReportsDuringScan(t *testing.T) {
} }
func TestRunStdoutClosedIsFatal(t *testing.T) { func TestRunStdoutClosedIsFatal(t *testing.T) {
// README §Error handling: a failed write to stdout is a fatal error, // README §Error handling: a stdout write failure exits 1, reported
// exit 1, reported on stderr starting with "sfdupes:". // in one line on stderr.
for _, name := range []string{cmdReport, cmdTrees, cmdInfo} { for _, name := range []string{cmdReport, cmdTrees, cmdInfo} {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
t.Setenv(databaseEnv, testDBPath(t)) t.Setenv(databaseEnv, testDBPath(t))