Author SHA1 Message Date
sneak a4fdad43a3 Rewrite README.md sentence by sentence for a newcomer
check / check (push) Canceled after 0s
The old README was a specification written as one dense document:
sentences past 30 words, lists chained with commas, exceptions ahead
of the rules, and release history in the user text. This rewrite
leads with what the program is, what it is for and how to use it,
then gives a reference for the settings, each subcommand, the
database, the output formats, the progress display and the exit
codes. Non-goals come before the reference; the constraints, the
entrypoints and the build come last. Every statement was checked
against the code on next and against what the built binary prints on
a small tree. The section names the code comments cite are kept, and
three comments in main_test.go now cite Settings. TODO.md records the
step.

Model: fable-5-1
2026-10-26 10:35:40 +00:00
7 changed files with 257 additions and 462 deletions
+216 -284
View File
@@ -18,11 +18,11 @@ a strong candidate, not proof.
`sfdupes` is meant for multi-terabyte servers, where reading every byte of every
file takes too long. It has four subcommands. `scan` walks the filesystem and
records the files it finds in an SQLite database. SQLite keeps a whole database
in one file. The database is kept between runs. A rescan reads only new and
changed files. That makes `scan` cheap enough to run every day from `cron`, the
system's job scheduler. `report`, `trees` and `info` read the database alone and
can run at any time.
records every file in an SQLite database. SQLite keeps a whole database in one
file. The database is kept between runs. A rescan reads only new and changed
files. That makes `scan` cheap enough to run every day from `cron`, the system's
job scheduler. `report`, `trees` and `info` read the database alone and can run
at any time.
## Getting Started
@@ -38,18 +38,16 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
`scan` walks the trees named by its operands, the paths given after `scan` on
the command line. The walk is the listing of every directory under them. `scan`
keeps one database record per regular file. A regular file is an ordinary file,
as opposed to a directory, a symlink or a device node. A symlink is a file that
holds the path of another file and stands in for it. A device node is a file
that stands for a piece of hardware, such as a disk or a terminal. `report`
prints the file-level duplicates. `trees` prints the duplicate directory trees.
`info` prints the database's path, size and record count.
as opposed to a directory, a symlink or a device. `report` prints the file-level
duplicates. `trees` prints the duplicate directory trees. `info` prints the
database's path, size and record count.
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
another file. The first `scan` creates it.
`report`, `trees` and `info` write their data to stdout. Progress, warnings and
errors go to stderr. stdout and stderr are a program's two output streams. The
shell shows both on the terminal, and `>` redirects stdout alone.
Every command writes its data to stdout. Everything else, such as progress and
warnings, goes to stderr. stdout and stderr are a program's two output streams.
The shell shows both on the terminal, and `>` redirects stdout alone.
### Install
@@ -69,12 +67,11 @@ The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set.
`sfdupes --version` prints `sfdupes VERSION` and exits. For a binary from
`go install`, `VERSION` is the version Go recorded in it. For `@latest` that is
a release tag such as `v0.0.3`. For `@main` it is the commit's tag when it has
one. Otherwise it is a pseudo-version, Go's version string for an untagged
commit. That string ends in the commit's hash. A binary built from a clone
carries the output of `git describe --tags --always --dirty` in that clone, such
as `v0.0.3` on a tagged commit. `VERSION` is `dev` when no version is known. To
see everything Go recorded in an installed binary:
a release tag such as `v0.0.3`. For `@main` it is a pseudo-version, Go's version
string for an untagged commit. That string ends in the commit's hash. A binary
built from a clone carries the `git` tag or commit it was built from. `VERSION`
is `dev` when no version is known. To see everything Go recorded in an installed
binary:
```sh
go version -m "$(command -v sfdupes)"
@@ -108,10 +105,9 @@ docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
### Daily scan from `cron`
Run `scan` as root, so that it can read every file. `scan` warns about a
directory it cannot read, and about a new or changed file it cannot read. It
skips the path and deletes the records at or below it (see "Rules for the
walk"). As the file `/etc/cron.d/sfdupes`:
Run `scan` as root, so that it can read every file. A path it cannot read is
skipped with a warning and loses its database record. As the file
`/etc/cron.d/sfdupes`:
```
30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log
@@ -125,11 +121,8 @@ walk"). As the file `/etc/cron.d/sfdupes`:
- one line naming the database;
- progress lines, as described under "Progress";
- a warning for each path it skips;
- a line for each system or temporary directory the walk leaves out (see
"Rules for the walk");
- a line for each path left out because its filesystem is one the kernel,
the core of the operating system, makes up or keeps in memory, such as
`tmpfs`;
- a line for each path the walk leaves out as a system or temporary
directory (see "Rules for the walk");
- the summary line;
- a warning when a report still has the database open as the scan ends (see
"Concurrent reads").
@@ -139,8 +132,8 @@ walk"). As the file `/etc/cron.d/sfdupes`:
- A scan that fails exits 1, with the reason among the last lines of the log.
`tail` prints those lines, and `cron` mails them to root if the host can send
mail.
- A running scan carries on when the next one starts. The new one fails at once,
and the lines that `cron` mails include
- A scan still running when the next one starts carries on. The new one fails at
once, and the lines `cron` mails include
`sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
- `report`, `trees` and `info` need only read access to the database. The umask
is the setting that removes permissions from the files a process creates.
@@ -158,8 +151,8 @@ to keep is your choice.
A row is a candidate, not proof:
- The reports show each file as it was when a scan last recorded it. A file may
have changed or gone since.
- The reports show the files as they were at the last scan. A file may have
changed or gone since.
- A file of 100 MiB or more is compared only on samples of its content (see
"Duplicate detection").
- Two paths that are hard links, two names for one file, are listed as
@@ -177,8 +170,7 @@ cmp /srv/a/big.iso /srv/b/big-copy.iso && echo identical
where the files differ. The second line prints `hard links` when both paths are
the same file.
A path holding a special character, such as a tab, is escaped in the reports.
The character is written as a backslash and one more character, `\t` for a tab
A path holding a special character, such as a tab, is escaped in the reports
(see "Report output format"). Undo the escapes before using such a path.
`printf '%b'` does that, because every backslash in an escaped path starts an
escape. The shell drops trailing newlines from the output of `$(...)`. To keep a
@@ -206,9 +198,9 @@ two ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it
makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
of reads, at the price of a comparison that is not exact.
The hashes live in a database that is kept between runs. A rescan reads only new
files, and files whose size changed or whose modification time (mtime) is newer
than recorded. All analysis then works from the database alone.
The hashes live in a database that is kept between runs. A rescan reads only the
files that are new or whose size or modification time (mtime) changed. All
analysis then works from the database alone.
The goal is whole duplicated trees, not only single files. A download extracted
twice, or a project copied once too often, is such a tree. It can be removed as
@@ -222,16 +214,17 @@ In order:
1. **Find whole duplicate trees, not just files.** The aim is to find places
where the same set of files and directories exists at two or more paths. A
whole subtree can then be removed at once. Tree-level detection is built on
file-level detection.
whole subtree can then be removed at once. File-level detection comes first.
Tree-level detection is built on top of it.
2. **Bound the reads per file.** A file is read only when it is new or has
changed. A file under 100 MiB is then read in full, and a larger one only at
its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read
in full. The target is a filesystem of about ten million files and 150 TB, on
disks that may be slow or busy. During a scan, one small record per file is
held in memory. The hashes stay in the database. `report` holds the duplicate
paths of one file size at a time. `trees` holds one entry per directory, plus
the files of the directories it is reading.
held in memory. It holds the path and the two values the walk compares, the
size and the mtime. The hashes stay in the database. `report` holds the
duplicate paths of one file size at a time. `trees` holds one entry per
directory, plus the files of the directories it is reading.
3. **Rescan cheaply, analyze from the database.** The scan keeps its database
between runs, and an unchanged file is never read again. `report`, `trees`
and `info` work from the database alone and never touch the scanned
@@ -257,22 +250,17 @@ sfdupes --version
sfdupes [command] --help
```
A missing or unknown subcommand is a usage error, a mistake in the command line.
`sfdupes` then prints the usage text, a summary of the commands and their flags,
on stderr and exits 2.
A missing or unknown subcommand prints the usage text on stderr and exits 2.
The reference below is also the program's specification.
## Non-goals
- No byte-for-byte compare.
- No deleting of duplicates.
- No replacing a copy with a hard link to another copy. The reports are
advisory. Acting on them is your job.
- No byte-for-byte compare, and no deleting or linking of duplicates. The
reports are advisory. Acting on them is your job.
- The SQLite database is the only stored data. There are no export or import
formats.
- `sfdupes` does not run in the background or watch the filesystem for changes.
Scheduling rescans is `cron`'s job.
- No daemon or filesystem watcher. Scheduling rescans is `cron`'s job.
## Reference
@@ -287,9 +275,9 @@ The reference below is also the program's specification.
hashes them. The default is the number of CPUs. `N` must be at least 1. A
smaller value is a usage error, reported in one line on stderr with exit 2
before anything is scanned.
- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of
directories on other filesystems, as the same flag does for `du` and `rsync`.
By default the walk also enters directories on other filesystems.
- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand on that
operand's filesystem, as the same flag does for `du` and `rsync`. By default
the walk also enters directories on other filesystems.
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
Those are the system and temporary directories the walk otherwise leaves out,
such as `/var/tmp` on Linux (see "Rules for the walk").
@@ -348,52 +336,32 @@ default. An operand may be a directory or a regular file.
- `scan` with no operand is a usage error. The usage text goes to stderr, and
the exit code is 2.
- An operand that does not exist is a fatal error. `scan` then stops before it
walks anything, and exits 1.
- Each operand is made absolute before the walk. A relative operand is joined to
the working directory as `pwd` prints it, with its symlinks kept. A `.` or
`..` in it is resolved as text, and symlinks are not followed. Each record is
stored under that absolute path.
- An operand that does not exist is a fatal error, exit 1.
- Each operand is made absolute before the walk. A `.` or `..` in it is resolved
as text, and symlinks are not followed. Each record is stored under that
absolute path, so results do not depend on the working directory.
- All operands belong to one scan, and one walk covers them all at once (see
"Concurrency").
- An operand that repeats another is dropped before the walk.
- An operand under another operand is dropped the same way, unless one of the
cases below applies. No path is then walked twice.
- An operand under another is kept, and walked on its own, when the inner
operand or a directory between the two is one of these:
- An operand that repeats another, or lies under another, is dropped before the
walk, without a line. Every file is then reached once. An operand under
another is kept only when the inner operand, or a directory between the two,
is one of these:
- a default exclude (see "Rules for the walk");
- a directory on a filesystem that is not persistent, one the kernel makes
up or keeps in memory (see "Rules for the walk");
- with `-x`, a directory on another filesystem than the outer operand.
It is also kept when a directory between the two is on a filesystem that is
not persistent. Such a filesystem is one the kernel makes up or keeps in
memory (see "Rules for the walk"). The outer operand's walk does not enter
such a directory, so the inner operand is walked on its own.
The outer operand's walk does not enter such a directory, so the inner
operand is walked on its own.
- No other case keeps an operand under another. An operand reached through a
symlinked directory below the outer operand is dropped too. The outer
operand's walk does not follow the symlink, so no file under the inner operand
is scanned. The records that an earlier scan left under it are deleted. The
`-x` case above also covers an operand reached that way.
- An operand that is neither a regular file nor a directory is not scanned.
`scan` prints a warning naming the path and what it is. It counts the operand
as skipped and drops it. An operand beneath a symlink operand is scanned all
the same. The scan keeps the records beneath a dropped operand, unless another
operand covers them. After a `cd` through a symlink, `scan .` names the
symlink and is skipped as a symlink operand. `scan sub` in that directory
records its files under the symlink's path. `report` can then list each as a
duplicate of the same file recorded under its real path.
- An operand at or inside a directory named `.zfs` is never scanned, whether
given alone or under another operand. The check looks at the path as given and
at the path with its symlinks resolved. `scan` prints a warning naming the
path, counts the operand as skipped and drops it. The records beneath it are
kept, unless it lies under another operand. Then they are deleted, because
that operand's walk never enters `.zfs` (see "Rules for the walk").
- An operand that is neither a regular file nor a directory is not scanned, and
neither is a directory named `.zfs`. `scan` prints a warning naming the path
and what it is. It counts the operand as skipped and drops it. An operand
beneath a dropped one is scanned all the same. The scan keeps the records
beneath a dropped operand, unless another operand covers them.
- An operand on a filesystem that is not persistent is dropped too. `scan`
prints a line naming it and its filesystem type, and does not count it as
skipped.
- `scan` prints these warnings, and the line for an operand that is not on a
persistent filesystem, even when the operand repeats another or lies under
another.
- A scan whose every operand is dropped walks nothing and exits 0.
#### What a scan does to the database
@@ -405,28 +373,27 @@ operands:
- A file already in the database is not read again when its size and mtime match
its record. The mtime matches when it is not newer than the recorded one. This
is what makes a daily rescan cheap.
- `scan` hashes a file again when its size differs or its mtime is newer. The
mtime is compared to the nanosecond, as finely as the filesystem records it. A
- A file whose size differs, or whose mtime is newer, is hashed again. The mtime
is compared to the nanosecond, as finely as the filesystem records it. A
same-size rewrite counts as a change whenever the filesystem gives it a later
mtime, even within the same second.
- The scan deletes each record under a scanned operand whose file it could not
verify. That covers deleted files, and paths the scan could not stat or read
this run. To stat a file is to read its details, such as its size, without
opening it. So after a complete scan, the database holds only what that scan
could verify under its operands. A later successful scan adds such files
again.
opening it. So the database holds only what the latest scan over each path
could verify. A later successful scan adds such files again.
- Records outside the scanned operands are left alone, so separate trees can be
scanned on different schedules into one database.
- The scan keeps every record at or below a path that the walk leaves out for
one of these reasons:
- it is a default exclude;
- it is not on a persistent filesystem;
- with `-x`, it is a directory on another filesystem than its operand.
- The scan keeps every record at or below a path that the walk leaves out on
purpose. Such records count as outside the scanned operands. The walk leaves
out:
- a default exclude;
- a path that is not on a persistent filesystem;
- with `-x`, a directory on another filesystem than its operand.
Such records count as outside the scanned operands. So with `-x`, a
filesystem mounted under an operand can be scanned on its own schedule into
the same database. An operand may itself lie at or inside such a path, as
"Operands" describes. Its walk verifies the records below it, so they are
So with `-x`, a filesystem mounted under an operand can be scanned on its
own schedule into the same database. An operand at or inside such a path is
walked on its own. Its walk verifies the records below it, so they are
deleted when their files are gone.
#### Phases
@@ -437,13 +404,12 @@ phases.
1. `load` reads the existing records. Each record under the operands is kept in
memory by path, with its size and mtime, for the walk to compare against.
2. `walk` lists the trees under all operands. Several workers share one queue of
directories to read. A worker is one thread of the program. A thread is one
of several parts of a program that run at the same time and share its memory.
"Concurrency" says how many workers there are. Each worker reads one
directory at a time. It hands the subdirectories it finds back to the queue,
and it stats each regular file as it finds it. The walk compares each file
with its record. An unchanged file needs nothing more. Every new or changed
file is passed to the hash phase.
directories to read. A worker is one thread of the program, and "Concurrency"
says how many there are. Each worker reads one directory at a time. It hands
the subdirectories it finds back to the queue, and it stats each regular file
as it finds it. The walk compares each file with its record. An unchanged
file needs nothing more. Every new or changed file is passed to the hash
phase.
3. `hash` reads every new or changed file and computes its three hashes. Files
are read in inode order. An inode is the filesystem's own record of a file.
Files with nearby inode numbers usually lie near each other on disk. Reading
@@ -462,11 +428,10 @@ walk practical on large or busy storage pools.
#### Rules for the walk
- Only regular files are recorded. Every other kind of file, such as a symlink
or a device node, is passed over. The walk follows no symlink it finds.
or a device node, is passed over. Symlinks are never followed.
- The walk never enters a directory named `.zfs`. Such a directory holds the
filesystem's ZFS snapshots, which are its saved earlier states. Walking it
would list every file once per snapshot. `scan` never starts a walk at or
inside one either (see "Operands").
would list every file once per snapshot.
- The walk never records the scan's own database files:
- the database file;
- its `.lock` file (see "The scan lock");
@@ -480,14 +445,14 @@ walk practical on large or busy storage pools.
operand at the path given, even when that path runs through a symlinked
directory. Such an operand can lead to the database files, and the walk
passes over them there as well. The walk prints no line for the database
files and does not count them as skipped. A record that an earlier scan left
for one of them is deleted.
files and does not count them as skipped. A record an earlier scan left for
one of them is deleted.
- By default the walk enters directories on other filesystems than its
operand's. With `-x`, it never enters a directory on a different filesystem
than its operand. `scan` prints no line for a directory that only `-x` leaves
out, and does not count it as skipped. The records below it are kept, as
described under "What a scan does to the database".
than its operand. `scan` prints no line for that directory and does not count
it as skipped. The records below it are kept, as described under "What a scan
does to the database".
- The walk never records anything on a filesystem that is not persistent. A
filesystem is not persistent when the kernel makes it up or keeps it in
memory. These are the types, as `sfdupes scan --help` lists them on Linux and
@@ -504,11 +469,10 @@ walk practical on large or busy storage pools.
starts, `scan` lists the mounted filesystems and notes the device ID of each
one of those types. On Linux the list comes from `/proc/self/mountinfo`,
which gives each filesystem's own device ID. On macOS it comes from the
system's mount list. There the device ID is read from the mount point, the
directory the filesystem is mounted on. Where two filesystems share a mount
point on macOS, the one mounted last counts. On any other system the list is
empty. A list that `scan` cannot read is a fatal error, and nothing is
scanned. No flag turns this rule off.
system's mount list, and the device ID is read from the mount point. Where
two filesystems share a mount point on macOS, the one mounted last counts.
On any other system the list is empty. A list that `scan` cannot read is a
fatal error, and nothing is scanned. No flag turns this rule off.
The walk never enters a directory on one of those devices, and never records
a file on one. That holds however the walk reaches the path, through a bind
@@ -523,10 +487,11 @@ walk practical on large or busy storage pools.
The path is not counted as skipped. The records below it are kept, as
described under "What a scan does to the database".
- Below an operand, the walk does not enter a default exclude. On Linux, the
default excludes hold temporary files and the kernel's view of the running
system. They depend on the system, and `sfdupes scan --help` lists those of
the system it runs on. On Linux they are:
- Below an operand, the walk does not enter a default exclude. The default
excludes hold the kernel's view of the running system and temporary files, so
that `sfdupes scan /` walks only the files worth comparing. They depend on the
system, and `sfdupes scan --help` lists those of the system it runs on. On
Linux they are:
```
/proc, /sys, /dev, /run, /tmp, /var/tmp
@@ -555,27 +520,21 @@ walk practical on large or busy storage pools.
scan: skipped /var/tmp (default exclude)
```
The directory is not counted as skipped. `scan` prints the filesystem line
instead for a default exclude that is not on a persistent filesystem, such
as `/proc`. One that is itself an operand gets no line, because it is walked
as that operand. The scan keeps the records below a default exclude that the
The directory is not counted as skipped. A default exclude on a filesystem
that is not persistent, such as `/proc`, gets the line for its filesystem
instead. One that is itself an operand gets no line, because it is walked as
that operand. The scan keeps the records below a default exclude that the
walk did not enter, as described under "What a scan does to the database".
- `scan` skips each of these, with a one-line warning on stderr:
- a directory it cannot read;
- a path it cannot stat;
- a new or changed file it cannot read.
An unchanged file is not opened, so it keeps its record even when it has
become unreadable. The scan continues, and the summary line says how many
paths were skipped. A skipped path that has a record from an earlier scan
loses that record, and so does every path under an unreadable directory.
- A path that cannot be read, for lack of permission or any other error, gets a
one-line warning on stderr and is skipped. The scan continues, and the summary
line says how many paths were skipped. A skipped path that has a record from
an earlier scan loses that record, and so does every path under an unreadable
directory. The database mirrors what the latest scan could verify.
- A file or directory that disappears while the scan runs is not an error. The
scan prints no line for it and does not count it as skipped. A path gone when
the walk reaches it gets no record, and so does a file gone when the hash
phase opens it. The records that an earlier scan left at or below it are
deleted.
phase opens it. The records an earlier scan left at or below it are deleted.
#### Concurrency
@@ -593,8 +552,7 @@ from a thread of its own and never blocks the workers.
- one line naming the database (see "Database");
- for each dropped operand, in operand order, the warning or the line described
under "Operands". `scan` prints neither for an operand it drops only because
it repeats another or lies under another;
under "Operands". A repeated operand, or one under another, gets neither;
- the progress display of each phase (see "Progress"). `scan` prints each
warning for a skipped path as it happens. It prints a line the same way for
each path left out as a default exclude or as not on a persistent filesystem;
@@ -621,14 +579,13 @@ signal `kill` sends by default:
- The records hashed but not yet written are written, so the next scan does not
hash them again.
- Nothing else is written or deleted. Records are deleted only after a complete
walk. So the records under paths that an interrupted walk never reached are
kept.
- The database is closed and the lock released.
walk. So the records under paths an interrupted walk never reached are kept.
- The database is closed and the lock released, as on any other exit.
- The line `scan: interrupted after N files` goes to stderr. `N` is the number
of files the walk reached. The exit code is 1.
The next scan reads none of the unchanged files already recorded, and brings the
database up to date as usual.
The next scan reads none of the files already recorded and brings the database
up to date as usual.
After the first signal `scan` stops catching signals. A second one ends it at
once, as any uncaught signal does. The records not yet written are lost, and the
@@ -638,12 +595,12 @@ and `scan` leaves it ignored.
### `report` mode
`report` reads every record from the database. It takes no positional arguments.
A positional argument is anything on the command line that is not a flag, such
as the paths `scan` takes. `report` never touches the scanned filesystem. It
neither stats nor opens any path in the records. It reads the database and
writes to stdout and stderr, and nothing else. Its output is the same whether or
not the scanned filesystem is still mounted.
`report` reads every record from the database. It takes no positional arguments,
meaning nothing may follow `report` on the command line but flags. It never
touches the scanned filesystem. It neither stats nor opens any path in the
records. It reads the database and writes to stdout and stderr, and nothing
else. Its output is the same whether or not the scanned filesystem is still
mounted.
Processing:
@@ -687,8 +644,8 @@ Four bytes are written as escapes:
Every other byte is written unchanged, including bytes that are not valid UTF-8,
the usual text encoding. Undoing the four escapes gives back the stored path.
Grouping and ordering use the stored path. The warnings about skipped paths that
`scan` prints are escaped the same way, so each stays one line.
Grouping and ordering use the stored path. The warnings `scan` prints are
escaped the same way, so each warning stays one line.
The summary line on stderr gives:
@@ -726,11 +683,11 @@ Processing:
- `trees` builds the directory tree and computes every directory's digest. It
then groups the directories by digest. Every group with two or more
directories is a duplicate-tree group.
- `trees` leaves out a group that a group of its members' parents already
implies. A group is left out when its members' parents are distinct
directories that all have the same digest. The parents' group, or an
ancestor's, already implies it. A group is always reported when it holds
sibling directories, or when its members' parents have different digests.
- Only the outermost duplicate trees are reported. A group is left out when its
members' parents are distinct directories that all have the same digest. The
parents' group, or an ancestor's, already implies it. A group holding sibling
directories, or members whose parents have different digests, is always
reported.
- Within a group, paths are sorted in byte order. The first is `first`, and
every other path is a `dupe`.
- Groups are ordered by total tree size, largest first, then by `first` path.
@@ -742,8 +699,8 @@ Two limitations are accepted:
They share storage, so removing one frees nothing. The scan uses the inode to
avoid reading a file twice, and does not store it.
- Only the regular files in the database define a tree. Empty directories are
invisible. When the scan skips a file in one copy but not in the other, the
two trees differ.
invisible. A file skipped during the scan in one copy but not in the other
makes two otherwise identical trees differ.
`trees` has four phases, each with its own progress display on stderr (see
"Progress"). `open` opens the database. `read` counts the records, then reads
@@ -809,10 +766,9 @@ size of recorded files 150000000000000 (136.4 TiB)
`info` has two phases, each with its own progress display on stderr (see
"Progress"). `open` opens the database. `count` reads the schema version and
counts the records in one transaction. A transaction is a group of reads or
writes that SQLite carries out as one step, against the database as it stands at
one moment. So both describe the database at one moment, even while a scan is
writing. `info` writes its lines after the `count` display has ended.
counts the records in one transaction, a read of the database at one moment. So
the two agree with each other even while a scan is writing. `info` writes its
lines after the `count` display has ended.
### Database
@@ -820,7 +776,7 @@ All four subcommands work on one SQLite database file.
#### Location
The database is the file named by `SFDUPES_DATABASE`, when that is set and not
The database is the file `SFDUPES_DATABASE` names, when that is set and not
empty, and otherwise `/var/lib/sfdupes/db.sqlite`.
Each subcommand prints one line on stderr naming the database it uses, before
@@ -854,21 +810,20 @@ the error for a missing database comes after it.
#### The scan lock
Only one `scan` at a time can hold the lock.
Only one `scan` runs against a database at a time.
- For its whole run, `scan` holds a lock on a lock file beside the database. The
lock is taken with the `flock(2)` system call, a request a program makes to
the kernel, and only one process can hold it. The lock file's name is the
database's path, with symlinks resolved, plus `.lock`. By default that is
`/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the same
lock file.
lock is taken with the `flock(2)` system call, and only one process can hold
it. The lock file's name is the database's path, with symlinks resolved, plus
`.lock`. By default that is `/var/lib/sfdupes/db.sqlite.lock`. Every path to
one database therefore names the same lock file.
- `scan` takes the lock before it walks the filesystem or opens the database.
Just before that, it creates the database file, empty, if it is missing. So
the lock file is named the same way before the first scan as after. A symlink
at the database path is followed, and its target is created.
- A second `scan` that finds the lock held does not wait. It fails at once with
an error naming the lock file, and exits 1. It has walked nothing and has not
opened the database. The running scan carries on.
- A second `scan` against the same database does not wait. It fails at once with
a one-line error naming the lock file, and exits 1. It has walked nothing and
opened nothing. The running scan carries on.
- The lock file is created on first use and never deleted. Only its owner may
open it. A leftover file blocks nothing. The lock ends with the process that
holds it, however that process ends. Deleting the file while a scan runs would
@@ -881,9 +836,8 @@ Only one `scan` at a time can hold the lock.
SQLite appends changes to a `-wal` file beside the database, and keeps a
lookup table for that file in a `-shm` file. Readers do not block the writer,
so running a report during a scan is safe.
- Every subcommand opens the database with a busy timeout of 10 seconds. That is
how long it waits for a database another process has locked before it gives
up.
- Every connection has a busy timeout of 10 seconds. That is how long it waits
for a database another connection has locked before it gives up.
- The filesystem is authoritative, and the database follows it. Hashed records
are written 10,000 at a time while the scan runs, so a report may see a scan's
changes partly applied. A scan that dies partway leaves a valid database
@@ -908,8 +862,7 @@ Only one `scan` at a time can hold the lock.
- Then, before the walk as well, it writes:
- the `-wal` and `-shm` files, which stay beside the database while the scan
runs;
- in a new database, the table, the index and the schema version described
under "Schema".
- in a new database, the table and the index described under "Schema".
- During the `load` and `walk` phases, no records are written. A first scan's
database holds no records until the hash phase, however long its walk runs.
- During the `hash` phase, records are written 10,000 at a time.
@@ -921,15 +874,13 @@ Only one `scan` at a time can hold the lock.
#### Schema
The schema is the database's layout, its table and its index.
`PRAGMA user_version` holds the schema version, currently 1. An empty database
is at version 0, and `scan` sets it up. Any other version is a fatal error.
`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal
error too. `scan` creates the table and the index and sets the version in one
transaction (see "`info` mode"). So the three land together or not at all. A
first scan stopped while doing so leaves an empty database, which the next scan
sets up. A database at version 0 that already has a `files` table was not made
by `sfdupes`. Every subcommand refuses it, with an error telling you to remove
the file and rescan.
`PRAGMA user_version` holds the schema version, currently 1. A database with any
other version is a fatal error. `scan` creates the table and the index and sets
the version in one transaction, a group of writes that lands whole or not at
all. A first scan stopped while doing so leaves an empty database, which the
next scan sets up. A database at version 0 that already has a `files` table was
not made by `sfdupes`. Every subcommand refuses it, with an error telling you to
remove the file and rescan.
```sql
CREATE TABLE files (
@@ -952,16 +903,15 @@ CREATE INDEX files_signature ON files (size, head, tail, content);
count of nanoseconds cannot hold those. They serve change detection only and
are not part of the signature.
- `head`, `tail` and `content` each hold the raw 32 bytes of a SHA-256 (see
"Duplicate detection"). `report` and `trees` fail on a record whose hash is
not 32 bytes.
"Duplicate detection"). A record whose hash is not 32 bytes is an error.
- `files_signature` lets `report` read the records in signature order without
sorting the table.
### Progress
Each phase of every subcommand gets a display of its own on stderr from the
moment it starts. The display names the phase and shows its progress. The parts
of its line are listed below.
Every subcommand always shows that it is working. Each phase gets a display of
its own on stderr from the moment it starts. The display names the phase and
shows its progress. The parts of its line are listed below.
The phases, and what each display counts:
@@ -984,7 +934,7 @@ The phases, and what each display counts:
- `trees` has four phases:
- `[1/4] open` and `[2/4] read` are as for `report`, except that `read`
writes nothing.
- `[3/4] group` has the directories that `read` found as its total.
- `[3/4] group` has the directories `read` found as its total.
- `[4/4] write` has the duplicate-tree groups as its total.
- `info` has two phases:
- `[1/2] open` is as for `report`.
@@ -996,9 +946,8 @@ Each display is one line holding:
- the phase and its place among the phases;
- the progress, written `[m/n] x%` for `m` items done of a total `n`, which is
`x` percent of it. A phase that has no total shows the items done instead, as
`1234 files`. A phase waiting for its total shows what it waits for, as
`counting the records`, and no rate;
`x` percent of it. Before a phase has a total, the line shows the items done
instead, or what the phase is waiting for;
- the current rate, in the items the phase counts, such as files or records. It
covers the items completed over the last 5 seconds, or, when stderr is not a
terminal, since the line before. It falls to 0 while nothing completes. A
@@ -1022,9 +971,9 @@ phase waits, in place of the rate. That happens while one large file is read, or
while a disk or the database is slow. Each phase names the places it may wait
in, and a phase that may wait in two places names both, as `disk/db`:
- `walk` says `disk`, because it reads directories and stats files.
- `hash` says `disk/db`, because it reads files and writes their records to the
database between them.
- `walk` says `disk`, for files.
- `hash` says `disk/db`: it reads files and writes their records to the database
between them.
- `load`, `update` and the `read` phase of `trees` say `db`.
- The `read` phase of `report` says `db/out`, because it writes the report to
stdout while it reads.
@@ -1050,18 +999,18 @@ On a terminal:
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
```
- No line reaches the terminal's last column. A line too wide for the terminal
first leaves out its rate, the words of a stalled phase, or what the phase is
waiting for. It leaves out its bar next. It is cut short at the edge only when
the rest does not fit on its own. The rest is:
- No line reaches the terminal's last column, however narrow the terminal is. A
line too wide for the terminal first leaves out its rate, or the words shown
in the rate's place. It leaves out its bar next. It is cut short at the edge
only when the four remaining parts do not fit on their own. Those are:
- the phase and its place;
- the progress, when the phase shows one;
- the progress;
- the elapsed time;
- the time remaining.
On a terminal 80 columns wide, no line is cut short, even hours into a scan
of millions of files. A terminal made narrower during a phase is fitted at
the next draw.
That happens hours into a scan of millions of files, on a terminal narrower
than about 75 columns. A terminal made narrower while a phase runs is fitted
at the next draw.
- A warning printed during a phase lands on a line of its own, never inside the
display. So does every line written to stdout when stdout and stderr are the
@@ -1077,8 +1026,8 @@ Each phase prints its line:
- every 5 seconds while it runs, whether or not an item has completed;
- once more as it ends.
When a phase stops short of its total, as an interrupted one does, it keeps
showing the count it stopped at.
A phase that stops short of its total, as an interrupted one does, keeps showing
the count it stopped at.
### Error handling and exit codes
@@ -1087,15 +1036,13 @@ showing the count it stopped at.
error is one of these:
- an operand that does not exist, or whose details `scan` cannot read;
- a list of mounted filesystems that `scan` cannot read;
- another `scan` holding the lock (see "The scan lock");
- a database, its directory or its lock file that cannot be created or
opened;
- a read or write of the database that fails;
- another `scan` already running against the same database;
- a database that cannot be created, opened, read or written, its directory
and its lock file included;
- a relative database path when the working directory cannot be read;
- a database at a schema version other than 0 or 1;
- a database at version 0 read by `report`, `trees` or `info`;
- a database at version 0 that already has a `files` table (see "Schema");
- a record with a hash that is not 32 bytes, read by `report` or `trees`;
- a database with another schema version, or one at version 0 that already
has a `files` table;
- a record whose `head`, `tail` or `content` is not 32 bytes;
- a missing database for `report`, `trees` or `info`;
- a failed write to stdout.
- `2`: a usage error, which is one of these:
@@ -1107,9 +1054,9 @@ showing the count it stopped at.
- `--workers` below 1;
- `report`, `trees` or `info` with a positional argument.
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
stdout, such as a full disk, is reported the same way. Two cases never reach
`sfdupes` as a failed write:
A fatal error is reported in one line on stderr that starts with `sfdupes:`. A
failed write to stdout, such as a full disk, is reported the same way. Two cases
never reach `sfdupes` as a failed write:
- The reader of a stdout pipe may exit early, as `head` does in
`sfdupes report | head`. The next write then ends `sfdupes` with `SIGPIPE`,
@@ -1126,8 +1073,8 @@ This repository follows the
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
standard. The executables in `script/` are the entrypoints of the development
workflow, and the `Makefile` targets call them, except `build` and `clean` (see
"Build"). Every script runs under plain `sh`, the standard Unix shell. It finds
the repository root itself, so it can be run from any directory.
"Build"). Every script is POSIX `sh`. It finds the repository root itself, so it
can be run from any directory.
- `script/bootstrap` installs what building and developing need. It can be run
again at any time, and installed tools are kept. `git`, `make` and `go` are
@@ -1139,16 +1086,14 @@ the repository root itself, so it can be run from any directory.
- `apk`.
Then come `node` and `yarn`. `prettier`, the Markdown formatter, is a `node`
program. `yarn` is the package manager that installs it:
program, and `yarn` is the package manager that installs it:
- An installed `node` is used when its major version is 22. A `yarn` already
on `PATH` is then used as it is. When `yarn` is missing, `yarn` 1.22.22 is
activated through `corepack`, the `yarn` installer that ships with `node`.
When there is no `corepack` either, the script installs `yarn` with `npm`,
the package manager that ships with `node`.
- Otherwise the script installs `node` 22.17.0 through `nvm`, a `node`
version manager, and activates `yarn` 1.22.22 under that `node` through
`corepack`. When `nvm` is missing, the script first downloads it and
checks the archive's SHA-256.
When there is no `corepack` either, `npm` installs it.
- Otherwise the script downloads `nvm`, a `node` version manager, and checks
the archive's SHA-256. It installs `node` 22.17.0 through `nvm`, and
activates `yarn` 1.22.22 under that `node` through `corepack`.
`yarn install --frozen-lockfile` then installs the `prettier` that
`package.json` and `yarn.lock` pin. The linter is never installed. It runs
@@ -1164,22 +1109,18 @@ the repository root itself, so it can be run from any directory.
unsynchronized access to memory from several threads. The suite runs with a
90-second timeout, and its coverage of the code is measured. On failure it
reruns the suite verbosely, so that the log names the failing test. It builds
only the `test` stage of the `Dockerfile` and writes no image. A stage is one
named part of the `Dockerfile`, which starts from an image of its own and can
be built alone. The race detector needs cgo, Go's bridge to C code, and a C
compiler. The stage therefore starts from a Debian `golang` image, which has
the C compiler `gcc`. The image is pinned by its digest, its content hash. The
tests run as the user `nobody`, because several of them make a file
unreadable, and root reads it anyway.
only the `test` stage of the `Dockerfile` and writes no image. The race
detector needs cgo, Go's bridge to C code, and a C compiler. The stage
therefore starts from a Debian `golang` image, which has `gcc`. The image is
pinned by its digest, its content hash. The tests run as `nobody`, because
several of them make a file unreadable, and root reads it anyway.
- `script/lint` runs the linter, `golangci-lint`, which checks the Go sources
for common mistakes. It builds only the `lint` stage of the `Dockerfile` and
writes no image. The stage starts from the `golangci/golangci-lint` image,
pinned by its digest, and runs these checks as build steps:
- the `gofmt` check, which fails when `gofmt`, Go's own formatter, would
change a Go source;
- `golangci-lint config verify`, which checks the linter's settings file
`.golangci.yml`;
- `golangci-lint run`, which lints the Go sources.
pinned by its digest, and runs three build steps:
- the `gofmt` check;
- `golangci-lint config verify`;
- `golangci-lint run`.
A successful build is a clean lint. The linter never runs on the host, so a
working `docker` is the one thing linting needs.
@@ -1188,8 +1129,8 @@ the repository root itself, so it can be run from any directory.
with `prettier`, in place. `prettier` indents by 4 spaces and wraps prose, as
`.prettierrc` sets and as the script also passes on the command line. It wraps
at 80 columns, its default width. Both run on the host. `prettier` runs
through `yarn` at the version that `yarn.lock` pins. When the `node` on `PATH`
is not `node` 22, the script loads the `node` 22.17.0 that `script/bootstrap`
through `yarn` at the version `yarn.lock` pins. When the `node` on `PATH` is
not `node` 22, the script loads the `node` 22.17.0 that `script/bootstrap`
installed.
- `script/fmt-check` is the read-only form of `script/fmt`. It names any
unformatted file and exits non-zero instead of writing. `gofmt` and `prettier`
@@ -1199,19 +1140,17 @@ the repository root itself, so it can be run from any directory.
- `script/test` and `script/lint`, which need `docker`;
- `script/fmt-check`, which needs what `script/bootstrap` installs.
- `script/docker` builds the Docker image, tagged with the name from
`script/projectname`. It passes the output of
`git describe --tags --always --dirty`, or `unknown` when that is empty, to
the `Dockerfile` as the build argument `VERSION`. A build argument is a value
that a `docker build` hands to the `Dockerfile`. The build stage depends on
the `lint` and `test` stages, so this also runs the tests and the linter.
`script/projectname`. It passes `git describe --tags --always --dirty`, or
`unknown` when that is empty, as the `VERSION` build argument. The build stage
depends on the `lint` and `test` stages, so this also runs the tests and the
linter.
- `script/cibuild` runs `script/bootstrap` and then `script/check`. Then it
builds the image as `script/docker` does. The Gitea workflow in
`.gitea/workflows/check.yml` runs it on every push. A successful run means
every check passed. The tests and the linter run twice, once in `script/check`
and again as stages of the image build.
- `script/precommit` runs from the `git` pre-commit hook, a script that `git`
runs before each commit. It runs `go mod tidy`, and fails the commit when that
changes `go.mod` or `go.sum`. Then it runs `script/check`.
builds the image as `script/docker` does. The repository's CI workflow on
Gitea runs it on every push, and a successful run means every check passed.
The tests and the linter run twice, once in `script/check` and again as stages
of the image build.
- `script/precommit` runs from the `git` pre-commit hook. It fails the commit
when `go mod tidy` would change anything. Then it runs `script/check`.
- `script/install-precommit` installs `.git/hooks/pre-commit`, which runs
`script/precommit`.
@@ -1221,9 +1160,9 @@ install system packages and download the Go modules. Every step that checks or
compiles the code comes after it. Every `RUN` step after it runs again on each
build. Without that, Docker would serve the test and lint steps from its cache
on an unchanged tree, and the build would pass without running them. The steps
above `ARG CACHEBUST` stay cached until a change to something they depend on,
such as `go.mod`. Go's build cache is kept between builds, so only what changed
is compiled again. The tests run with `-count=1`, so no test reports a stored
above `ARG CACHEBUST` stay cached until something they depend on changes, such
as `go.mod`. Go's build cache is kept between builds, so only what changed is
compiled again. The tests run with `-count=1`, so no test reports a stored
result. `golangci-lint run` keeps the linter's own cache as well, so an
unchanged package is not linted again.
@@ -1255,19 +1194,15 @@ commands of their own:
- The language is Go. The module is `sneak.berlin/go/sfdupes`, and the binary is
`sfdupes`.
- The code imports these modules beyond the standard library:
- The dependencies beyond the standard library are:
- `github.com/spf13/cobra`, for the command line;
- `github.com/schollz/progressbar/v3`, for the progress display;
- `golang.org/x/term`, to tell whether stderr is a terminal;
- `modernc.org/sqlite`, the SQLite driver;
- `golang.org/x/sys`, for the scan lock and the list of mounted filesystems.
No other dependency outside the standard library is added.
- The code stays pure Go, with no C code. Building on one system for another is
not a goal. `modernc.org/sqlite` is pure Go too, so the binary builds without
cgo. The `Makefile` exports `CGO_ENABLED=0`. The test build is the one
exception, because the race detector needs cgo (see `script/test`).
- `modernc.org/sqlite` is pure Go, so the binary builds without cgo. The
`Makefile` exports `CGO_ENABLED=0`. The test build is the one exception,
because the race detector needs cgo (see `script/test`).
- `report` and `trees` give the same output for the same database contents,
whatever order the records were inserted in.
@@ -1277,15 +1212,12 @@ All of the following, run in this directory, must pass:
1. `make check` passes.
2. `make docker` succeeds.
3. The test below behaves as described. It builds a throwaway tree in a new
directory under `/var/tmp`, which is on a persistent filesystem. On many
systems `/tmp` is a `tmpfs`, which `scan` leaves out. Never test against real
data.
3. The test below behaves as described. It builds a throwaway tree in a
temporary directory. Never test against real data.
```sh
make build
d=$(mktemp -d /var/tmp/sfdupes.XXXXXX)
export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite"
d=$(mktemp -d)
export SFDUPES_DATABASE="$(mktemp -d)/db.sqlite"
mkdir -p "$d/a" "$d/b"
head -c 2000 /dev/urandom > "$d/a/one.bin"
cp "$d/a/one.bin" "$d/b/copy.bin"
@@ -1336,11 +1268,11 @@ All of the following, run in this directory, must pass:
## TODO
Tracked in [`TODO.md`](TODO.md).
Tracked in [TODO.md](TODO.md).
## License
MIT. See [`LICENSE`](LICENSE).
MIT. See [LICENSE](LICENSE).
## Author
+3 -9
View File
@@ -32,15 +32,9 @@
- `README.md` rewritten from scratch, sentence by sentence, as a newcomer reads
it. It leads with what the program is and how to use it. A reference section
covers every setting and every subcommand. Development and testing come last.
Every statement was checked against the code on `next`. Five comments in
`main_test.go` cite "Settings", where their rules now live. One comment each
in `scan_test.go`, `trees_test.go` and `exclude_test.go` cites what the README
now says (2026-10-26, https://git.eeqj.de/sneak/sfdupes/issues/135)
- an operand at or inside a directory named `.zfs` is never scanned, whether
given alone or under another operand. `scan` warns about it, counts it as
skipped and does not walk it (2026-10-26,
https://git.eeqj.de/sneak/sfdupes/issues/142)
Every statement was checked against the code on `next`. Three comments in
`main_test.go` cite "Settings", where their rules now live (2026-10-26,
https://git.eeqj.de/sneak/sfdupes/issues/135)
- a file or directory removed while the scan runs is not an error, in the walk
or in the hash phase. The scan prints no line for it and does not count it as
+1 -3
View File
@@ -17,9 +17,7 @@ import (
func TestDefaultExcludes(t *testing.T) {
t.Parallel()
// README "Rules for the walk": the Linux and macOS lists, and none
// on any other system. The Windows list exists in the code only: the
// binary does not build for Windows, so the README leaves it out.
// README "Rules for the walk": each system's list.
cases := map[string][]string{
goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
goosDarwin: {
+21 -132
View File
@@ -393,7 +393,7 @@ func TestRunNamesRelativeDatabase(t *testing.T) {
}
func TestRunUsesDatabaseThroughSymlinkedDirectory(t *testing.T) {
// README §Settings: the path names the file exactly. With link a
// README §Database: the path names the file exactly. With link a
// symlink to the directory real/sub, link/../db.sqlite names
// real/db.sqlite, not db.sqlite beside link. Every subcommand must
// use that file, relative or absolute, and its first line must name
@@ -773,8 +773,8 @@ func TestBuildVersion(t *testing.T) {
func TestRunVersionWriteFailureIsFatal(t *testing.T) {
t.Parallel()
// README §Error handling: a failed write to stdout is a fatal error,
// exit 1, reported on stderr starting with "sfdupes:".
// README §Error handling: a stdout write failure exits 1, reported
// in one line on stderr.
var stderr bytes.Buffer
code := run([]string{"--version"}, failingWriter{}, &stderr)
@@ -928,21 +928,24 @@ func TestRunScanWalksOperandUnderSymlinkOperand(t *testing.T) {
}
func TestRunScanSkipsZFSOperand(t *testing.T) {
zfs := filepath.Join(t.TempDir(), ".zfs")
f := writeFile(t, zfs, "snapshot/hourly/f", pattern(1, 10))
// No scan records a file inside a .zfs directory, so the record
// beneath the operand is written straight into the database.
path := seedDatabase(t, []scanRec{{size: 10, path: f}})
path := testDBPath(t)
t.Setenv(databaseEnv, path)
zfs := filepath.Join(t.TempDir(), ".zfs")
snapshot := filepath.Join(zfs, "snapshot", "hourly")
f := writeFile(t, snapshot, "f", pattern(1, 10))
// An operand beneath a .zfs directory is walked, because it is not
// itself named .zfs.
scanOK(t, snapshot)
assertOperandSkipped(t, path, zfs, ".zfs directory", f)
}
// assertOperandSkipped scans operand alone and checks that it is skipped
// as kind: a warning naming it, one skip in the summary, exit 0, and the
// record for kept, stored beneath operand before this scan, still in the
// database at dbPath.
// record for kept, which an earlier scan stored beneath operand, still
// in the database at dbPath.
func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
kept string,
) {
@@ -971,121 +974,6 @@ func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
recordByPath(t, dbRecords(t, db), kept)
}
// TestRunScanSkipsOperandInsideZFS scans a directory inside a .zfs
// directory, first alone and then under another operand. Each time it
// is warned about and counted as skipped, and nothing in it is walked.
// The other operand is scanned as usual.
func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := t.TempDir()
hourly := filepath.Join(dir, ".zfs", "snapshot", "hourly")
writeFile(t, hourly, "f", pattern(1, 10))
f := writeFile(t, dir, "f", pattern(2, 10))
warning := "walk " + hourly + ": skipping operand inside a .zfs directory\n"
for _, tc := range []struct {
operands []string
summary string
}{
{
[]string{hourly},
"scan: 0 files seen (0 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n",
},
{
[]string{dir, hourly},
"scan: 1 files seen (1 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n",
},
} {
stderr := scanOK(t, tc.operands...)
if !strings.Contains(stderr, warning) ||
!strings.Contains(stderr, tc.summary) {
t.Errorf("scan %q: stderr = %q, want %q and %q", tc.operands,
stderr, warning, tc.summary)
}
}
db, err := openDB(path, reportParams)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = db.Close() })
if got := recordPaths(dbRecords(t, db)); !slices.Equal(got, []string{f}) {
t.Errorf("records = %q, want %q", got, []string{f})
}
}
// TestRunScanSkipsOperandThroughSymlinkIntoZFS scans snaps/daily, where
// snaps is a symlink to pool/.zfs/snapshot. The operand is warned about
// by the path given, counted as skipped, and not walked.
func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
t.Setenv(databaseEnv, testDBPath(t))
dir := t.TempDir()
writeFile(t, dir, "pool/.zfs/snapshot/daily/f", pattern(1, 10))
snaps := filepath.Join(dir, "snaps")
err := os.Symlink(filepath.Join(dir, "pool", ".zfs", "snapshot"), snaps)
if err != nil {
t.Fatal(err)
}
daily := filepath.Join(snaps, "daily")
stderr := scanOK(t, daily)
warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
"0 unchanged), 1 skipped\n"
if !strings.Contains(stderr, warning) ||
!strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q and %q", stderr, warning, summary)
}
}
// TestRunScanWalksOperandThroughSymlink scans data/daily, where data is
// a symlink to a directory outside any .zfs directory. The operand is
// scanned as usual, and its file is recorded by the path given.
func TestRunScanWalksOperandThroughSymlink(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
dir := t.TempDir()
writeFile(t, dir, "pool/data/daily/f", pattern(1, 10))
data := filepath.Join(dir, "data")
err := os.Symlink(filepath.Join(dir, "pool", "data"), data)
if err != nil {
t.Fatal(err)
}
daily := filepath.Join(data, "daily")
stderr := scanOK(t, daily)
summary := "scan: 1 files seen (1 added, 0 updated, 0 removed, " +
"0 unchanged), 0 skipped\n"
if !strings.Contains(stderr, summary) {
t.Errorf("stderr = %q, want %q", stderr, summary)
}
db, err := openDB(path, reportParams)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = db.Close() })
recordByPath(t, dbRecords(t, db), filepath.Join(daily, "f"))
}
func TestRunReportSucceeds(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
@@ -1234,7 +1122,7 @@ func assertRunsUseDatabase(t *testing.T, dir, name string) {
}
func TestRunDatabasePathUsedAsGiven(t *testing.T) {
// README §Settings: the path names the database file exactly. In
// README §Database: the path names the database file exactly. In
// SQLite's connection string an unescaped ? or # would end the file
// name and % would start an escape, and a path starting with //
// could be read as a host name. %25 is a valid escape, so unescaped
@@ -1279,8 +1167,9 @@ func holdScanLock(t *testing.T, path string) {
}
func TestRunSecondScanFails(t *testing.T) {
// README §Database: a second scan that finds the lock held fails
// at once, naming the lock file, without opening the database.
// README §Database: while one scan holds the lock, a second scan
// fails at once, naming the lock file, without opening the
// database.
path := testDBPath(t)
t.Setenv(databaseEnv, path)
@@ -1339,7 +1228,7 @@ func assertSecondScanFails(t *testing.T, path string) {
}
func TestRunSecondScanThroughSymlinkFails(t *testing.T) {
// README §Database: a path through a symlink names the same lock
// README §Database: every path to one database names the same lock
// file, so while a scan holds it through one path, a scan through
// another fails, naming the lock file beside the database's real
// path.
@@ -1444,8 +1333,8 @@ func TestRunReportsDuringScan(t *testing.T) {
}
func TestRunStdoutClosedIsFatal(t *testing.T) {
// README §Error handling: a failed write to stdout is a fatal error,
// exit 1, reported on stderr starting with "sfdupes:".
// README §Error handling: a stdout write failure exits 1, reported
// in one line on stderr.
for _, name := range []string{cmdReport, cmdTrees, cmdInfo} {
t.Run(name, func(t *testing.T) {
t.Setenv(databaseEnv, testDBPath(t))
+12 -28
View File
@@ -383,8 +383,7 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
) error {
// Types and filesystems are checked before pruning so that an
// operand under a dropped one is still scanned, not dropped as lying
// under it, and so that an operand inside a .zfs directory gets its
// warning even when it lies under another operand.
// under it.
roots = pruneRoots(s.walkableRoots(roots, rules), oneFS, rules,
deviceOfPath)
@@ -412,15 +411,15 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
}
// walkableRoots returns the operands the walk can start from: regular
// files, and directories not named .zfs, none of them inside a
// directory named .zfs. Every other operand is warned about, counted as
// skipped, and dropped. A dropped operand is no longer a root, so the
// records stored beneath it are treated as if it had not been given:
// those under no other root count as outside the roots and are not
// deleted as unverified. An operand on a filesystem in rules that is not
// persistent is dropped the same way, with a line saying so that is not
// a warning, and is not counted as skipped. An operand that fails lstat
// here is kept, and the walk warns about it unless it no longer exists.
// files, and directories not named .zfs. Every other operand is warned
// about, counted as skipped, and dropped. A dropped operand is no
// longer a root, so the records stored beneath it are treated as if it
// had not been given: those under no other root count as outside the
// roots and are not deleted as unverified. An operand on a filesystem
// in rules that is not persistent is dropped the same way, with a line
// saying so that is not a warning, and is not counted as skipped. An
// operand that fails lstat here is kept, and the walk warns about it
// unless it no longer exists.
func (s *scanState) walkableRoots(roots []string, rules walkRules) []string {
kept := make([]string, 0, len(roots))
@@ -875,24 +874,9 @@ func sendEvent(ctx context.Context, events chan<- walkEvent,
// operandWarning returns the one-line warning for an operand the walk
// does not start from, naming the path and what it is, or "" for one it
// does: a regular file, or a directory not named .zfs, that is not
// inside a directory named .zfs, neither as given nor with its symlinks
// resolved. Symlinks are never followed, including as operands.
// does: a regular file, or a directory not named .zfs. Symlinks are
// never followed, including as operands.
func operandWarning(root string, fi fs.FileInfo) string {
// The walk never enters a .zfs directory, so it starts from nothing
// inside one either, not even through a symlink. An operand whose
// symlinks cannot be resolved is checked as given.
resolved, err := filepath.EvalSymlinks(root)
if err != nil {
resolved = root
}
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") ||
slices.Contains(strings.Split(filepath.Dir(resolved), "/"), ".zfs") {
return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
root)
}
var kind string
switch mode := fi.Mode(); {
+2 -3
View File
@@ -1064,9 +1064,8 @@ func TestScanOneFilesystemWalksNestedFileOperand(t *testing.T) {
}
}
// buildSmokeTree recreates the tree of the test under README
// "Definition of done" with deterministic content and returns the tree
// root.
// buildSmokeTree recreates the README smoke-test filesystem layout
// with deterministic content and returns the tree root.
func buildSmokeTree(t *testing.T) string {
t.Helper()
+2 -3
View File
@@ -15,9 +15,8 @@ import (
const treesOpenLine = "[1/4] open: opening the database, elapsed 0s, " +
"remain unknown"
// smokeTreeRecs mirrors the tree of the test under README "Definition
// of done": /d/t1 and /d/t2 are identical, /d/t3 differs from them only
// by one filename.
// smokeTreeRecs mirrors the README smoke-test tree layout: /d/t1 and
// /d/t2 are identical, /d/t3 differs from them only by one filename.
func smokeTreeRecs() []scanRec {
f1Head, f1Tail, f1Content := testHash("f1h"), testHash("f1t"), testHash("f1c")
f2Head, f2Tail, f2Content := testHash("f2h"), testHash("f2t"), testHash("f2c")