Compare commits

..
1 Commits
Author SHA1 Message Date
sneak b641024fc0 Rewrite README.md sentence by sentence for a newcomer
check / check (push) Canceled after 0s
Rewrites `README.md` from scratch for
#135. Every statement was checked
against the code on `next` and against what the built binary prints on
small trees under `/var/tmp`. The headings that the code comments cite
are kept.

Where the old text and the repository disagreed, the new text states
what the code does: `@latest` installs the newest tagged release, ten
million files in both "Rationale" and "Design", no Windows excludes, a
relative operand is joined to the working directory as `pwd` prints it,
and an operand reached through a symlinked directory below another
operand is dropped (#144).

Dropped from "Constraints": the permission to add `github.com/spf13/viper`
if configuration-file support is ever needed.

Model: fable-5-1
2026-10-26 20:38:16 +00:00
+33 -32
View File
@@ -154,8 +154,8 @@ to keep is your choice.
A row is a candidate, not proof:
- The reports show the files as they were at the last scan. A file may have
changed or gone since.
- The reports show each file as it was when a scan last recorded it. A file may
have changed or gone since.
- A file of 100 MiB or more is compared only on samples of its content (see
"Duplicate detection").
- Two paths that are hard links, two names for one file, are listed as
@@ -225,10 +225,9 @@ In order:
its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read
in full. The target is a filesystem of about ten million files and 150 TB, on
disks that may be slow or busy. During a scan, one small record per file is
held in memory. It holds the path and the two values the walk compares, the
size and the mtime. The hashes stay in the database. `report` holds the
duplicate paths of one file size at a time. `trees` holds one entry per
directory, plus the files of the directories it is reading.
held in memory. The hashes stay in the database. `report` holds the duplicate
paths of one file size at a time. `trees` holds one entry per directory, plus
the files of the directories it is reading.
3. **Rescan cheaply, analyze from the database.** The scan keeps its database
between runs, and an unchanged file is never read again. `report`, `trees`
and `info` work from the database alone and never touch the scanned
@@ -344,9 +343,13 @@ default. An operand may be a directory or a regular file.
the exit code is 2.
- An operand that does not exist is a fatal error. `scan` then stops before it
walks anything, and exits 1.
- Each operand is made absolute before the walk. A `.` or `..` in it is resolved
as text, and symlinks are not followed. Each record is stored under that
absolute path, so results do not depend on the working directory.
- Each operand is made absolute before the walk. A relative operand is joined to
the working directory as `pwd` prints it, with its symlinks kept. A `.` or
`..` in it is resolved as text, and symlinks are not followed. Each record is
stored under that absolute path. After a `cd` through a symlink, `scan .`
names the symlink and is skipped as a symlink operand. `scan sub` in that
directory records its files under the symlink's path. `report` can then list
each as a duplicate of the same file recorded under its real path.
- All operands belong to one scan, and one walk covers them all at once (see
"Concurrency").
- An operand that repeats another is dropped before the walk, without a line.
@@ -360,8 +363,7 @@ default. An operand may be a directory or a regular file.
It is also kept when a directory between the two is on a filesystem that is
not persistent. Such a filesystem is one the kernel makes up or keeps in
memory (see "Rules for the walk"). The outer operand's walk does not enter
such a directory, so the inner operand is walked on its own. For an inner
operand that is a regular file, the check starts at its directory.
such a directory, so the inner operand is walked on its own.
- No other case keeps an operand under another. An operand reached through a
symlinked directory below the outer operand is dropped too, without a line.
@@ -401,8 +403,8 @@ operands:
- The scan deletes each record under a scanned operand whose file it could not
verify. That covers deleted files, and paths the scan could not stat or read
this run. To stat a file is to read its details, such as its size, without
opening it. So the database holds only what the latest scan over each path
could verify. A later successful scan adds such files again.
opening it. So the database holds only what the latest complete scan over each
path could verify. A later successful scan adds such files again.
- Records outside the scanned operands are left alone, so separate trees can be
scanned on different schedules into one database.
- The scan keeps every record at or below a path that the walk leaves out for
@@ -413,10 +415,9 @@ operands:
Such records count as outside the scanned operands. So with `-x`, a
filesystem mounted under an operand can be scanned on its own schedule into
the same database. An operand may itself lie at or inside a default exclude.
With `-x`, it may also lie on another filesystem than an operand above it.
Such an operand is walked on its own. Its walk verifies the records below
it, so they are deleted when their files are gone.
the same database. An operand may itself lie at or inside such a path, as
"Operands" describes. Its walk verifies the records below it, so they are
deleted when their files are gone.
#### Phases
@@ -450,7 +451,7 @@ walk practical on large or busy storage pools.
#### Rules for the walk
- Only regular files are recorded. Every other kind of file, such as a symlink
or a device node, is passed over. Symlinks are never followed.
or a device node, is passed over. The walk follows no symlink it finds.
- The walk never enters a directory named `.zfs`. Such a directory holds the
filesystem's ZFS snapshots, which are its saved earlier states. Walking it
would list every file once per snapshot. `scan` never starts a walk at or
@@ -554,7 +555,7 @@ walk practical on large or busy storage pools.
one-line warning on stderr and is skipped. The scan continues, and the summary
line says how many paths were skipped. A skipped path that has a record from
an earlier scan loses that record, and so does every path under an unreadable
directory. The database mirrors what the latest scan could verify.
directory. The database mirrors what the latest complete scan could verify.
- A file or directory that disappears while the scan runs is not an error. The
scan prints no line for it and does not count it as skipped. A path gone when
the walk reaches it gets no record, and so does a file gone when the hash
@@ -607,7 +608,7 @@ signal `kill` sends by default:
- Nothing else is written or deleted. Records are deleted only after a complete
walk. So the records under paths that an interrupted walk never reached are
kept.
- The database is closed and the lock released, as on any other exit.
- The database is closed and the lock released.
- The line `scan: interrupted after N files` goes to stderr. `N` is the number
of files the walk reached. The exit code is 1.
@@ -941,9 +942,9 @@ CREATE INDEX files_signature ON files (size, head, tail, content);
### Progress
Every subcommand shows that it is working: each phase gets a display of its own
on stderr from the moment it starts. The display names the phase and shows its
progress. The parts of its line are listed below.
Each phase of every subcommand gets a display of its own on stderr from the
moment it starts. The display names the phase and shows its progress. The parts
of its line are listed below.
The phases, and what each display counts:
@@ -1032,11 +1033,10 @@ On a terminal:
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
```
- No line reaches the terminal's last column, however narrow the terminal is. A
line too wide for the terminal first leaves out its rate, the words of a
stalled phase, or what the phase is waiting for. It leaves out its bar next.
It is cut short at the edge only when the rest does not fit on its own. The
rest is:
- No line reaches the terminal's last column. A line too wide for the terminal
first leaves out its rate, the words of a stalled phase, or what the phase is
waiting for. It leaves out its bar next. It is cut short at the edge only when
the rest does not fit on its own. The rest is:
- the phase and its place;
- the progress, when the phase shows one;
- the elapsed time;
@@ -1070,13 +1070,13 @@ the count it stopped at.
error is one of these:
- an operand that does not exist, or whose details `scan` cannot read;
- a list of mounted filesystems that `scan` cannot read;
- another `scan` already running against the same database;
- another `scan` holding the lock (see "The scan lock");
- a database that cannot be created or opened, or whose reads or writes
fail, its directory and its lock file included;
- a relative database path when the working directory cannot be read;
- a database with another schema version, or one at version 0 that `scan` is
not setting up: a database read by `report`, `trees` or `info`, or one
that already has a `files` table (see "Schema");
- a database at a schema version other than 0 or 1;
- a database at version 0 read by `report`, `trees` or `info`;
- a database at version 0 that already has a `files` table (see "Schema");
- a record with a hash that is not 32 bytes, read by `report` or `trees`;
- a missing database for `report`, `trees` or `info`;
- a failed write to stdout.
@@ -1265,6 +1265,7 @@ All of the following, run in this directory, must pass:
data.
```sh
make build
d=$(mktemp -d /var/tmp/sfdupes.XXXXXX)
export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite"
mkdir -p "$d/a" "$d/b"