Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
b641024fc0 |
@@ -154,8 +154,8 @@ to keep is your choice.
|
|||||||
|
|
||||||
A row is a candidate, not proof:
|
A row is a candidate, not proof:
|
||||||
|
|
||||||
- The reports show the files as they were at the last scan. A file may have
|
- The reports show each file as it was when a scan last recorded it. A file may
|
||||||
changed or gone since.
|
have changed or gone since.
|
||||||
- A file of 100 MiB or more is compared only on samples of its content (see
|
- A file of 100 MiB or more is compared only on samples of its content (see
|
||||||
"Duplicate detection").
|
"Duplicate detection").
|
||||||
- Two paths that are hard links, two names for one file, are listed as
|
- Two paths that are hard links, two names for one file, are listed as
|
||||||
@@ -225,10 +225,9 @@ In order:
|
|||||||
its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read
|
its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read
|
||||||
in full. The target is a filesystem of about ten million files and 150 TB, on
|
in full. The target is a filesystem of about ten million files and 150 TB, on
|
||||||
disks that may be slow or busy. During a scan, one small record per file is
|
disks that may be slow or busy. During a scan, one small record per file is
|
||||||
held in memory. It holds the path and the two values the walk compares, the
|
held in memory. The hashes stay in the database. `report` holds the duplicate
|
||||||
size and the mtime. The hashes stay in the database. `report` holds the
|
paths of one file size at a time. `trees` holds one entry per directory, plus
|
||||||
duplicate paths of one file size at a time. `trees` holds one entry per
|
the files of the directories it is reading.
|
||||||
directory, plus the files of the directories it is reading.
|
|
||||||
3. **Rescan cheaply, analyze from the database.** The scan keeps its database
|
3. **Rescan cheaply, analyze from the database.** The scan keeps its database
|
||||||
between runs, and an unchanged file is never read again. `report`, `trees`
|
between runs, and an unchanged file is never read again. `report`, `trees`
|
||||||
and `info` work from the database alone and never touch the scanned
|
and `info` work from the database alone and never touch the scanned
|
||||||
@@ -344,9 +343,13 @@ default. An operand may be a directory or a regular file.
|
|||||||
the exit code is 2.
|
the exit code is 2.
|
||||||
- An operand that does not exist is a fatal error. `scan` then stops before it
|
- An operand that does not exist is a fatal error. `scan` then stops before it
|
||||||
walks anything, and exits 1.
|
walks anything, and exits 1.
|
||||||
- Each operand is made absolute before the walk. A `.` or `..` in it is resolved
|
- Each operand is made absolute before the walk. A relative operand is joined to
|
||||||
as text, and symlinks are not followed. Each record is stored under that
|
the working directory as `pwd` prints it, with its symlinks kept. A `.` or
|
||||||
absolute path, so results do not depend on the working directory.
|
`..` in it is resolved as text, and symlinks are not followed. Each record is
|
||||||
|
stored under that absolute path. After a `cd` through a symlink, `scan .`
|
||||||
|
names the symlink and is skipped as a symlink operand. `scan sub` in that
|
||||||
|
directory records its files under the symlink's path. `report` can then list
|
||||||
|
each as a duplicate of the same file recorded under its real path.
|
||||||
- All operands belong to one scan, and one walk covers them all at once (see
|
- All operands belong to one scan, and one walk covers them all at once (see
|
||||||
"Concurrency").
|
"Concurrency").
|
||||||
- An operand that repeats another is dropped before the walk, without a line.
|
- An operand that repeats another is dropped before the walk, without a line.
|
||||||
@@ -360,8 +363,7 @@ default. An operand may be a directory or a regular file.
|
|||||||
It is also kept when a directory between the two is on a filesystem that is
|
It is also kept when a directory between the two is on a filesystem that is
|
||||||
not persistent. Such a filesystem is one the kernel makes up or keeps in
|
not persistent. Such a filesystem is one the kernel makes up or keeps in
|
||||||
memory (see "Rules for the walk"). The outer operand's walk does not enter
|
memory (see "Rules for the walk"). The outer operand's walk does not enter
|
||||||
such a directory, so the inner operand is walked on its own. For an inner
|
such a directory, so the inner operand is walked on its own.
|
||||||
operand that is a regular file, the check starts at its directory.
|
|
||||||
|
|
||||||
- No other case keeps an operand under another. An operand reached through a
|
- No other case keeps an operand under another. An operand reached through a
|
||||||
symlinked directory below the outer operand is dropped too, without a line.
|
symlinked directory below the outer operand is dropped too, without a line.
|
||||||
@@ -401,8 +403,8 @@ operands:
|
|||||||
- The scan deletes each record under a scanned operand whose file it could not
|
- The scan deletes each record under a scanned operand whose file it could not
|
||||||
verify. That covers deleted files, and paths the scan could not stat or read
|
verify. That covers deleted files, and paths the scan could not stat or read
|
||||||
this run. To stat a file is to read its details, such as its size, without
|
this run. To stat a file is to read its details, such as its size, without
|
||||||
opening it. So the database holds only what the latest scan over each path
|
opening it. So the database holds only what the latest complete scan over each
|
||||||
could verify. A later successful scan adds such files again.
|
path could verify. A later successful scan adds such files again.
|
||||||
- Records outside the scanned operands are left alone, so separate trees can be
|
- Records outside the scanned operands are left alone, so separate trees can be
|
||||||
scanned on different schedules into one database.
|
scanned on different schedules into one database.
|
||||||
- The scan keeps every record at or below a path that the walk leaves out for
|
- The scan keeps every record at or below a path that the walk leaves out for
|
||||||
@@ -413,10 +415,9 @@ operands:
|
|||||||
|
|
||||||
Such records count as outside the scanned operands. So with `-x`, a
|
Such records count as outside the scanned operands. So with `-x`, a
|
||||||
filesystem mounted under an operand can be scanned on its own schedule into
|
filesystem mounted under an operand can be scanned on its own schedule into
|
||||||
the same database. An operand may itself lie at or inside a default exclude.
|
the same database. An operand may itself lie at or inside such a path, as
|
||||||
With `-x`, it may also lie on another filesystem than an operand above it.
|
"Operands" describes. Its walk verifies the records below it, so they are
|
||||||
Such an operand is walked on its own. Its walk verifies the records below
|
deleted when their files are gone.
|
||||||
it, so they are deleted when their files are gone.
|
|
||||||
|
|
||||||
#### Phases
|
#### Phases
|
||||||
|
|
||||||
@@ -450,7 +451,7 @@ walk practical on large or busy storage pools.
|
|||||||
#### Rules for the walk
|
#### Rules for the walk
|
||||||
|
|
||||||
- Only regular files are recorded. Every other kind of file, such as a symlink
|
- Only regular files are recorded. Every other kind of file, such as a symlink
|
||||||
or a device node, is passed over. Symlinks are never followed.
|
or a device node, is passed over. The walk follows no symlink it finds.
|
||||||
- The walk never enters a directory named `.zfs`. Such a directory holds the
|
- The walk never enters a directory named `.zfs`. Such a directory holds the
|
||||||
filesystem's ZFS snapshots, which are its saved earlier states. Walking it
|
filesystem's ZFS snapshots, which are its saved earlier states. Walking it
|
||||||
would list every file once per snapshot. `scan` never starts a walk at or
|
would list every file once per snapshot. `scan` never starts a walk at or
|
||||||
@@ -554,7 +555,7 @@ walk practical on large or busy storage pools.
|
|||||||
one-line warning on stderr and is skipped. The scan continues, and the summary
|
one-line warning on stderr and is skipped. The scan continues, and the summary
|
||||||
line says how many paths were skipped. A skipped path that has a record from
|
line says how many paths were skipped. A skipped path that has a record from
|
||||||
an earlier scan loses that record, and so does every path under an unreadable
|
an earlier scan loses that record, and so does every path under an unreadable
|
||||||
directory. The database mirrors what the latest scan could verify.
|
directory. The database mirrors what the latest complete scan could verify.
|
||||||
- A file or directory that disappears while the scan runs is not an error. The
|
- A file or directory that disappears while the scan runs is not an error. The
|
||||||
scan prints no line for it and does not count it as skipped. A path gone when
|
scan prints no line for it and does not count it as skipped. A path gone when
|
||||||
the walk reaches it gets no record, and so does a file gone when the hash
|
the walk reaches it gets no record, and so does a file gone when the hash
|
||||||
@@ -607,7 +608,7 @@ signal `kill` sends by default:
|
|||||||
- Nothing else is written or deleted. Records are deleted only after a complete
|
- Nothing else is written or deleted. Records are deleted only after a complete
|
||||||
walk. So the records under paths that an interrupted walk never reached are
|
walk. So the records under paths that an interrupted walk never reached are
|
||||||
kept.
|
kept.
|
||||||
- The database is closed and the lock released, as on any other exit.
|
- The database is closed and the lock released.
|
||||||
- The line `scan: interrupted after N files` goes to stderr. `N` is the number
|
- The line `scan: interrupted after N files` goes to stderr. `N` is the number
|
||||||
of files the walk reached. The exit code is 1.
|
of files the walk reached. The exit code is 1.
|
||||||
|
|
||||||
@@ -941,9 +942,9 @@ CREATE INDEX files_signature ON files (size, head, tail, content);
|
|||||||
|
|
||||||
### Progress
|
### Progress
|
||||||
|
|
||||||
Every subcommand shows that it is working: each phase gets a display of its own
|
Each phase of every subcommand gets a display of its own on stderr from the
|
||||||
on stderr from the moment it starts. The display names the phase and shows its
|
moment it starts. The display names the phase and shows its progress. The parts
|
||||||
progress. The parts of its line are listed below.
|
of its line are listed below.
|
||||||
|
|
||||||
The phases, and what each display counts:
|
The phases, and what each display counts:
|
||||||
|
|
||||||
@@ -1032,11 +1033,10 @@ On a terminal:
|
|||||||
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
|
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
|
||||||
```
|
```
|
||||||
|
|
||||||
- No line reaches the terminal's last column, however narrow the terminal is. A
|
- No line reaches the terminal's last column. A line too wide for the terminal
|
||||||
line too wide for the terminal first leaves out its rate, the words of a
|
first leaves out its rate, the words of a stalled phase, or what the phase is
|
||||||
stalled phase, or what the phase is waiting for. It leaves out its bar next.
|
waiting for. It leaves out its bar next. It is cut short at the edge only when
|
||||||
It is cut short at the edge only when the rest does not fit on its own. The
|
the rest does not fit on its own. The rest is:
|
||||||
rest is:
|
|
||||||
- the phase and its place;
|
- the phase and its place;
|
||||||
- the progress, when the phase shows one;
|
- the progress, when the phase shows one;
|
||||||
- the elapsed time;
|
- the elapsed time;
|
||||||
@@ -1070,13 +1070,13 @@ the count it stopped at.
|
|||||||
error is one of these:
|
error is one of these:
|
||||||
- an operand that does not exist, or whose details `scan` cannot read;
|
- an operand that does not exist, or whose details `scan` cannot read;
|
||||||
- a list of mounted filesystems that `scan` cannot read;
|
- a list of mounted filesystems that `scan` cannot read;
|
||||||
- another `scan` already running against the same database;
|
- another `scan` holding the lock (see "The scan lock");
|
||||||
- a database that cannot be created or opened, or whose reads or writes
|
- a database that cannot be created or opened, or whose reads or writes
|
||||||
fail, its directory and its lock file included;
|
fail, its directory and its lock file included;
|
||||||
- a relative database path when the working directory cannot be read;
|
- a relative database path when the working directory cannot be read;
|
||||||
- a database with another schema version, or one at version 0 that `scan` is
|
- a database at a schema version other than 0 or 1;
|
||||||
not setting up: a database read by `report`, `trees` or `info`, or one
|
- a database at version 0 read by `report`, `trees` or `info`;
|
||||||
that already has a `files` table (see "Schema");
|
- a database at version 0 that already has a `files` table (see "Schema");
|
||||||
- a record with a hash that is not 32 bytes, read by `report` or `trees`;
|
- a record with a hash that is not 32 bytes, read by `report` or `trees`;
|
||||||
- a missing database for `report`, `trees` or `info`;
|
- a missing database for `report`, `trees` or `info`;
|
||||||
- a failed write to stdout.
|
- a failed write to stdout.
|
||||||
@@ -1265,6 +1265,7 @@ All of the following, run in this directory, must pass:
|
|||||||
data.
|
data.
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
|
make build
|
||||||
d=$(mktemp -d /var/tmp/sfdupes.XXXXXX)
|
d=$(mktemp -d /var/tmp/sfdupes.XXXXXX)
|
||||||
export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite"
|
export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite"
|
||||||
mkdir -p "$d/a" "$d/b"
|
mkdir -p "$d/a" "$d/b"
|
||||||
|
|||||||
Reference in new issue
Block a user