Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f6230f410c |
@@ -154,8 +154,8 @@ to keep is your choice.
|
||||
|
||||
A row is a candidate, not proof:
|
||||
|
||||
- The reports show each file as it was when a scan last recorded it. A file may
|
||||
have changed or gone since.
|
||||
- The reports show the files as they were at the last scan. A file may have
|
||||
changed or gone since.
|
||||
- A file of 100 MiB or more is compared only on samples of its content (see
|
||||
"Duplicate detection").
|
||||
- Two paths that are hard links, two names for one file, are listed as
|
||||
@@ -225,9 +225,10 @@ In order:
|
||||
its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read
|
||||
in full. The target is a filesystem of about ten million files and 150 TB, on
|
||||
disks that may be slow or busy. During a scan, one small record per file is
|
||||
held in memory. The hashes stay in the database. `report` holds the duplicate
|
||||
paths of one file size at a time. `trees` holds one entry per directory, plus
|
||||
the files of the directories it is reading.
|
||||
held in memory. It holds the path and the two values the walk compares, the
|
||||
size and the mtime. The hashes stay in the database. `report` holds the
|
||||
duplicate paths of one file size at a time. `trees` holds one entry per
|
||||
directory, plus the files of the directories it is reading.
|
||||
3. **Rescan cheaply, analyze from the database.** The scan keeps its database
|
||||
between runs, and an unchanged file is never read again. `report`, `trees`
|
||||
and `info` work from the database alone and never touch the scanned
|
||||
@@ -343,13 +344,9 @@ default. An operand may be a directory or a regular file.
|
||||
the exit code is 2.
|
||||
- An operand that does not exist is a fatal error. `scan` then stops before it
|
||||
walks anything, and exits 1.
|
||||
- Each operand is made absolute before the walk. A relative operand is joined to
|
||||
the working directory as `pwd` prints it, with its symlinks kept. A `.` or
|
||||
`..` in it is resolved as text, and symlinks are not followed. Each record is
|
||||
stored under that absolute path. After a `cd` through a symlink, `scan .`
|
||||
names the symlink and is skipped as a symlink operand. `scan sub` in that
|
||||
directory records its files under the symlink's path. `report` can then list
|
||||
each as a duplicate of the same file recorded under its real path.
|
||||
- Each operand is made absolute before the walk. A `.` or `..` in it is resolved
|
||||
as text, and symlinks are not followed. Each record is stored under that
|
||||
absolute path, so results do not depend on the working directory.
|
||||
- All operands belong to one scan, and one walk covers them all at once (see
|
||||
"Concurrency").
|
||||
- An operand that repeats another is dropped before the walk, without a line.
|
||||
@@ -363,7 +360,8 @@ default. An operand may be a directory or a regular file.
|
||||
It is also kept when a directory between the two is on a filesystem that is
|
||||
not persistent. Such a filesystem is one the kernel makes up or keeps in
|
||||
memory (see "Rules for the walk"). The outer operand's walk does not enter
|
||||
such a directory, so the inner operand is walked on its own.
|
||||
such a directory, so the inner operand is walked on its own. For an inner
|
||||
operand that is a regular file, the check starts at its directory.
|
||||
|
||||
- No other case keeps an operand under another. An operand reached through a
|
||||
symlinked directory below the outer operand is dropped too, without a line.
|
||||
@@ -403,8 +401,8 @@ operands:
|
||||
- The scan deletes each record under a scanned operand whose file it could not
|
||||
verify. That covers deleted files, and paths the scan could not stat or read
|
||||
this run. To stat a file is to read its details, such as its size, without
|
||||
opening it. So the database holds only what the latest complete scan over each
|
||||
path could verify. A later successful scan adds such files again.
|
||||
opening it. So the database holds only what the latest scan over each path
|
||||
could verify. A later successful scan adds such files again.
|
||||
- Records outside the scanned operands are left alone, so separate trees can be
|
||||
scanned on different schedules into one database.
|
||||
- The scan keeps every record at or below a path that the walk leaves out for
|
||||
@@ -415,9 +413,10 @@ operands:
|
||||
|
||||
Such records count as outside the scanned operands. So with `-x`, a
|
||||
filesystem mounted under an operand can be scanned on its own schedule into
|
||||
the same database. An operand may itself lie at or inside such a path, as
|
||||
"Operands" describes. Its walk verifies the records below it, so they are
|
||||
deleted when their files are gone.
|
||||
the same database. An operand may itself lie at or inside a default exclude.
|
||||
With `-x`, it may also lie on another filesystem than an operand above it.
|
||||
Such an operand is walked on its own. Its walk verifies the records below
|
||||
it, so they are deleted when their files are gone.
|
||||
|
||||
#### Phases
|
||||
|
||||
@@ -451,7 +450,7 @@ walk practical on large or busy storage pools.
|
||||
#### Rules for the walk
|
||||
|
||||
- Only regular files are recorded. Every other kind of file, such as a symlink
|
||||
or a device node, is passed over. The walk follows no symlink it finds.
|
||||
or a device node, is passed over. Symlinks are never followed.
|
||||
- The walk never enters a directory named `.zfs`. Such a directory holds the
|
||||
filesystem's ZFS snapshots, which are its saved earlier states. Walking it
|
||||
would list every file once per snapshot. `scan` never starts a walk at or
|
||||
@@ -555,7 +554,7 @@ walk practical on large or busy storage pools.
|
||||
one-line warning on stderr and is skipped. The scan continues, and the summary
|
||||
line says how many paths were skipped. A skipped path that has a record from
|
||||
an earlier scan loses that record, and so does every path under an unreadable
|
||||
directory. The database mirrors what the latest complete scan could verify.
|
||||
directory. The database mirrors what the latest scan could verify.
|
||||
- A file or directory that disappears while the scan runs is not an error. The
|
||||
scan prints no line for it and does not count it as skipped. A path gone when
|
||||
the walk reaches it gets no record, and so does a file gone when the hash
|
||||
@@ -608,7 +607,7 @@ signal `kill` sends by default:
|
||||
- Nothing else is written or deleted. Records are deleted only after a complete
|
||||
walk. So the records under paths that an interrupted walk never reached are
|
||||
kept.
|
||||
- The database is closed and the lock released.
|
||||
- The database is closed and the lock released, as on any other exit.
|
||||
- The line `scan: interrupted after N files` goes to stderr. `N` is the number
|
||||
of files the walk reached. The exit code is 1.
|
||||
|
||||
@@ -942,9 +941,9 @@ CREATE INDEX files_signature ON files (size, head, tail, content);
|
||||
|
||||
### Progress
|
||||
|
||||
Each phase of every subcommand gets a display of its own on stderr from the
|
||||
moment it starts. The display names the phase and shows its progress. The parts
|
||||
of its line are listed below.
|
||||
Every subcommand shows that it is working: each phase gets a display of its own
|
||||
on stderr from the moment it starts. The display names the phase and shows its
|
||||
progress. The parts of its line are listed below.
|
||||
|
||||
The phases, and what each display counts:
|
||||
|
||||
@@ -1033,10 +1032,11 @@ On a terminal:
|
||||
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
|
||||
```
|
||||
|
||||
- No line reaches the terminal's last column. A line too wide for the terminal
|
||||
first leaves out its rate, the words of a stalled phase, or what the phase is
|
||||
waiting for. It leaves out its bar next. It is cut short at the edge only when
|
||||
the rest does not fit on its own. The rest is:
|
||||
- No line reaches the terminal's last column, however narrow the terminal is. A
|
||||
line too wide for the terminal first leaves out its rate, the words of a
|
||||
stalled phase, or what the phase is waiting for. It leaves out its bar next.
|
||||
It is cut short at the edge only when the rest does not fit on its own. The
|
||||
rest is:
|
||||
- the phase and its place;
|
||||
- the progress, when the phase shows one;
|
||||
- the elapsed time;
|
||||
@@ -1070,13 +1070,13 @@ the count it stopped at.
|
||||
error is one of these:
|
||||
- an operand that does not exist, or whose details `scan` cannot read;
|
||||
- a list of mounted filesystems that `scan` cannot read;
|
||||
- another `scan` holding the lock (see "The scan lock");
|
||||
- another `scan` already running against the same database;
|
||||
- a database that cannot be created or opened, or whose reads or writes
|
||||
fail, its directory and its lock file included;
|
||||
- a relative database path when the working directory cannot be read;
|
||||
- a database at a schema version other than 0 or 1;
|
||||
- a database at version 0 read by `report`, `trees` or `info`;
|
||||
- a database at version 0 that already has a `files` table (see "Schema");
|
||||
- a database with another schema version, or one at version 0 that `scan` is
|
||||
not setting up: a database read by `report`, `trees` or `info`, or one
|
||||
that already has a `files` table (see "Schema");
|
||||
- a record with a hash that is not 32 bytes, read by `report` or `trees`;
|
||||
- a missing database for `report`, `trees` or `info`;
|
||||
- a failed write to stdout.
|
||||
@@ -1265,7 +1265,6 @@ All of the following, run in this directory, must pass:
|
||||
data.
|
||||
|
||||
```sh
|
||||
make build
|
||||
d=$(mktemp -d /var/tmp/sfdupes.XXXXXX)
|
||||
export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite"
|
||||
mkdir -p "$d/a" "$d/b"
|
||||
|
||||
Reference in new issue
Block a user