|
|
|
@@ -18,11 +18,11 @@ a strong candidate, not proof.
|
|
|
|
|
|
|
|
|
|
`sfdupes` is meant for multi-terabyte servers, where reading every byte of every
|
|
|
|
|
file takes too long. It has four subcommands. `scan` walks the filesystem and
|
|
|
|
|
records the files it finds in an SQLite database. SQLite keeps a whole database
|
|
|
|
|
in one file. The database is kept between runs. A rescan reads only new and
|
|
|
|
|
changed files. That makes `scan` cheap enough to run every day from `cron`, the
|
|
|
|
|
system's job scheduler. `report`, `trees` and `info` read the database alone and
|
|
|
|
|
can run at any time.
|
|
|
|
|
records every file in an SQLite database. SQLite keeps a whole database in one
|
|
|
|
|
file. The database is kept between runs. A rescan reads only new and changed
|
|
|
|
|
files. That makes `scan` cheap enough to run every day from `cron`, the system's
|
|
|
|
|
job scheduler. `report`, `trees` and `info` read the database alone and can run
|
|
|
|
|
at any time.
|
|
|
|
|
|
|
|
|
|
## Getting Started
|
|
|
|
|
|
|
|
|
@@ -38,18 +38,16 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
|
|
|
|
|
`scan` walks the trees named by its operands, the paths given after `scan` on
|
|
|
|
|
the command line. The walk is the listing of every directory under them. `scan`
|
|
|
|
|
keeps one database record per regular file. A regular file is an ordinary file,
|
|
|
|
|
as opposed to a directory, a symlink or a device node. A symlink is a file that
|
|
|
|
|
holds the path of another file and stands in for it. A device node is a file
|
|
|
|
|
that stands for a piece of hardware, such as a disk or a terminal. `report`
|
|
|
|
|
prints the file-level duplicates. `trees` prints the duplicate directory trees.
|
|
|
|
|
`info` prints the database's path, size and record count.
|
|
|
|
|
as opposed to a directory, a symlink or a device. `report` prints the file-level
|
|
|
|
|
duplicates. `trees` prints the duplicate directory trees. `info` prints the
|
|
|
|
|
database's path, size and record count.
|
|
|
|
|
|
|
|
|
|
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
|
|
|
|
|
another file. The first `scan` creates it.
|
|
|
|
|
|
|
|
|
|
`report`, `trees` and `info` write their data to stdout. Progress, warnings and
|
|
|
|
|
errors go to stderr. stdout and stderr are a program's two output streams. The
|
|
|
|
|
shell shows both on the terminal, and `>` redirects stdout alone.
|
|
|
|
|
Every command writes its data to stdout. Everything else, such as progress and
|
|
|
|
|
warnings, goes to stderr. stdout and stderr are a program's two output streams.
|
|
|
|
|
The shell shows both on the terminal, and `>` redirects stdout alone.
|
|
|
|
|
|
|
|
|
|
### Install
|
|
|
|
|
|
|
|
|
@@ -69,12 +67,11 @@ The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set.
|
|
|
|
|
|
|
|
|
|
`sfdupes --version` prints `sfdupes VERSION` and exits. For a binary from
|
|
|
|
|
`go install`, `VERSION` is the version Go recorded in it. For `@latest` that is
|
|
|
|
|
a release tag such as `v0.0.3`. For `@main` it is the commit's tag when it has
|
|
|
|
|
one. Otherwise it is a pseudo-version, Go's version string for an untagged
|
|
|
|
|
commit. That string ends in the commit's hash. A binary built from a clone
|
|
|
|
|
carries the output of `git describe --tags --always --dirty` in that clone, such
|
|
|
|
|
as `v0.0.3` on a tagged commit. `VERSION` is `dev` when no version is known. To
|
|
|
|
|
see everything Go recorded in an installed binary:
|
|
|
|
|
a release tag such as `v0.0.3`. For `@main` it is a pseudo-version, Go's version
|
|
|
|
|
string for an untagged commit. That string ends in the commit's hash. A binary
|
|
|
|
|
built from a clone carries the `git` tag or commit it was built from. `VERSION`
|
|
|
|
|
is `dev` when no version is known. To see everything Go recorded in an installed
|
|
|
|
|
binary:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
go version -m "$(command -v sfdupes)"
|
|
|
|
@@ -108,10 +105,9 @@ docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
|
|
|
|
|
|
|
|
|
|
### Daily scan from `cron`
|
|
|
|
|
|
|
|
|
|
Run `scan` as root, so that it can read every file. `scan` warns about a
|
|
|
|
|
directory it cannot read, and about a new or changed file it cannot read. It
|
|
|
|
|
skips the path and deletes the records at or below it (see "Rules for the
|
|
|
|
|
walk"). As the file `/etc/cron.d/sfdupes`:
|
|
|
|
|
Run `scan` as root, so that it can read every file. A path it cannot read is
|
|
|
|
|
skipped with a warning and loses its database record. As the file
|
|
|
|
|
`/etc/cron.d/sfdupes`:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log
|
|
|
|
@@ -125,11 +121,8 @@ walk"). As the file `/etc/cron.d/sfdupes`:
|
|
|
|
|
- one line naming the database;
|
|
|
|
|
- progress lines, as described under "Progress";
|
|
|
|
|
- a warning for each path it skips;
|
|
|
|
|
- a line for each system or temporary directory the walk leaves out (see
|
|
|
|
|
"Rules for the walk");
|
|
|
|
|
- a line for each path left out because its filesystem is one the kernel,
|
|
|
|
|
the core of the operating system, makes up or keeps in memory, such as
|
|
|
|
|
`tmpfs`;
|
|
|
|
|
- a line for each path the walk leaves out as a system or temporary
|
|
|
|
|
directory (see "Rules for the walk");
|
|
|
|
|
- the summary line;
|
|
|
|
|
- a warning when a report still has the database open as the scan ends (see
|
|
|
|
|
"Concurrent reads").
|
|
|
|
@@ -139,8 +132,8 @@ walk"). As the file `/etc/cron.d/sfdupes`:
|
|
|
|
|
- A scan that fails exits 1, with the reason among the last lines of the log.
|
|
|
|
|
`tail` prints those lines, and `cron` mails them to root if the host can send
|
|
|
|
|
mail.
|
|
|
|
|
- A running scan carries on when the next one starts. The new one fails at once,
|
|
|
|
|
and the lines that `cron` mails include
|
|
|
|
|
- A scan still running when the next one starts carries on. The new one fails at
|
|
|
|
|
once, and the lines `cron` mails include
|
|
|
|
|
`sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
|
|
|
|
|
- `report`, `trees` and `info` need only read access to the database. The umask
|
|
|
|
|
is the setting that removes permissions from the files a process creates.
|
|
|
|
@@ -158,8 +151,8 @@ to keep is your choice.
|
|
|
|
|
|
|
|
|
|
A row is a candidate, not proof:
|
|
|
|
|
|
|
|
|
|
- The reports show each file as it was when a scan last recorded it. A file may
|
|
|
|
|
have changed or gone since.
|
|
|
|
|
- The reports show the files as they were at the last scan. A file may have
|
|
|
|
|
changed or gone since.
|
|
|
|
|
- A file of 100 MiB or more is compared only on samples of its content (see
|
|
|
|
|
"Duplicate detection").
|
|
|
|
|
- Two paths that are hard links, two names for one file, are listed as
|
|
|
|
@@ -177,8 +170,7 @@ cmp /srv/a/big.iso /srv/b/big-copy.iso && echo identical
|
|
|
|
|
where the files differ. The second line prints `hard links` when both paths are
|
|
|
|
|
the same file.
|
|
|
|
|
|
|
|
|
|
A path holding a special character, such as a tab, is escaped in the reports.
|
|
|
|
|
The character is written as a backslash and one more character, `\t` for a tab
|
|
|
|
|
A path holding a special character, such as a tab, is escaped in the reports
|
|
|
|
|
(see "Report output format"). Undo the escapes before using such a path.
|
|
|
|
|
`printf '%b'` does that, because every backslash in an escaped path starts an
|
|
|
|
|
escape. The shell drops trailing newlines from the output of `$(...)`. To keep a
|
|
|
|
@@ -206,9 +198,9 @@ two ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it
|
|
|
|
|
makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
|
|
|
|
|
of reads, at the price of a comparison that is not exact.
|
|
|
|
|
|
|
|
|
|
The hashes live in a database that is kept between runs. A rescan reads only new
|
|
|
|
|
files, and files whose size changed or whose modification time (mtime) is newer
|
|
|
|
|
than recorded. All analysis then works from the database alone.
|
|
|
|
|
The hashes live in a database that is kept between runs. A rescan reads only the
|
|
|
|
|
files that are new or whose size or modification time (mtime) changed. All
|
|
|
|
|
analysis then works from the database alone.
|
|
|
|
|
|
|
|
|
|
The goal is whole duplicated trees, not only single files. A download extracted
|
|
|
|
|
twice, or a project copied once too often, is such a tree. It can be removed as
|
|
|
|
@@ -222,16 +214,17 @@ In order:
|
|
|
|
|
|
|
|
|
|
1. **Find whole duplicate trees, not just files.** The aim is to find places
|
|
|
|
|
where the same set of files and directories exists at two or more paths. A
|
|
|
|
|
whole subtree can then be removed at once. Tree-level detection is built on
|
|
|
|
|
file-level detection.
|
|
|
|
|
whole subtree can then be removed at once. File-level detection comes first.
|
|
|
|
|
Tree-level detection is built on top of it.
|
|
|
|
|
2. **Bound the reads per file.** A file is read only when it is new or has
|
|
|
|
|
changed. A file under 100 MiB is then read in full, and a larger one only at
|
|
|
|
|
its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read
|
|
|
|
|
in full. The target is a filesystem of about ten million files and 150 TB, on
|
|
|
|
|
disks that may be slow or busy. During a scan, one small record per file is
|
|
|
|
|
held in memory. The hashes stay in the database. `report` holds the duplicate
|
|
|
|
|
paths of one file size at a time. `trees` holds one entry per directory, plus
|
|
|
|
|
the files of the directories it is reading.
|
|
|
|
|
held in memory. It holds the path and the two values the walk compares, the
|
|
|
|
|
size and the mtime. The hashes stay in the database. `report` holds the
|
|
|
|
|
duplicate paths of one file size at a time. `trees` holds one entry per
|
|
|
|
|
directory, plus the files of the directories it is reading.
|
|
|
|
|
3. **Rescan cheaply, analyze from the database.** The scan keeps its database
|
|
|
|
|
between runs, and an unchanged file is never read again. `report`, `trees`
|
|
|
|
|
and `info` work from the database alone and never touch the scanned
|
|
|
|
@@ -257,22 +250,17 @@ sfdupes --version
|
|
|
|
|
sfdupes [command] --help
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
A missing or unknown subcommand is a usage error, a mistake in the command line.
|
|
|
|
|
`sfdupes` then prints the usage text, a summary of the commands and their flags,
|
|
|
|
|
on stderr and exits 2.
|
|
|
|
|
A missing or unknown subcommand prints the usage text on stderr and exits 2.
|
|
|
|
|
|
|
|
|
|
The reference below is also the program's specification.
|
|
|
|
|
|
|
|
|
|
## Non-goals
|
|
|
|
|
|
|
|
|
|
- No byte-for-byte compare.
|
|
|
|
|
- No deleting of duplicates.
|
|
|
|
|
- No replacing a copy with a hard link to another copy. The reports are
|
|
|
|
|
advisory. Acting on them is your job.
|
|
|
|
|
- No byte-for-byte compare, and no deleting or linking of duplicates. The
|
|
|
|
|
reports are advisory. Acting on them is your job.
|
|
|
|
|
- The SQLite database is the only stored data. There are no export or import
|
|
|
|
|
formats.
|
|
|
|
|
- `sfdupes` does not run in the background or watch the filesystem for changes.
|
|
|
|
|
Scheduling rescans is `cron`'s job.
|
|
|
|
|
- No daemon or filesystem watcher. Scheduling rescans is `cron`'s job.
|
|
|
|
|
|
|
|
|
|
## Reference
|
|
|
|
|
|
|
|
|
@@ -287,9 +275,9 @@ The reference below is also the program's specification.
|
|
|
|
|
hashes them. The default is the number of CPUs. `N` must be at least 1. A
|
|
|
|
|
smaller value is a usage error, reported in one line on stderr with exit 2
|
|
|
|
|
before anything is scanned.
|
|
|
|
|
- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of
|
|
|
|
|
directories on other filesystems, as the same flag does for `du` and `rsync`.
|
|
|
|
|
By default the walk also enters directories on other filesystems.
|
|
|
|
|
- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand on that
|
|
|
|
|
operand's filesystem, as the same flag does for `du` and `rsync`. By default
|
|
|
|
|
the walk also enters directories on other filesystems.
|
|
|
|
|
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
|
|
|
|
|
Those are the system and temporary directories the walk otherwise leaves out,
|
|
|
|
|
such as `/var/tmp` on Linux (see "Rules for the walk").
|
|
|
|
@@ -348,52 +336,32 @@ default. An operand may be a directory or a regular file.
|
|
|
|
|
|
|
|
|
|
- `scan` with no operand is a usage error. The usage text goes to stderr, and
|
|
|
|
|
the exit code is 2.
|
|
|
|
|
- An operand that does not exist is a fatal error. `scan` then stops before it
|
|
|
|
|
walks anything, and exits 1.
|
|
|
|
|
- Each operand is made absolute before the walk. A relative operand is joined to
|
|
|
|
|
the working directory as `pwd` prints it, with its symlinks kept. A `.` or
|
|
|
|
|
`..` in it is resolved as text, and symlinks are not followed. Each record is
|
|
|
|
|
stored under that absolute path.
|
|
|
|
|
- An operand that does not exist is a fatal error, exit 1.
|
|
|
|
|
- Each operand is made absolute before the walk. A `.` or `..` in it is resolved
|
|
|
|
|
as text, and symlinks are not followed. Each record is stored under that
|
|
|
|
|
absolute path, so results do not depend on the working directory.
|
|
|
|
|
- All operands belong to one scan, and one walk covers them all at once (see
|
|
|
|
|
"Concurrency").
|
|
|
|
|
- An operand that repeats another is dropped before the walk.
|
|
|
|
|
- An operand under another operand is dropped the same way, unless one of the
|
|
|
|
|
cases below applies. No path is then walked twice.
|
|
|
|
|
- An operand under another is kept, and walked on its own, when the inner
|
|
|
|
|
operand or a directory between the two is one of these:
|
|
|
|
|
- An operand that repeats another, or lies under another, is dropped before the
|
|
|
|
|
walk, without a line. Every file is then reached once. An operand under
|
|
|
|
|
another is kept only when the inner operand, or a directory between the two,
|
|
|
|
|
is one of these:
|
|
|
|
|
- a default exclude (see "Rules for the walk");
|
|
|
|
|
- a directory on a filesystem that is not persistent, one the kernel makes
|
|
|
|
|
up or keeps in memory (see "Rules for the walk");
|
|
|
|
|
- with `-x`, a directory on another filesystem than the outer operand.
|
|
|
|
|
|
|
|
|
|
It is also kept when a directory between the two is on a filesystem that is
|
|
|
|
|
not persistent. Such a filesystem is one the kernel makes up or keeps in
|
|
|
|
|
memory (see "Rules for the walk"). The outer operand's walk does not enter
|
|
|
|
|
such a directory, so the inner operand is walked on its own.
|
|
|
|
|
The outer operand's walk does not enter such a directory, so the inner
|
|
|
|
|
operand is walked on its own.
|
|
|
|
|
|
|
|
|
|
- No other case keeps an operand under another. An operand reached through a
|
|
|
|
|
symlinked directory below the outer operand is dropped too. The outer
|
|
|
|
|
operand's walk does not follow the symlink, so no file under the inner operand
|
|
|
|
|
is scanned. The records that an earlier scan left under it are deleted. The
|
|
|
|
|
`-x` case above also covers an operand reached that way.
|
|
|
|
|
- An operand that is neither a regular file nor a directory is not scanned.
|
|
|
|
|
`scan` prints a warning naming the path and what it is. It counts the operand
|
|
|
|
|
as skipped and drops it. An operand beneath a symlink operand is scanned all
|
|
|
|
|
the same. The scan keeps the records beneath a dropped operand, unless another
|
|
|
|
|
operand covers them. After a `cd` through a symlink, `scan .` names the
|
|
|
|
|
symlink and is skipped as a symlink operand. `scan sub` in that directory
|
|
|
|
|
records its files under the symlink's path. `report` can then list each as a
|
|
|
|
|
duplicate of the same file recorded under its real path.
|
|
|
|
|
- An operand at or inside a directory named `.zfs` is never scanned, whether
|
|
|
|
|
given alone or under another operand. The check looks at the path as given and
|
|
|
|
|
at the path with its symlinks resolved. `scan` prints a warning naming the
|
|
|
|
|
path, counts the operand as skipped and drops it. The records beneath it are
|
|
|
|
|
kept, unless it lies under another operand. Then they are deleted, because
|
|
|
|
|
that operand's walk never enters `.zfs` (see "Rules for the walk").
|
|
|
|
|
- An operand that is neither a regular file nor a directory is not scanned, and
|
|
|
|
|
neither is a directory named `.zfs`. `scan` prints a warning naming the path
|
|
|
|
|
and what it is. It counts the operand as skipped and drops it. An operand
|
|
|
|
|
beneath a dropped one is scanned all the same. The scan keeps the records
|
|
|
|
|
beneath a dropped operand, unless another operand covers them.
|
|
|
|
|
- An operand on a filesystem that is not persistent is dropped too. `scan`
|
|
|
|
|
prints a line naming it and its filesystem type, and does not count it as
|
|
|
|
|
skipped.
|
|
|
|
|
- `scan` prints these warnings, and the line for an operand that is not on a
|
|
|
|
|
persistent filesystem, even when the operand repeats another or lies under
|
|
|
|
|
another.
|
|
|
|
|
- A scan whose every operand is dropped walks nothing and exits 0.
|
|
|
|
|
|
|
|
|
|
#### What a scan does to the database
|
|
|
|
@@ -405,28 +373,27 @@ operands:
|
|
|
|
|
- A file already in the database is not read again when its size and mtime match
|
|
|
|
|
its record. The mtime matches when it is not newer than the recorded one. This
|
|
|
|
|
is what makes a daily rescan cheap.
|
|
|
|
|
- `scan` hashes a file again when its size differs or its mtime is newer. The
|
|
|
|
|
mtime is compared to the nanosecond, as finely as the filesystem records it. A
|
|
|
|
|
- A file whose size differs, or whose mtime is newer, is hashed again. The mtime
|
|
|
|
|
is compared to the nanosecond, as finely as the filesystem records it. A
|
|
|
|
|
same-size rewrite counts as a change whenever the filesystem gives it a later
|
|
|
|
|
mtime, even within the same second.
|
|
|
|
|
- The scan deletes each record under a scanned operand whose file it could not
|
|
|
|
|
verify. That covers deleted files, and paths the scan could not stat or read
|
|
|
|
|
this run. To stat a file is to read its details, such as its size, without
|
|
|
|
|
opening it. So after a complete scan, the database holds only what that scan
|
|
|
|
|
could verify under its operands. A later successful scan adds such files
|
|
|
|
|
again.
|
|
|
|
|
opening it. So the database holds only what the latest scan over each path
|
|
|
|
|
could verify. A later successful scan adds such files again.
|
|
|
|
|
- Records outside the scanned operands are left alone, so separate trees can be
|
|
|
|
|
scanned on different schedules into one database.
|
|
|
|
|
- The scan keeps every record at or below a path that the walk leaves out for
|
|
|
|
|
one of these reasons:
|
|
|
|
|
- it is a default exclude;
|
|
|
|
|
- it is not on a persistent filesystem;
|
|
|
|
|
- with `-x`, it is a directory on another filesystem than its operand.
|
|
|
|
|
- The scan keeps every record at or below a path that the walk leaves out on
|
|
|
|
|
purpose. Such records count as outside the scanned operands. The walk leaves
|
|
|
|
|
out:
|
|
|
|
|
- a default exclude;
|
|
|
|
|
- a path that is not on a persistent filesystem;
|
|
|
|
|
- with `-x`, a directory on another filesystem than its operand.
|
|
|
|
|
|
|
|
|
|
Such records count as outside the scanned operands. So with `-x`, a
|
|
|
|
|
filesystem mounted under an operand can be scanned on its own schedule into
|
|
|
|
|
the same database. An operand may itself lie at or inside such a path, as
|
|
|
|
|
"Operands" describes. Its walk verifies the records below it, so they are
|
|
|
|
|
So with `-x`, a filesystem mounted under an operand can be scanned on its
|
|
|
|
|
own schedule into the same database. An operand at or inside such a path is
|
|
|
|
|
walked on its own. Its walk verifies the records below it, so they are
|
|
|
|
|
deleted when their files are gone.
|
|
|
|
|
|
|
|
|
|
#### Phases
|
|
|
|
@@ -437,13 +404,12 @@ phases.
|
|
|
|
|
1. `load` reads the existing records. Each record under the operands is kept in
|
|
|
|
|
memory by path, with its size and mtime, for the walk to compare against.
|
|
|
|
|
2. `walk` lists the trees under all operands. Several workers share one queue of
|
|
|
|
|
directories to read. A worker is one thread of the program. A thread is one
|
|
|
|
|
of several parts of a program that run at the same time and share its memory.
|
|
|
|
|
"Concurrency" says how many workers there are. Each worker reads one
|
|
|
|
|
directory at a time. It hands the subdirectories it finds back to the queue,
|
|
|
|
|
and it stats each regular file as it finds it. The walk compares each file
|
|
|
|
|
with its record. An unchanged file needs nothing more. Every new or changed
|
|
|
|
|
file is passed to the hash phase.
|
|
|
|
|
directories to read. A worker is one thread of the program, and "Concurrency"
|
|
|
|
|
says how many there are. Each worker reads one directory at a time. It hands
|
|
|
|
|
the subdirectories it finds back to the queue, and it stats each regular file
|
|
|
|
|
as it finds it. The walk compares each file with its record. An unchanged
|
|
|
|
|
file needs nothing more. Every new or changed file is passed to the hash
|
|
|
|
|
phase.
|
|
|
|
|
3. `hash` reads every new or changed file and computes its three hashes. Files
|
|
|
|
|
are read in inode order. An inode is the filesystem's own record of a file.
|
|
|
|
|
Files with nearby inode numbers usually lie near each other on disk. Reading
|
|
|
|
@@ -462,11 +428,10 @@ walk practical on large or busy storage pools.
|
|
|
|
|
#### Rules for the walk
|
|
|
|
|
|
|
|
|
|
- Only regular files are recorded. Every other kind of file, such as a symlink
|
|
|
|
|
or a device node, is passed over. The walk follows no symlink it finds.
|
|
|
|
|
or a device node, is passed over. Symlinks are never followed.
|
|
|
|
|
- The walk never enters a directory named `.zfs`. Such a directory holds the
|
|
|
|
|
filesystem's ZFS snapshots, which are its saved earlier states. Walking it
|
|
|
|
|
would list every file once per snapshot. `scan` never starts a walk at or
|
|
|
|
|
inside one either (see "Operands").
|
|
|
|
|
would list every file once per snapshot.
|
|
|
|
|
- The walk never records the scan's own database files:
|
|
|
|
|
- the database file;
|
|
|
|
|
- its `.lock` file (see "The scan lock");
|
|
|
|
@@ -480,14 +445,14 @@ walk practical on large or busy storage pools.
|
|
|
|
|
operand at the path given, even when that path runs through a symlinked
|
|
|
|
|
directory. Such an operand can lead to the database files, and the walk
|
|
|
|
|
passes over them there as well. The walk prints no line for the database
|
|
|
|
|
files and does not count them as skipped. A record that an earlier scan left
|
|
|
|
|
for one of them is deleted.
|
|
|
|
|
files and does not count them as skipped. A record an earlier scan left for
|
|
|
|
|
one of them is deleted.
|
|
|
|
|
|
|
|
|
|
- By default the walk enters directories on other filesystems than its
|
|
|
|
|
operand's. With `-x`, it never enters a directory on a different filesystem
|
|
|
|
|
than its operand. `scan` prints no line for a directory that only `-x` leaves
|
|
|
|
|
out, and does not count it as skipped. The records below it are kept, as
|
|
|
|
|
described under "What a scan does to the database".
|
|
|
|
|
than its operand. `scan` prints no line for that directory and does not count
|
|
|
|
|
it as skipped. The records below it are kept, as described under "What a scan
|
|
|
|
|
does to the database".
|
|
|
|
|
- The walk never records anything on a filesystem that is not persistent. A
|
|
|
|
|
filesystem is not persistent when the kernel makes it up or keeps it in
|
|
|
|
|
memory. These are the types, as `sfdupes scan --help` lists them on Linux and
|
|
|
|
@@ -504,11 +469,10 @@ walk practical on large or busy storage pools.
|
|
|
|
|
starts, `scan` lists the mounted filesystems and notes the device ID of each
|
|
|
|
|
one of those types. On Linux the list comes from `/proc/self/mountinfo`,
|
|
|
|
|
which gives each filesystem's own device ID. On macOS it comes from the
|
|
|
|
|
system's mount list. There the device ID is read from the mount point, the
|
|
|
|
|
directory the filesystem is mounted on. Where two filesystems share a mount
|
|
|
|
|
point on macOS, the one mounted last counts. On any other system the list is
|
|
|
|
|
empty. A list that `scan` cannot read is a fatal error, and nothing is
|
|
|
|
|
scanned. No flag turns this rule off.
|
|
|
|
|
system's mount list, and the device ID is read from the mount point. Where
|
|
|
|
|
two filesystems share a mount point on macOS, the one mounted last counts.
|
|
|
|
|
On any other system the list is empty. A list that `scan` cannot read is a
|
|
|
|
|
fatal error, and nothing is scanned. No flag turns this rule off.
|
|
|
|
|
|
|
|
|
|
The walk never enters a directory on one of those devices, and never records
|
|
|
|
|
a file on one. That holds however the walk reaches the path, through a bind
|
|
|
|
@@ -523,10 +487,11 @@ walk practical on large or busy storage pools.
|
|
|
|
|
The path is not counted as skipped. The records below it are kept, as
|
|
|
|
|
described under "What a scan does to the database".
|
|
|
|
|
|
|
|
|
|
- Below an operand, the walk does not enter a default exclude. On Linux, the
|
|
|
|
|
default excludes hold temporary files and the kernel's view of the running
|
|
|
|
|
system. They depend on the system, and `sfdupes scan --help` lists those of
|
|
|
|
|
the system it runs on. On Linux they are:
|
|
|
|
|
- Below an operand, the walk does not enter a default exclude. The default
|
|
|
|
|
excludes hold the kernel's view of the running system and temporary files, so
|
|
|
|
|
that `sfdupes scan /` walks only the files worth comparing. They depend on the
|
|
|
|
|
system, and `sfdupes scan --help` lists those of the system it runs on. On
|
|
|
|
|
Linux they are:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
/proc, /sys, /dev, /run, /tmp, /var/tmp
|
|
|
|
@@ -555,27 +520,21 @@ walk practical on large or busy storage pools.
|
|
|
|
|
scan: skipped /var/tmp (default exclude)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The directory is not counted as skipped. `scan` prints the filesystem line
|
|
|
|
|
instead for a default exclude that is not on a persistent filesystem, such
|
|
|
|
|
as `/proc`. One that is itself an operand gets no line, because it is walked
|
|
|
|
|
as that operand. The scan keeps the records below a default exclude that the
|
|
|
|
|
The directory is not counted as skipped. A default exclude on a filesystem
|
|
|
|
|
that is not persistent, such as `/proc`, gets the line for its filesystem
|
|
|
|
|
instead. One that is itself an operand gets no line, because it is walked as
|
|
|
|
|
that operand. The scan keeps the records below a default exclude that the
|
|
|
|
|
walk did not enter, as described under "What a scan does to the database".
|
|
|
|
|
|
|
|
|
|
- `scan` skips each of these, with a one-line warning on stderr:
|
|
|
|
|
- a directory it cannot read;
|
|
|
|
|
- a path it cannot stat;
|
|
|
|
|
- a new or changed file it cannot read.
|
|
|
|
|
|
|
|
|
|
An unchanged file is not opened, so it keeps its record even when it has
|
|
|
|
|
become unreadable. The scan continues, and the summary line says how many
|
|
|
|
|
paths were skipped. A skipped path that has a record from an earlier scan
|
|
|
|
|
loses that record, and so does every path under an unreadable directory.
|
|
|
|
|
|
|
|
|
|
- A path that cannot be read, for lack of permission or any other error, gets a
|
|
|
|
|
one-line warning on stderr and is skipped. The scan continues, and the summary
|
|
|
|
|
line says how many paths were skipped. A skipped path that has a record from
|
|
|
|
|
an earlier scan loses that record, and so does every path under an unreadable
|
|
|
|
|
directory. The database mirrors what the latest scan could verify.
|
|
|
|
|
- A file or directory that disappears while the scan runs is not an error. The
|
|
|
|
|
scan prints no line for it and does not count it as skipped. A path gone when
|
|
|
|
|
the walk reaches it gets no record, and so does a file gone when the hash
|
|
|
|
|
phase opens it. The records that an earlier scan left at or below it are
|
|
|
|
|
deleted.
|
|
|
|
|
phase opens it. The records an earlier scan left at or below it are deleted.
|
|
|
|
|
|
|
|
|
|
#### Concurrency
|
|
|
|
|
|
|
|
|
@@ -593,8 +552,7 @@ from a thread of its own and never blocks the workers.
|
|
|
|
|
|
|
|
|
|
- one line naming the database (see "Database");
|
|
|
|
|
- for each dropped operand, in operand order, the warning or the line described
|
|
|
|
|
under "Operands". `scan` prints neither for an operand it drops only because
|
|
|
|
|
it repeats another or lies under another;
|
|
|
|
|
under "Operands". A repeated operand, or one under another, gets neither;
|
|
|
|
|
- the progress display of each phase (see "Progress"). `scan` prints each
|
|
|
|
|
warning for a skipped path as it happens. It prints a line the same way for
|
|
|
|
|
each path left out as a default exclude or as not on a persistent filesystem;
|
|
|
|
@@ -621,14 +579,13 @@ signal `kill` sends by default:
|
|
|
|
|
- The records hashed but not yet written are written, so the next scan does not
|
|
|
|
|
hash them again.
|
|
|
|
|
- Nothing else is written or deleted. Records are deleted only after a complete
|
|
|
|
|
walk. So the records under paths that an interrupted walk never reached are
|
|
|
|
|
kept.
|
|
|
|
|
- The database is closed and the lock released.
|
|
|
|
|
walk. So the records under paths an interrupted walk never reached are kept.
|
|
|
|
|
- The database is closed and the lock released, as on any other exit.
|
|
|
|
|
- The line `scan: interrupted after N files` goes to stderr. `N` is the number
|
|
|
|
|
of files the walk reached. The exit code is 1.
|
|
|
|
|
|
|
|
|
|
The next scan reads none of the unchanged files already recorded, and brings the
|
|
|
|
|
database up to date as usual.
|
|
|
|
|
The next scan reads none of the files already recorded and brings the database
|
|
|
|
|
up to date as usual.
|
|
|
|
|
|
|
|
|
|
After the first signal `scan` stops catching signals. A second one ends it at
|
|
|
|
|
once, as any uncaught signal does. The records not yet written are lost, and the
|
|
|
|
@@ -638,12 +595,12 @@ and `scan` leaves it ignored.
|
|
|
|
|
|
|
|
|
|
### `report` mode
|
|
|
|
|
|
|
|
|
|
`report` reads every record from the database. It takes no positional arguments.
|
|
|
|
|
A positional argument is anything on the command line that is not a flag, such
|
|
|
|
|
as the paths `scan` takes. `report` never touches the scanned filesystem. It
|
|
|
|
|
neither stats nor opens any path in the records. It reads the database and
|
|
|
|
|
writes to stdout and stderr, and nothing else. Its output is the same whether or
|
|
|
|
|
not the scanned filesystem is still mounted.
|
|
|
|
|
`report` reads every record from the database. It takes no positional arguments,
|
|
|
|
|
meaning nothing may follow `report` on the command line but flags. It never
|
|
|
|
|
touches the scanned filesystem. It neither stats nor opens any path in the
|
|
|
|
|
records. It reads the database and writes to stdout and stderr, and nothing
|
|
|
|
|
else. Its output is the same whether or not the scanned filesystem is still
|
|
|
|
|
mounted.
|
|
|
|
|
|
|
|
|
|
Processing:
|
|
|
|
|
|
|
|
|
@@ -687,8 +644,8 @@ Four bytes are written as escapes:
|
|
|
|
|
|
|
|
|
|
Every other byte is written unchanged, including bytes that are not valid UTF-8,
|
|
|
|
|
the usual text encoding. Undoing the four escapes gives back the stored path.
|
|
|
|
|
Grouping and ordering use the stored path. The warnings about skipped paths that
|
|
|
|
|
`scan` prints are escaped the same way, so each stays one line.
|
|
|
|
|
Grouping and ordering use the stored path. The warnings `scan` prints are
|
|
|
|
|
escaped the same way, so each warning stays one line.
|
|
|
|
|
|
|
|
|
|
The summary line on stderr gives:
|
|
|
|
|
|
|
|
|
@@ -726,11 +683,11 @@ Processing:
|
|
|
|
|
- `trees` builds the directory tree and computes every directory's digest. It
|
|
|
|
|
then groups the directories by digest. Every group with two or more
|
|
|
|
|
directories is a duplicate-tree group.
|
|
|
|
|
- `trees` leaves out a group that a group of its members' parents already
|
|
|
|
|
implies. A group is left out when its members' parents are distinct
|
|
|
|
|
directories that all have the same digest. The parents' group, or an
|
|
|
|
|
ancestor's, already implies it. A group is always reported when it holds
|
|
|
|
|
sibling directories, or when its members' parents have different digests.
|
|
|
|
|
- Only the outermost duplicate trees are reported. A group is left out when its
|
|
|
|
|
members' parents are distinct directories that all have the same digest. The
|
|
|
|
|
parents' group, or an ancestor's, already implies it. A group holding sibling
|
|
|
|
|
directories, or members whose parents have different digests, is always
|
|
|
|
|
reported.
|
|
|
|
|
- Within a group, paths are sorted in byte order. The first is `first`, and
|
|
|
|
|
every other path is a `dupe`.
|
|
|
|
|
- Groups are ordered by total tree size, largest first, then by `first` path.
|
|
|
|
@@ -742,8 +699,8 @@ Two limitations are accepted:
|
|
|
|
|
They share storage, so removing one frees nothing. The scan uses the inode to
|
|
|
|
|
avoid reading a file twice, and does not store it.
|
|
|
|
|
- Only the regular files in the database define a tree. Empty directories are
|
|
|
|
|
invisible. When the scan skips a file in one copy but not in the other, the
|
|
|
|
|
two trees differ.
|
|
|
|
|
invisible. A file skipped during the scan in one copy but not in the other
|
|
|
|
|
makes two otherwise identical trees differ.
|
|
|
|
|
|
|
|
|
|
`trees` has four phases, each with its own progress display on stderr (see
|
|
|
|
|
"Progress"). `open` opens the database. `read` counts the records, then reads
|
|
|
|
@@ -809,10 +766,9 @@ size of recorded files 150000000000000 (136.4 TiB)
|
|
|
|
|
|
|
|
|
|
`info` has two phases, each with its own progress display on stderr (see
|
|
|
|
|
"Progress"). `open` opens the database. `count` reads the schema version and
|
|
|
|
|
counts the records in one transaction. A transaction is a group of reads or
|
|
|
|
|
writes that SQLite carries out as one step, against the database as it stands at
|
|
|
|
|
one moment. So both describe the database at one moment, even while a scan is
|
|
|
|
|
writing. `info` writes its lines after the `count` display has ended.
|
|
|
|
|
counts the records in one transaction, a read of the database at one moment. So
|
|
|
|
|
the two agree with each other even while a scan is writing. `info` writes its
|
|
|
|
|
lines after the `count` display has ended.
|
|
|
|
|
|
|
|
|
|
### Database
|
|
|
|
|
|
|
|
|
@@ -820,7 +776,7 @@ All four subcommands work on one SQLite database file.
|
|
|
|
|
|
|
|
|
|
#### Location
|
|
|
|
|
|
|
|
|
|
The database is the file named by `SFDUPES_DATABASE`, when that is set and not
|
|
|
|
|
The database is the file `SFDUPES_DATABASE` names, when that is set and not
|
|
|
|
|
empty, and otherwise `/var/lib/sfdupes/db.sqlite`.
|
|
|
|
|
|
|
|
|
|
Each subcommand prints one line on stderr naming the database it uses, before
|
|
|
|
@@ -854,21 +810,20 @@ the error for a missing database comes after it.
|
|
|
|
|
|
|
|
|
|
#### The scan lock
|
|
|
|
|
|
|
|
|
|
Only one `scan` at a time can hold the lock.
|
|
|
|
|
Only one `scan` runs against a database at a time.
|
|
|
|
|
|
|
|
|
|
- For its whole run, `scan` holds a lock on a lock file beside the database. The
|
|
|
|
|
lock is taken with the `flock(2)` system call, a request a program makes to
|
|
|
|
|
the kernel, and only one process can hold it. The lock file's name is the
|
|
|
|
|
database's path, with symlinks resolved, plus `.lock`. By default that is
|
|
|
|
|
`/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the same
|
|
|
|
|
lock file.
|
|
|
|
|
lock is taken with the `flock(2)` system call, and only one process can hold
|
|
|
|
|
it. The lock file's name is the database's path, with symlinks resolved, plus
|
|
|
|
|
`.lock`. By default that is `/var/lib/sfdupes/db.sqlite.lock`. Every path to
|
|
|
|
|
one database therefore names the same lock file.
|
|
|
|
|
- `scan` takes the lock before it walks the filesystem or opens the database.
|
|
|
|
|
Just before that, it creates the database file, empty, if it is missing. So
|
|
|
|
|
the lock file is named the same way before the first scan as after. A symlink
|
|
|
|
|
at the database path is followed, and its target is created.
|
|
|
|
|
- A second `scan` that finds the lock held does not wait. It fails at once with
|
|
|
|
|
an error naming the lock file, and exits 1. It has walked nothing and has not
|
|
|
|
|
opened the database. The running scan carries on.
|
|
|
|
|
- A second `scan` against the same database does not wait. It fails at once with
|
|
|
|
|
a one-line error naming the lock file, and exits 1. It has walked nothing and
|
|
|
|
|
opened nothing. The running scan carries on.
|
|
|
|
|
- The lock file is created on first use and never deleted. Only its owner may
|
|
|
|
|
open it. A leftover file blocks nothing. The lock ends with the process that
|
|
|
|
|
holds it, however that process ends. Deleting the file while a scan runs would
|
|
|
|
@@ -881,9 +836,8 @@ Only one `scan` at a time can hold the lock.
|
|
|
|
|
SQLite appends changes to a `-wal` file beside the database, and keeps a
|
|
|
|
|
lookup table for that file in a `-shm` file. Readers do not block the writer,
|
|
|
|
|
so running a report during a scan is safe.
|
|
|
|
|
- Every subcommand opens the database with a busy timeout of 10 seconds. That is
|
|
|
|
|
how long it waits for a database another process has locked before it gives
|
|
|
|
|
up.
|
|
|
|
|
- Every connection has a busy timeout of 10 seconds. That is how long it waits
|
|
|
|
|
for a database another connection has locked before it gives up.
|
|
|
|
|
- The filesystem is authoritative, and the database follows it. Hashed records
|
|
|
|
|
are written 10,000 at a time while the scan runs, so a report may see a scan's
|
|
|
|
|
changes partly applied. A scan that dies partway leaves a valid database
|
|
|
|
@@ -908,8 +862,7 @@ Only one `scan` at a time can hold the lock.
|
|
|
|
|
- Then, before the walk as well, it writes:
|
|
|
|
|
- the `-wal` and `-shm` files, which stay beside the database while the scan
|
|
|
|
|
runs;
|
|
|
|
|
- in a new database, the table, the index and the schema version described
|
|
|
|
|
under "Schema".
|
|
|
|
|
- in a new database, the table and the index described under "Schema".
|
|
|
|
|
- During the `load` and `walk` phases, no records are written. A first scan's
|
|
|
|
|
database holds no records until the hash phase, however long its walk runs.
|
|
|
|
|
- During the `hash` phase, records are written 10,000 at a time.
|
|
|
|
@@ -921,15 +874,13 @@ Only one `scan` at a time can hold the lock.
|
|
|
|
|
#### Schema
|
|
|
|
|
|
|
|
|
|
The schema is the database's layout, its table and its index.
|
|
|
|
|
`PRAGMA user_version` holds the schema version, currently 1. An empty database
|
|
|
|
|
is at version 0, and `scan` sets it up. Any other version is a fatal error.
|
|
|
|
|
`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal
|
|
|
|
|
error too. `scan` creates the table and the index and sets the version in one
|
|
|
|
|
transaction (see "`info` mode"). So the three land together or not at all. A
|
|
|
|
|
first scan stopped while doing so leaves an empty database, which the next scan
|
|
|
|
|
sets up. A database at version 0 that already has a `files` table was not made
|
|
|
|
|
by `sfdupes`. Every subcommand refuses it, with an error telling you to remove
|
|
|
|
|
the file and rescan.
|
|
|
|
|
`PRAGMA user_version` holds the schema version, currently 1. A database with any
|
|
|
|
|
other version is a fatal error. `scan` creates the table and the index and sets
|
|
|
|
|
the version in one transaction, a group of writes that lands whole or not at
|
|
|
|
|
all. A first scan stopped while doing so leaves an empty database, which the
|
|
|
|
|
next scan sets up. A database at version 0 that already has a `files` table was
|
|
|
|
|
not made by `sfdupes`. Every subcommand refuses it, with an error telling you to
|
|
|
|
|
remove the file and rescan.
|
|
|
|
|
|
|
|
|
|
```sql
|
|
|
|
|
CREATE TABLE files (
|
|
|
|
@@ -952,16 +903,15 @@ CREATE INDEX files_signature ON files (size, head, tail, content);
|
|
|
|
|
count of nanoseconds cannot hold those. They serve change detection only and
|
|
|
|
|
are not part of the signature.
|
|
|
|
|
- `head`, `tail` and `content` each hold the raw 32 bytes of a SHA-256 (see
|
|
|
|
|
"Duplicate detection"). `report` and `trees` fail on a record whose hash is
|
|
|
|
|
not 32 bytes.
|
|
|
|
|
"Duplicate detection"). A record whose hash is not 32 bytes is an error.
|
|
|
|
|
- `files_signature` lets `report` read the records in signature order without
|
|
|
|
|
sorting the table.
|
|
|
|
|
|
|
|
|
|
### Progress
|
|
|
|
|
|
|
|
|
|
Each phase of every subcommand gets a display of its own on stderr from the
|
|
|
|
|
moment it starts. The display names the phase and shows its progress. The parts
|
|
|
|
|
of its line are listed below.
|
|
|
|
|
Every subcommand always shows that it is working. Each phase gets a display of
|
|
|
|
|
its own on stderr from the moment it starts. The display names the phase and
|
|
|
|
|
shows its progress. The parts of its line are listed below.
|
|
|
|
|
|
|
|
|
|
The phases, and what each display counts:
|
|
|
|
|
|
|
|
|
@@ -984,7 +934,7 @@ The phases, and what each display counts:
|
|
|
|
|
- `trees` has four phases:
|
|
|
|
|
- `[1/4] open` and `[2/4] read` are as for `report`, except that `read`
|
|
|
|
|
writes nothing.
|
|
|
|
|
- `[3/4] group` has the directories that `read` found as its total.
|
|
|
|
|
- `[3/4] group` has the directories `read` found as its total.
|
|
|
|
|
- `[4/4] write` has the duplicate-tree groups as its total.
|
|
|
|
|
- `info` has two phases:
|
|
|
|
|
- `[1/2] open` is as for `report`.
|
|
|
|
@@ -996,9 +946,8 @@ Each display is one line holding:
|
|
|
|
|
|
|
|
|
|
- the phase and its place among the phases;
|
|
|
|
|
- the progress, written `[m/n] x%` for `m` items done of a total `n`, which is
|
|
|
|
|
`x` percent of it. A phase that has no total shows the items done instead, as
|
|
|
|
|
`1234 files`. A phase waiting for its total shows what it waits for, as
|
|
|
|
|
`counting the records`, and no rate;
|
|
|
|
|
`x` percent of it. Before a phase has a total, the line shows the items done
|
|
|
|
|
instead, or what the phase is waiting for;
|
|
|
|
|
- the current rate, in the items the phase counts, such as files or records. It
|
|
|
|
|
covers the items completed over the last 5 seconds, or, when stderr is not a
|
|
|
|
|
terminal, since the line before. It falls to 0 while nothing completes. A
|
|
|
|
@@ -1022,9 +971,9 @@ phase waits, in place of the rate. That happens while one large file is read, or
|
|
|
|
|
while a disk or the database is slow. Each phase names the places it may wait
|
|
|
|
|
in, and a phase that may wait in two places names both, as `disk/db`:
|
|
|
|
|
|
|
|
|
|
- `walk` says `disk`, because it reads directories and stats files.
|
|
|
|
|
- `hash` says `disk/db`, because it reads files and writes their records to the
|
|
|
|
|
database between them.
|
|
|
|
|
- `walk` says `disk`, for files.
|
|
|
|
|
- `hash` says `disk/db`: it reads files and writes their records to the database
|
|
|
|
|
between them.
|
|
|
|
|
- `load`, `update` and the `read` phase of `trees` say `db`.
|
|
|
|
|
- The `read` phase of `report` says `db/out`, because it writes the report to
|
|
|
|
|
stdout while it reads.
|
|
|
|
@@ -1050,18 +999,18 @@ On a terminal:
|
|
|
|
|
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- No line reaches the terminal's last column. A line too wide for the terminal
|
|
|
|
|
first leaves out its rate, the words of a stalled phase, or what the phase is
|
|
|
|
|
waiting for. It leaves out its bar next. It is cut short at the edge only when
|
|
|
|
|
the rest does not fit on its own. The rest is:
|
|
|
|
|
- No line reaches the terminal's last column, however narrow the terminal is. A
|
|
|
|
|
line too wide for the terminal first leaves out its rate, or the words shown
|
|
|
|
|
in the rate's place. It leaves out its bar next. It is cut short at the edge
|
|
|
|
|
only when the four remaining parts do not fit on their own. Those are:
|
|
|
|
|
- the phase and its place;
|
|
|
|
|
- the progress, when the phase shows one;
|
|
|
|
|
- the progress;
|
|
|
|
|
- the elapsed time;
|
|
|
|
|
- the time remaining.
|
|
|
|
|
|
|
|
|
|
On a terminal 80 columns wide, no line is cut short, even hours into a scan
|
|
|
|
|
of millions of files. A terminal made narrower during a phase is fitted at
|
|
|
|
|
the next draw.
|
|
|
|
|
That happens hours into a scan of millions of files, on a terminal narrower
|
|
|
|
|
than about 75 columns. A terminal made narrower while a phase runs is fitted
|
|
|
|
|
at the next draw.
|
|
|
|
|
|
|
|
|
|
- A warning printed during a phase lands on a line of its own, never inside the
|
|
|
|
|
display. So does every line written to stdout when stdout and stderr are the
|
|
|
|
@@ -1077,8 +1026,8 @@ Each phase prints its line:
|
|
|
|
|
- every 5 seconds while it runs, whether or not an item has completed;
|
|
|
|
|
- once more as it ends.
|
|
|
|
|
|
|
|
|
|
When a phase stops short of its total, as an interrupted one does, it keeps
|
|
|
|
|
showing the count it stopped at.
|
|
|
|
|
A phase that stops short of its total, as an interrupted one does, keeps showing
|
|
|
|
|
the count it stopped at.
|
|
|
|
|
|
|
|
|
|
### Error handling and exit codes
|
|
|
|
|
|
|
|
|
@@ -1087,15 +1036,13 @@ showing the count it stopped at.
|
|
|
|
|
error is one of these:
|
|
|
|
|
- an operand that does not exist, or whose details `scan` cannot read;
|
|
|
|
|
- a list of mounted filesystems that `scan` cannot read;
|
|
|
|
|
- another `scan` holding the lock (see "The scan lock");
|
|
|
|
|
- a database, its directory or its lock file that cannot be created or
|
|
|
|
|
opened;
|
|
|
|
|
- a read or write of the database that fails;
|
|
|
|
|
- another `scan` already running against the same database;
|
|
|
|
|
- a database that cannot be created, opened, read or written, its directory
|
|
|
|
|
and its lock file included;
|
|
|
|
|
- a relative database path when the working directory cannot be read;
|
|
|
|
|
- a database at a schema version other than 0 or 1;
|
|
|
|
|
- a database at version 0 read by `report`, `trees` or `info`;
|
|
|
|
|
- a database at version 0 that already has a `files` table (see "Schema");
|
|
|
|
|
- a record with a hash that is not 32 bytes, read by `report` or `trees`;
|
|
|
|
|
- a database with another schema version, or one at version 0 that already
|
|
|
|
|
has a `files` table;
|
|
|
|
|
- a record whose `head`, `tail` or `content` is not 32 bytes;
|
|
|
|
|
- a missing database for `report`, `trees` or `info`;
|
|
|
|
|
- a failed write to stdout.
|
|
|
|
|
- `2`: a usage error, which is one of these:
|
|
|
|
@@ -1107,9 +1054,9 @@ showing the count it stopped at.
|
|
|
|
|
- `--workers` below 1;
|
|
|
|
|
- `report`, `trees` or `info` with a positional argument.
|
|
|
|
|
|
|
|
|
|
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
|
|
|
|
stdout, such as a full disk, is reported the same way. Two cases never reach
|
|
|
|
|
`sfdupes` as a failed write:
|
|
|
|
|
A fatal error is reported in one line on stderr that starts with `sfdupes:`. A
|
|
|
|
|
failed write to stdout, such as a full disk, is reported the same way. Two cases
|
|
|
|
|
never reach `sfdupes` as a failed write:
|
|
|
|
|
|
|
|
|
|
- The reader of a stdout pipe may exit early, as `head` does in
|
|
|
|
|
`sfdupes report | head`. The next write then ends `sfdupes` with `SIGPIPE`,
|
|
|
|
@@ -1126,8 +1073,8 @@ This repository follows the
|
|
|
|
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
|
|
|
|
standard. The executables in `script/` are the entrypoints of the development
|
|
|
|
|
workflow, and the `Makefile` targets call them, except `build` and `clean` (see
|
|
|
|
|
"Build"). Every script runs under plain `sh`, the standard Unix shell. It finds
|
|
|
|
|
the repository root itself, so it can be run from any directory.
|
|
|
|
|
"Build"). Every script is POSIX `sh`. It finds the repository root itself, so it
|
|
|
|
|
can be run from any directory.
|
|
|
|
|
|
|
|
|
|
- `script/bootstrap` installs what building and developing need. It can be run
|
|
|
|
|
again at any time, and installed tools are kept. `git`, `make` and `go` are
|
|
|
|
@@ -1139,16 +1086,14 @@ the repository root itself, so it can be run from any directory.
|
|
|
|
|
- `apk`.
|
|
|
|
|
|
|
|
|
|
Then come `node` and `yarn`. `prettier`, the Markdown formatter, is a `node`
|
|
|
|
|
program. `yarn` is the package manager that installs it:
|
|
|
|
|
program, and `yarn` is the package manager that installs it:
|
|
|
|
|
- An installed `node` is used when its major version is 22. A `yarn` already
|
|
|
|
|
on `PATH` is then used as it is. When `yarn` is missing, `yarn` 1.22.22 is
|
|
|
|
|
activated through `corepack`, the `yarn` installer that ships with `node`.
|
|
|
|
|
When there is no `corepack` either, the script installs `yarn` with `npm`,
|
|
|
|
|
the package manager that ships with `node`.
|
|
|
|
|
- Otherwise the script installs `node` 22.17.0 through `nvm`, a `node`
|
|
|
|
|
version manager, and activates `yarn` 1.22.22 under that `node` through
|
|
|
|
|
`corepack`. When `nvm` is missing, the script first downloads it and
|
|
|
|
|
checks the archive's SHA-256.
|
|
|
|
|
When there is no `corepack` either, `npm` installs it.
|
|
|
|
|
- Otherwise the script downloads `nvm`, a `node` version manager, and checks
|
|
|
|
|
the archive's SHA-256. It installs `node` 22.17.0 through `nvm`, and
|
|
|
|
|
activates `yarn` 1.22.22 under that `node` through `corepack`.
|
|
|
|
|
|
|
|
|
|
`yarn install --frozen-lockfile` then installs the `prettier` that
|
|
|
|
|
`package.json` and `yarn.lock` pin. The linter is never installed. It runs
|
|
|
|
@@ -1164,22 +1109,18 @@ the repository root itself, so it can be run from any directory.
|
|
|
|
|
unsynchronized access to memory from several threads. The suite runs with a
|
|
|
|
|
90-second timeout, and its coverage of the code is measured. On failure it
|
|
|
|
|
reruns the suite verbosely, so that the log names the failing test. It builds
|
|
|
|
|
only the `test` stage of the `Dockerfile` and writes no image. A stage is one
|
|
|
|
|
named part of the `Dockerfile`, which starts from an image of its own and can
|
|
|
|
|
be built alone. The race detector needs cgo, Go's bridge to C code, and a C
|
|
|
|
|
compiler. The stage therefore starts from a Debian `golang` image, which has
|
|
|
|
|
the C compiler `gcc`. The image is pinned by its digest, its content hash. The
|
|
|
|
|
tests run as the user `nobody`, because several of them make a file
|
|
|
|
|
unreadable, and root reads it anyway.
|
|
|
|
|
only the `test` stage of the `Dockerfile` and writes no image. The race
|
|
|
|
|
detector needs cgo, Go's bridge to C code, and a C compiler. The stage
|
|
|
|
|
therefore starts from a Debian `golang` image, which has `gcc`. The image is
|
|
|
|
|
pinned by its digest, its content hash. The tests run as `nobody`, because
|
|
|
|
|
several of them make a file unreadable, and root reads it anyway.
|
|
|
|
|
- `script/lint` runs the linter, `golangci-lint`, which checks the Go sources
|
|
|
|
|
for common mistakes. It builds only the `lint` stage of the `Dockerfile` and
|
|
|
|
|
writes no image. The stage starts from the `golangci/golangci-lint` image,
|
|
|
|
|
pinned by its digest, and runs these checks as build steps:
|
|
|
|
|
- the `gofmt` check, which fails when `gofmt`, Go's own formatter, would
|
|
|
|
|
change a Go source;
|
|
|
|
|
- `golangci-lint config verify`, which checks the linter's settings file
|
|
|
|
|
`.golangci.yml`;
|
|
|
|
|
- `golangci-lint run`, which lints the Go sources.
|
|
|
|
|
pinned by its digest, and runs three build steps:
|
|
|
|
|
- the `gofmt` check;
|
|
|
|
|
- `golangci-lint config verify`;
|
|
|
|
|
- `golangci-lint run`.
|
|
|
|
|
|
|
|
|
|
A successful build is a clean lint. The linter never runs on the host, so a
|
|
|
|
|
working `docker` is the one thing linting needs.
|
|
|
|
@@ -1188,8 +1129,8 @@ the repository root itself, so it can be run from any directory.
|
|
|
|
|
with `prettier`, in place. `prettier` indents by 4 spaces and wraps prose, as
|
|
|
|
|
`.prettierrc` sets and as the script also passes on the command line. It wraps
|
|
|
|
|
at 80 columns, its default width. Both run on the host. `prettier` runs
|
|
|
|
|
through `yarn` at the version that `yarn.lock` pins. When the `node` on `PATH`
|
|
|
|
|
is not `node` 22, the script loads the `node` 22.17.0 that `script/bootstrap`
|
|
|
|
|
through `yarn` at the version `yarn.lock` pins. When the `node` on `PATH` is
|
|
|
|
|
not `node` 22, the script loads the `node` 22.17.0 that `script/bootstrap`
|
|
|
|
|
installed.
|
|
|
|
|
- `script/fmt-check` is the read-only form of `script/fmt`. It names any
|
|
|
|
|
unformatted file and exits non-zero instead of writing. `gofmt` and `prettier`
|
|
|
|
@@ -1199,19 +1140,17 @@ the repository root itself, so it can be run from any directory.
|
|
|
|
|
- `script/test` and `script/lint`, which need `docker`;
|
|
|
|
|
- `script/fmt-check`, which needs what `script/bootstrap` installs.
|
|
|
|
|
- `script/docker` builds the Docker image, tagged with the name from
|
|
|
|
|
`script/projectname`. It passes the output of
|
|
|
|
|
`git describe --tags --always --dirty`, or `unknown` when that is empty, to
|
|
|
|
|
the `Dockerfile` as the build argument `VERSION`. A build argument is a value
|
|
|
|
|
that a `docker build` hands to the `Dockerfile`. The build stage depends on
|
|
|
|
|
the `lint` and `test` stages, so this also runs the tests and the linter.
|
|
|
|
|
`script/projectname`. It passes `git describe --tags --always --dirty`, or
|
|
|
|
|
`unknown` when that is empty, as the `VERSION` build argument. The build stage
|
|
|
|
|
depends on the `lint` and `test` stages, so this also runs the tests and the
|
|
|
|
|
linter.
|
|
|
|
|
- `script/cibuild` runs `script/bootstrap` and then `script/check`. Then it
|
|
|
|
|
builds the image as `script/docker` does. The Gitea workflow in
|
|
|
|
|
`.gitea/workflows/check.yml` runs it on every push. A successful run means
|
|
|
|
|
every check passed. The tests and the linter run twice, once in `script/check`
|
|
|
|
|
and again as stages of the image build.
|
|
|
|
|
- `script/precommit` runs from the `git` pre-commit hook, a script that `git`
|
|
|
|
|
runs before each commit. It runs `go mod tidy`, and fails the commit when that
|
|
|
|
|
changes `go.mod` or `go.sum`. Then it runs `script/check`.
|
|
|
|
|
builds the image as `script/docker` does. The repository's CI workflow on
|
|
|
|
|
Gitea runs it on every push, and a successful run means every check passed.
|
|
|
|
|
The tests and the linter run twice, once in `script/check` and again as stages
|
|
|
|
|
of the image build.
|
|
|
|
|
- `script/precommit` runs from the `git` pre-commit hook. It fails the commit
|
|
|
|
|
when `go mod tidy` would change anything. Then it runs `script/check`.
|
|
|
|
|
- `script/install-precommit` installs `.git/hooks/pre-commit`, which runs
|
|
|
|
|
`script/precommit`.
|
|
|
|
|
|
|
|
|
@@ -1221,9 +1160,9 @@ install system packages and download the Go modules. Every step that checks or
|
|
|
|
|
compiles the code comes after it. Every `RUN` step after it runs again on each
|
|
|
|
|
build. Without that, Docker would serve the test and lint steps from its cache
|
|
|
|
|
on an unchanged tree, and the build would pass without running them. The steps
|
|
|
|
|
above `ARG CACHEBUST` stay cached until a change to something they depend on,
|
|
|
|
|
such as `go.mod`. Go's build cache is kept between builds, so only what changed
|
|
|
|
|
is compiled again. The tests run with `-count=1`, so no test reports a stored
|
|
|
|
|
above `ARG CACHEBUST` stay cached until something they depend on changes, such
|
|
|
|
|
as `go.mod`. Go's build cache is kept between builds, so only what changed is
|
|
|
|
|
compiled again. The tests run with `-count=1`, so no test reports a stored
|
|
|
|
|
result. `golangci-lint run` keeps the linter's own cache as well, so an
|
|
|
|
|
unchanged package is not linted again.
|
|
|
|
|
|
|
|
|
@@ -1255,19 +1194,15 @@ commands of their own:
|
|
|
|
|
|
|
|
|
|
- The language is Go. The module is `sneak.berlin/go/sfdupes`, and the binary is
|
|
|
|
|
`sfdupes`.
|
|
|
|
|
- The code imports these modules beyond the standard library:
|
|
|
|
|
- The dependencies beyond the standard library are:
|
|
|
|
|
- `github.com/spf13/cobra`, for the command line;
|
|
|
|
|
- `github.com/schollz/progressbar/v3`, for the progress display;
|
|
|
|
|
- `golang.org/x/term`, to tell whether stderr is a terminal;
|
|
|
|
|
- `modernc.org/sqlite`, the SQLite driver;
|
|
|
|
|
- `golang.org/x/sys`, for the scan lock and the list of mounted filesystems.
|
|
|
|
|
|
|
|
|
|
No other dependency outside the standard library is added.
|
|
|
|
|
|
|
|
|
|
- The code stays pure Go, with no C code. Building on one system for another is
|
|
|
|
|
not a goal. `modernc.org/sqlite` is pure Go too, so the binary builds without
|
|
|
|
|
cgo. The `Makefile` exports `CGO_ENABLED=0`. The test build is the one
|
|
|
|
|
exception, because the race detector needs cgo (see `script/test`).
|
|
|
|
|
- `modernc.org/sqlite` is pure Go, so the binary builds without cgo. The
|
|
|
|
|
`Makefile` exports `CGO_ENABLED=0`. The test build is the one exception,
|
|
|
|
|
because the race detector needs cgo (see `script/test`).
|
|
|
|
|
- `report` and `trees` give the same output for the same database contents,
|
|
|
|
|
whatever order the records were inserted in.
|
|
|
|
|
|
|
|
|
@@ -1277,15 +1212,12 @@ All of the following, run in this directory, must pass:
|
|
|
|
|
|
|
|
|
|
1. `make check` passes.
|
|
|
|
|
2. `make docker` succeeds.
|
|
|
|
|
3. The test below behaves as described. It builds a throwaway tree in a new
|
|
|
|
|
directory under `/var/tmp`, which is on a persistent filesystem. On many
|
|
|
|
|
systems `/tmp` is a `tmpfs`, which `scan` leaves out. Never test against real
|
|
|
|
|
data.
|
|
|
|
|
3. The test below behaves as described. It builds a throwaway tree in a
|
|
|
|
|
temporary directory. Never test against real data.
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
make build
|
|
|
|
|
d=$(mktemp -d /var/tmp/sfdupes.XXXXXX)
|
|
|
|
|
export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite"
|
|
|
|
|
d=$(mktemp -d)
|
|
|
|
|
export SFDUPES_DATABASE="$(mktemp -d)/db.sqlite"
|
|
|
|
|
mkdir -p "$d/a" "$d/b"
|
|
|
|
|
head -c 2000 /dev/urandom > "$d/a/one.bin"
|
|
|
|
|
cp "$d/a/one.bin" "$d/b/copy.bin"
|
|
|
|
@@ -1336,11 +1268,11 @@ All of the following, run in this directory, must pass:
|
|
|
|
|
|
|
|
|
|
## TODO
|
|
|
|
|
|
|
|
|
|
Tracked in [`TODO.md`](TODO.md).
|
|
|
|
|
Tracked in [TODO.md](TODO.md).
|
|
|
|
|
|
|
|
|
|
## License
|
|
|
|
|
|
|
|
|
|
MIT. See [`LICENSE`](LICENSE).
|
|
|
|
|
MIT. See [LICENSE](LICENSE).
|
|
|
|
|
|
|
|
|
|
## Author
|
|
|
|
|
|
|
|
|
|