Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2b46d374b1 |
@@ -38,9 +38,11 @@ export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
|
||||
`scan` walks the trees named by its operands, the paths given after `scan` on
|
||||
the command line. The walk is the listing of every directory under them. `scan`
|
||||
keeps one database record per regular file. A regular file is an ordinary file,
|
||||
as opposed to a directory, a symlink or a device. `report` prints the file-level
|
||||
duplicates. `trees` prints the duplicate directory trees. `info` prints the
|
||||
database's path, size and record count.
|
||||
as opposed to a directory, a symlink or a device node. A symlink is a file that
|
||||
holds the path of another file and stands in for it. A device node is a file
|
||||
that stands for a piece of hardware, such as a disk or a terminal. `report`
|
||||
prints the file-level duplicates. `trees` prints the duplicate directory trees.
|
||||
`info` prints the database's path, size and record count.
|
||||
|
||||
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
|
||||
another file. The first `scan` creates it.
|
||||
@@ -67,11 +69,12 @@ The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set.
|
||||
|
||||
`sfdupes --version` prints `sfdupes VERSION` and exits. For a binary from
|
||||
`go install`, `VERSION` is the version Go recorded in it. For `@latest` that is
|
||||
a release tag such as `v0.0.3`. For `@main` it is a pseudo-version, Go's version
|
||||
string for an untagged commit. That string ends in the commit's hash. A binary
|
||||
built from a clone carries the version `git describe --tags --always --dirty`
|
||||
prints in that clone, such as `v0.0.3` on a tagged commit. `VERSION` is `dev`
|
||||
when no version is known. To see everything Go recorded in an installed binary:
|
||||
a release tag such as `v0.0.3`. For `@main` it is the commit's tag when it has
|
||||
one. Otherwise it is a pseudo-version, Go's version string for an untagged
|
||||
commit. That string ends in the commit's hash. A binary built from a clone
|
||||
carries the output of `git describe --tags --always --dirty` in that clone, such
|
||||
as `v0.0.3` on a tagged commit. `VERSION` is `dev` when no version is known. To
|
||||
see everything Go recorded in an installed binary:
|
||||
|
||||
```sh
|
||||
go version -m "$(command -v sfdupes)"
|
||||
@@ -105,9 +108,10 @@ docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
|
||||
|
||||
### Daily scan from `cron`
|
||||
|
||||
Run `scan` as root, so that it can read every file. A path it cannot read is
|
||||
skipped with a warning and loses its database record. As the file
|
||||
`/etc/cron.d/sfdupes`:
|
||||
Run `scan` as root, so that it can read every file. `scan` warns about a
|
||||
directory it cannot read, and about a new or changed file it cannot read. It
|
||||
skips the path and deletes the records at or below it (see "Rules for the
|
||||
walk"). As the file `/etc/cron.d/sfdupes`:
|
||||
|
||||
```
|
||||
30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log
|
||||
@@ -123,8 +127,9 @@ skipped with a warning and loses its database record. As the file
|
||||
- a warning for each path it skips;
|
||||
- a line for each system or temporary directory the walk leaves out (see
|
||||
"Rules for the walk");
|
||||
- a line for each path left out because its filesystem is one the kernel
|
||||
makes up or keeps in memory, such as `tmpfs`;
|
||||
- a line for each path left out because its filesystem is one the kernel,
|
||||
the core of the operating system, makes up or keeps in memory, such as
|
||||
`tmpfs`;
|
||||
- the summary line;
|
||||
- a warning when a report still has the database open as the scan ends (see
|
||||
"Concurrent reads").
|
||||
@@ -134,8 +139,8 @@ skipped with a warning and loses its database record. As the file
|
||||
- A scan that fails exits 1, with the reason among the last lines of the log.
|
||||
`tail` prints those lines, and `cron` mails them to root if the host can send
|
||||
mail.
|
||||
- A scan still running when the next one starts carries on. The new one fails at
|
||||
once, and the lines `cron` mails include
|
||||
- A running scan carries on when the next one starts. The new one fails at once,
|
||||
and the lines that `cron` mails include
|
||||
`sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
|
||||
- `report`, `trees` and `info` need only read access to the database. The umask
|
||||
is the setting that removes permissions from the files a process creates.
|
||||
@@ -153,8 +158,8 @@ to keep is your choice.
|
||||
|
||||
A row is a candidate, not proof:
|
||||
|
||||
- The reports show the files as they were at the last scan. A file may have
|
||||
changed or gone since.
|
||||
- The reports show each file as it was when a scan last recorded it. A file may
|
||||
have changed or gone since.
|
||||
- A file of 100 MiB or more is compared only on samples of its content (see
|
||||
"Duplicate detection").
|
||||
- Two paths that are hard links, two names for one file, are listed as
|
||||
@@ -201,9 +206,9 @@ two ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it
|
||||
makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
|
||||
of reads, at the price of a comparison that is not exact.
|
||||
|
||||
The hashes live in a database that is kept between runs. A rescan reads only the
|
||||
files that are new or whose size or modification time (mtime) changed. All
|
||||
analysis then works from the database alone.
|
||||
The hashes live in a database that is kept between runs. A rescan reads only new
|
||||
files, and files whose size changed or whose modification time (mtime) is newer
|
||||
than recorded. All analysis then works from the database alone.
|
||||
|
||||
The goal is whole duplicated trees, not only single files. A download extracted
|
||||
twice, or a project copied once too often, is such a tree. It can be removed as
|
||||
@@ -224,10 +229,9 @@ In order:
|
||||
its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read
|
||||
in full. The target is a filesystem of about ten million files and 150 TB, on
|
||||
disks that may be slow or busy. During a scan, one small record per file is
|
||||
held in memory. It holds the path and the two values the walk compares, the
|
||||
size and the mtime. The hashes stay in the database. `report` holds the
|
||||
duplicate paths of one file size at a time. `trees` holds one entry per
|
||||
directory, plus the files of the directories it is reading.
|
||||
held in memory. The hashes stay in the database. `report` holds the duplicate
|
||||
paths of one file size at a time. `trees` holds one entry per directory, plus
|
||||
the files of the directories it is reading.
|
||||
3. **Rescan cheaply, analyze from the database.** The scan keeps its database
|
||||
between runs, and an unchanged file is never read again. `report`, `trees`
|
||||
and `info` work from the database alone and never touch the scanned
|
||||
@@ -261,11 +265,14 @@ The reference below is also the program's specification.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No byte-for-byte compare, and no deleting or linking of duplicates. The
|
||||
reports are advisory. Acting on them is your job.
|
||||
- No byte-for-byte compare.
|
||||
- No deleting of duplicates.
|
||||
- No replacing a copy with a hard link to another copy. The reports are
|
||||
advisory. Acting on them is your job.
|
||||
- The SQLite database is the only stored data. There are no export or import
|
||||
formats.
|
||||
- No daemon or filesystem watcher. Scheduling rescans is `cron`'s job.
|
||||
- `sfdupes` does not run in the background or watch the filesystem for changes.
|
||||
Scheduling rescans is `cron`'s job.
|
||||
|
||||
## Reference
|
||||
|
||||
@@ -280,9 +287,9 @@ The reference below is also the program's specification.
|
||||
hashes them. The default is the number of CPUs. `N` must be at least 1. A
|
||||
smaller value is a usage error, reported in one line on stderr with exit 2
|
||||
before anything is scanned.
|
||||
- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand on that
|
||||
operand's filesystem, as the same flag does for `du` and `rsync`. By default
|
||||
the walk also enters directories on other filesystems.
|
||||
- `-x`, `--one-file-system` (`scan`) keeps the walk of each operand out of
|
||||
directories on other filesystems, as the same flag does for `du` and `rsync`.
|
||||
By default the walk also enters directories on other filesystems.
|
||||
- `--no-default-excludes` (`scan`) lets the walk enter the default excludes.
|
||||
Those are the system and temporary directories the walk otherwise leaves out,
|
||||
such as `/var/tmp` on Linux (see "Rules for the walk").
|
||||
@@ -343,14 +350,15 @@ default. An operand may be a directory or a regular file.
|
||||
the exit code is 2.
|
||||
- An operand that does not exist is a fatal error. `scan` then stops before it
|
||||
walks anything, and exits 1.
|
||||
- Each operand is made absolute before the walk. A `.` or `..` in it is resolved
|
||||
as text, and symlinks are not followed. Each record is stored under that
|
||||
absolute path, so results do not depend on the working directory.
|
||||
- Each operand is made absolute before the walk. A relative operand is joined to
|
||||
the working directory as `pwd` prints it, with its symlinks kept. A `.` or
|
||||
`..` in it is resolved as text, and symlinks are not followed. Each record is
|
||||
stored under that absolute path.
|
||||
- All operands belong to one scan, and one walk covers them all at once (see
|
||||
"Concurrency").
|
||||
- An operand that repeats another is dropped before the walk, without a line.
|
||||
- An operand that repeats another is dropped before the walk.
|
||||
- An operand under another operand is dropped the same way, unless one of the
|
||||
cases below applies. No file is then reached twice.
|
||||
cases below applies. No path is then walked twice.
|
||||
- An operand under another is kept, and walked on its own, when the inner
|
||||
operand or a directory between the two is one of these:
|
||||
- a default exclude (see "Rules for the walk");
|
||||
@@ -359,19 +367,21 @@ default. An operand may be a directory or a regular file.
|
||||
It is also kept when a directory between the two is on a filesystem that is
|
||||
not persistent. Such a filesystem is one the kernel makes up or keeps in
|
||||
memory (see "Rules for the walk"). The outer operand's walk does not enter
|
||||
such a directory, so the inner operand is walked on its own. For an inner
|
||||
operand that is a regular file, the check starts at its directory.
|
||||
such a directory, so the inner operand is walked on its own.
|
||||
|
||||
- No other case keeps an operand under another. An operand reached through a
|
||||
symlinked directory below the outer operand is dropped too, without a line.
|
||||
The outer operand's walk does not follow the symlink, so no file under the
|
||||
inner operand is scanned. The records an earlier scan left under it are
|
||||
deleted.
|
||||
symlinked directory below the outer operand is dropped too. The outer
|
||||
operand's walk does not follow the symlink, so no file under the inner operand
|
||||
is scanned. The records that an earlier scan left under it are deleted. The
|
||||
`-x` case above also covers an operand reached that way.
|
||||
- An operand that is neither a regular file nor a directory is not scanned.
|
||||
`scan` prints a warning naming the path and what it is. It counts the operand
|
||||
as skipped and drops it. An operand beneath a dropped one is scanned all the
|
||||
same. The scan keeps the records beneath a dropped operand, unless another
|
||||
operand covers them.
|
||||
as skipped and drops it. An operand beneath a symlink operand is scanned all
|
||||
the same. The scan keeps the records beneath a dropped operand, unless another
|
||||
operand covers them. After a `cd` through a symlink, `scan .` names the
|
||||
symlink and is skipped as a symlink operand. `scan sub` in that directory
|
||||
records its files under the symlink's path. `report` can then list each as a
|
||||
duplicate of the same file recorded under its real path.
|
||||
- An operand at or inside a directory named `.zfs` is never scanned, whether
|
||||
given alone or under another operand. The check looks at the path as given and
|
||||
at the path with its symlinks resolved. `scan` prints a warning naming the
|
||||
@@ -381,6 +391,9 @@ default. An operand may be a directory or a regular file.
|
||||
- An operand on a filesystem that is not persistent is dropped too. `scan`
|
||||
prints a line naming it and its filesystem type, and does not count it as
|
||||
skipped.
|
||||
- `scan` prints these warnings, and the line for an operand that is not on a
|
||||
persistent filesystem, even when the operand repeats another or lies under
|
||||
another.
|
||||
- A scan whose every operand is dropped walks nothing and exits 0.
|
||||
|
||||
#### What a scan does to the database
|
||||
@@ -392,15 +405,16 @@ operands:
|
||||
- A file already in the database is not read again when its size and mtime match
|
||||
its record. The mtime matches when it is not newer than the recorded one. This
|
||||
is what makes a daily rescan cheap.
|
||||
- A file whose size differs, or whose mtime is newer, is hashed again. The mtime
|
||||
is compared to the nanosecond, as finely as the filesystem records it. A
|
||||
- `scan` hashes a file again when its size differs or its mtime is newer. The
|
||||
mtime is compared to the nanosecond, as finely as the filesystem records it. A
|
||||
same-size rewrite counts as a change whenever the filesystem gives it a later
|
||||
mtime, even within the same second.
|
||||
- The scan deletes each record under a scanned operand whose file it could not
|
||||
verify. That covers deleted files, and paths the scan could not stat or read
|
||||
this run. To stat a file is to read its details, such as its size, without
|
||||
opening it. So the database holds only what the latest scan over each path
|
||||
could verify. A later successful scan adds such files again.
|
||||
opening it. So after a complete scan, the database holds only what that scan
|
||||
could verify under its operands. A later successful scan adds such files
|
||||
again.
|
||||
- Records outside the scanned operands are left alone, so separate trees can be
|
||||
scanned on different schedules into one database.
|
||||
- The scan keeps every record at or below a path that the walk leaves out for
|
||||
@@ -411,9 +425,8 @@ operands:
|
||||
|
||||
Such records count as outside the scanned operands. So with `-x`, a
|
||||
filesystem mounted under an operand can be scanned on its own schedule into
|
||||
the same database. An operand may lie at or inside a default exclude, or,
|
||||
with `-x`, at or inside a directory on another filesystem. Such an operand
|
||||
is walked on its own. Its walk verifies the records below it, so they are
|
||||
the same database. An operand may itself lie at or inside such a path, as
|
||||
"Operands" describes. Its walk verifies the records below it, so they are
|
||||
deleted when their files are gone.
|
||||
|
||||
#### Phases
|
||||
@@ -424,12 +437,13 @@ phases.
|
||||
1. `load` reads the existing records. Each record under the operands is kept in
|
||||
memory by path, with its size and mtime, for the walk to compare against.
|
||||
2. `walk` lists the trees under all operands. Several workers share one queue of
|
||||
directories to read. A worker is one thread of the program, and "Concurrency"
|
||||
says how many there are. Each worker reads one directory at a time. It hands
|
||||
the subdirectories it finds back to the queue, and it stats each regular file
|
||||
as it finds it. The walk compares each file with its record. An unchanged
|
||||
file needs nothing more. Every new or changed file is passed to the hash
|
||||
phase.
|
||||
directories to read. A worker is one thread of the program. A thread is one
|
||||
of several parts of a program that run at the same time and share its memory.
|
||||
"Concurrency" says how many workers there are. Each worker reads one
|
||||
directory at a time. It hands the subdirectories it finds back to the queue,
|
||||
and it stats each regular file as it finds it. The walk compares each file
|
||||
with its record. An unchanged file needs nothing more. Every new or changed
|
||||
file is passed to the hash phase.
|
||||
3. `hash` reads every new or changed file and computes its three hashes. Files
|
||||
are read in inode order. An inode is the filesystem's own record of a file.
|
||||
Files with nearby inode numbers usually lie near each other on disk. Reading
|
||||
@@ -448,7 +462,7 @@ walk practical on large or busy storage pools.
|
||||
#### Rules for the walk
|
||||
|
||||
- Only regular files are recorded. Every other kind of file, such as a symlink
|
||||
or a device node, is passed over. Symlinks are never followed.
|
||||
or a device node, is passed over. The walk follows no symlink it finds.
|
||||
- The walk never enters a directory named `.zfs`. Such a directory holds the
|
||||
filesystem's ZFS snapshots, which are its saved earlier states. Walking it
|
||||
would list every file once per snapshot. `scan` never starts a walk at or
|
||||
@@ -466,14 +480,14 @@ walk practical on large or busy storage pools.
|
||||
operand at the path given, even when that path runs through a symlinked
|
||||
directory. Such an operand can lead to the database files, and the walk
|
||||
passes over them there as well. The walk prints no line for the database
|
||||
files and does not count them as skipped. A record an earlier scan left for
|
||||
one of them is deleted.
|
||||
files and does not count them as skipped. A record that an earlier scan left
|
||||
for one of them is deleted.
|
||||
|
||||
- By default the walk enters directories on other filesystems than its
|
||||
operand's. With `-x`, it never enters a directory on a different filesystem
|
||||
than its operand. `scan` prints no line for that directory and does not count
|
||||
it as skipped. The records below it are kept, as described under "What a scan
|
||||
does to the database".
|
||||
than its operand. `scan` prints no line for a directory that only `-x` leaves
|
||||
out, and does not count it as skipped. The records below it are kept, as
|
||||
described under "What a scan does to the database".
|
||||
- The walk never records anything on a filesystem that is not persistent. A
|
||||
filesystem is not persistent when the kernel makes it up or keeps it in
|
||||
memory. These are the types, as `sfdupes scan --help` lists them on Linux and
|
||||
@@ -509,11 +523,10 @@ walk practical on large or busy storage pools.
|
||||
The path is not counted as skipped. The records below it are kept, as
|
||||
described under "What a scan does to the database".
|
||||
|
||||
- Below an operand, the walk does not enter a default exclude. The default
|
||||
excludes hold the kernel's view of the running system and temporary files, so
|
||||
that `sfdupes scan /` walks only the files worth comparing. They depend on the
|
||||
system, and `sfdupes scan --help` lists those of the system it runs on. On
|
||||
Linux they are:
|
||||
- Below an operand, the walk does not enter a default exclude. On Linux, the
|
||||
default excludes hold temporary files and the kernel's view of the running
|
||||
system. They depend on the system, and `sfdupes scan --help` lists those of
|
||||
the system it runs on. On Linux they are:
|
||||
|
||||
```
|
||||
/proc, /sys, /dev, /run, /tmp, /var/tmp
|
||||
@@ -542,21 +555,27 @@ walk practical on large or busy storage pools.
|
||||
scan: skipped /var/tmp (default exclude)
|
||||
```
|
||||
|
||||
The directory is not counted as skipped. A default exclude on a filesystem
|
||||
that is not persistent, such as `/proc`, gets the line for its filesystem
|
||||
instead. One that is itself an operand gets no line, because it is walked as
|
||||
that operand. The scan keeps the records below a default exclude that the
|
||||
The directory is not counted as skipped. `scan` prints the filesystem line
|
||||
instead for a default exclude that is not on a persistent filesystem, such
|
||||
as `/proc`. One that is itself an operand gets no line, because it is walked
|
||||
as that operand. The scan keeps the records below a default exclude that the
|
||||
walk did not enter, as described under "What a scan does to the database".
|
||||
|
||||
- A path that cannot be read, for lack of permission or any other error, gets a
|
||||
one-line warning on stderr and is skipped. The scan continues, and the summary
|
||||
line says how many paths were skipped. A skipped path that has a record from
|
||||
an earlier scan loses that record, and so does every path under an unreadable
|
||||
directory. The database mirrors what the latest scan could verify.
|
||||
- `scan` skips each of these, with a one-line warning on stderr:
|
||||
- a directory it cannot read;
|
||||
- a path it cannot stat;
|
||||
- a new or changed file it cannot read.
|
||||
|
||||
An unchanged file is not opened, so it keeps its record even when it has
|
||||
become unreadable. The scan continues, and the summary line says how many
|
||||
paths were skipped. A skipped path that has a record from an earlier scan
|
||||
loses that record, and so does every path under an unreadable directory.
|
||||
|
||||
- A file or directory that disappears while the scan runs is not an error. The
|
||||
scan prints no line for it and does not count it as skipped. A path gone when
|
||||
the walk reaches it gets no record, and so does a file gone when the hash
|
||||
phase opens it. The records an earlier scan left at or below it are deleted.
|
||||
phase opens it. The records that an earlier scan left at or below it are
|
||||
deleted.
|
||||
|
||||
#### Concurrency
|
||||
|
||||
@@ -574,8 +593,8 @@ from a thread of its own and never blocks the workers.
|
||||
|
||||
- one line naming the database (see "Database");
|
||||
- for each dropped operand, in operand order, the warning or the line described
|
||||
under "Operands". An operand dropped as a repeat of another, or as lying under
|
||||
another, gets neither;
|
||||
under "Operands". `scan` prints neither for an operand it drops only because
|
||||
it repeats another or lies under another;
|
||||
- the progress display of each phase (see "Progress"). `scan` prints each
|
||||
warning for a skipped path as it happens. It prints a line the same way for
|
||||
each path left out as a default exclude or as not on a persistent filesystem;
|
||||
@@ -602,13 +621,14 @@ signal `kill` sends by default:
|
||||
- The records hashed but not yet written are written, so the next scan does not
|
||||
hash them again.
|
||||
- Nothing else is written or deleted. Records are deleted only after a complete
|
||||
walk. So the records under paths an interrupted walk never reached are kept.
|
||||
- The database is closed and the lock released, as on any other exit.
|
||||
walk. So the records under paths that an interrupted walk never reached are
|
||||
kept.
|
||||
- The database is closed and the lock released.
|
||||
- The line `scan: interrupted after N files` goes to stderr. `N` is the number
|
||||
of files the walk reached. The exit code is 1.
|
||||
|
||||
The next scan reads none of the files already recorded and brings the database
|
||||
up to date as usual.
|
||||
The next scan reads none of the unchanged files already recorded, and brings the
|
||||
database up to date as usual.
|
||||
|
||||
After the first signal `scan` stops catching signals. A second one ends it at
|
||||
once, as any uncaught signal does. The records not yet written are lost, and the
|
||||
@@ -667,8 +687,8 @@ Four bytes are written as escapes:
|
||||
|
||||
Every other byte is written unchanged, including bytes that are not valid UTF-8,
|
||||
the usual text encoding. Undoing the four escapes gives back the stored path.
|
||||
Grouping and ordering use the stored path. The warnings `scan` prints are
|
||||
escaped the same way, so each warning stays one line.
|
||||
Grouping and ordering use the stored path. The warnings about skipped paths that
|
||||
`scan` prints are escaped the same way, so each stays one line.
|
||||
|
||||
The summary line on stderr gives:
|
||||
|
||||
@@ -706,11 +726,11 @@ Processing:
|
||||
- `trees` builds the directory tree and computes every directory's digest. It
|
||||
then groups the directories by digest. Every group with two or more
|
||||
directories is a duplicate-tree group.
|
||||
- Only the outermost duplicate trees are reported. A group is left out when its
|
||||
members' parents are distinct directories that all have the same digest. The
|
||||
parents' group, or an ancestor's, already implies it. A group holding sibling
|
||||
directories, or members whose parents have different digests, is always
|
||||
reported.
|
||||
- `trees` leaves out a group that a group of its members' parents already
|
||||
implies. A group is left out when its members' parents are distinct
|
||||
directories that all have the same digest. The parents' group, or an
|
||||
ancestor's, already implies it. A group is always reported when it holds
|
||||
sibling directories, or when its members' parents have different digests.
|
||||
- Within a group, paths are sorted in byte order. The first is `first`, and
|
||||
every other path is a `dupe`.
|
||||
- Groups are ordered by total tree size, largest first, then by `first` path.
|
||||
@@ -722,8 +742,8 @@ Two limitations are accepted:
|
||||
They share storage, so removing one frees nothing. The scan uses the inode to
|
||||
avoid reading a file twice, and does not store it.
|
||||
- Only the regular files in the database define a tree. Empty directories are
|
||||
invisible. A file skipped during the scan in one copy but not in the other
|
||||
makes two otherwise identical trees differ.
|
||||
invisible. When the scan skips a file in one copy but not in the other, the
|
||||
two trees differ.
|
||||
|
||||
`trees` has four phases, each with its own progress display on stderr (see
|
||||
"Progress"). `open` opens the database. `read` counts the records, then reads
|
||||
@@ -800,7 +820,7 @@ All four subcommands work on one SQLite database file.
|
||||
|
||||
#### Location
|
||||
|
||||
The database is the file `SFDUPES_DATABASE` names, when that is set and not
|
||||
The database is the file named by `SFDUPES_DATABASE`, when that is set and not
|
||||
empty, and otherwise `/var/lib/sfdupes/db.sqlite`.
|
||||
|
||||
Each subcommand prints one line on stderr naming the database it uses, before
|
||||
@@ -834,20 +854,21 @@ the error for a missing database comes after it.
|
||||
|
||||
#### The scan lock
|
||||
|
||||
Only one `scan` runs against a database at a time.
|
||||
Only one `scan` at a time can hold the lock.
|
||||
|
||||
- For its whole run, `scan` holds a lock on a lock file beside the database. The
|
||||
lock is taken with the `flock(2)` system call, and only one process can hold
|
||||
it. The lock file's name is the database's path, with symlinks resolved, plus
|
||||
`.lock`. By default that is `/var/lib/sfdupes/db.sqlite.lock`. Every path to
|
||||
one database therefore names the same lock file.
|
||||
lock is taken with the `flock(2)` system call, a request a program makes to
|
||||
the kernel, and only one process can hold it. The lock file's name is the
|
||||
database's path, with symlinks resolved, plus `.lock`. By default that is
|
||||
`/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the same
|
||||
lock file.
|
||||
- `scan` takes the lock before it walks the filesystem or opens the database.
|
||||
Just before that, it creates the database file, empty, if it is missing. So
|
||||
the lock file is named the same way before the first scan as after. A symlink
|
||||
at the database path is followed, and its target is created.
|
||||
- A second `scan` against the same database does not wait. It fails at once with
|
||||
a one-line error naming the lock file, and exits 1. It has walked nothing and
|
||||
opened nothing. The running scan carries on.
|
||||
- A second `scan` that finds the lock held does not wait. It fails at once with
|
||||
an error naming the lock file, and exits 1. It has walked nothing and has not
|
||||
opened the database. The running scan carries on.
|
||||
- The lock file is created on first use and never deleted. Only its owner may
|
||||
open it. A leftover file blocks nothing. The lock ends with the process that
|
||||
holds it, however that process ends. Deleting the file while a scan runs would
|
||||
@@ -860,8 +881,9 @@ Only one `scan` runs against a database at a time.
|
||||
SQLite appends changes to a `-wal` file beside the database, and keeps a
|
||||
lookup table for that file in a `-shm` file. Readers do not block the writer,
|
||||
so running a report during a scan is safe.
|
||||
- Every connection has a busy timeout of 10 seconds. That is how long it waits
|
||||
for a database another connection has locked before it gives up.
|
||||
- Every subcommand opens the database with a busy timeout of 10 seconds. That is
|
||||
how long it waits for a database another process has locked before it gives
|
||||
up.
|
||||
- The filesystem is authoritative, and the database follows it. Hashed records
|
||||
are written 10,000 at a time while the scan runs, so a report may see a scan's
|
||||
changes partly applied. A scan that dies partway leaves a valid database
|
||||
@@ -899,13 +921,15 @@ Only one `scan` runs against a database at a time.
|
||||
#### Schema
|
||||
|
||||
The schema is the database's layout, its table and its index.
|
||||
`PRAGMA user_version` holds the schema version, currently 1. A database with any
|
||||
other version is a fatal error. `scan` creates the table and the index and sets
|
||||
the version in one transaction (see "`info` mode"). So the three land together
|
||||
or not at all. A first scan stopped while doing so leaves an empty database,
|
||||
which the next scan sets up. A database at version 0 that already has a `files`
|
||||
table was not made by `sfdupes`. Every subcommand refuses it, with an error
|
||||
telling you to remove the file and rescan.
|
||||
`PRAGMA user_version` holds the schema version, currently 1. An empty database
|
||||
is at version 0, and `scan` sets it up. Any other version is a fatal error.
|
||||
`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal
|
||||
error too. `scan` creates the table and the index and sets the version in one
|
||||
transaction (see "`info` mode"). So the three land together or not at all. A
|
||||
first scan stopped while doing so leaves an empty database, which the next scan
|
||||
sets up. A database at version 0 that already has a `files` table was not made
|
||||
by `sfdupes`. Every subcommand refuses it, with an error telling you to remove
|
||||
the file and rescan.
|
||||
|
||||
```sql
|
||||
CREATE TABLE files (
|
||||
@@ -935,9 +959,9 @@ CREATE INDEX files_signature ON files (size, head, tail, content);
|
||||
|
||||
### Progress
|
||||
|
||||
Every subcommand always shows that it is working. Each phase gets a display of
|
||||
its own on stderr from the moment it starts. The display names the phase and
|
||||
shows its progress. The parts of its line are listed below.
|
||||
Each phase of every subcommand gets a display of its own on stderr from the
|
||||
moment it starts. The display names the phase and shows its progress. The parts
|
||||
of its line are listed below.
|
||||
|
||||
The phases, and what each display counts:
|
||||
|
||||
@@ -960,7 +984,7 @@ The phases, and what each display counts:
|
||||
- `trees` has four phases:
|
||||
- `[1/4] open` and `[2/4] read` are as for `report`, except that `read`
|
||||
writes nothing.
|
||||
- `[3/4] group` has the directories `read` found as its total.
|
||||
- `[3/4] group` has the directories that `read` found as its total.
|
||||
- `[4/4] write` has the duplicate-tree groups as its total.
|
||||
- `info` has two phases:
|
||||
- `[1/2] open` is as for `report`.
|
||||
@@ -998,7 +1022,7 @@ phase waits, in place of the rate. That happens while one large file is read, or
|
||||
while a disk or the database is slow. Each phase names the places it may wait
|
||||
in, and a phase that may wait in two places names both, as `disk/db`:
|
||||
|
||||
- `walk` says `disk`, for files.
|
||||
- `walk` says `disk`, because it reads directories and stats files.
|
||||
- `hash` says `disk/db`, because it reads files and writes their records to the
|
||||
database between them.
|
||||
- `load`, `update` and the `read` phase of `trees` say `db`.
|
||||
@@ -1026,19 +1050,18 @@ On a terminal:
|
||||
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
|
||||
```
|
||||
|
||||
- No line reaches the terminal's last column, however narrow the terminal is. A
|
||||
line too wide for the terminal first leaves out its rate, the words of a
|
||||
stalled phase, or what the phase is waiting for. It leaves out its bar next.
|
||||
It is cut short at the edge only when the rest does not fit on its own. The
|
||||
rest is:
|
||||
- No line reaches the terminal's last column. A line too wide for the terminal
|
||||
first leaves out its rate, the words of a stalled phase, or what the phase is
|
||||
waiting for. It leaves out its bar next. It is cut short at the edge only when
|
||||
the rest does not fit on its own. The rest is:
|
||||
- the phase and its place;
|
||||
- the progress, when the phase shows one;
|
||||
- the elapsed time;
|
||||
- the time remaining.
|
||||
|
||||
That happens hours into a scan of millions of files, on a terminal narrower
|
||||
than about 75 columns. A terminal made narrower while a phase runs is fitted
|
||||
at the next draw.
|
||||
On a terminal 80 columns wide, no line is cut short, even hours into a scan
|
||||
of millions of files. A terminal made narrower during a phase is fitted at
|
||||
the next draw.
|
||||
|
||||
- A warning printed during a phase lands on a line of its own, never inside the
|
||||
display. So does every line written to stdout when stdout and stderr are the
|
||||
@@ -1054,8 +1077,8 @@ Each phase prints its line:
|
||||
- every 5 seconds while it runs, whether or not an item has completed;
|
||||
- once more as it ends.
|
||||
|
||||
A phase that stops short of its total, as an interrupted one does, keeps showing
|
||||
the count it stopped at.
|
||||
When a phase stops short of its total, as an interrupted one does, it keeps
|
||||
showing the count it stopped at.
|
||||
|
||||
### Error handling and exit codes
|
||||
|
||||
@@ -1064,12 +1087,14 @@ the count it stopped at.
|
||||
error is one of these:
|
||||
- an operand that does not exist, or whose details `scan` cannot read;
|
||||
- a list of mounted filesystems that `scan` cannot read;
|
||||
- another `scan` already running against the same database;
|
||||
- a database that cannot be created or opened, or whose reads or writes
|
||||
fail, its directory and its lock file included;
|
||||
- another `scan` holding the lock (see "The scan lock");
|
||||
- a database, its directory or its lock file that cannot be created or
|
||||
opened;
|
||||
- a read or write of the database that fails;
|
||||
- a relative database path when the working directory cannot be read;
|
||||
- a database with another schema version, or one at version 0 that already
|
||||
has a `files` table;
|
||||
- a database at a schema version other than 0 or 1;
|
||||
- a database at version 0 read by `report`, `trees` or `info`;
|
||||
- a database at version 0 that already has a `files` table (see "Schema");
|
||||
- a record with a hash that is not 32 bytes, read by `report` or `trees`;
|
||||
- a missing database for `report`, `trees` or `info`;
|
||||
- a failed write to stdout.
|
||||
@@ -1082,9 +1107,9 @@ the count it stopped at.
|
||||
- `--workers` below 1;
|
||||
- `report`, `trees` or `info` with a positional argument.
|
||||
|
||||
A fatal error is reported in one line on stderr that starts with `sfdupes:`. A
|
||||
failed write to stdout, such as a full disk, is reported the same way. Two cases
|
||||
never reach `sfdupes` as a failed write:
|
||||
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
||||
stdout, such as a full disk, is reported the same way. Two cases never reach
|
||||
`sfdupes` as a failed write:
|
||||
|
||||
- The reader of a stdout pipe may exit early, as `head` does in
|
||||
`sfdupes report | head`. The next write then ends `sfdupes` with `SIGPIPE`,
|
||||
@@ -1118,8 +1143,8 @@ the repository root itself, so it can be run from any directory.
|
||||
- An installed `node` is used when its major version is 22. A `yarn` already
|
||||
on `PATH` is then used as it is. When `yarn` is missing, `yarn` 1.22.22 is
|
||||
activated through `corepack`, the `yarn` installer that ships with `node`.
|
||||
When there is no `corepack` either, `npm`, the package manager that ships
|
||||
with `node`, installs it.
|
||||
When there is no `corepack` either, the script installs `yarn` with `npm`,
|
||||
the package manager that ships with `node`.
|
||||
- Otherwise the script installs `node` 22.17.0 through `nvm`, a `node`
|
||||
version manager, and activates `yarn` 1.22.22 under that `node` through
|
||||
`corepack`. When `nvm` is missing, the script first downloads it and
|
||||
@@ -1149,9 +1174,9 @@ the repository root itself, so it can be run from any directory.
|
||||
- `script/lint` runs the linter, `golangci-lint`, which checks the Go sources
|
||||
for common mistakes. It builds only the `lint` stage of the `Dockerfile` and
|
||||
writes no image. The stage starts from the `golangci/golangci-lint` image,
|
||||
pinned by its digest, and runs three build steps:
|
||||
- the `gofmt` check, which fails on a Go source that `gofmt`, Go's own
|
||||
formatter, would change;
|
||||
pinned by its digest, and runs these checks as build steps:
|
||||
- the `gofmt` check, which fails when `gofmt`, Go's own formatter, would
|
||||
change a Go source;
|
||||
- `golangci-lint config verify`, which checks the linter's settings file
|
||||
`.golangci.yml`;
|
||||
- `golangci-lint run`, which lints the Go sources.
|
||||
@@ -1163,8 +1188,8 @@ the repository root itself, so it can be run from any directory.
|
||||
with `prettier`, in place. `prettier` indents by 4 spaces and wraps prose, as
|
||||
`.prettierrc` sets and as the script also passes on the command line. It wraps
|
||||
at 80 columns, its default width. Both run on the host. `prettier` runs
|
||||
through `yarn` at the version `yarn.lock` pins. When the `node` on `PATH` is
|
||||
not `node` 22, the script loads the `node` 22.17.0 that `script/bootstrap`
|
||||
through `yarn` at the version that `yarn.lock` pins. When the `node` on `PATH`
|
||||
is not `node` 22, the script loads the `node` 22.17.0 that `script/bootstrap`
|
||||
installed.
|
||||
- `script/fmt-check` is the read-only form of `script/fmt`. It names any
|
||||
unformatted file and exits non-zero instead of writing. `gofmt` and `prettier`
|
||||
@@ -1185,8 +1210,8 @@ the repository root itself, so it can be run from any directory.
|
||||
every check passed. The tests and the linter run twice, once in `script/check`
|
||||
and again as stages of the image build.
|
||||
- `script/precommit` runs from the `git` pre-commit hook, a script that `git`
|
||||
runs before each commit. It fails the commit when `go mod tidy` would change
|
||||
anything. Then it runs `script/check`.
|
||||
runs before each commit. It runs `go mod tidy`, and fails the commit when that
|
||||
changes `go.mod` or `go.sum`. Then it runs `script/check`.
|
||||
- `script/install-precommit` installs `.git/hooks/pre-commit`, which runs
|
||||
`script/precommit`.
|
||||
|
||||
@@ -1196,9 +1221,9 @@ install system packages and download the Go modules. Every step that checks or
|
||||
compiles the code comes after it. Every `RUN` step after it runs again on each
|
||||
build. Without that, Docker would serve the test and lint steps from its cache
|
||||
on an unchanged tree, and the build would pass without running them. The steps
|
||||
above `ARG CACHEBUST` stay cached until something they depend on changes, such
|
||||
as `go.mod`. Go's build cache is kept between builds, so only what changed is
|
||||
compiled again. The tests run with `-count=1`, so no test reports a stored
|
||||
above `ARG CACHEBUST` stay cached until a change to something they depend on,
|
||||
such as `go.mod`. Go's build cache is kept between builds, so only what changed
|
||||
is compiled again. The tests run with `-count=1`, so no test reports a stored
|
||||
result. `golangci-lint run` keeps the linter's own cache as well, so an
|
||||
unchanged package is not linted again.
|
||||
|
||||
@@ -1258,6 +1283,7 @@ All of the following, run in this directory, must pass:
|
||||
data.
|
||||
|
||||
```sh
|
||||
make build
|
||||
d=$(mktemp -d /var/tmp/sfdupes.XXXXXX)
|
||||
export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite"
|
||||
mkdir -p "$d/a" "$d/b"
|
||||
|
||||
+7
-8
@@ -773,8 +773,8 @@ func TestBuildVersion(t *testing.T) {
|
||||
func TestRunVersionWriteFailureIsFatal(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
// README §Error handling: a stdout write failure exits 1, reported
|
||||
// in one line on stderr.
|
||||
// README §Error handling: a failed write to stdout is a fatal error,
|
||||
// exit 1, reported on stderr starting with "sfdupes:".
|
||||
var stderr bytes.Buffer
|
||||
|
||||
code := run([]string{"--version"}, failingWriter{}, &stderr)
|
||||
@@ -1279,9 +1279,8 @@ func holdScanLock(t *testing.T, path string) {
|
||||
}
|
||||
|
||||
func TestRunSecondScanFails(t *testing.T) {
|
||||
// README §Database: while one scan holds the lock, a second scan
|
||||
// fails at once, naming the lock file, without opening the
|
||||
// database.
|
||||
// README §Database: a second scan that finds the lock held fails
|
||||
// at once, naming the lock file, without opening the database.
|
||||
path := testDBPath(t)
|
||||
t.Setenv(databaseEnv, path)
|
||||
|
||||
@@ -1340,7 +1339,7 @@ func assertSecondScanFails(t *testing.T, path string) {
|
||||
}
|
||||
|
||||
func TestRunSecondScanThroughSymlinkFails(t *testing.T) {
|
||||
// README §Database: every path to one database names the same lock
|
||||
// README §Database: a path through a symlink names the same lock
|
||||
// file, so while a scan holds it through one path, a scan through
|
||||
// another fails, naming the lock file beside the database's real
|
||||
// path.
|
||||
@@ -1445,8 +1444,8 @@ func TestRunReportsDuringScan(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestRunStdoutClosedIsFatal(t *testing.T) {
|
||||
// README §Error handling: a stdout write failure exits 1, reported
|
||||
// in one line on stderr.
|
||||
// README §Error handling: a failed write to stdout is a fatal error,
|
||||
// exit 1, reported on stderr starting with "sfdupes:".
|
||||
for _, name := range []string{cmdReport, cmdTrees, cmdInfo} {
|
||||
t.Run(name, func(t *testing.T) {
|
||||
t.Setenv(databaseEnv, testDBPath(t))
|
||||
|
||||
Reference in New Issue
Block a user