|
|
|
@@ -18,11 +18,11 @@ a strong candidate, not proof.
|
|
|
|
|
|
|
|
|
|
`sfdupes` is meant for multi-terabyte servers, where reading every byte of every
|
|
|
|
|
file takes too long. It has four subcommands. `scan` walks the filesystem and
|
|
|
|
|
records every file in an SQLite database. SQLite keeps a whole database in one
|
|
|
|
|
file. The database is kept between runs. A rescan reads only new and changed
|
|
|
|
|
files. That makes `scan` cheap enough to run every day from `cron`, the system's
|
|
|
|
|
job scheduler. `report`, `trees` and `info` read the database alone and can run
|
|
|
|
|
at any time.
|
|
|
|
|
records the files it finds in an SQLite database. SQLite keeps a whole database
|
|
|
|
|
in one file. The database is kept between runs. A rescan reads only new and
|
|
|
|
|
changed files. That makes `scan` cheap enough to run every day from `cron`, the
|
|
|
|
|
system's job scheduler. `report`, `trees` and `info` read the database alone and
|
|
|
|
|
can run at any time.
|
|
|
|
|
|
|
|
|
|
## Getting Started
|
|
|
|
|
|
|
|
|
@@ -45,10 +45,9 @@ database's path, size and record count.
|
|
|
|
|
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
|
|
|
|
|
another file. The first `scan` creates it.
|
|
|
|
|
|
|
|
|
|
`report`, `trees` and `info` write their data to stdout. Everything else, such
|
|
|
|
|
as progress and warnings, goes to stderr. stdout and stderr are a program's two
|
|
|
|
|
output streams. The shell shows both on the terminal, and `>` redirects stdout
|
|
|
|
|
alone.
|
|
|
|
|
`report`, `trees` and `info` write their data to stdout. Progress, warnings and
|
|
|
|
|
errors go to stderr. stdout and stderr are a program's two output streams. The
|
|
|
|
|
shell shows both on the terminal, and `>` redirects stdout alone.
|
|
|
|
|
|
|
|
|
|
### Install
|
|
|
|
|
|
|
|
|
@@ -68,11 +67,12 @@ The binary goes to `$(go env GOPATH)/bin`, or to `$GOBIN` when that is set.
|
|
|
|
|
|
|
|
|
|
`sfdupes --version` prints `sfdupes VERSION` and exits. For a binary from
|
|
|
|
|
`go install`, `VERSION` is the version Go recorded in it. For `@latest` that is
|
|
|
|
|
a release tag such as `v0.0.3`. For `@main` it is a pseudo-version, Go's version
|
|
|
|
|
string for an untagged commit. That string ends in the commit's hash. A binary
|
|
|
|
|
built from a clone carries the version `git describe --tags --always --dirty`
|
|
|
|
|
prints in that clone, such as `v0.0.3` on a tagged commit. `VERSION` is `dev`
|
|
|
|
|
when no version is known. To see everything Go recorded in an installed binary:
|
|
|
|
|
a release tag such as `v0.0.3`. For `@main` it is the commit's tag when it has
|
|
|
|
|
one. Otherwise it is a pseudo-version, Go's version string for an untagged
|
|
|
|
|
commit. That string ends in the commit's hash. A binary built from a clone
|
|
|
|
|
carries the output of `git describe --tags --always --dirty` in that clone, such
|
|
|
|
|
as `v0.0.3` on a tagged commit. `VERSION` is `dev` when no version is known. To
|
|
|
|
|
see everything Go recorded in an installed binary:
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
go version -m "$(command -v sfdupes)"
|
|
|
|
@@ -106,9 +106,10 @@ docker run --rm -v /var/lib/sfdupes:/var/lib/sfdupes sfdupes report > dupes.tsv
|
|
|
|
|
|
|
|
|
|
### Daily scan from `cron`
|
|
|
|
|
|
|
|
|
|
Run `scan` as root, so that it can read every file. A path it cannot read is
|
|
|
|
|
skipped with a warning and loses its database record. As the file
|
|
|
|
|
`/etc/cron.d/sfdupes`:
|
|
|
|
|
Run `scan` as root, so that it can read every file. `scan` warns about a
|
|
|
|
|
directory it cannot read, and about a new or changed file it cannot read. It
|
|
|
|
|
skips the path and deletes the records at or below it (see "Rules for the
|
|
|
|
|
walk"). As the file `/etc/cron.d/sfdupes`:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log
|
|
|
|
@@ -122,8 +123,10 @@ skipped with a warning and loses its database record. As the file
|
|
|
|
|
- one line naming the database;
|
|
|
|
|
- progress lines, as described under "Progress";
|
|
|
|
|
- a warning for each path it skips;
|
|
|
|
|
- a line for each path the walk leaves out as a system or temporary
|
|
|
|
|
directory (see "Rules for the walk");
|
|
|
|
|
- a line for each system or temporary directory the walk leaves out (see
|
|
|
|
|
"Rules for the walk");
|
|
|
|
|
- a line for each path left out because its filesystem is one the kernel
|
|
|
|
|
makes up or keeps in memory, such as `tmpfs`;
|
|
|
|
|
- the summary line;
|
|
|
|
|
- a warning when a report still has the database open as the scan ends (see
|
|
|
|
|
"Concurrent reads").
|
|
|
|
@@ -133,8 +136,8 @@ skipped with a warning and loses its database record. As the file
|
|
|
|
|
- A scan that fails exits 1, with the reason among the last lines of the log.
|
|
|
|
|
`tail` prints those lines, and `cron` mails them to root if the host can send
|
|
|
|
|
mail.
|
|
|
|
|
- A scan still running when the next one starts carries on. The new one fails at
|
|
|
|
|
once, and the lines `cron` mails include
|
|
|
|
|
- A running scan carries on when the next one starts. The new one fails at once,
|
|
|
|
|
and the lines that `cron` mails include
|
|
|
|
|
`sfdupes: another scan is running (lock held on /var/lib/sfdupes/db.sqlite.lock)`.
|
|
|
|
|
- `report`, `trees` and `info` need only read access to the database. The umask
|
|
|
|
|
is the setting that removes permissions from the files a process creates.
|
|
|
|
@@ -152,8 +155,8 @@ to keep is your choice.
|
|
|
|
|
|
|
|
|
|
A row is a candidate, not proof:
|
|
|
|
|
|
|
|
|
|
- The reports show the files as they were at the last scan. A file may have
|
|
|
|
|
changed or gone since.
|
|
|
|
|
- The reports show each file as it was when a scan last recorded it. A file may
|
|
|
|
|
have changed or gone since.
|
|
|
|
|
- A file of 100 MiB or more is compared only on samples of its content (see
|
|
|
|
|
"Duplicate detection").
|
|
|
|
|
- Two paths that are hard links, two names for one file, are listed as
|
|
|
|
@@ -172,8 +175,8 @@ where the files differ. The second line prints `hard links` when both paths are
|
|
|
|
|
the same file.
|
|
|
|
|
|
|
|
|
|
A path holding a special character, such as a tab, is escaped in the reports.
|
|
|
|
|
The character is written as a backslash and a letter, `\t` for a tab (see
|
|
|
|
|
"Report output format"). Undo the escapes before using such a path.
|
|
|
|
|
The character is written as a backslash and one more character, `\t` for a tab
|
|
|
|
|
(see "Report output format"). Undo the escapes before using such a path.
|
|
|
|
|
`printf '%b'` does that, because every backslash in an escaped path starts an
|
|
|
|
|
escape. The shell drops trailing newlines from the output of `$(...)`. To keep a
|
|
|
|
|
newline at the end of a path, print an `x` after the path and remove it
|
|
|
|
@@ -200,9 +203,9 @@ two ends and at 1 MiB per gigabyte. A full read of a small file is cheap, and it
|
|
|
|
|
makes the comparison exact. Reading samples cuts a 150 GiB file to about 150 MiB
|
|
|
|
|
of reads, at the price of a comparison that is not exact.
|
|
|
|
|
|
|
|
|
|
The hashes live in a database that is kept between runs. A rescan reads only the
|
|
|
|
|
files that are new or whose size or modification time (mtime) changed. All
|
|
|
|
|
analysis then works from the database alone.
|
|
|
|
|
The hashes live in a database that is kept between runs. A rescan reads only new
|
|
|
|
|
files, and files whose size changed or whose modification time (mtime) is newer
|
|
|
|
|
than recorded. All analysis then works from the database alone.
|
|
|
|
|
|
|
|
|
|
The goal is whole duplicated trees, not only single files. A download extracted
|
|
|
|
|
twice, or a project copied once too often, is such a tree. It can be removed as
|
|
|
|
@@ -223,10 +226,9 @@ In order:
|
|
|
|
|
its ends and at 1 MiB per gigabyte. No file of 100 MiB or more is ever read
|
|
|
|
|
in full. The target is a filesystem of about ten million files and 150 TB, on
|
|
|
|
|
disks that may be slow or busy. During a scan, one small record per file is
|
|
|
|
|
held in memory. It holds the path and the two values the walk compares, the
|
|
|
|
|
size and the mtime. The hashes stay in the database. `report` holds the
|
|
|
|
|
duplicate paths of one file size at a time. `trees` holds one entry per
|
|
|
|
|
directory, plus the files of the directories it is reading.
|
|
|
|
|
held in memory. The hashes stay in the database. `report` holds the duplicate
|
|
|
|
|
paths of one file size at a time. `trees` holds one entry per directory, plus
|
|
|
|
|
the files of the directories it is reading.
|
|
|
|
|
3. **Rescan cheaply, analyze from the database.** The scan keeps its database
|
|
|
|
|
between runs, and an unchanged file is never read again. `report`, `trees`
|
|
|
|
|
and `info` work from the database alone and never touch the scanned
|
|
|
|
@@ -260,11 +262,14 @@ The reference below is also the program's specification.
|
|
|
|
|
|
|
|
|
|
## Non-goals
|
|
|
|
|
|
|
|
|
|
- No byte-for-byte compare, and no deleting or linking of duplicates. The
|
|
|
|
|
reports are advisory. Acting on them is your job.
|
|
|
|
|
- No byte-for-byte compare.
|
|
|
|
|
- No deleting of duplicates.
|
|
|
|
|
- No replacing a copy with a hard link to another copy. The reports are
|
|
|
|
|
advisory. Acting on them is your job.
|
|
|
|
|
- The SQLite database is the only stored data. There are no export or import
|
|
|
|
|
formats.
|
|
|
|
|
- No daemon or filesystem watcher. Scheduling rescans is `cron`'s job.
|
|
|
|
|
- `sfdupes` does not run in the background or watch the filesystem for changes.
|
|
|
|
|
Scheduling rescans is `cron`'s job.
|
|
|
|
|
|
|
|
|
|
## Reference
|
|
|
|
|
|
|
|
|
@@ -342,28 +347,46 @@ default. An operand may be a directory or a regular file.
|
|
|
|
|
the exit code is 2.
|
|
|
|
|
- An operand that does not exist is a fatal error. `scan` then stops before it
|
|
|
|
|
walks anything, and exits 1.
|
|
|
|
|
- Each operand is made absolute before the walk. A `.` or `..` in it is resolved
|
|
|
|
|
as text, and symlinks are not followed. Each record is stored under that
|
|
|
|
|
absolute path, so results do not depend on the working directory.
|
|
|
|
|
- Each operand is made absolute before the walk. A relative operand is joined to
|
|
|
|
|
the working directory as `pwd` prints it, with its symlinks kept. A `.` or
|
|
|
|
|
`..` in it is resolved as text, and symlinks are not followed. Each record is
|
|
|
|
|
stored under that absolute path.
|
|
|
|
|
- All operands belong to one scan, and one walk covers them all at once (see
|
|
|
|
|
"Concurrency").
|
|
|
|
|
- An operand that repeats another, or lies under another, is dropped before the
|
|
|
|
|
walk, without a line. Every file is then reached once. An operand under
|
|
|
|
|
another is kept when the outer operand's walk would not reach it. That is the
|
|
|
|
|
case when the inner operand, or a directory between the two, is one of these:
|
|
|
|
|
- The warnings and the line described below come first, even for an operand that
|
|
|
|
|
repeats or lies under another. After them, an operand that repeats another is
|
|
|
|
|
dropped before the walk, without a line.
|
|
|
|
|
- An operand under another operand is dropped the same way, unless one of the
|
|
|
|
|
cases below applies. No path is then walked twice.
|
|
|
|
|
- An operand under another is kept, and walked on its own, when the inner
|
|
|
|
|
operand or a directory between the two is one of these:
|
|
|
|
|
- a default exclude (see "Rules for the walk");
|
|
|
|
|
- with `-x`, a directory on another filesystem than the outer operand.
|
|
|
|
|
|
|
|
|
|
It is also the case when a directory between the two is on a filesystem that
|
|
|
|
|
is not persistent. Such a filesystem is one the kernel makes up or keeps in
|
|
|
|
|
It is also kept when a directory between the two is on a filesystem that is
|
|
|
|
|
not persistent. Such a filesystem is one the kernel makes up or keeps in
|
|
|
|
|
memory (see "Rules for the walk"). The outer operand's walk does not enter
|
|
|
|
|
such a directory, so the inner operand is walked on its own.
|
|
|
|
|
|
|
|
|
|
- An operand that is neither a regular file nor a directory is not scanned, and
|
|
|
|
|
neither is a directory named `.zfs`. `scan` prints a warning naming the path
|
|
|
|
|
and what it is. It counts the operand as skipped and drops it. An operand
|
|
|
|
|
beneath a dropped one is scanned all the same. The scan keeps the records
|
|
|
|
|
beneath a dropped operand, unless another operand covers them.
|
|
|
|
|
- No other case keeps an operand under another. An operand reached through a
|
|
|
|
|
symlinked directory below the outer operand is dropped too. The outer
|
|
|
|
|
operand's walk does not follow the symlink, so no file under the inner operand
|
|
|
|
|
is scanned. The records that an earlier scan left under it are deleted. The
|
|
|
|
|
`-x` case above also covers an operand reached that way.
|
|
|
|
|
- An operand that is neither a regular file nor a directory is not scanned.
|
|
|
|
|
`scan` prints a warning naming the path and what it is. It counts the operand
|
|
|
|
|
as skipped and drops it. An operand beneath a symlink operand is scanned all
|
|
|
|
|
the same. The scan keeps the records beneath a dropped operand, unless another
|
|
|
|
|
operand covers them. After a `cd` through a symlink, `scan .` names the
|
|
|
|
|
symlink and is skipped as a symlink operand. `scan sub` in that directory
|
|
|
|
|
records its files under the symlink's path. `report` can then list each as a
|
|
|
|
|
duplicate of the same file recorded under its real path.
|
|
|
|
|
- An operand at or inside a directory named `.zfs` is never scanned, whether
|
|
|
|
|
given alone or under another operand. The check looks at the path as given and
|
|
|
|
|
at the path with its symlinks resolved. `scan` prints a warning naming the
|
|
|
|
|
path, counts the operand as skipped and drops it. The records beneath it are
|
|
|
|
|
kept, unless it lies under another operand. Then they are deleted, because
|
|
|
|
|
that operand's walk never enters `.zfs` (see "Rules for the walk").
|
|
|
|
|
- An operand on a filesystem that is not persistent is dropped too. `scan`
|
|
|
|
|
prints a line naming it and its filesystem type, and does not count it as
|
|
|
|
|
skipped.
|
|
|
|
@@ -385,8 +408,9 @@ operands:
|
|
|
|
|
- The scan deletes each record under a scanned operand whose file it could not
|
|
|
|
|
verify. That covers deleted files, and paths the scan could not stat or read
|
|
|
|
|
this run. To stat a file is to read its details, such as its size, without
|
|
|
|
|
opening it. So the database holds only what the latest scan over each path
|
|
|
|
|
could verify. A later successful scan adds such files again.
|
|
|
|
|
opening it. So after a complete scan, the database holds only what that scan
|
|
|
|
|
could verify under its operands. A later successful scan adds such files
|
|
|
|
|
again.
|
|
|
|
|
- Records outside the scanned operands are left alone, so separate trees can be
|
|
|
|
|
scanned on different schedules into one database.
|
|
|
|
|
- The scan keeps every record at or below a path that the walk leaves out for
|
|
|
|
@@ -397,9 +421,8 @@ operands:
|
|
|
|
|
|
|
|
|
|
Such records count as outside the scanned operands. So with `-x`, a
|
|
|
|
|
filesystem mounted under an operand can be scanned on its own schedule into
|
|
|
|
|
the same database. An operand may lie at or inside a default exclude, or,
|
|
|
|
|
with `-x`, at or inside a directory on another filesystem. Such an operand
|
|
|
|
|
is walked on its own. Its walk verifies the records below it, so they are
|
|
|
|
|
the same database. An operand may itself lie at or inside such a path, as
|
|
|
|
|
"Operands" describes. Its walk verifies the records below it, so they are
|
|
|
|
|
deleted when their files are gone.
|
|
|
|
|
|
|
|
|
|
#### Phases
|
|
|
|
@@ -410,12 +433,13 @@ phases.
|
|
|
|
|
1. `load` reads the existing records. Each record under the operands is kept in
|
|
|
|
|
memory by path, with its size and mtime, for the walk to compare against.
|
|
|
|
|
2. `walk` lists the trees under all operands. Several workers share one queue of
|
|
|
|
|
directories to read. A worker is one thread of the program, and "Concurrency"
|
|
|
|
|
says how many there are. Each worker reads one directory at a time. It hands
|
|
|
|
|
the subdirectories it finds back to the queue, and it stats each regular file
|
|
|
|
|
as it finds it. The walk compares each file with its record. An unchanged
|
|
|
|
|
file needs nothing more. Every new or changed file is passed to the hash
|
|
|
|
|
phase.
|
|
|
|
|
directories to read. A worker is one thread of the program. A thread is one
|
|
|
|
|
of several parts of a program that run at the same time and share its memory.
|
|
|
|
|
"Concurrency" says how many workers there are. Each worker reads one
|
|
|
|
|
directory at a time. It hands the subdirectories it finds back to the queue,
|
|
|
|
|
and it stats each regular file as it finds it. The walk compares each file
|
|
|
|
|
with its record. An unchanged file needs nothing more. Every new or changed
|
|
|
|
|
file is passed to the hash phase.
|
|
|
|
|
3. `hash` reads every new or changed file and computes its three hashes. Files
|
|
|
|
|
are read in inode order. An inode is the filesystem's own record of a file.
|
|
|
|
|
Files with nearby inode numbers usually lie near each other on disk. Reading
|
|
|
|
@@ -434,10 +458,11 @@ walk practical on large or busy storage pools.
|
|
|
|
|
#### Rules for the walk
|
|
|
|
|
|
|
|
|
|
- Only regular files are recorded. Every other kind of file, such as a symlink
|
|
|
|
|
or a device node, is passed over. Symlinks are never followed.
|
|
|
|
|
or a device node, is passed over. The walk follows no symlink it finds.
|
|
|
|
|
- The walk never enters a directory named `.zfs`. Such a directory holds the
|
|
|
|
|
filesystem's ZFS snapshots, which are its saved earlier states. Walking it
|
|
|
|
|
would list every file once per snapshot.
|
|
|
|
|
would list every file once per snapshot. `scan` never starts a walk at or
|
|
|
|
|
inside one either (see "Operands").
|
|
|
|
|
- The walk never records the scan's own database files:
|
|
|
|
|
- the database file;
|
|
|
|
|
- its `.lock` file (see "The scan lock");
|
|
|
|
@@ -451,14 +476,14 @@ walk practical on large or busy storage pools.
|
|
|
|
|
operand at the path given, even when that path runs through a symlinked
|
|
|
|
|
directory. Such an operand can lead to the database files, and the walk
|
|
|
|
|
passes over them there as well. The walk prints no line for the database
|
|
|
|
|
files and does not count them as skipped. A record an earlier scan left for
|
|
|
|
|
one of them is deleted.
|
|
|
|
|
files and does not count them as skipped. A record that an earlier scan left
|
|
|
|
|
for one of them is deleted.
|
|
|
|
|
|
|
|
|
|
- By default the walk enters directories on other filesystems than its
|
|
|
|
|
operand's. With `-x`, it never enters a directory on a different filesystem
|
|
|
|
|
than its operand. `scan` prints no line for that directory and does not count
|
|
|
|
|
it as skipped. The records below it are kept, as described under "What a scan
|
|
|
|
|
does to the database".
|
|
|
|
|
than its operand. `scan` prints no line for a directory that only `-x` leaves
|
|
|
|
|
out, and does not count it as skipped. The records below it are kept, as
|
|
|
|
|
described under "What a scan does to the database".
|
|
|
|
|
- The walk never records anything on a filesystem that is not persistent. A
|
|
|
|
|
filesystem is not persistent when the kernel makes it up or keeps it in
|
|
|
|
|
memory. These are the types, as `sfdupes scan --help` lists them on Linux and
|
|
|
|
@@ -494,11 +519,10 @@ walk practical on large or busy storage pools.
|
|
|
|
|
The path is not counted as skipped. The records below it are kept, as
|
|
|
|
|
described under "What a scan does to the database".
|
|
|
|
|
|
|
|
|
|
- Below an operand, the walk does not enter a default exclude. The default
|
|
|
|
|
excludes hold the kernel's view of the running system and temporary files, so
|
|
|
|
|
that `sfdupes scan /` walks only the files worth comparing. They depend on the
|
|
|
|
|
system, and `sfdupes scan --help` lists those of the system it runs on. On
|
|
|
|
|
Linux they are:
|
|
|
|
|
- Below an operand, the walk does not enter a default exclude. On Linux, the
|
|
|
|
|
default excludes hold temporary files and the kernel's view of the running
|
|
|
|
|
system. They depend on the system, and `sfdupes scan --help` lists those of
|
|
|
|
|
the system it runs on. On Linux they are:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
/proc, /sys, /dev, /run, /tmp, /var/tmp
|
|
|
|
@@ -527,21 +551,27 @@ walk practical on large or busy storage pools.
|
|
|
|
|
scan: skipped /var/tmp (default exclude)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
The directory is not counted as skipped. A default exclude on a filesystem
|
|
|
|
|
that is not persistent, such as `/proc`, gets the line for its filesystem
|
|
|
|
|
instead. One that is itself an operand gets no line, because it is walked as
|
|
|
|
|
that operand. The scan keeps the records below a default exclude that the
|
|
|
|
|
The directory is not counted as skipped. `scan` prints the filesystem line
|
|
|
|
|
instead for a default exclude that is not on a persistent filesystem, such
|
|
|
|
|
as `/proc`. One that is itself an operand gets no line, because it is walked
|
|
|
|
|
as that operand. The scan keeps the records below a default exclude that the
|
|
|
|
|
walk did not enter, as described under "What a scan does to the database".
|
|
|
|
|
|
|
|
|
|
- A path that cannot be read, for lack of permission or any other error, gets a
|
|
|
|
|
one-line warning on stderr and is skipped. The scan continues, and the summary
|
|
|
|
|
line says how many paths were skipped. A skipped path that has a record from
|
|
|
|
|
an earlier scan loses that record, and so does every path under an unreadable
|
|
|
|
|
directory. The database mirrors what the latest scan could verify.
|
|
|
|
|
- `scan` skips each of these, with a one-line warning on stderr:
|
|
|
|
|
- a directory it cannot read;
|
|
|
|
|
- a path it cannot stat;
|
|
|
|
|
- a new or changed file it cannot read.
|
|
|
|
|
|
|
|
|
|
An unchanged file is not opened, so it keeps its record even when it has
|
|
|
|
|
become unreadable. The scan continues, and the summary line says how many
|
|
|
|
|
paths were skipped. A skipped path that has a record from an earlier scan
|
|
|
|
|
loses that record, and so does every path under an unreadable directory.
|
|
|
|
|
|
|
|
|
|
- A file or directory that disappears while the scan runs is not an error. The
|
|
|
|
|
scan prints no line for it and does not count it as skipped. A path gone when
|
|
|
|
|
the walk reaches it gets no record, and so does a file gone when the hash
|
|
|
|
|
phase opens it. The records an earlier scan left at or below it are deleted.
|
|
|
|
|
phase opens it. The records that an earlier scan left at or below it are
|
|
|
|
|
deleted.
|
|
|
|
|
|
|
|
|
|
#### Concurrency
|
|
|
|
|
|
|
|
|
@@ -559,7 +589,8 @@ from a thread of its own and never blocks the workers.
|
|
|
|
|
|
|
|
|
|
- one line naming the database (see "Database");
|
|
|
|
|
- for each dropped operand, in operand order, the warning or the line described
|
|
|
|
|
under "Operands". A repeated operand, or one under another, gets neither;
|
|
|
|
|
under "Operands". An operand dropped only because it repeats another or lies
|
|
|
|
|
under another gets neither;
|
|
|
|
|
- the progress display of each phase (see "Progress"). `scan` prints each
|
|
|
|
|
warning for a skipped path as it happens. It prints a line the same way for
|
|
|
|
|
each path left out as a default exclude or as not on a persistent filesystem;
|
|
|
|
@@ -586,13 +617,14 @@ signal `kill` sends by default:
|
|
|
|
|
- The records hashed but not yet written are written, so the next scan does not
|
|
|
|
|
hash them again.
|
|
|
|
|
- Nothing else is written or deleted. Records are deleted only after a complete
|
|
|
|
|
walk. So the records under paths an interrupted walk never reached are kept.
|
|
|
|
|
- The database is closed and the lock released, as on any other exit.
|
|
|
|
|
walk. So the records under paths that an interrupted walk never reached are
|
|
|
|
|
kept.
|
|
|
|
|
- The database is closed and the lock released.
|
|
|
|
|
- The line `scan: interrupted after N files` goes to stderr. `N` is the number
|
|
|
|
|
of files the walk reached. The exit code is 1.
|
|
|
|
|
|
|
|
|
|
The next scan reads none of the files already recorded and brings the database
|
|
|
|
|
up to date as usual.
|
|
|
|
|
The next scan reads none of the unchanged files already recorded, and brings the
|
|
|
|
|
database up to date as usual.
|
|
|
|
|
|
|
|
|
|
After the first signal `scan` stops catching signals. A second one ends it at
|
|
|
|
|
once, as any uncaught signal does. The records not yet written are lost, and the
|
|
|
|
@@ -602,12 +634,12 @@ and `scan` leaves it ignored.
|
|
|
|
|
|
|
|
|
|
### `report` mode
|
|
|
|
|
|
|
|
|
|
`report` reads every record from the database. It takes no positional arguments,
|
|
|
|
|
meaning nothing may follow `report` on the command line but flags. It never
|
|
|
|
|
touches the scanned filesystem. It neither stats nor opens any path in the
|
|
|
|
|
records. It reads the database and writes to stdout and stderr, and nothing
|
|
|
|
|
else. Its output is the same whether or not the scanned filesystem is still
|
|
|
|
|
mounted.
|
|
|
|
|
`report` reads every record from the database. It takes no positional arguments.
|
|
|
|
|
A positional argument is anything on the command line that is not a flag, such
|
|
|
|
|
as the paths `scan` takes. `report` never touches the scanned filesystem. It
|
|
|
|
|
neither stats nor opens any path in the records. It reads the database and
|
|
|
|
|
writes to stdout and stderr, and nothing else. Its output is the same whether or
|
|
|
|
|
not the scanned filesystem is still mounted.
|
|
|
|
|
|
|
|
|
|
Processing:
|
|
|
|
|
|
|
|
|
@@ -651,8 +683,8 @@ Four bytes are written as escapes:
|
|
|
|
|
|
|
|
|
|
Every other byte is written unchanged, including bytes that are not valid UTF-8,
|
|
|
|
|
the usual text encoding. Undoing the four escapes gives back the stored path.
|
|
|
|
|
Grouping and ordering use the stored path. The warnings `scan` prints are
|
|
|
|
|
escaped the same way, so each warning stays one line.
|
|
|
|
|
Grouping and ordering use the stored path. The warnings about skipped paths that
|
|
|
|
|
`scan` prints are escaped the same way, so each stays one line.
|
|
|
|
|
|
|
|
|
|
The summary line on stderr gives:
|
|
|
|
|
|
|
|
|
@@ -692,9 +724,9 @@ Processing:
|
|
|
|
|
directories is a duplicate-tree group.
|
|
|
|
|
- Only the outermost duplicate trees are reported. A group is left out when its
|
|
|
|
|
members' parents are distinct directories that all have the same digest. The
|
|
|
|
|
parents' group, or an ancestor's, already implies it. A group holding sibling
|
|
|
|
|
directories, or members whose parents have different digests, is always
|
|
|
|
|
reported.
|
|
|
|
|
parents' group, or an ancestor's, already implies it. A group is always
|
|
|
|
|
reported when it holds sibling directories, or when its members' parents have
|
|
|
|
|
different digests.
|
|
|
|
|
- Within a group, paths are sorted in byte order. The first is `first`, and
|
|
|
|
|
every other path is a `dupe`.
|
|
|
|
|
- Groups are ordered by total tree size, largest first, then by `first` path.
|
|
|
|
@@ -706,8 +738,8 @@ Two limitations are accepted:
|
|
|
|
|
They share storage, so removing one frees nothing. The scan uses the inode to
|
|
|
|
|
avoid reading a file twice, and does not store it.
|
|
|
|
|
- Only the regular files in the database define a tree. Empty directories are
|
|
|
|
|
invisible. A file skipped during the scan in one copy but not in the other
|
|
|
|
|
makes two otherwise identical trees differ.
|
|
|
|
|
invisible. When the scan skips a file in one copy but not in the other, the
|
|
|
|
|
two trees differ.
|
|
|
|
|
|
|
|
|
|
`trees` has four phases, each with its own progress display on stderr (see
|
|
|
|
|
"Progress"). `open` opens the database. `read` counts the records, then reads
|
|
|
|
@@ -773,9 +805,10 @@ size of recorded files 150000000000000 (136.4 TiB)
|
|
|
|
|
|
|
|
|
|
`info` has two phases, each with its own progress display on stderr (see
|
|
|
|
|
"Progress"). `open` opens the database. `count` reads the schema version and
|
|
|
|
|
counts the records in one transaction, a read of the database at one moment. So
|
|
|
|
|
the two agree with each other even while a scan is writing. `info` writes its
|
|
|
|
|
lines after the `count` display has ended.
|
|
|
|
|
counts the records in one transaction. A transaction is a group of reads or
|
|
|
|
|
writes that SQLite carries out as one step, against the database as it stands at
|
|
|
|
|
one moment. So both describe the database at one moment, even while a scan is
|
|
|
|
|
writing. `info` writes its lines after the `count` display has ended.
|
|
|
|
|
|
|
|
|
|
### Database
|
|
|
|
|
|
|
|
|
@@ -783,7 +816,7 @@ All four subcommands work on one SQLite database file.
|
|
|
|
|
|
|
|
|
|
#### Location
|
|
|
|
|
|
|
|
|
|
The database is the file `SFDUPES_DATABASE` names, when that is set and not
|
|
|
|
|
The database is the file named by `SFDUPES_DATABASE`, when that is set and not
|
|
|
|
|
empty, and otherwise `/var/lib/sfdupes/db.sqlite`.
|
|
|
|
|
|
|
|
|
|
Each subcommand prints one line on stderr naming the database it uses, before
|
|
|
|
@@ -817,20 +850,20 @@ the error for a missing database comes after it.
|
|
|
|
|
|
|
|
|
|
#### The scan lock
|
|
|
|
|
|
|
|
|
|
Only one `scan` runs against a database at a time.
|
|
|
|
|
Only one `scan` at a time can hold the lock.
|
|
|
|
|
|
|
|
|
|
- For its whole run, `scan` holds a lock on a lock file beside the database. The
|
|
|
|
|
lock is taken with the `flock(2)` system call, and only one process can hold
|
|
|
|
|
it. The lock file's name is the database's path, with symlinks resolved, plus
|
|
|
|
|
`.lock`. By default that is `/var/lib/sfdupes/db.sqlite.lock`. Every path to
|
|
|
|
|
one database therefore names the same lock file.
|
|
|
|
|
`.lock`. By default that is `/var/lib/sfdupes/db.sqlite.lock`. A path through
|
|
|
|
|
a symlink names the same lock file.
|
|
|
|
|
- `scan` takes the lock before it walks the filesystem or opens the database.
|
|
|
|
|
Just before that, it creates the database file, empty, if it is missing. So
|
|
|
|
|
the lock file is named the same way before the first scan as after. A symlink
|
|
|
|
|
at the database path is followed, and its target is created.
|
|
|
|
|
- A second `scan` against the same database does not wait. It fails at once with
|
|
|
|
|
a one-line error naming the lock file, and exits 1. It has walked nothing and
|
|
|
|
|
opened nothing. The running scan carries on.
|
|
|
|
|
- A second `scan` that finds the lock held does not wait. It fails at once with
|
|
|
|
|
an error naming the lock file, and exits 1. It has walked nothing and has not
|
|
|
|
|
opened the database. The running scan carries on.
|
|
|
|
|
- The lock file is created on first use and never deleted. Only its owner may
|
|
|
|
|
open it. A leftover file blocks nothing. The lock ends with the process that
|
|
|
|
|
holds it, however that process ends. Deleting the file while a scan runs would
|
|
|
|
@@ -843,8 +876,9 @@ Only one `scan` runs against a database at a time.
|
|
|
|
|
SQLite appends changes to a `-wal` file beside the database, and keeps a
|
|
|
|
|
lookup table for that file in a `-shm` file. Readers do not block the writer,
|
|
|
|
|
so running a report during a scan is safe.
|
|
|
|
|
- Every connection has a busy timeout of 10 seconds. That is how long it waits
|
|
|
|
|
for a database another connection has locked before it gives up.
|
|
|
|
|
- Every subcommand opens the database with a busy timeout of 10 seconds. That is
|
|
|
|
|
how long it waits for a database another process has locked before it gives
|
|
|
|
|
up.
|
|
|
|
|
- The filesystem is authoritative, and the database follows it. Hashed records
|
|
|
|
|
are written 10,000 at a time while the scan runs, so a report may see a scan's
|
|
|
|
|
changes partly applied. A scan that dies partway leaves a valid database
|
|
|
|
@@ -882,13 +916,15 @@ Only one `scan` runs against a database at a time.
|
|
|
|
|
#### Schema
|
|
|
|
|
|
|
|
|
|
The schema is the database's layout, its table and its index.
|
|
|
|
|
`PRAGMA user_version` holds the schema version, currently 1. A database with any
|
|
|
|
|
other version is a fatal error. `scan` creates the table and the index and sets
|
|
|
|
|
the version in one transaction, a group of writes that lands whole or not at
|
|
|
|
|
all. A first scan stopped while doing so leaves an empty database, which the
|
|
|
|
|
next scan sets up. A database at version 0 that already has a `files` table was
|
|
|
|
|
not made by `sfdupes`. Every subcommand refuses it, with an error telling you to
|
|
|
|
|
remove the file and rescan.
|
|
|
|
|
`PRAGMA user_version` holds the schema version, currently 1. An empty database
|
|
|
|
|
is at version 0, and `scan` sets it up. Any other version is a fatal error.
|
|
|
|
|
`report`, `trees` and `info` set nothing up, so for them version 0 is a fatal
|
|
|
|
|
error too. `scan` creates the table and the index and sets the version in one
|
|
|
|
|
transaction (see "`info` mode"). So the three land together or not at all. A
|
|
|
|
|
first scan stopped while doing so leaves an empty database, which the next scan
|
|
|
|
|
sets up. A database at version 0 that already has a `files` table was not made
|
|
|
|
|
by `sfdupes`. Every subcommand refuses it, with an error telling you to remove
|
|
|
|
|
the file and rescan.
|
|
|
|
|
|
|
|
|
|
```sql
|
|
|
|
|
CREATE TABLE files (
|
|
|
|
@@ -918,9 +954,9 @@ CREATE INDEX files_signature ON files (size, head, tail, content);
|
|
|
|
|
|
|
|
|
|
### Progress
|
|
|
|
|
|
|
|
|
|
Every subcommand always shows that it is working. Each phase gets a display of
|
|
|
|
|
its own on stderr from the moment it starts. The display names the phase and
|
|
|
|
|
shows its progress. The parts of its line are listed below.
|
|
|
|
|
Each phase of every subcommand gets a display of its own on stderr from the
|
|
|
|
|
moment it starts. The display names the phase and shows its progress. The parts
|
|
|
|
|
of its line are listed below.
|
|
|
|
|
|
|
|
|
|
The phases, and what each display counts:
|
|
|
|
|
|
|
|
|
@@ -943,7 +979,7 @@ The phases, and what each display counts:
|
|
|
|
|
- `trees` has four phases:
|
|
|
|
|
- `[1/4] open` and `[2/4] read` are as for `report`, except that `read`
|
|
|
|
|
writes nothing.
|
|
|
|
|
- `[3/4] group` has the directories `read` found as its total.
|
|
|
|
|
- `[3/4] group` has the directories that `read` found as its total.
|
|
|
|
|
- `[4/4] write` has the duplicate-tree groups as its total.
|
|
|
|
|
- `info` has two phases:
|
|
|
|
|
- `[1/2] open` is as for `report`.
|
|
|
|
@@ -981,9 +1017,9 @@ phase waits, in place of the rate. That happens while one large file is read, or
|
|
|
|
|
while a disk or the database is slow. Each phase names the places it may wait
|
|
|
|
|
in, and a phase that may wait in two places names both, as `disk/db`:
|
|
|
|
|
|
|
|
|
|
- `walk` says `disk`, for files.
|
|
|
|
|
- `hash` says `disk/db`: it reads files and writes their records to the database
|
|
|
|
|
between them.
|
|
|
|
|
- `walk` says `disk`, because it reads directories and stats files.
|
|
|
|
|
- `hash` says `disk/db`, because it reads files and writes their records to the
|
|
|
|
|
database between them.
|
|
|
|
|
- `load`, `update` and the `read` phase of `trees` say `db`.
|
|
|
|
|
- The `read` phase of `report` says `db/out`, because it writes the report to
|
|
|
|
|
stdout while it reads.
|
|
|
|
@@ -1009,19 +1045,18 @@ On a terminal:
|
|
|
|
|
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- No line reaches the terminal's last column, however narrow the terminal is. A
|
|
|
|
|
line too wide for the terminal first leaves out its rate, the words of a
|
|
|
|
|
stalled phase, or what the phase is waiting for. It leaves out its bar next.
|
|
|
|
|
It is cut short at the edge only when the rest does not fit on its own. The
|
|
|
|
|
rest is:
|
|
|
|
|
- No line reaches the terminal's last column. A line too wide for the terminal
|
|
|
|
|
first leaves out its rate, the words of a stalled phase, or what the phase is
|
|
|
|
|
waiting for. It leaves out its bar next. It is cut short at the edge only when
|
|
|
|
|
the rest does not fit on its own. The rest is:
|
|
|
|
|
- the phase and its place;
|
|
|
|
|
- the progress, when the phase shows one;
|
|
|
|
|
- the elapsed time;
|
|
|
|
|
- the time remaining.
|
|
|
|
|
|
|
|
|
|
That happens hours into a scan of millions of files, on a terminal narrower
|
|
|
|
|
than about 75 columns. A terminal made narrower while a phase runs is fitted
|
|
|
|
|
at the next draw.
|
|
|
|
|
On a terminal 80 columns wide, no line is cut short, even hours into a scan
|
|
|
|
|
of millions of files. A terminal made narrower during a phase is fitted at
|
|
|
|
|
the next draw.
|
|
|
|
|
|
|
|
|
|
- A warning printed during a phase lands on a line of its own, never inside the
|
|
|
|
|
display. So does every line written to stdout when stdout and stderr are the
|
|
|
|
@@ -1037,8 +1072,8 @@ Each phase prints its line:
|
|
|
|
|
- every 5 seconds while it runs, whether or not an item has completed;
|
|
|
|
|
- once more as it ends.
|
|
|
|
|
|
|
|
|
|
A phase that stops short of its total, as an interrupted one does, keeps showing
|
|
|
|
|
the count it stopped at.
|
|
|
|
|
When a phase stops short of its total, as an interrupted one does, it keeps
|
|
|
|
|
showing the count it stopped at.
|
|
|
|
|
|
|
|
|
|
### Error handling and exit codes
|
|
|
|
|
|
|
|
|
@@ -1047,12 +1082,14 @@ the count it stopped at.
|
|
|
|
|
error is one of these:
|
|
|
|
|
- an operand that does not exist, or whose details `scan` cannot read;
|
|
|
|
|
- a list of mounted filesystems that `scan` cannot read;
|
|
|
|
|
- another `scan` already running against the same database;
|
|
|
|
|
- a database that cannot be created or opened, or whose reads or writes
|
|
|
|
|
fail, its directory and its lock file included;
|
|
|
|
|
- another `scan` holding the lock (see "The scan lock");
|
|
|
|
|
- a database, its directory or its lock file that cannot be created or
|
|
|
|
|
opened;
|
|
|
|
|
- a read or write of the database that fails;
|
|
|
|
|
- a relative database path when the working directory cannot be read;
|
|
|
|
|
- a database with another schema version, or one at version 0 that already
|
|
|
|
|
has a `files` table;
|
|
|
|
|
- a database at a schema version other than 0 or 1;
|
|
|
|
|
- a database at version 0 read by `report`, `trees` or `info`;
|
|
|
|
|
- a database at version 0 that already has a `files` table (see "Schema");
|
|
|
|
|
- a record with a hash that is not 32 bytes, read by `report` or `trees`;
|
|
|
|
|
- a missing database for `report`, `trees` or `info`;
|
|
|
|
|
- a failed write to stdout.
|
|
|
|
@@ -1065,9 +1102,9 @@ the count it stopped at.
|
|
|
|
|
- `--workers` below 1;
|
|
|
|
|
- `report`, `trees` or `info` with a positional argument.
|
|
|
|
|
|
|
|
|
|
A fatal error is reported in one line on stderr that starts with `sfdupes:`. A
|
|
|
|
|
failed write to stdout, such as a full disk, is reported the same way. Two cases
|
|
|
|
|
never reach `sfdupes` as a failed write:
|
|
|
|
|
A fatal error is reported on stderr, starting with `sfdupes:`. A failed write to
|
|
|
|
|
stdout, such as a full disk, is reported the same way. Two cases never reach
|
|
|
|
|
`sfdupes` as a failed write:
|
|
|
|
|
|
|
|
|
|
- The reader of a stdout pipe may exit early, as `head` does in
|
|
|
|
|
`sfdupes report | head`. The next write then ends `sfdupes` with `SIGPIPE`,
|
|
|
|
@@ -1084,9 +1121,8 @@ This repository follows the
|
|
|
|
|
[Scripts to Rule Them All](https://github.com/github/scripts-to-rule-them-all)
|
|
|
|
|
standard. The executables in `script/` are the entrypoints of the development
|
|
|
|
|
workflow, and the `Makefile` targets call them, except `build` and `clean` (see
|
|
|
|
|
"Build"). Every script runs under plain `sh`, the standard Unix shell, and uses
|
|
|
|
|
no `bash` feature. It finds the repository root itself, so it can be run from
|
|
|
|
|
any directory.
|
|
|
|
|
"Build"). Every script runs under plain `sh`, the standard Unix shell. It finds
|
|
|
|
|
the repository root itself, so it can be run from any directory.
|
|
|
|
|
|
|
|
|
|
- `script/bootstrap` installs what building and developing need. It can be run
|
|
|
|
|
again at any time, and installed tools are kept. `git`, `make` and `go` are
|
|
|
|
@@ -1102,8 +1138,8 @@ any directory.
|
|
|
|
|
- An installed `node` is used when its major version is 22. A `yarn` already
|
|
|
|
|
on `PATH` is then used as it is. When `yarn` is missing, `yarn` 1.22.22 is
|
|
|
|
|
activated through `corepack`, the `yarn` installer that ships with `node`.
|
|
|
|
|
When there is no `corepack` either, `npm`, the package manager that ships
|
|
|
|
|
with `node`, installs it.
|
|
|
|
|
When there is no `corepack` either, the script installs `yarn` with `npm`,
|
|
|
|
|
the package manager that ships with `node`.
|
|
|
|
|
- Otherwise the script installs `node` 22.17.0 through `nvm`, a `node`
|
|
|
|
|
version manager, and activates `yarn` 1.22.22 under that `node` through
|
|
|
|
|
`corepack`. When `nvm` is missing, the script first downloads it and
|
|
|
|
@@ -1134,8 +1170,8 @@ any directory.
|
|
|
|
|
for common mistakes. It builds only the `lint` stage of the `Dockerfile` and
|
|
|
|
|
writes no image. The stage starts from the `golangci/golangci-lint` image,
|
|
|
|
|
pinned by its digest, and runs three build steps:
|
|
|
|
|
- the `gofmt` check, which fails on a Go source that `gofmt`, Go's own
|
|
|
|
|
formatter, would change;
|
|
|
|
|
- the `gofmt` check, which fails when `gofmt`, Go's own formatter, would
|
|
|
|
|
change a Go source;
|
|
|
|
|
- `golangci-lint config verify`, which checks the linter's settings file
|
|
|
|
|
`.golangci.yml`;
|
|
|
|
|
- `golangci-lint run`, which lints the Go sources.
|
|
|
|
@@ -1147,8 +1183,8 @@ any directory.
|
|
|
|
|
with `prettier`, in place. `prettier` indents by 4 spaces and wraps prose, as
|
|
|
|
|
`.prettierrc` sets and as the script also passes on the command line. It wraps
|
|
|
|
|
at 80 columns, its default width. Both run on the host. `prettier` runs
|
|
|
|
|
through `yarn` at the version `yarn.lock` pins. When the `node` on `PATH` is
|
|
|
|
|
not `node` 22, the script loads the `node` 22.17.0 that `script/bootstrap`
|
|
|
|
|
through `yarn` at the version that `yarn.lock` pins. When the `node` on `PATH`
|
|
|
|
|
is not `node` 22, the script loads the `node` 22.17.0 that `script/bootstrap`
|
|
|
|
|
installed.
|
|
|
|
|
- `script/fmt-check` is the read-only form of `script/fmt`. It names any
|
|
|
|
|
unformatted file and exits non-zero instead of writing. `gofmt` and `prettier`
|
|
|
|
@@ -1169,8 +1205,8 @@ any directory.
|
|
|
|
|
every check passed. The tests and the linter run twice, once in `script/check`
|
|
|
|
|
and again as stages of the image build.
|
|
|
|
|
- `script/precommit` runs from the `git` pre-commit hook, a script that `git`
|
|
|
|
|
runs before each commit. It fails the commit when `go mod tidy` would change
|
|
|
|
|
anything. Then it runs `script/check`.
|
|
|
|
|
runs before each commit. It runs `go mod tidy`, and fails the commit when that
|
|
|
|
|
changes `go.mod` or `go.sum`. Then it runs `script/check`.
|
|
|
|
|
- `script/install-precommit` installs `.git/hooks/pre-commit`, which runs
|
|
|
|
|
`script/precommit`.
|
|
|
|
|
|
|
|
|
@@ -1180,9 +1216,9 @@ install system packages and download the Go modules. Every step that checks or
|
|
|
|
|
compiles the code comes after it. Every `RUN` step after it runs again on each
|
|
|
|
|
build. Without that, Docker would serve the test and lint steps from its cache
|
|
|
|
|
on an unchanged tree, and the build would pass without running them. The steps
|
|
|
|
|
above `ARG CACHEBUST` stay cached until something they depend on changes, such
|
|
|
|
|
as `go.mod`. Go's build cache is kept between builds, so only what changed is
|
|
|
|
|
compiled again. The tests run with `-count=1`, so no test reports a stored
|
|
|
|
|
above `ARG CACHEBUST` stay cached until a change to something they depend on,
|
|
|
|
|
such as `go.mod`. Go's build cache is kept between builds, so only what changed
|
|
|
|
|
is compiled again. The tests run with `-count=1`, so no test reports a stored
|
|
|
|
|
result. `golangci-lint run` keeps the linter's own cache as well, so an
|
|
|
|
|
unchanged package is not linted again.
|
|
|
|
|
|
|
|
|
@@ -1223,10 +1259,10 @@ commands of their own:
|
|
|
|
|
|
|
|
|
|
No other dependency outside the standard library is added.
|
|
|
|
|
|
|
|
|
|
- The code stays pure Go, with no C code. Cross-compilation is not a concern.
|
|
|
|
|
`modernc.org/sqlite` is pure Go too, so the binary builds without cgo. The
|
|
|
|
|
`Makefile` exports `CGO_ENABLED=0`. The test build is the one exception,
|
|
|
|
|
because the race detector needs cgo (see `script/test`).
|
|
|
|
|
- The code stays pure Go, with no C code. Building on one system for another is
|
|
|
|
|
not a goal. `modernc.org/sqlite` is pure Go too, so the binary builds without
|
|
|
|
|
cgo. The `Makefile` exports `CGO_ENABLED=0`. The test build is the one
|
|
|
|
|
exception, because the race detector needs cgo (see `script/test`).
|
|
|
|
|
- `report` and `trees` give the same output for the same database contents,
|
|
|
|
|
whatever order the records were inserted in.
|
|
|
|
|
|
|
|
|
@@ -1242,6 +1278,7 @@ All of the following, run in this directory, must pass:
|
|
|
|
|
data.
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
make build
|
|
|
|
|
d=$(mktemp -d /var/tmp/sfdupes.XXXXXX)
|
|
|
|
|
export SFDUPES_DATABASE="$(mktemp -d /var/tmp/sfdupes.XXXXXX)/db.sqlite"
|
|
|
|
|
mkdir -p "$d/a" "$d/b"
|
|
|
|
@@ -1294,11 +1331,11 @@ All of the following, run in this directory, must pass:
|
|
|
|
|
|
|
|
|
|
## TODO
|
|
|
|
|
|
|
|
|
|
Tracked in [TODO.md](TODO.md).
|
|
|
|
|
Tracked in [`TODO.md`](TODO.md).
|
|
|
|
|
|
|
|
|
|
## License
|
|
|
|
|
|
|
|
|
|
MIT. See [LICENSE](LICENSE).
|
|
|
|
|
MIT. See [`LICENSE`](LICENSE).
|
|
|
|
|
|
|
|
|
|
## Author
|
|
|
|
|
|
|
|
|
|