Compare commits
7
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
e16c46804d | ||
|
|
317abb834c | ||
|
|
652df1c382 | ||
|
|
37893d2d5a | ||
|
|
95aac3d804 | ||
|
|
d4f90f43ff | ||
|
|
58d27a6df3 |
No files matched your search
@@ -29,7 +29,6 @@ first look at a new machine with one command.
|
|||||||
|
|
||||||
```sh
|
```sh
|
||||||
make build
|
make build
|
||||||
export SFDUPES_DATABASE="$HOME/.local/share/sfdupes/db.sqlite"
|
|
||||||
./sfdupes scan /srv
|
./sfdupes scan /srv
|
||||||
./sfdupes report > dupes.tsv
|
./sfdupes report > dupes.tsv
|
||||||
./sfdupes trees > dupetrees.tsv
|
./sfdupes trees > dupetrees.tsv
|
||||||
@@ -47,8 +46,10 @@ Both cover the whole database, or only the paths given after them, as in
|
|||||||
`sfdupes report /srv/photos`. `info` prints the database's path, size and record
|
`sfdupes report /srv/photos`. `info` prints the database's path, size and record
|
||||||
count.
|
count.
|
||||||
|
|
||||||
The database is `/var/lib/sfdupes/db.sqlite` unless `SFDUPES_DATABASE` names
|
Root keeps the database in `/var/lib/sfdupes/db.sqlite`. Any other user keeps it
|
||||||
another file. The first `scan` creates it.
|
in `sfdupes/db.sqlite` in the XDG state directory, `~/.local/state` by default
|
||||||
|
on Linux. `SFDUPES_DATABASE` names another file (see "Location"). The first
|
||||||
|
`scan` creates the database.
|
||||||
|
|
||||||
`report`, `trees` and `info` write their data to stdout. Progress, warnings and
|
`report`, `trees` and `info` write their data to stdout. Progress, warnings and
|
||||||
errors go to stderr. stdout and stderr are a program's two output streams. The
|
errors go to stderr. stdout and stderr are a program's two output streams. The
|
||||||
@@ -116,22 +117,27 @@ docker run --rm -v /srv:/srv:ro -v /var/lib/sfdupes:/var/lib/sfdupes \
|
|||||||
Run `scan` as root, so that it can read every file. `scan` warns about a
|
Run `scan` as root, so that it can read every file. `scan` warns about a
|
||||||
directory it cannot read, and about a new or changed file it cannot read. It
|
directory it cannot read, and about a new or changed file it cannot read. It
|
||||||
skips the path and deletes the records at or below it (see "Rules for the
|
skips the path and deletes the records at or below it (see "Rules for the
|
||||||
walk"). As the file `/etc/cron.d/sfdupes`:
|
walk").
|
||||||
|
|
||||||
|
An unmounted disk leaves its mount point empty, so `scan` deletes every record
|
||||||
|
under it, and the next scan reads every file on the disk again. Test the mount
|
||||||
|
point of each disk at or below `/srv` with its own `mountpoint -q`, joined with
|
||||||
|
`&&`, and test `/srv` only when it is one. As the file `/etc/cron.d/sfdupes`:
|
||||||
|
|
||||||
```
|
```
|
||||||
30 3 * * * root /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log
|
30 3 * * * root if mountpoint -q /srv; then /usr/local/bin/sfdupes scan /srv 2>>/var/log/sfdupes.log || tail -n 3 /var/log/sfdupes.log; else echo "a disk is not mounted; scan of /srv skipped"; fi
|
||||||
```
|
```
|
||||||
|
|
||||||
- The first scan creates `/var/lib/sfdupes/db.sqlite` and its directory. To keep
|
- The first scan creates `/var/lib/sfdupes/db.sqlite` and its directory. To keep
|
||||||
the database elsewhere, put `SFDUPES_DATABASE=/path/to/db.sqlite` before the
|
the database elsewhere, put `SFDUPES_DATABASE=/path/to/db.sqlite` directly
|
||||||
command on the same line.
|
before `/usr/local/bin/sfdupes`.
|
||||||
- `scan` writes nothing to stdout. Its stderr, appended here to
|
- `scan` writes nothing to stdout. Its stderr, appended here to
|
||||||
`/var/log/sfdupes.log`, holds:
|
`/var/log/sfdupes.log`, holds:
|
||||||
- one line naming the database;
|
- one line naming the database;
|
||||||
- progress lines, as described under "Progress";
|
- progress lines, as described under "Progress";
|
||||||
- a warning for each path it skips;
|
- a warning for each path it skips;
|
||||||
- a line for each system or temporary directory the walk leaves out (see
|
- a line for each system, temporary or cache directory the walk leaves out
|
||||||
"Rules for the walk");
|
(see "Rules for the walk");
|
||||||
- a line for each path left out because its filesystem is one the kernel,
|
- a line for each path left out because its filesystem is one the kernel,
|
||||||
the core of the operating system, makes up or keeps in memory, such as
|
the core of the operating system, makes up or keeps in memory, such as
|
||||||
`tmpfs`;
|
`tmpfs`;
|
||||||
@@ -150,8 +156,9 @@ walk"). As the file `/etc/cron.d/sfdupes`:
|
|||||||
- `report`, `trees` and `info` need only read access to the database. The umask
|
- `report`, `trees` and `info` need only read access to the database. The umask
|
||||||
is the setting that removes permissions from the files a process creates.
|
is the setting that removes permissions from the files a process creates.
|
||||||
Under the usual umask of `022`, the first scan creates the database readable
|
Under the usual umask of `022`, the first scan creates the database readable
|
||||||
by every user. An unprivileged user can then run those three commands against
|
by every user. Another user can then run those three commands against root's
|
||||||
root's database.
|
database, with `SFDUPES_DATABASE=/var/lib/sfdupes/db.sqlite` before the
|
||||||
|
command.
|
||||||
|
|
||||||
### Reading the reports
|
### Reading the reports
|
||||||
|
|
||||||
@@ -290,9 +297,9 @@ The reference below is also the program's specification.
|
|||||||
### Settings
|
### Settings
|
||||||
|
|
||||||
- `SFDUPES_DATABASE` names the database file. When it is unset or empty, the
|
- `SFDUPES_DATABASE` names the database file. When it is unset or empty, the
|
||||||
database is `/var/lib/sfdupes/db.sqlite`. There is no command-line flag for
|
database is at its default path (see "Location"). There is no command-line
|
||||||
it. The path names the file exactly, whatever characters it holds. A relative
|
flag for it. The path names the file exactly, whatever characters it holds. A
|
||||||
path is relative to the working directory.
|
relative path is relative to the working directory.
|
||||||
- `--workers N` (`scan`, `dwim`) sets how many directories `scan` reads at once
|
- `--workers N` (`scan`, `dwim`) sets how many directories `scan` reads at once
|
||||||
while it lists the trees. It also sets how many files `scan` reads at once
|
while it lists the trees. It also sets how many files `scan` reads at once
|
||||||
while it hashes them. The default is the number of CPUs. `N` must be at
|
while it hashes them. The default is the number of CPUs. `N` must be at
|
||||||
@@ -302,8 +309,9 @@ The reference below is also the program's specification.
|
|||||||
of directories on other filesystems, as the same flag does for `du` and
|
of directories on other filesystems, as the same flag does for `du` and
|
||||||
`rsync`. By default the walk also enters directories on other filesystems.
|
`rsync`. By default the walk also enters directories on other filesystems.
|
||||||
- `--no-default-excludes` (`scan`, `dwim`) lets the walk enter the default
|
- `--no-default-excludes` (`scan`, `dwim`) lets the walk enter the default
|
||||||
excludes. Those are the system and temporary directories the walk otherwise
|
excludes. Those are the system, temporary and cache directories the walk
|
||||||
leaves out, such as `/var/tmp` on Linux (see "Rules for the walk").
|
otherwise leaves out, such as `/var/tmp` and `/var/cache` on Linux (see "Rules
|
||||||
|
for the walk").
|
||||||
- `--exclude PATH` (`scan`, `dwim`) keeps the walk out of `PATH` and everything
|
- `--exclude PATH` (`scan`, `dwim`) keeps the walk out of `PATH` and everything
|
||||||
below it. `PATH` may be a directory or a file. The flag may be given more than
|
below it. `PATH` may be a directory or a file. The flag may be given more than
|
||||||
once. The walk knows `PATH` by its device ID and inode number, as it knows the
|
once. The walk knows `PATH` by its device ID and inode number, as it knows the
|
||||||
@@ -555,7 +563,7 @@ walk practical on large or busy storage pools.
|
|||||||
`scan` prints a line on stderr naming the path and its filesystem type:
|
`scan` prints a line on stderr naming the path and its filesystem type:
|
||||||
|
|
||||||
```
|
```
|
||||||
scan: skipped /dev/shm (tmpfs, not a persistent filesystem)
|
scan: left out /dev/shm (tmpfs, not a persistent filesystem)
|
||||||
```
|
```
|
||||||
|
|
||||||
The path is not counted as skipped. The records below it are kept, as
|
The path is not counted as skipped. The records below it are kept, as
|
||||||
@@ -565,7 +573,7 @@ walk practical on large or busy storage pools.
|
|||||||
For each one it reaches, `scan` prints a line on stderr naming it:
|
For each one it reaches, `scan` prints a line on stderr naming it:
|
||||||
|
|
||||||
```
|
```
|
||||||
scan: skipped /srv/cache (--exclude)
|
scan: left out /srv/cache (--exclude)
|
||||||
```
|
```
|
||||||
|
|
||||||
The path is not counted as skipped. The scan keeps the records at and below
|
The path is not counted as skipped. The scan keeps the records at and below
|
||||||
@@ -573,12 +581,17 @@ walk practical on large or busy storage pools.
|
|||||||
`--no-default-excludes` does not change this rule.
|
`--no-default-excludes` does not change this rule.
|
||||||
|
|
||||||
- Below an operand, the walk does not enter a default exclude. On Linux, the
|
- Below an operand, the walk does not enter a default exclude. On Linux, the
|
||||||
default excludes hold temporary files and the kernel's view of the running
|
default excludes hold the kernel's view of the running system, temporary
|
||||||
system. They depend on the system, and `sfdupes scan --help` lists those of
|
files, and caches. A cache here is data that programs and package managers
|
||||||
the system it runs on. On Linux they are:
|
download or build again on their own. The default excludes depend on the
|
||||||
|
system, and `sfdupes scan --help` lists those of the system it runs on. In the
|
||||||
|
lists, `*` stands for any one directory name, so `/home/*/.cache` matches
|
||||||
|
`/home/alice/.cache`. On Linux they are:
|
||||||
|
|
||||||
```
|
```
|
||||||
/proc, /sys, /dev, /run, /tmp, /var/tmp
|
/proc, /sys, /dev, /run, /tmp, /var/tmp, /var/cache
|
||||||
|
/root/.cache, /root/.npm, /root/.cargo/registry, /root/go/pkg/mod
|
||||||
|
/home/*/.cache, /home/*/.npm, /home/*/.cargo/registry, /home/*/go/pkg/mod
|
||||||
```
|
```
|
||||||
|
|
||||||
On macOS they are:
|
On macOS they are:
|
||||||
@@ -586,13 +599,21 @@ walk practical on large or busy storage pools.
|
|||||||
```
|
```
|
||||||
/dev, /private/tmp, /private/var/tmp, /private/var/vm
|
/dev, /private/tmp, /private/var/tmp, /private/var/vm
|
||||||
/System/Volumes/VM, /System/Volumes/Data
|
/System/Volumes/VM, /System/Volumes/Data
|
||||||
|
/private/var/folders, /Library/Caches
|
||||||
|
/private/var/root/Library/Caches, /private/var/root/.cache
|
||||||
|
/private/var/root/.npm, /private/var/root/.cargo/registry
|
||||||
|
/private/var/root/go/pkg/mod
|
||||||
|
/Users/*/Library/Caches, /Users/*/.cache, /Users/*/.npm
|
||||||
|
/Users/*/.cargo/registry, /Users/*/go/pkg/mod
|
||||||
```
|
```
|
||||||
|
|
||||||
A walk of `/` on macOS reaches everything in `/System/Volumes/Data` already,
|
`/private/var/folders` holds each user's temporary files and caches on
|
||||||
through firmlinks. Firmlinks are the links macOS uses to show the data
|
macOS. Root's home on macOS is `/var/root`, which the walk reaches as
|
||||||
volume inside the system volume. Walking `/System/Volumes/Data` as well
|
`/private/var/root`. A walk of `/` on macOS reaches everything in
|
||||||
would record each of those files twice. Any other system has no default
|
`/System/Volumes/Data` already, through firmlinks. Firmlinks are the links
|
||||||
excludes.
|
macOS uses to show the data volume inside the system volume. Walking
|
||||||
|
`/System/Volumes/Data` as well would record each of those files twice. Any
|
||||||
|
other system has no default excludes.
|
||||||
|
|
||||||
An operand at or inside a default exclude is walked, so
|
An operand at or inside a default exclude is walked, so
|
||||||
`sfdupes scan /tmp/x` works when `/tmp/x` is on a persistent filesystem.
|
`sfdupes scan /tmp/x` works when `/tmp/x` is on a persistent filesystem.
|
||||||
@@ -601,7 +622,7 @@ walk practical on large or busy storage pools.
|
|||||||
not enter, `scan` prints a line on stderr naming it:
|
not enter, `scan` prints a line on stderr naming it:
|
||||||
|
|
||||||
```
|
```
|
||||||
scan: skipped /var/tmp (default exclude)
|
scan: left out /var/tmp (default exclude)
|
||||||
```
|
```
|
||||||
|
|
||||||
The directory is not counted as skipped. `scan` prints the filesystem line
|
The directory is not counted as skipped. `scan` prints the filesystem line
|
||||||
@@ -661,6 +682,18 @@ scan: 123400 files seen (1200 added, 34 updated, 56 removed, 122166 unchanged),
|
|||||||
|
|
||||||
`removed` counts deleted records, which are not part of the files seen.
|
`removed` counts deleted records, which are not part of the files seen.
|
||||||
|
|
||||||
|
The warning for a skipped path names the path once and gives the reason:
|
||||||
|
|
||||||
|
```
|
||||||
|
scan: skipped /srv/a/closed (permission denied)
|
||||||
|
scan: skipped /srv/a/pipe (a FIFO)
|
||||||
|
```
|
||||||
|
|
||||||
|
The summary counts each path named in such a warning. A line for a path left out
|
||||||
|
on purpose starts with `scan: left out` instead, and is not counted. A search of
|
||||||
|
the log for `scan: skipped` therefore finds the paths the summary counts. When a
|
||||||
|
file with several hard links cannot be read, each of its paths gets a warning.
|
||||||
|
|
||||||
#### Stopping a scan
|
#### Stopping a scan
|
||||||
|
|
||||||
`scan` stops cleanly on `SIGINT`, the signal Ctrl-C sends, or `SIGTERM`, the
|
`scan` stops cleanly on `SIGINT`, the signal Ctrl-C sends, or `SIGTERM`, the
|
||||||
@@ -880,7 +913,8 @@ The summary line on stderr gives:
|
|||||||
operands;
|
operands;
|
||||||
- the duplicate-tree groups;
|
- the duplicate-tree groups;
|
||||||
- the dupe trees, which is the number of rows;
|
- the dupe trees, which is the number of rows;
|
||||||
- the reclaimable bytes, the sum of `size` over all rows.
|
- the reclaimable bytes, the sum of `size` over all rows, leaving out a row
|
||||||
|
whose `dupe` lies inside another row's `dupe`.
|
||||||
|
|
||||||
```
|
```
|
||||||
trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable
|
trees: 14 records read, 1 duplicate tree groups, 1 dupe trees, 3.0 KiB reclaimable
|
||||||
@@ -971,7 +1005,22 @@ All five subcommands work on one SQLite database file.
|
|||||||
#### Location
|
#### Location
|
||||||
|
|
||||||
The database is the file named by `SFDUPES_DATABASE`, when that is set and not
|
The database is the file named by `SFDUPES_DATABASE`, when that is set and not
|
||||||
empty, and otherwise `/var/lib/sfdupes/db.sqlite`.
|
empty. Otherwise it is at its default path, which depends on the user:
|
||||||
|
|
||||||
|
- For root, it is `/var/lib/sfdupes/db.sqlite`.
|
||||||
|
- For any other user, it is `sfdupes/db.sqlite` in the XDG state directory. That
|
||||||
|
directory is `$XDG_STATE_HOME` when that is set and not empty. Otherwise it is
|
||||||
|
`~/.local/state` on Linux and `~/Library/Application Support` on macOS. An
|
||||||
|
`XDG_STATE_HOME` that is not an absolute path is a fatal error, exit 1, for
|
||||||
|
every subcommand. The error names `XDG_STATE_HOME`.
|
||||||
|
|
||||||
|
`scan` creates each missing directory on the way to the database with mode 755.
|
||||||
|
At the default path of a user other than root, it uses mode 700 instead, as the
|
||||||
|
XDG Base Directory specification asks. Other users cannot enter a directory with
|
||||||
|
mode 700, so they cannot read a database in it.
|
||||||
|
|
||||||
|
All five subcommands use the same default, so a user's `report` reads the
|
||||||
|
database that user's `scan` wrote.
|
||||||
|
|
||||||
Each subcommand prints one line on stderr naming the database it uses, before
|
Each subcommand prints one line on stderr naming the database it uses, before
|
||||||
anything else and whether or not stderr is a terminal. The line gives the path
|
anything else and whether or not stderr is a terminal. The line gives the path
|
||||||
@@ -1011,9 +1060,9 @@ Only one `scan` at a time can hold the lock.
|
|||||||
- For its whole run, `scan` holds a lock on a lock file beside the database. The
|
- For its whole run, `scan` holds a lock on a lock file beside the database. The
|
||||||
lock is taken with the `flock(2)` system call, a request a program makes to
|
lock is taken with the `flock(2)` system call, a request a program makes to
|
||||||
the kernel, and only one process can hold it. The lock file's name is the
|
the kernel, and only one process can hold it. The lock file's name is the
|
||||||
database's path, with symlinks resolved, plus `.lock`. By default that is
|
database's path, with symlinks resolved, plus `.lock`. For root, by default,
|
||||||
`/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the same
|
that is `/var/lib/sfdupes/db.sqlite.lock`. A path through a symlink names the
|
||||||
lock file.
|
same lock file.
|
||||||
- `scan` takes the lock before it walks the filesystem or opens the database.
|
- `scan` takes the lock before it walks the filesystem or opens the database.
|
||||||
Just before that, it creates the database file, empty, if it is missing. So
|
Just before that, it creates the database file, empty, if it is missing. So
|
||||||
the lock file is named the same way before the first scan as after. A symlink
|
the lock file is named the same way before the first scan as after. A symlink
|
||||||
@@ -1235,6 +1284,14 @@ On a terminal:
|
|||||||
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
|
⠹ [3/4] hash: [12345/98765], 92 files/s, elapsed 2m32s, remain 17m44s 12% |█ |
|
||||||
```
|
```
|
||||||
|
|
||||||
|
- During `scan`'s `walk`, the line below the display shows the path of the last
|
||||||
|
file the walk found. Each draw shows the path of that moment, cut short before
|
||||||
|
the terminal's last column. The path is escaped as in the reports (see "Report
|
||||||
|
output format"). Each control character left after that is written as `\x` and
|
||||||
|
its code in two hexadecimal digits, such as `\x1b` for the escape character. A
|
||||||
|
control character is one that a terminal acts on instead of showing. The line
|
||||||
|
is erased when the walk ends, and the next phase is drawn on it.
|
||||||
|
|
||||||
- No line reaches the terminal's last column. A line too wide for the terminal
|
- No line reaches the terminal's last column. A line too wide for the terminal
|
||||||
first leaves out its rate, the words of a stalled phase, or what the phase is
|
first leaves out its rate, the words of a stalled phase, or what the phase is
|
||||||
waiting for. It leaves out its bar next. It is cut short at the edge only when
|
waiting for. It leaves out its bar next. It is cut short at the edge only when
|
||||||
|
|||||||
+4
-2
@@ -5,8 +5,10 @@
|
|||||||
// KiB, and of the content. The content hash covers the whole file under
|
// KiB, and of the content. The content hash covers the whole file under
|
||||||
// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large
|
// 100 MiB, and 1 MiB at each gigabyte at 100 MiB or more, so a large
|
||||||
// file is never read in full. scan maintains a persistent SQLite
|
// file is never read in full. scan maintains a persistent SQLite
|
||||||
// database of file signatures (SFDUPES_DATABASE, default
|
// database of file signatures that the other subcommands read. The
|
||||||
// /var/lib/sfdupes/db.sqlite) that the other subcommands read.
|
// database is SFDUPES_DATABASE, or by default /var/lib/sfdupes/db.sqlite
|
||||||
|
// for root and sfdupes/db.sqlite in the XDG state directory for any
|
||||||
|
// other user.
|
||||||
//
|
//
|
||||||
// Usage:
|
// Usage:
|
||||||
//
|
//
|
||||||
|
|||||||
@@ -3,6 +3,8 @@ module sneak.berlin/go/sfdupes
|
|||||||
go 1.25.7
|
go 1.25.7
|
||||||
|
|
||||||
require (
|
require (
|
||||||
|
github.com/adrg/xdg v0.5.3
|
||||||
|
github.com/rivo/uniseg v0.4.7
|
||||||
github.com/schollz/progressbar/v3 v3.19.1
|
github.com/schollz/progressbar/v3 v3.19.1
|
||||||
github.com/spf13/cobra v1.10.2
|
github.com/spf13/cobra v1.10.2
|
||||||
golang.org/x/sys v0.46.0
|
golang.org/x/sys v0.46.0
|
||||||
@@ -18,7 +20,6 @@ require (
|
|||||||
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
|
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
|
||||||
github.com/ncruces/go-strftime v1.0.0 // indirect
|
github.com/ncruces/go-strftime v1.0.0 // indirect
|
||||||
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
|
||||||
github.com/rivo/uniseg v0.4.7 // indirect
|
|
||||||
github.com/spf13/pflag v1.0.9 // indirect
|
github.com/spf13/pflag v1.0.9 // indirect
|
||||||
modernc.org/libc v1.74.1 // indirect
|
modernc.org/libc v1.74.1 // indirect
|
||||||
modernc.org/mathutil v1.7.1 // indirect
|
modernc.org/mathutil v1.7.1 // indirect
|
||||||
|
|||||||
@@ -1,3 +1,5 @@
|
|||||||
|
github.com/adrg/xdg v0.5.3 h1:xRnxJXne7+oWDatRhR1JLnvuccuIeCoBu2rtuLqQB78=
|
||||||
|
github.com/adrg/xdg v0.5.3/go.mod h1:nlTsY+NNiCBGCK2tpm09vRqfVzrc2fLmXGpBLF0zlTQ=
|
||||||
github.com/chengxilo/virtualterm v1.0.4 h1:Z6IpERbRVlfB8WkOmtbHiDbBANU7cimRIof7mk9/PwM=
|
github.com/chengxilo/virtualterm v1.0.4 h1:Z6IpERbRVlfB8WkOmtbHiDbBANU7cimRIof7mk9/PwM=
|
||||||
github.com/chengxilo/virtualterm v1.0.4/go.mod h1:DyxxBZz/x1iqJjFxTFcr6/x+jSpqN0iwWCOK1q10rlY=
|
github.com/chengxilo/virtualterm v1.0.4/go.mod h1:DyxxBZz/x1iqJjFxTFcr6/x+jSpqN0iwWCOK1q10rlY=
|
||||||
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
|
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
|
||||||
|
|||||||
@@ -34,10 +34,14 @@ const (
|
|||||||
|
|
||||||
// databaseHelp is the paragraph of the root and scan help that names
|
// databaseHelp is the paragraph of the root and scan help that names
|
||||||
// the database (README "Location").
|
// the database (README "Location").
|
||||||
const databaseHelp = "scan and dwim store what they find in a database " +
|
const databaseHelp = `scan and dwim store what they find in a database file. report,
|
||||||
"file. report, trees\nand info read it. The file is $" + databaseEnv +
|
trees and info read it. The file is $SFDUPES_DATABASE when that is set
|
||||||
" when that is set and\nnot empty, and otherwise " +
|
and not empty. Otherwise root uses /var/lib/sfdupes/db.sqlite. Any other
|
||||||
defaultDatabasePath + ", where creating it\nneeds root.\n"
|
user uses sfdupes/db.sqlite in the XDG state directory: $XDG_STATE_HOME,
|
||||||
|
or ~/.local/state when that is unset or empty
|
||||||
|
(~/Library/Application Support on macOS). An XDG_STATE_HOME that is
|
||||||
|
not an absolute path is an error.
|
||||||
|
`
|
||||||
|
|
||||||
// escapesHelp is the paragraph of the report and trees help on how
|
// escapesHelp is the paragraph of the report and trees help on how
|
||||||
// paths are escaped (README "Report output format").
|
// paths are escaped (README "Report output format").
|
||||||
|
|||||||
+131
-24
@@ -15,6 +15,7 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
|
|
||||||
|
"github.com/adrg/xdg"
|
||||||
"golang.org/x/sys/unix"
|
"golang.org/x/sys/unix"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -559,13 +560,17 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
|||||||
// line.
|
// line.
|
||||||
t.Setenv(databaseEnv, "")
|
t.Setenv(databaseEnv, "")
|
||||||
|
|
||||||
// Whether scan calls the database new depends on that machine too.
|
dbPath, err := defaultDatabasePath(os.Geteuid())
|
||||||
scanWant := "scan: database " + defaultDatabasePath + " (default)\n"
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
_, err := os.Stat(defaultDatabasePath)
|
// Whether scan calls the database new depends on that machine too.
|
||||||
|
scanWant := "scan: database " + dbPath + " (default)\n"
|
||||||
|
|
||||||
|
_, err = os.Stat(dbPath)
|
||||||
if errors.Is(err, fs.ErrNotExist) {
|
if errors.Is(err, fs.ErrNotExist) {
|
||||||
scanWant = "scan: new database " + defaultDatabasePath +
|
scanWant = "scan: new database " + dbPath + " (default)\n"
|
||||||
" (default)\n"
|
|
||||||
}
|
}
|
||||||
|
|
||||||
missing := filepath.Join(t.TempDir(), "nope")
|
missing := filepath.Join(t.TempDir(), "nope")
|
||||||
@@ -585,21 +590,21 @@ func TestRunNamesDefaultDatabase(t *testing.T) {
|
|||||||
run: func() error {
|
run: func() error {
|
||||||
return runReport(cancelledContext(t), nil, io.Discard)
|
return runReport(cancelledContext(t), nil, io.Discard)
|
||||||
},
|
},
|
||||||
want: "report: database " + defaultDatabasePath + " (default)\n" +
|
want: "report: database " + dbPath + " (default)\n" +
|
||||||
reportOpenLine + "\n" + reportOpenLine + "\n",
|
reportOpenLine + "\n" + reportOpenLine + "\n",
|
||||||
},
|
},
|
||||||
cmdTrees: {
|
cmdTrees: {
|
||||||
run: func() error {
|
run: func() error {
|
||||||
return runTrees(cancelledContext(t), nil, io.Discard)
|
return runTrees(cancelledContext(t), nil, io.Discard)
|
||||||
},
|
},
|
||||||
want: "trees: database " + defaultDatabasePath + " (default)\n" +
|
want: "trees: database " + dbPath + " (default)\n" +
|
||||||
treesOpenLine + "\n" + treesOpenLine + "\n",
|
treesOpenLine + "\n" + treesOpenLine + "\n",
|
||||||
},
|
},
|
||||||
cmdInfo: {
|
cmdInfo: {
|
||||||
run: func() error {
|
run: func() error {
|
||||||
return runInfo(cancelledContext(t), io.Discard)
|
return runInfo(cancelledContext(t), io.Discard)
|
||||||
},
|
},
|
||||||
want: "info: database " + defaultDatabasePath + " (default)\n" +
|
want: "info: database " + dbPath + " (default)\n" +
|
||||||
infoOpenLine + "\n" + infoOpenLine + "\n",
|
infoOpenLine + "\n" + infoOpenLine + "\n",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -716,6 +721,19 @@ func TestRunHelp(t *testing.T) {
|
|||||||
t.Errorf("run(%v) stdout = %q, want nothing (data only)",
|
t.Errorf("run(%v) stdout = %q, want nothing (data only)",
|
||||||
args, got)
|
args, got)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// README §Location: the help of sfdupes and of scan says where
|
||||||
|
// the database is, and that XDG_STATE_HOME must be an absolute
|
||||||
|
// path.
|
||||||
|
for _, want := range []string{
|
||||||
|
"scan and dwim store", databaseEnv, rootDatabasePath,
|
||||||
|
stateHomeEnv, "~/.local/state", "not an absolute path is an error",
|
||||||
|
} {
|
||||||
|
if !strings.Contains(stderr.String(), want) {
|
||||||
|
t.Errorf("run(%v) stderr = %q, want it to name %q",
|
||||||
|
args, stderr.String(), want)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// The scan help lists --exclude with its argument.
|
// The scan help lists --exclude with its argument.
|
||||||
@@ -758,28 +776,24 @@ func TestRunScanUsageLine(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestRunHelpDescribesDatabaseAndOutput(t *testing.T) {
|
func TestRunHelpDescribesOutput(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
// README §Settings: the help names the database, and says what
|
// README §Settings: the help of report, trees, info and dwim says
|
||||||
// report, trees, info and dwim print. The help of report and trees
|
// what each prints. The help of report and trees also explains
|
||||||
// also explains their PATH operands.
|
// their PATH operands. TestRunHelp checks the database paragraph
|
||||||
database := []string{
|
// of the root and scan help.
|
||||||
"scan and dwim store", databaseEnv, defaultDatabasePath, "needs root",
|
|
||||||
}
|
|
||||||
escapes := []string{`\\`, `\t`, `\n`, `\r`}
|
escapes := []string{`\\`, `\t`, `\n`, `\r`}
|
||||||
operands := []string{
|
operands := []string{
|
||||||
"With PATH operands", "at or below them", "working directory",
|
"With PATH operands", "at or below them", "working directory",
|
||||||
"No PATH is looked up", "not followed",
|
"No PATH is looked up", "not followed",
|
||||||
}
|
}
|
||||||
|
|
||||||
// The subcommand whose help is printed; none for the root.
|
// The subcommand whose help is printed.
|
||||||
cases := []struct {
|
cases := []struct {
|
||||||
subcommand []string
|
subcommand []string
|
||||||
want []string
|
want []string
|
||||||
}{
|
}{
|
||||||
{nil, database},
|
|
||||||
{[]string{cmdScan}, database},
|
|
||||||
{
|
{
|
||||||
[]string{cmdReport},
|
[]string{cmdReport},
|
||||||
slices.Concat([]string{
|
slices.Concat([]string{
|
||||||
@@ -999,8 +1013,7 @@ func TestRunScanSucceedsDespiteWarnings(t *testing.T) {
|
|||||||
|
|
||||||
stderr := scanOK(t, dir)
|
stderr := scanOK(t, dir)
|
||||||
|
|
||||||
warning := "hash " + unreadable + ": open " + unreadable +
|
warning := "scan: skipped " + unreadable + " (permission denied)\n"
|
||||||
": permission denied\n"
|
|
||||||
if !strings.Contains(stderr, warning) {
|
if !strings.Contains(stderr, warning) {
|
||||||
t.Errorf("stderr = %q, want %q", stderr, warning)
|
t.Errorf("stderr = %q, want %q", stderr, warning)
|
||||||
}
|
}
|
||||||
@@ -1272,7 +1285,7 @@ func TestRunScanSkipsSymlinkToFIFOOperand(t *testing.T) {
|
|||||||
|
|
||||||
stderr := scanOK(t, link)
|
stderr := scanOK(t, link)
|
||||||
|
|
||||||
warning := "walk " + fifo + ": skipping FIFO operand\n"
|
warning := "scan: skipped " + fifo + " (a FIFO)\n"
|
||||||
if !strings.Contains(stderr, warning) {
|
if !strings.Contains(stderr, warning) {
|
||||||
t.Errorf("stderr = %q, want %q", stderr, warning)
|
t.Errorf("stderr = %q, want %q", stderr, warning)
|
||||||
}
|
}
|
||||||
@@ -1307,7 +1320,7 @@ func assertOperandSkipped(t *testing.T, dbPath, operand, kind,
|
|||||||
|
|
||||||
stderr := scanOK(t, operand)
|
stderr := scanOK(t, operand)
|
||||||
|
|
||||||
warning := "walk " + operand + ": skipping " + kind + " operand\n"
|
warning := "scan: skipped " + operand + " (a " + kind + ")\n"
|
||||||
if !strings.Contains(stderr, warning) {
|
if !strings.Contains(stderr, warning) {
|
||||||
t.Errorf("stderr = %q, want %q", stderr, warning)
|
t.Errorf("stderr = %q, want %q", stderr, warning)
|
||||||
}
|
}
|
||||||
@@ -1341,7 +1354,7 @@ func TestRunScanSkipsOperandInsideZFS(t *testing.T) {
|
|||||||
writeFile(t, hourly, "f", pattern(1, 10))
|
writeFile(t, hourly, "f", pattern(1, 10))
|
||||||
f := writeFile(t, dir, "f", pattern(2, 10))
|
f := writeFile(t, dir, "f", pattern(2, 10))
|
||||||
|
|
||||||
warning := "walk " + hourly + ": skipping operand inside a .zfs directory\n"
|
warning := "scan: skipped " + hourly + " (inside a .zfs directory)\n"
|
||||||
|
|
||||||
for _, tc := range []struct {
|
for _, tc := range []struct {
|
||||||
operands []string
|
operands []string
|
||||||
@@ -1393,7 +1406,7 @@ func TestRunScanSkipsOperandThroughSymlinkIntoZFS(t *testing.T) {
|
|||||||
stderr := scanOK(t, filepath.Join(snaps, "daily"))
|
stderr := scanOK(t, filepath.Join(snaps, "daily"))
|
||||||
|
|
||||||
daily := filepath.Join(dir, "pool", ".zfs", "snapshot", "daily")
|
daily := filepath.Join(dir, "pool", ".zfs", "snapshot", "daily")
|
||||||
warning := "walk " + daily + ": skipping operand inside a .zfs directory\n"
|
warning := "scan: skipped " + daily + " (inside a .zfs directory)\n"
|
||||||
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
|
summary := "scan: 0 files seen (0 added, 0 updated, 0 removed, " +
|
||||||
"0 unchanged), 1 skipped\n"
|
"0 unchanged), 1 skipped\n"
|
||||||
|
|
||||||
@@ -1612,6 +1625,100 @@ func TestRunDatabasePathUsedAsGiven(t *testing.T) {
|
|||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRunNonRootUsesStateDirectory(t *testing.T) {
|
||||||
|
// README §Location: without SFDUPES_DATABASE, a user other than
|
||||||
|
// root keeps the database in the XDG state directory. scan creates
|
||||||
|
// the missing directories with mode 0700, and report and trees read
|
||||||
|
// the same file.
|
||||||
|
if os.Geteuid() == 0 {
|
||||||
|
t.Skip("root uses " + rootDatabasePath)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Cleanup(xdg.Reload)
|
||||||
|
|
||||||
|
state := filepath.Join(t.TempDir(), "state")
|
||||||
|
t.Setenv(stateHomeEnv, state)
|
||||||
|
t.Setenv(databaseEnv, "")
|
||||||
|
xdg.Reload()
|
||||||
|
|
||||||
|
assertRunsUseDatabase(t, filepath.Join(state, "sfdupes"), "db.sqlite")
|
||||||
|
|
||||||
|
for _, dir := range []string{state, filepath.Join(state, "sfdupes")} {
|
||||||
|
fi, err := os.Stat(dir)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := fi.Mode().Perm(); got != stateDirPerm {
|
||||||
|
t.Errorf("%s mode = %#o, want %#o", dir, got, stateDirPerm)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunNonRootDatabaseEnvDirectoryMode(t *testing.T) {
|
||||||
|
// README §Location: for SFDUPES_DATABASE, scan creates each missing
|
||||||
|
// directory with mode 755, for a user other than root too.
|
||||||
|
if os.Geteuid() == 0 {
|
||||||
|
t.Skip("needs a user other than root")
|
||||||
|
}
|
||||||
|
|
||||||
|
top := filepath.Join(t.TempDir(), "missing")
|
||||||
|
sub := filepath.Join(top, "sub")
|
||||||
|
t.Setenv(databaseEnv, filepath.Join(sub, "db.sqlite"))
|
||||||
|
|
||||||
|
scanOK(t, t.TempDir())
|
||||||
|
|
||||||
|
for _, dir := range []string{top, sub} {
|
||||||
|
fi, err := os.Stat(dir)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := fi.Mode().Perm(); got != dbDirPerm {
|
||||||
|
t.Errorf("%s mode = %#o, want %#o", dir, got, dbDirPerm)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRunRelativeStateHomeFails(t *testing.T) {
|
||||||
|
// README §Location: without SFDUPES_DATABASE, a user other than
|
||||||
|
// root whose XDG_STATE_HOME is not an absolute path gets an error
|
||||||
|
// naming it from every subcommand, which exits 1 before it does
|
||||||
|
// anything else.
|
||||||
|
if os.Geteuid() == 0 {
|
||||||
|
t.Skip("root does not use " + stateHomeEnv)
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Setenv(stateHomeEnv, "relative/state")
|
||||||
|
t.Setenv(databaseEnv, "")
|
||||||
|
|
||||||
|
want := "sfdupes: " + stateHomeEnv +
|
||||||
|
" must be an absolute path, got \"relative/state\"\n"
|
||||||
|
|
||||||
|
// A scan that got past the check would stop at this missing
|
||||||
|
// operand, before it creates anything.
|
||||||
|
missing := filepath.Join(t.TempDir(), "missing")
|
||||||
|
|
||||||
|
for _, args := range [][]string{
|
||||||
|
{cmdScan, missing}, {cmdReport}, {cmdTrees}, {cmdInfo},
|
||||||
|
} {
|
||||||
|
var stdout, stderr bytes.Buffer
|
||||||
|
|
||||||
|
code := run(args, &stdout, &stderr)
|
||||||
|
if code != exitFatal || stderr.String() != want {
|
||||||
|
t.Errorf("run(%v) = %d with stderr %q, want %d and %q",
|
||||||
|
args, code, stderr.String(), exitFatal, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// dwim scans /, so runDwim gets the missing operand instead.
|
||||||
|
err := runDwim(t.Context(), missing, scanOptions{workers: 1}, io.Discard)
|
||||||
|
if !errors.Is(err, errStateHomeNotAbsolute) {
|
||||||
|
t.Errorf("runDwim: error = %v, want %v", err,
|
||||||
|
errStateHomeNotAbsolute)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// holdScanLock takes the lock on the database at path, as a running
|
// holdScanLock takes the lock on the database at path, as a running
|
||||||
// scan does, and holds it until the test ends. It fails the test when
|
// scan does, and holds it until the test ends. It fails the test when
|
||||||
// the lock is already held.
|
// the lock is already held.
|
||||||
|
|||||||
+68
-15
@@ -15,28 +15,40 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"github.com/adrg/xdg"
|
||||||
"golang.org/x/sys/unix"
|
"golang.org/x/sys/unix"
|
||||||
// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
|
// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
|
||||||
// disabled.
|
// disabled.
|
||||||
_ "modernc.org/sqlite"
|
_ "modernc.org/sqlite"
|
||||||
)
|
)
|
||||||
|
|
||||||
// defaultDatabasePath is where the persistent scan database lives when
|
// rootDatabasePath is the database root uses when SFDUPES_DATABASE is
|
||||||
// SFDUPES_DATABASE is not set.
|
// not set.
|
||||||
const defaultDatabasePath = "/var/lib/sfdupes/db.sqlite"
|
const rootDatabasePath = "/var/lib/sfdupes/db.sqlite"
|
||||||
|
|
||||||
// databaseEnv is the environment variable that overrides the database
|
// databaseEnv is the environment variable that overrides the database
|
||||||
// path.
|
// path.
|
||||||
const databaseEnv = "SFDUPES_DATABASE"
|
const databaseEnv = "SFDUPES_DATABASE"
|
||||||
|
|
||||||
|
// stateHomeEnv is the environment variable that names the XDG state
|
||||||
|
// directory.
|
||||||
|
const stateHomeEnv = "XDG_STATE_HOME"
|
||||||
|
|
||||||
// schemaVersion is the database schema version this build reads and
|
// schemaVersion is the database schema version this build reads and
|
||||||
// writes, stored in PRAGMA user_version.
|
// writes, stored in PRAGMA user_version.
|
||||||
const schemaVersion = 1
|
const schemaVersion = 1
|
||||||
|
|
||||||
// dbDirPerm is the mode for a database parent directory created by
|
// dbDirPerm is the mode for a directory scan creates for root's
|
||||||
// scan.
|
// default database or for SFDUPES_DATABASE.
|
||||||
const dbDirPerm = 0o755
|
const dbDirPerm = 0o755
|
||||||
|
|
||||||
|
// stateDirPerm is the mode for a directory scan creates for the
|
||||||
|
// database at a user's default path in the XDG state directory. It is
|
||||||
|
// the mode the XDG Base Directory specification asks for, and it keeps
|
||||||
|
// other users from reading the database, which lists every scanned
|
||||||
|
// path.
|
||||||
|
const stateDirPerm = 0o700
|
||||||
|
|
||||||
// dbFilePerm is the mode for a database file created by scan, the mode
|
// dbFilePerm is the mode for a database file created by scan, the mode
|
||||||
// SQLite gives a database file it creates.
|
// SQLite gives a database file it creates.
|
||||||
const dbFilePerm = 0o644
|
const dbFilePerm = 0o644
|
||||||
@@ -144,15 +156,48 @@ var errScanRunning = errors.New("another scan is running")
|
|||||||
// 32 bytes of a SHA-256.
|
// 32 bytes of a SHA-256.
|
||||||
var errHashSize = errors.New("hash is not 32 bytes")
|
var errHashSize = errors.New("hash is not 32 bytes")
|
||||||
|
|
||||||
|
// errStateHomeNotAbsolute reports an XDG_STATE_HOME that is set to a
|
||||||
|
// path that is not absolute.
|
||||||
|
var errStateHomeNotAbsolute = errors.New(
|
||||||
|
stateHomeEnv + " must be an absolute path")
|
||||||
|
|
||||||
|
// defaultDatabasePath is the database used when SFDUPES_DATABASE is not
|
||||||
|
// set: rootDatabasePath for root (euid 0), and db.sqlite in the
|
||||||
|
// directory sfdupes under the XDG state directory for any other user.
|
||||||
|
// xdg reads XDG_STATE_HOME and HOME once, when the program starts. For
|
||||||
|
// any other user, an XDG_STATE_HOME that is set and not an absolute
|
||||||
|
// path is an error, where xdg would use its default instead. The path
|
||||||
|
// is joined as text, not cleaned, for the reason absolutePath gives.
|
||||||
|
func defaultDatabasePath(euid int) (string, error) {
|
||||||
|
if euid == 0 {
|
||||||
|
return rootDatabasePath, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
stateHome := os.Getenv(stateHomeEnv)
|
||||||
|
if stateHome != "" && !filepath.IsAbs(stateHome) {
|
||||||
|
return "", fmt.Errorf("%w, got %q", errStateHomeNotAbsolute,
|
||||||
|
stateHome)
|
||||||
|
}
|
||||||
|
|
||||||
|
stateDir := strings.TrimSuffix(xdg.StateHome, "/")
|
||||||
|
|
||||||
|
return stateDir + "/sfdupes/db.sqlite", nil
|
||||||
|
}
|
||||||
|
|
||||||
// databasePath resolves the database location: SFDUPES_DATABASE when
|
// databasePath resolves the database location: SFDUPES_DATABASE when
|
||||||
// set and non-empty, the compiled-in default otherwise. It returns the
|
// set and non-empty, defaultDatabasePath otherwise. It returns the
|
||||||
// path, made absolute by absolutePath, and where the path came from,
|
// path, made absolute by absolutePath, and where the path came from,
|
||||||
// "default" or "from SFDUPES_DATABASE", for the line each subcommand
|
// "default" or "from SFDUPES_DATABASE", for the line each subcommand
|
||||||
// prints first on stderr.
|
// prints first on stderr.
|
||||||
func databasePath() (string, string, error) {
|
func databasePath() (string, string, error) {
|
||||||
path, source := defaultDatabasePath, "default"
|
path, source := os.Getenv(databaseEnv), "from "+databaseEnv
|
||||||
if p := os.Getenv(databaseEnv); p != "" {
|
if path == "" {
|
||||||
path, source = p, "from "+databaseEnv
|
def, err := defaultDatabasePath(os.Geteuid())
|
||||||
|
if err != nil {
|
||||||
|
return "", "", err
|
||||||
|
}
|
||||||
|
|
||||||
|
path, source = def, "default"
|
||||||
}
|
}
|
||||||
|
|
||||||
abs, err := absolutePath(path)
|
abs, err := absolutePath(path)
|
||||||
@@ -188,15 +233,23 @@ func absolutePath(path string) (string, error) {
|
|||||||
|
|
||||||
// createDatabaseDirectory creates the directory the database at path
|
// createDatabaseDirectory creates the directory the database at path
|
||||||
// goes in, and any directory above it, if missing. path is absolute, as
|
// goes in, and any directory above it, if missing. path is absolute, as
|
||||||
// databasePath returns it. The directory is path with its last element
|
// databasePath returns it. Each directory gets mode dbDirPerm, or
|
||||||
// cut off as text, by filepath.Split. filepath.Dir would clean it,
|
// stateDirPerm when databasePath uses a user's default path in the XDG
|
||||||
// dropping each ".." together with the directory name before it; when
|
// state directory: the user is not root and SFDUPES_DATABASE is unset
|
||||||
// that name is a symlink, the cleaned path names another directory than
|
// or empty. The directory is path with its last element cut off as
|
||||||
// the one the system reaches when it opens path.
|
// text, by filepath.Split. filepath.Dir would clean it, dropping each
|
||||||
|
// ".." together with the directory name before it; when that name is a
|
||||||
|
// symlink, the cleaned path names another directory than the one the
|
||||||
|
// system reaches when it opens path.
|
||||||
func createDatabaseDirectory(path string) error {
|
func createDatabaseDirectory(path string) error {
|
||||||
dir, _ := filepath.Split(path)
|
dir, _ := filepath.Split(path)
|
||||||
|
|
||||||
err := os.MkdirAll(dir, dbDirPerm)
|
perm := os.FileMode(dbDirPerm)
|
||||||
|
if os.Geteuid() != 0 && os.Getenv(databaseEnv) == "" {
|
||||||
|
perm = stateDirPerm
|
||||||
|
}
|
||||||
|
|
||||||
|
err := os.MkdirAll(dir, perm)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return fmt.Errorf("create database directory: %w", err)
|
return fmt.Errorf("create database directory: %w", err)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,10 +9,13 @@ import (
|
|||||||
"io/fs"
|
"io/fs"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"runtime"
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
|
"github.com/adrg/xdg"
|
||||||
)
|
)
|
||||||
|
|
||||||
// testDBPath returns a database path inside a fresh temp dir, with
|
// testDBPath returns a database path inside a fresh temp dir, with
|
||||||
@@ -56,11 +59,16 @@ func TestDatabasePath(t *testing.T) {
|
|||||||
|
|
||||||
t.Chdir(dir)
|
t.Chdir(dir)
|
||||||
|
|
||||||
|
defaultPath, err := defaultDatabasePath(os.Geteuid())
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
fromEnv := "from " + databaseEnv
|
fromEnv := "from " + databaseEnv
|
||||||
cases := []struct {
|
cases := []struct {
|
||||||
env, path, source string
|
env, path, source string
|
||||||
}{
|
}{
|
||||||
{"", defaultDatabasePath, "default"},
|
{"", defaultPath, "default"},
|
||||||
{"/custom/place.sqlite", "/custom/place.sqlite", fromEnv},
|
{"/custom/place.sqlite", "/custom/place.sqlite", fromEnv},
|
||||||
{"/custom/link/../place.sqlite", "/custom/link/../place.sqlite",
|
{"/custom/link/../place.sqlite", "/custom/link/../place.sqlite",
|
||||||
fromEnv},
|
fromEnv},
|
||||||
@@ -89,6 +97,48 @@ func TestDatabasePath(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestDefaultDatabasePath(t *testing.T) {
|
||||||
|
// README §Location: root's database is /var/lib/sfdupes/db.sqlite.
|
||||||
|
// Any other user's is db.sqlite in sfdupes under XDG_STATE_HOME, or
|
||||||
|
// under ~/.local/state when XDG_STATE_HOME is unset or empty. For
|
||||||
|
// that user, an XDG_STATE_HOME that is not an absolute path is an
|
||||||
|
// error.
|
||||||
|
if runtime.GOOS != goosLinux {
|
||||||
|
t.Skip("macOS has another default state directory")
|
||||||
|
}
|
||||||
|
|
||||||
|
t.Cleanup(xdg.Reload)
|
||||||
|
t.Setenv("HOME", "/home/someone")
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
euid int
|
||||||
|
stateHome string
|
||||||
|
want string
|
||||||
|
wantErr error
|
||||||
|
}{
|
||||||
|
{0, "", rootDatabasePath, nil},
|
||||||
|
{0, "/elsewhere/state", rootDatabasePath, nil},
|
||||||
|
{0, "relative/state", rootDatabasePath, nil},
|
||||||
|
{1000, "", "/home/someone/.local/state/sfdupes/db.sqlite", nil},
|
||||||
|
{1000, "relative/state", "", errStateHomeNotAbsolute},
|
||||||
|
{1000, "~/state", "", errStateHomeNotAbsolute},
|
||||||
|
{1000, "/elsewhere/state", "/elsewhere/state/sfdupes/db.sqlite", nil},
|
||||||
|
{1000, "/elsewhere/state/", "/elsewhere/state/sfdupes/db.sqlite", nil},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range cases {
|
||||||
|
t.Setenv(stateHomeEnv, tc.stateHome)
|
||||||
|
xdg.Reload()
|
||||||
|
|
||||||
|
got, err := defaultDatabasePath(tc.euid)
|
||||||
|
if got != tc.want || !errors.Is(err, tc.wantErr) {
|
||||||
|
t.Errorf("defaultDatabasePath(%d) with %s=%q = %q, %v; want %q, %v",
|
||||||
|
tc.euid, stateHomeEnv, tc.stateHome, got, err, tc.want,
|
||||||
|
tc.wantErr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestOpenScanDatabaseCreates(t *testing.T) {
|
func TestOpenScanDatabaseCreates(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
|
|||||||
@@ -121,9 +121,9 @@ func TestRunDwimExcludeLeavesTreeOut(t *testing.T) {
|
|||||||
t.Errorf("group = %q, want it to start %q", lines[1], want)
|
t.Errorf("group = %q, want it to start %q", lines[1], want)
|
||||||
}
|
}
|
||||||
|
|
||||||
skipped := "scan: skipped " + excluded + " (--exclude)\n"
|
leftOut := "scan: left out " + excluded + " (--exclude)\n"
|
||||||
if got := stderr(); !strings.Contains(got, skipped) {
|
if got := stderr(); !strings.Contains(got, leftOut) {
|
||||||
t.Errorf("stderr = %q, want it to contain %q", got, skipped)
|
t.Errorf("stderr = %q, want it to contain %q", got, leftOut)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -72,14 +72,31 @@ type walkRules struct {
|
|||||||
func defaultExcludes(goos string) []string {
|
func defaultExcludes(goos string) []string {
|
||||||
switch goos {
|
switch goos {
|
||||||
case goosLinux:
|
case goosLinux:
|
||||||
return []string{"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"}
|
// The kernel's view of the running system, temporary files, and
|
||||||
|
// caches that programs and package managers fill again on their
|
||||||
|
// own: the system's, root's and those in each home in /home.
|
||||||
|
return []string{
|
||||||
|
"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
|
||||||
|
"/root/.cache", "/root/.npm", "/root/.cargo/registry",
|
||||||
|
"/root/go/pkg/mod",
|
||||||
|
"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
|
||||||
|
"/home/*/go/pkg/mod",
|
||||||
|
}
|
||||||
case goosDarwin:
|
case goosDarwin:
|
||||||
// A walk of / reaches everything in /System/Volumes/Data through
|
// A walk of / reaches everything in /System/Volumes/Data through
|
||||||
// the firmlinks, so walking it as well would record each of
|
// the firmlinks, so walking it as well would record each of
|
||||||
// those files twice.
|
// those files twice. /private/var/folders holds each user's
|
||||||
|
// temporary files and caches. The walk reaches root's home,
|
||||||
|
// /var/root, as /private/var/root.
|
||||||
return []string{
|
return []string{
|
||||||
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
|
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
|
||||||
"/System/Volumes/VM", "/System/Volumes/Data",
|
"/System/Volumes/VM", "/System/Volumes/Data",
|
||||||
|
"/private/var/folders", "/Library/Caches",
|
||||||
|
"/private/var/root/Library/Caches", "/private/var/root/.cache",
|
||||||
|
"/private/var/root/.npm", "/private/var/root/.cargo/registry",
|
||||||
|
"/private/var/root/go/pkg/mod",
|
||||||
|
"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
|
||||||
|
"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
|
||||||
}
|
}
|
||||||
case goosWindows:
|
case goosWindows:
|
||||||
// Relative to the root of the drive an operand lies on. The
|
// Relative to the root of the drive an operand lies on. The
|
||||||
@@ -186,7 +203,7 @@ func (rules walkRules) excludePathLine(path string, fi fs.FileInfo) string {
|
|||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
|
|
||||||
return fmt.Sprintf("scan: skipped %s (--exclude)", path)
|
return fmt.Sprintf("scan: left out %s (--exclude)", path)
|
||||||
}
|
}
|
||||||
|
|
||||||
// nonPersistentFilesystemTypes returns the filesystem types that are
|
// nonPersistentFilesystemTypes returns the filesystem types that are
|
||||||
@@ -216,7 +233,7 @@ func (rules walkRules) nonPersistentLine(path string, fi fs.FileInfo) string {
|
|||||||
return ""
|
return ""
|
||||||
}
|
}
|
||||||
|
|
||||||
return fmt.Sprintf("scan: skipped %s (%s, not a persistent filesystem)",
|
return fmt.Sprintf("scan: left out %s (%s, not a persistent filesystem)",
|
||||||
path, fsType)
|
path, fsType)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -24,10 +24,22 @@ func TestDefaultExcludes(t *testing.T) {
|
|||||||
// on any other system. The Windows list exists in the code only: the
|
// on any other system. The Windows list exists in the code only: the
|
||||||
// binary does not build for Windows, so the README leaves it out.
|
// binary does not build for Windows, so the README leaves it out.
|
||||||
cases := map[string][]string{
|
cases := map[string][]string{
|
||||||
goosLinux: {"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp"},
|
goosLinux: {
|
||||||
|
"/proc", "/sys", "/dev", "/run", "/tmp", "/var/tmp", "/var/cache",
|
||||||
|
"/root/.cache", "/root/.npm", "/root/.cargo/registry",
|
||||||
|
"/root/go/pkg/mod",
|
||||||
|
"/home/*/.cache", "/home/*/.npm", "/home/*/.cargo/registry",
|
||||||
|
"/home/*/go/pkg/mod",
|
||||||
|
},
|
||||||
goosDarwin: {
|
goosDarwin: {
|
||||||
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
|
"/dev", "/private/tmp", "/private/var/tmp", "/private/var/vm",
|
||||||
"/System/Volumes/VM", "/System/Volumes/Data",
|
"/System/Volumes/VM", "/System/Volumes/Data",
|
||||||
|
"/private/var/folders", "/Library/Caches",
|
||||||
|
"/private/var/root/Library/Caches", "/private/var/root/.cache",
|
||||||
|
"/private/var/root/.npm", "/private/var/root/.cargo/registry",
|
||||||
|
"/private/var/root/go/pkg/mod",
|
||||||
|
"/Users/*/Library/Caches", "/Users/*/.cache", "/Users/*/.npm",
|
||||||
|
"/Users/*/.cargo/registry", "/Users/*/go/pkg/mod",
|
||||||
},
|
},
|
||||||
goosWindows: {
|
goosWindows: {
|
||||||
"pagefile.sys", "swapfile.sys", "hiberfil.sys",
|
"pagefile.sys", "swapfile.sys", "hiberfil.sys",
|
||||||
@@ -63,6 +75,29 @@ func TestIsDefaultExclude(t *testing.T) {
|
|||||||
t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
|
t.Errorf("isDefaultExclude(%q) = %v, want %v", path, got, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The Linux list matches the cache directories in /root and in each
|
||||||
|
// home in /home, but nothing below them and no .cache directory
|
||||||
|
// elsewhere.
|
||||||
|
linux := map[string]bool{
|
||||||
|
"/home/u/.cache": true,
|
||||||
|
"/home/u/go/pkg/mod": true,
|
||||||
|
"/root/.cache": true,
|
||||||
|
"/var/cache": true,
|
||||||
|
"/home/u/.cache/pip": false,
|
||||||
|
"/home/u/src/.cache": false,
|
||||||
|
"/srv/home/u/.cache": false,
|
||||||
|
"/home/u/.cargo/bin": false,
|
||||||
|
"/home/u/go/pkg/mod/cache": false,
|
||||||
|
}
|
||||||
|
|
||||||
|
for path, want := range linux {
|
||||||
|
got := isDefaultExclude(path, defaultExcludes(goosLinux))
|
||||||
|
if got != want {
|
||||||
|
t.Errorf("isDefaultExclude(%q, Linux list) = %v, want %v", path,
|
||||||
|
got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestReadMountInfo(t *testing.T) {
|
func TestReadMountInfo(t *testing.T) {
|
||||||
@@ -207,7 +242,7 @@ func TestScanDoesNotEnterDefaultExclude(t *testing.T) {
|
|||||||
t.Errorf("%s recorded: the walk entered %s", unseen, excluded)
|
t.Errorf("%s recorded: the walk entered %s", unseen, excluded)
|
||||||
}
|
}
|
||||||
|
|
||||||
line := "\nscan: skipped " + excluded + " (default exclude)\n"
|
line := "\nscan: left out " + excluded + " (default exclude)\n"
|
||||||
if !strings.Contains(stderr(), line) {
|
if !strings.Contains(stderr(), line) {
|
||||||
t.Errorf("stderr = %q, want %q", stderr(), line)
|
t.Errorf("stderr = %q, want %q", stderr(), line)
|
||||||
}
|
}
|
||||||
@@ -276,7 +311,7 @@ func TestScanNamesNoDefaultExcludeThatIsAnOperand(t *testing.T) {
|
|||||||
|
|
||||||
recordByPath(t, dbRecords(t, db), f)
|
recordByPath(t, dbRecords(t, db), f)
|
||||||
|
|
||||||
if strings.Contains(stderr(), "scan: skipped") {
|
if strings.Contains(stderr(), "scan: left out") {
|
||||||
t.Errorf("stderr = %q, want no line for %s", stderr(), excluded)
|
t.Errorf("stderr = %q, want no line for %s", stderr(), excluded)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -290,15 +325,15 @@ func TestRunScanTempDirOperand(t *testing.T) {
|
|||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
dir := realTempDir(t)
|
dir := realTempDir(t)
|
||||||
if !underAnyRoot(dir, defaultExcludes(runtime.GOOS)) {
|
if !insideDefaultExclude(dir, defaultExcludes(runtime.GOOS)) {
|
||||||
t.Skip(dir + " is not inside a default exclude")
|
t.Skip(dir + " is not inside a default exclude")
|
||||||
}
|
}
|
||||||
|
|
||||||
f := writeFile(t, dir, "sub/f", pattern(1, 10))
|
f := writeFile(t, dir, "sub/f", pattern(1, 10))
|
||||||
|
|
||||||
stderr := scanOK(t, dir)
|
stderr := scanOK(t, dir)
|
||||||
if strings.Contains(stderr, "scan: skipped") {
|
if strings.Contains(stderr, "scan: left out") {
|
||||||
t.Errorf("stderr = %q, want no line saying a path was skipped",
|
t.Errorf("stderr = %q, want no line saying a path was left out",
|
||||||
stderr)
|
stderr)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -342,7 +377,7 @@ func TestRunScanNoDefaultExcludes(t *testing.T) {
|
|||||||
path := testDBPath(t)
|
path := testDBPath(t)
|
||||||
t.Setenv(databaseEnv, path)
|
t.Setenv(databaseEnv, path)
|
||||||
|
|
||||||
line := "scan: skipped /var/tmp (default exclude)\n"
|
line := "scan: left out /var/tmp (default exclude)\n"
|
||||||
|
|
||||||
if stderr := scanOK(t, "/var"); !strings.Contains(stderr, line) {
|
if stderr := scanOK(t, "/var"); !strings.Contains(stderr, line) {
|
||||||
t.Errorf("scan /var: stderr = %q, want %q", stderr, line)
|
t.Errorf("scan /var: stderr = %q, want %q", stderr, line)
|
||||||
@@ -383,7 +418,7 @@ func TestRunScanExcludeNested(t *testing.T) {
|
|||||||
|
|
||||||
stderr := scanOK(t, excludeFlag, excluded, dir)
|
stderr := scanOK(t, excludeFlag, excluded, dir)
|
||||||
|
|
||||||
line := "scan: skipped " + excluded + " (--exclude)\n"
|
line := "scan: left out " + excluded + " (--exclude)\n"
|
||||||
if !strings.Contains(stderr, line) {
|
if !strings.Contains(stderr, line) {
|
||||||
t.Errorf("stderr = %q, want %q", stderr, line)
|
t.Errorf("stderr = %q, want %q", stderr, line)
|
||||||
}
|
}
|
||||||
@@ -445,8 +480,8 @@ func TestRunScanExcludeReachedByAnotherPath(t *testing.T) {
|
|||||||
stderr := scanOK(t, excludeFlag, filepath.Join(link, "cache"),
|
stderr := scanOK(t, excludeFlag, filepath.Join(link, "cache"),
|
||||||
excludeFlag, dotDot, dir)
|
excludeFlag, dotDot, dir)
|
||||||
|
|
||||||
for _, skipped := range []string{filepath.Dir(inDir), file, hardLink} {
|
for _, leftOut := range []string{filepath.Dir(inDir), file, hardLink} {
|
||||||
line := "scan: skipped " + skipped + " (--exclude)\n"
|
line := "scan: left out " + leftOut + " (--exclude)\n"
|
||||||
if !strings.Contains(stderr, line) {
|
if !strings.Contains(stderr, line) {
|
||||||
t.Errorf("stderr = %q, want %q", stderr, line)
|
t.Errorf("stderr = %q, want %q", stderr, line)
|
||||||
}
|
}
|
||||||
@@ -580,7 +615,7 @@ func TestRunScanSkipsNonPersistentOperand(t *testing.T) {
|
|||||||
|
|
||||||
t.Setenv(databaseEnv, testDBPath(t))
|
t.Setenv(databaseEnv, testDBPath(t))
|
||||||
|
|
||||||
line := "scan: skipped " + shm + " (" + fsType +
|
line := "scan: left out " + shm + " (" + fsType +
|
||||||
", not a persistent filesystem)\n"
|
", not a persistent filesystem)\n"
|
||||||
summary := "scan: 0 files seen"
|
summary := "scan: 0 files seen"
|
||||||
|
|
||||||
@@ -634,7 +669,7 @@ func TestScanKeepsRecordsOnNonPersistent(t *testing.T) {
|
|||||||
t.Errorf("%s recorded", unseen)
|
t.Errorf("%s recorded", unseen)
|
||||||
}
|
}
|
||||||
|
|
||||||
line := "scan: skipped " + shm + " (" + fsType +
|
line := "scan: left out " + shm + " (" + fsType +
|
||||||
", not a persistent filesystem)\n"
|
", not a persistent filesystem)\n"
|
||||||
if !strings.Contains(stderr(), line) {
|
if !strings.Contains(stderr(), line) {
|
||||||
t.Errorf("stderr = %q, want %q", stderr(), line)
|
t.Errorf("stderr = %q, want %q", stderr(), line)
|
||||||
@@ -667,7 +702,7 @@ func TestWalkLeavesOutNonPersistent(t *testing.T) {
|
|||||||
|
|
||||||
leftOut = append(leftOut, ev.leftOut)
|
leftOut = append(leftOut, ev.leftOut)
|
||||||
|
|
||||||
want := "scan: skipped " + ev.leftOut + " (" + fsType +
|
want := "scan: left out " + ev.leftOut + " (" + fsType +
|
||||||
", not a persistent filesystem)"
|
", not a persistent filesystem)"
|
||||||
if ev.warn != want {
|
if ev.warn != want {
|
||||||
t.Errorf("line = %q, want %q", ev.warn, want)
|
t.Errorf("line = %q, want %q", ev.warn, want)
|
||||||
@@ -711,6 +746,20 @@ func shmTempDir(t *testing.T) (string, string, walkRules) {
|
|||||||
return dir, fsType, walkRules{nonPersistent: nonPersistent}
|
return dir, fsType, walkRules{nonPersistent: nonPersistent}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// insideDefaultExclude reports whether path, which is absolute and
|
||||||
|
// clean, or a directory above it is one of excludes.
|
||||||
|
func insideDefaultExclude(path string, excludes []string) bool {
|
||||||
|
for dir := path; ; dir = filepath.Dir(dir) {
|
||||||
|
if isDefaultExclude(dir, excludes) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
if dir == "/" {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// assertRecorded checks that the database at dbPath holds a record for
|
// assertRecorded checks that the database at dbPath holds a record for
|
||||||
// path when want is set, and none when it is not.
|
// path when want is set, and none when it is not.
|
||||||
func assertRecorded(t *testing.T, dbPath, path string, want bool) {
|
func assertRecorded(t *testing.T, dbPath, path string, want bool) {
|
||||||
|
|||||||
@@ -9,8 +9,10 @@ import (
|
|||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"time"
|
"time"
|
||||||
|
"unicode"
|
||||||
"unicode/utf8"
|
"unicode/utf8"
|
||||||
|
|
||||||
|
"github.com/rivo/uniseg"
|
||||||
"github.com/schollz/progressbar/v3"
|
"github.com/schollz/progressbar/v3"
|
||||||
"golang.org/x/term"
|
"golang.org/x/term"
|
||||||
)
|
)
|
||||||
@@ -116,7 +118,8 @@ type drawnCount struct {
|
|||||||
// terminal's width. A phase with an exact total ends the line with its
|
// terminal's width. A phase with an exact total ends the line with its
|
||||||
// percent and bar, at the terminal's right edge, when the terminal has
|
// percent and bar, at the terminal's right edge, when the terminal has
|
||||||
// room for them. It is drawn again every redrawInterval, so its elapsed
|
// room for them. It is drawn again every redrawInterval, so its elapsed
|
||||||
// time keeps moving while the phase waits for its next item. When
|
// time keeps moving while the phase waits for its next item. Each draw
|
||||||
|
// of the walk shows the path it reached last on the line below. When
|
||||||
// stderr is not a TTY it emits no ANSI redraws: it prints line as the
|
// stderr is not a TTY it emits no ANSI redraws: it prints line as the
|
||||||
// phase starts, every plainInterval while it runs, and as it ends.
|
// phase starts, every plainInterval while it runs, and as it ends.
|
||||||
//
|
//
|
||||||
@@ -141,6 +144,7 @@ type progress struct {
|
|||||||
estimate bool
|
estimate bool
|
||||||
waitingFor string
|
waitingFor string
|
||||||
stalledOn string
|
stalledOn string
|
||||||
|
path string // the path the walk reached last; "" for none
|
||||||
count int64
|
count int64
|
||||||
start time.Time // when the phase started, for the elapsed time
|
start time.Time // when the phase started, for the elapsed time
|
||||||
rateStart time.Time // when the count started, for the time remaining
|
rateStart time.Time // when the count started, for the time remaining
|
||||||
@@ -239,6 +243,82 @@ func (p *progress) draw() {
|
|||||||
// less the terminal's last column, which spinnerMargin keeps clear. A
|
// less the terminal's last column, which spinnerMargin keeps clear. A
|
||||||
// bar's cells take one column each but more than one byte.
|
// bar's cells take one column each but more than one byte.
|
||||||
p.widest = max(p.widest, utf8.RuneCountInString(text)+spinnerMargin-1)
|
p.widest = max(p.widest, utf8.RuneCountInString(text)+spinnerMargin-1)
|
||||||
|
|
||||||
|
if p.path != "" {
|
||||||
|
// The library draws the path on the line below and returns to
|
||||||
|
// the phase's line. The path is escaped as warnings are, and its
|
||||||
|
// other control characters are written out, so that the terminal
|
||||||
|
// shows them instead of acting on them. It stays clear of the
|
||||||
|
// last column, so that no terminal wraps it.
|
||||||
|
shown := escapeControlChars(escapePath(p.path))
|
||||||
|
_ = p.bar.AddDetail(cutToColumns(shown, width-1))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// escapeControlChars returns s with each control character written as
|
||||||
|
// \x and its code in two hex digits, as \x1b for the escape character.
|
||||||
|
// The control characters are U+0000 to U+001F, U+007F and U+0080 to
|
||||||
|
// U+009F, as unicode.IsControl reports them. A byte from 0x80 to 0x9F
|
||||||
|
// that is not part of a valid UTF-8 character is written the same way,
|
||||||
|
// as \x9b, since a terminal not in UTF-8 mode acts on it. Every other
|
||||||
|
// byte is kept as it is.
|
||||||
|
func escapeControlChars(s string) string {
|
||||||
|
var b strings.Builder
|
||||||
|
|
||||||
|
for rest := s; rest != ""; {
|
||||||
|
char, size := utf8.DecodeRuneInString(rest)
|
||||||
|
if char == utf8.RuneError && size == 1 {
|
||||||
|
// Not part of a valid UTF-8 character. A terminal not in
|
||||||
|
// UTF-8 mode reads the byte as the character with its code,
|
||||||
|
// so that 0x9b is a control character there and 0xe9 is not.
|
||||||
|
char = rune(rest[0])
|
||||||
|
}
|
||||||
|
|
||||||
|
if unicode.IsControl(char) {
|
||||||
|
fmt.Fprintf(&b, `\x%02x`, char)
|
||||||
|
} else {
|
||||||
|
b.WriteString(rest[:size])
|
||||||
|
}
|
||||||
|
|
||||||
|
rest = rest[size:]
|
||||||
|
}
|
||||||
|
|
||||||
|
return b.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
// cutToColumns returns s cut to at most columns terminal columns, at a
|
||||||
|
// character boundary. A wide character, as in Chinese, takes two
|
||||||
|
// columns.
|
||||||
|
func cutToColumns(s string, columns int) string {
|
||||||
|
used, end, state := 0, 0, -1
|
||||||
|
|
||||||
|
for rest := s; rest != ""; {
|
||||||
|
char, next, width, nextState := uniseg.FirstGraphemeClusterInString(
|
||||||
|
rest, state)
|
||||||
|
if used+width > columns {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
used += width
|
||||||
|
end += len(char)
|
||||||
|
rest, state = next, nextState
|
||||||
|
}
|
||||||
|
|
||||||
|
return s[:end]
|
||||||
|
}
|
||||||
|
|
||||||
|
// clear erases the display from the terminal: its line, and the path on
|
||||||
|
// the line below when one is shown. p.mu must be held.
|
||||||
|
func (p *progress) clear() {
|
||||||
|
if p.bar == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
if p.path != "" {
|
||||||
|
_ = p.bar.AddDetail("")
|
||||||
|
}
|
||||||
|
|
||||||
|
_ = p.bar.Clear()
|
||||||
}
|
}
|
||||||
|
|
||||||
// newBar has the library make what draw draws: its spinner, followed by
|
// newBar has the library make what draw draws: its spinner, followed by
|
||||||
@@ -263,6 +343,8 @@ func (p *progress) newBar(width int) {
|
|||||||
// The text carries the elapsed time and the time remaining.
|
// The text carries the elapsed time and the time remaining.
|
||||||
progressbar.OptionSetElapsedTime(false),
|
progressbar.OptionSetElapsedTime(false),
|
||||||
progressbar.OptionSetPredictTime(false),
|
progressbar.OptionSetPredictTime(false),
|
||||||
|
// The line below the phase's line, for the walk's path.
|
||||||
|
progressbar.OptionSetMaxDetailRow(1),
|
||||||
)
|
)
|
||||||
p.widest = 0
|
p.widest = 0
|
||||||
}
|
}
|
||||||
@@ -314,6 +396,19 @@ func (p *progress) increment() {
|
|||||||
p.lastItem = now
|
p.lastItem = now
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// setPath records the path the walk reached last, which a display on a
|
||||||
|
// terminal shows at its next draw.
|
||||||
|
func (p *progress) setPath(path string) {
|
||||||
|
if p == nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
p.mu.Lock()
|
||||||
|
defer p.mu.Unlock()
|
||||||
|
|
||||||
|
p.path = path
|
||||||
|
}
|
||||||
|
|
||||||
// warnf prints a one-line warning to stderr without corrupting the
|
// warnf prints a one-line warning to stderr without corrupting the
|
||||||
// display. The whole message is escaped like a report's path columns,
|
// display. The whole message is escaped like a report's path columns,
|
||||||
// so a path holding a newline cannot split the warning.
|
// so a path holding a newline cannot split the warning.
|
||||||
@@ -327,11 +422,9 @@ func (p *progress) warnf(format string, args ...any) {
|
|||||||
p.mu.Lock()
|
p.mu.Lock()
|
||||||
defer p.mu.Unlock()
|
defer p.mu.Unlock()
|
||||||
|
|
||||||
if p.bar != nil {
|
// The warning takes the display's line; the next draw shows the
|
||||||
// The warning takes the display's line; the next draw shows
|
// display again, on the line below.
|
||||||
// the display again, on the line below.
|
p.clear()
|
||||||
_ = p.bar.Clear()
|
|
||||||
}
|
|
||||||
|
|
||||||
fmt.Fprintln(os.Stderr, msg)
|
fmt.Fprintln(os.Stderr, msg)
|
||||||
}
|
}
|
||||||
@@ -392,9 +485,7 @@ func (w *progressWriter) Flush() error {
|
|||||||
w.p.mu.Lock()
|
w.p.mu.Lock()
|
||||||
defer w.p.mu.Unlock()
|
defer w.p.mu.Unlock()
|
||||||
|
|
||||||
if w.p.bar != nil {
|
w.p.clear()
|
||||||
_ = w.p.bar.Clear()
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
_, err := w.out.Write(w.buf)
|
_, err := w.out.Write(w.buf)
|
||||||
@@ -414,6 +505,13 @@ func (p *progress) finish() {
|
|||||||
close(p.stop)
|
close(p.stop)
|
||||||
<-p.done
|
<-p.done
|
||||||
|
|
||||||
|
// The walk's path is only for someone watching the walk. Its line is
|
||||||
|
// erased, and the next phase is drawn on it.
|
||||||
|
p.mu.Lock()
|
||||||
|
p.clear()
|
||||||
|
p.path = ""
|
||||||
|
p.mu.Unlock()
|
||||||
|
|
||||||
p.draw()
|
p.draw()
|
||||||
|
|
||||||
if p.width != nil {
|
if p.width != nil {
|
||||||
|
|||||||
@@ -6,6 +6,7 @@ import (
|
|||||||
"math"
|
"math"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
"slices"
|
"slices"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
@@ -1138,3 +1139,164 @@ func TestWarningAfterLastItemKeepsBar(t *testing.T) {
|
|||||||
t.Errorf("terminal shows %q, want a count of 2 of 2", shown)
|
t.Errorf("terminal shows %q, want a count of 2 of 2", shown)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestCutToColumns(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cases := []struct {
|
||||||
|
s string
|
||||||
|
columns int
|
||||||
|
want string
|
||||||
|
}{
|
||||||
|
{"/srv/a", 10, "/srv/a"},
|
||||||
|
{"/srv/photos", 8, "/srv/pho"},
|
||||||
|
// Each of these two fullwidth letters takes two columns.
|
||||||
|
{"/AB/a", 4, "/A"},
|
||||||
|
{"/AB/a", 5, "/AB"},
|
||||||
|
// An accent that combines with the letter before stays with it.
|
||||||
|
{"/café/a", 5, "/café"},
|
||||||
|
{"/srv", 0, ""},
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, tc := range cases {
|
||||||
|
if got := cutToColumns(tc.s, tc.columns); got != tc.want {
|
||||||
|
t.Errorf("cutToColumns(%q, %d) = %q, want %q",
|
||||||
|
tc.s, tc.columns, got, tc.want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestEscapeControlChars(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
cases := map[string]string{
|
||||||
|
"/srv/music": "/srv/music",
|
||||||
|
"/a\x1b[2Jb": `/a\x1b[2Jb`,
|
||||||
|
"/a\x00\a\x7fb": `/a\x00\x07\x7fb`,
|
||||||
|
// U+009B takes two bytes in UTF-8.
|
||||||
|
"/a\u009b2Jb": `/a\x9b2Jb`,
|
||||||
|
// Byte 0x9b alone is not valid UTF-8. A terminal not in UTF-8
|
||||||
|
// mode acts on it, as on every byte from 0x80 to 0x9f.
|
||||||
|
"/a\x9b2Jb": `/a\x9b2Jb`,
|
||||||
|
"/a\x80\x9f\xa0b": `/a\x80\x9f` + "\xa0b",
|
||||||
|
// In ě, which is c4 9b in UTF-8, byte 0x9b is part of a valid
|
||||||
|
// character.
|
||||||
|
"/aě2Jb": "/aě2Jb",
|
||||||
|
"/café/AB": "/café/AB",
|
||||||
|
"/latin1-caf\xe9": "/latin1-caf\xe9",
|
||||||
|
}
|
||||||
|
for in, want := range cases {
|
||||||
|
if got := escapeControlChars(in); got != want {
|
||||||
|
t.Errorf("escapeControlChars(%q) = %q, want %q", in, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// pathLines returns the text of each line the library wrote below a
|
||||||
|
// display's line. It writes one as an erase to the end of the line, a
|
||||||
|
// carriage return, the text and a newline.
|
||||||
|
func pathLines(out string) []string {
|
||||||
|
matches := regexp.MustCompile("\x1b\\[K\r([^\n]*)\n").
|
||||||
|
FindAllStringSubmatch(out, -1)
|
||||||
|
lines := make([]string, 0, len(matches))
|
||||||
|
|
||||||
|
for _, m := range matches {
|
||||||
|
lines = append(lines, m[1])
|
||||||
|
}
|
||||||
|
|
||||||
|
return lines
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTerminalWalkShowsPathBelow checks that the walk's display on a
|
||||||
|
// terminal 30 columns wide shows the path it reached last on the line
|
||||||
|
// below, cut to 29 columns. That line is erased before a warning is
|
||||||
|
// printed, and again when the walk ends, so the next phase is drawn on
|
||||||
|
// an empty line.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||||
|
func TestTerminalWalkShowsPathBelow(t *testing.T) {
|
||||||
|
stderr := captureStderr(t)
|
||||||
|
p := startProgress(progressParams{
|
||||||
|
label: scanPhase("walk"), unit: scanUnit, total: -1,
|
||||||
|
}, func() int { return 30 }, testRedraw, time.Now)
|
||||||
|
|
||||||
|
const shown = "/srv/photos/2026/holiday/beac"
|
||||||
|
|
||||||
|
p.setPath("/srv/photos/2026/holiday/beach.jpg")
|
||||||
|
p.increment()
|
||||||
|
awaitOutput(t, stderr, shown, 1)
|
||||||
|
|
||||||
|
p.warnf("warning")
|
||||||
|
|
||||||
|
before, _, _ := strings.Cut(stderr(), "warning")
|
||||||
|
if lines := pathLines(before); lines[len(lines)-1] != "" {
|
||||||
|
t.Errorf("line below shows %q as the warning is printed, want it "+
|
||||||
|
"empty", lines[len(lines)-1])
|
||||||
|
}
|
||||||
|
|
||||||
|
// Drawn again below the warning.
|
||||||
|
awaitOutput(t, stderr, shown, strings.Count(before, shown)+1)
|
||||||
|
p.finish()
|
||||||
|
|
||||||
|
lines := pathLines(stderr())
|
||||||
|
if !slices.Contains(lines, shown) {
|
||||||
|
t.Errorf("lines below the display = %q, want %q among them",
|
||||||
|
lines, shown)
|
||||||
|
}
|
||||||
|
|
||||||
|
if last := lines[len(lines)-1]; last != "" {
|
||||||
|
t.Errorf("line below the display shows %q after the walk, want it "+
|
||||||
|
"empty", last)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestTerminalWalkEscapesControlChars checks that a walked path holding
|
||||||
|
// escape characters, which would clear the screen and set the window
|
||||||
|
// title, puts none on the line below the walk's display. The line shows
|
||||||
|
// each control character written out, as \x1b.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||||
|
func TestTerminalWalkEscapesControlChars(t *testing.T) {
|
||||||
|
stderr := captureStderr(t)
|
||||||
|
p := startProgress(progressParams{
|
||||||
|
label: scanPhase("walk"), unit: scanUnit, total: -1,
|
||||||
|
}, func() int { return 80 }, testRedraw, time.Now)
|
||||||
|
|
||||||
|
p.setPath("/srv/\x1b[2J\x1b]0;title\a/beach.jpg")
|
||||||
|
p.increment()
|
||||||
|
awaitOutput(t, stderr, "beach.jpg", 1)
|
||||||
|
p.finish()
|
||||||
|
|
||||||
|
lines := pathLines(stderr())
|
||||||
|
for _, line := range lines {
|
||||||
|
if strings.Contains(line, "\x1b") {
|
||||||
|
t.Errorf("line below the display = %q, want no escape "+
|
||||||
|
"character", line)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const shown = `/srv/\x1b[2J\x1b]0;title\x07/beach.jpg`
|
||||||
|
if !slices.Contains(lines, shown) {
|
||||||
|
t.Errorf("lines below the display = %q, want %q among them",
|
||||||
|
lines, shown)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestPlainWalkShowsNoPath checks that the walk's display prints no path
|
||||||
|
// when stderr is not a terminal.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||||
|
func TestPlainWalkShowsNoPath(t *testing.T) {
|
||||||
|
stderr := captureStderr(t)
|
||||||
|
p := startProgress(progressParams{
|
||||||
|
label: scanPhase("walk"), unit: scanUnit, total: -1,
|
||||||
|
}, nil, testRedraw, time.Now)
|
||||||
|
|
||||||
|
p.setPath("/srv/a")
|
||||||
|
p.increment()
|
||||||
|
p.finish()
|
||||||
|
|
||||||
|
if strings.Contains(stderr(), "/srv/a") {
|
||||||
|
t.Errorf("stderr = %q, want no path", stderr())
|
||||||
|
}
|
||||||
|
}
|
||||||
+29
-12
@@ -562,6 +562,7 @@ func (s *scanState) walkPhase(events <-chan walkEvent, roots []string) {
|
|||||||
|
|
||||||
s.st.walked++
|
s.st.walked++
|
||||||
|
|
||||||
|
prog.setPath(ev.rec.path)
|
||||||
prog.increment()
|
prog.increment()
|
||||||
|
|
||||||
old, ok := s.existing[ev.rec.path]
|
old, ok := s.existing[ev.rec.path]
|
||||||
@@ -663,8 +664,8 @@ func isDatabaseFile(f fileRec, rules walkRules) bool {
|
|||||||
// inode run, committing completed records to the database in batches
|
// inode run, committing completed records to the database in batches
|
||||||
// as results arrive, so a long scan persists its progress as it goes
|
// as results arrive, so a long scan persists its progress as it goes
|
||||||
// (an interrupted scan resumes cheaply: the next run skips everything
|
// (an interrupted scan resumes cheaply: the next run skips everything
|
||||||
// already recorded). A run that fails to hash is warned about and
|
// already recorded). Each path of a run that fails to hash is warned
|
||||||
// counted as skipped; stale records for its paths, if any, are deleted
|
// about and counted as skipped; stale records for them, if any, are deleted
|
||||||
// by the update phase. A run whose file no longer exists gets no record
|
// by the update phase. A run whose file no longer exists gets no record
|
||||||
// and no warning, and is not counted as skipped. The total counts actual
|
// and no warning, and is not counted as skipped. The total counts actual
|
||||||
// reads, so the display's time remaining is meaningful.
|
// reads, so the display's time remaining is meaningful.
|
||||||
@@ -705,9 +706,12 @@ func (s *scanState) hashPhase(ctx context.Context, workers int) error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
if r.err != nil {
|
if r.err != nil {
|
||||||
s.st.skipped += len(r.run)
|
// Every path in the run is counted, so every path gets a line.
|
||||||
|
for _, rec := range r.run {
|
||||||
|
s.st.skipped++
|
||||||
|
|
||||||
prog.warnf("hash %s: %v", r.run[0].path, r.err)
|
prog.warnf("%s", skippedLine(rec.path, r.err))
|
||||||
|
}
|
||||||
|
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
@@ -914,8 +918,7 @@ func operandWarning(root string, fi fs.FileInfo) string {
|
|||||||
// The walk never enters a .zfs directory, so it starts from nothing
|
// The walk never enters a .zfs directory, so it starts from nothing
|
||||||
// inside one either.
|
// inside one either.
|
||||||
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") {
|
if slices.Contains(strings.Split(filepath.Dir(root), "/"), ".zfs") {
|
||||||
return fmt.Sprintf("walk %s: skipping operand inside a .zfs directory",
|
return fmt.Sprintf("scan: skipped %s (inside a .zfs directory)", root)
|
||||||
root)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
var kind string
|
var kind string
|
||||||
@@ -941,7 +944,21 @@ func operandWarning(root string, fi fs.FileInfo) string {
|
|||||||
kind = "non-regular file"
|
kind = "non-regular file"
|
||||||
}
|
}
|
||||||
|
|
||||||
return fmt.Sprintf("walk %s: skipping %s operand", root, kind)
|
return fmt.Sprintf("scan: skipped %s (a %s)", root, kind)
|
||||||
|
}
|
||||||
|
|
||||||
|
// skippedLine returns the warning for a path the scan skips because of
|
||||||
|
// err. The reason is the error without its path, which the line already
|
||||||
|
// names.
|
||||||
|
func skippedLine(path string, err error) string {
|
||||||
|
reason := err.Error()
|
||||||
|
|
||||||
|
var pathErr *fs.PathError
|
||||||
|
if errors.As(err, &pathErr) {
|
||||||
|
reason = pathErr.Err.Error()
|
||||||
|
}
|
||||||
|
|
||||||
|
return fmt.Sprintf("scan: skipped %s (%s)", path, reason)
|
||||||
}
|
}
|
||||||
|
|
||||||
// seedRoot turns one PATH operand into the walk's starting state: a
|
// seedRoot turns one PATH operand into the walk's starting state: a
|
||||||
@@ -964,7 +981,7 @@ func seedRoot(ctx context.Context, root string, rules walkRules,
|
|||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
sendEvent(ctx, events, walkEvent{
|
sendEvent(ctx, events, walkEvent{
|
||||||
warn: fmt.Sprintf("walk %s: %v", root, err),
|
warn: skippedLine(root, err),
|
||||||
fail: true,
|
fail: true,
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -1093,7 +1110,7 @@ func walkOneDir(ctx context.Context, job dirJob, oneFS bool,
|
|||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
sendEvent(ctx, events, walkEvent{
|
sendEvent(ctx, events, walkEvent{
|
||||||
warn: fmt.Sprintf("walk %s: %v", job.path, err),
|
warn: skippedLine(job.path, err),
|
||||||
fail: true,
|
fail: true,
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -1152,7 +1169,7 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
|
|||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
sendEvent(ctx, events, walkEvent{
|
sendEvent(ctx, events, walkEvent{
|
||||||
warn: fmt.Sprintf("stat %s: %v", p, err),
|
warn: skippedLine(p, err),
|
||||||
fail: true,
|
fail: true,
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -1213,7 +1230,7 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
|||||||
|
|
||||||
if err != nil {
|
if err != nil {
|
||||||
sendEvent(ctx, events, walkEvent{
|
sendEvent(ctx, events, walkEvent{
|
||||||
warn: fmt.Sprintf("walk %s: %v", p, err),
|
warn: skippedLine(p, err),
|
||||||
fail: true,
|
fail: true,
|
||||||
})
|
})
|
||||||
|
|
||||||
@@ -1234,7 +1251,7 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
|
|||||||
if isDefaultExclude(p, rules.defaultExcludes) {
|
if isDefaultExclude(p, rules.defaultExcludes) {
|
||||||
sendEvent(ctx, events, walkEvent{
|
sendEvent(ctx, events, walkEvent{
|
||||||
leftOut: p,
|
leftOut: p,
|
||||||
warn: fmt.Sprintf("scan: skipped %s (default exclude)", p),
|
warn: fmt.Sprintf("scan: left out %s (default exclude)", p),
|
||||||
})
|
})
|
||||||
|
|
||||||
return dirJob{}, false
|
return dirJob{}, false
|
||||||
|
|||||||
@@ -752,8 +752,9 @@ func TestWalkPassesOverRemovedPaths(t *testing.T) {
|
|||||||
|
|
||||||
// TestScanWarnsAboutUnreadablePaths checks that a permission error is
|
// TestScanWarnsAboutUnreadablePaths checks that a permission error is
|
||||||
// still warned about and counted as skipped, unlike a removed path. The
|
// still warned about and counted as skipped, unlike a removed path. The
|
||||||
// walk reaches an unreadable directory, and a file in a directory that
|
// walk reaches an unreadable directory. It also reaches a file and a
|
||||||
// can be read but not searched. Another such file is an operand.
|
// subdirectory in a directory that can be read but not searched. Another
|
||||||
|
// such file is an operand.
|
||||||
//
|
//
|
||||||
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||||
func TestScanWarnsAboutUnreadablePaths(t *testing.T) {
|
func TestScanWarnsAboutUnreadablePaths(t *testing.T) {
|
||||||
@@ -765,6 +766,7 @@ func TestScanWarnsAboutUnreadablePaths(t *testing.T) {
|
|||||||
dir := t.TempDir()
|
dir := t.TempDir()
|
||||||
tree := filepath.Join(dir, "tree")
|
tree := filepath.Join(dir, "tree")
|
||||||
walked := writeFile(t, tree, "unsearchable/f", []byte("x"))
|
walked := writeFile(t, tree, "unsearchable/f", []byte("x"))
|
||||||
|
subdir := filepath.Join(filepath.Dir(walked), "d")
|
||||||
operand := writeFile(t, dir, "unsearchable/f", []byte("y"))
|
operand := writeFile(t, dir, "unsearchable/f", []byte("y"))
|
||||||
unreadable := filepath.Join(tree, "unreadable")
|
unreadable := filepath.Join(tree, "unreadable")
|
||||||
|
|
||||||
@@ -773,6 +775,11 @@ func TestScanWarnsAboutUnreadablePaths(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
err = os.Mkdir(subdir, 0o750)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
modes := map[string]fs.FileMode{
|
modes := map[string]fs.FileMode{
|
||||||
unreadable: 0,
|
unreadable: 0,
|
||||||
filepath.Dir(walked): 0o400,
|
filepath.Dir(walked): 0o400,
|
||||||
@@ -793,15 +800,16 @@ func TestScanWarnsAboutUnreadablePaths(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
st := syncTree(t, openTestDB(t), tree, operand)
|
st := syncTree(t, openTestDB(t), tree, operand)
|
||||||
if st != (scanStats{skipped: 3}) {
|
if st != (scanStats{skipped: 4}) {
|
||||||
t.Errorf("stats = %+v, want 3 skipped and nothing else", st)
|
t.Errorf("stats = %+v, want 4 skipped and nothing else", st)
|
||||||
}
|
}
|
||||||
|
|
||||||
got := stderr()
|
got := stderr()
|
||||||
for _, warning := range []string{
|
for _, warning := range []string{
|
||||||
"walk " + unreadable + ": open " + unreadable + ": permission denied\n",
|
"scan: skipped " + unreadable + " (permission denied)\n",
|
||||||
"stat " + walked + ": lstat " + walked + ": permission denied\n",
|
"scan: skipped " + walked + " (permission denied)\n",
|
||||||
"walk " + operand + ": lstat " + operand + ": permission denied\n",
|
"scan: skipped " + subdir + " (permission denied)\n",
|
||||||
|
"scan: skipped " + operand + " (permission denied)\n",
|
||||||
} {
|
} {
|
||||||
if !strings.Contains(got, warning) {
|
if !strings.Contains(got, warning) {
|
||||||
t.Errorf("stderr = %q, want %q", got, warning)
|
t.Errorf("stderr = %q, want %q", got, warning)
|
||||||
@@ -1941,14 +1949,19 @@ func TestScanHardlinksReadOnce(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestScanHardlinkRunFailsTogether makes a file with two hard links
|
||||||
|
// unreadable. Both paths are counted as skipped, and each gets its own
|
||||||
|
// warning.
|
||||||
|
//
|
||||||
|
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
|
||||||
func TestScanHardlinkRunFailsTogether(t *testing.T) {
|
func TestScanHardlinkRunFailsTogether(t *testing.T) {
|
||||||
t.Parallel()
|
stderr := captureStderr(t)
|
||||||
|
|
||||||
dir := t.TempDir()
|
dir := t.TempDir()
|
||||||
db := openTestDB(t)
|
db := openTestDB(t)
|
||||||
a := writeFile(t, dir, "a.bin", pattern(1, 300))
|
a := writeFile(t, dir, "a.bin", pattern(1, 300))
|
||||||
|
b := filepath.Join(dir, "b.bin")
|
||||||
|
|
||||||
err := os.Link(a, filepath.Join(dir, "b.bin"))
|
err := os.Link(a, b)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -1964,6 +1977,15 @@ func TestScanHardlinkRunFailsTogether(t *testing.T) {
|
|||||||
if st.skipped != 2 || st.added != 0 {
|
if st.skipped != 2 || st.added != 0 {
|
||||||
t.Fatalf("stats = %+v, want both hardlink paths skipped", st)
|
t.Fatalf("stats = %+v, want both hardlink paths skipped", st)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
got := stderr()
|
||||||
|
|
||||||
|
for _, p := range []string{a, b} {
|
||||||
|
warning := "scan: skipped " + p + " (permission denied)\n"
|
||||||
|
if !strings.Contains(got, warning) {
|
||||||
|
t.Errorf("stderr = %q, want %q", got, warning)
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// TestHashPhaseSkipsFileReplacedByFIFO replaces a walked regular file
|
// TestHashPhaseSkipsFileReplacedByFIFO replaces a walked regular file
|
||||||
@@ -2001,7 +2023,7 @@ func TestHashPhaseSkipsFileReplacedByFIFO(t *testing.T) {
|
|||||||
s.st.skipped, len(s.batch))
|
s.st.skipped, len(s.batch))
|
||||||
}
|
}
|
||||||
|
|
||||||
warning := "hash " + p + ": not a regular file\n"
|
warning := "scan: skipped " + p + " (not a regular file)\n"
|
||||||
if got := stderr(); strings.Count(got, warning) != 1 {
|
if got := stderr(); strings.Count(got, warning) != 1 {
|
||||||
t.Errorf("stderr = %q, want %q once", got, warning)
|
t.Errorf("stderr = %q, want %q once", got, warning)
|
||||||
}
|
}
|
||||||
|
|||||||
+42
-12
@@ -108,7 +108,7 @@ func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
|
|||||||
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
|
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
|
||||||
stalledOn: "out",
|
stalledOn: "out",
|
||||||
})
|
})
|
||||||
dupeTrees, reclaimable, err := writeTreeGroups(stdout, dupes, prog)
|
dupeTrees, err := writeTreeGroups(stdout, dupes, prog)
|
||||||
|
|
||||||
prog.finish()
|
prog.finish()
|
||||||
|
|
||||||
@@ -119,7 +119,7 @@ func runTrees(ctx context.Context, args []string, stdout io.Writer) error {
|
|||||||
fmt.Fprintf(os.Stderr,
|
fmt.Fprintf(os.Stderr,
|
||||||
"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
|
"trees: %d records read, %d duplicate tree groups, %d dupe trees, "+
|
||||||
"%s reclaimable\n",
|
"%s reclaimable\n",
|
||||||
records, len(dupes), dupeTrees, humanBytes(reclaimable))
|
records, len(dupes), dupeTrees, humanBytes(reclaimableBytes(dupes)))
|
||||||
|
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
@@ -177,22 +177,19 @@ func readTree(ctx context.Context, db *sql.DB, operands []string,
|
|||||||
// a row for every member of each group but its first, counting each
|
// a row for every member of each group but its first, counting each
|
||||||
// group on prog once its rows are written. The disk_size column is the
|
// group on prog once its rows are written. The disk_size column is the
|
||||||
// dupe's own. stdout may be the terminal prog is drawn on, so the lines
|
// dupe's own. stdout may be the terminal prog is drawn on, so the lines
|
||||||
// go through prog's writer. It returns the number of rows (dupe trees)
|
// go through prog's writer. It returns the number of rows (dupe trees).
|
||||||
// and the bytes they could reclaim.
|
|
||||||
func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
||||||
prog *progress,
|
prog *progress,
|
||||||
) (int, int64, error) {
|
) (int, error) {
|
||||||
out := prog.writer(stdout)
|
out := prog.writer(stdout)
|
||||||
|
|
||||||
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize\tdisk_size")
|
_, err := fmt.Fprintln(out, "first\tdupe\tfiles\tsize\tdisk_size")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
return 0, fmt.Errorf("write stdout: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
dupeTrees := 0
|
dupeTrees := 0
|
||||||
|
|
||||||
var reclaimable int64
|
|
||||||
|
|
||||||
for _, g := range dupes {
|
for _, g := range dupes {
|
||||||
first := g[0]
|
first := g[0]
|
||||||
for _, n := range g[1:] {
|
for _, n := range g[1:] {
|
||||||
@@ -200,11 +197,10 @@ func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
|||||||
escapePath(first.path), escapePath(n.path),
|
escapePath(first.path), escapePath(n.path),
|
||||||
first.fileCount, first.totalSize, n.diskSize)
|
first.fileCount, first.totalSize, n.diskSize)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
return 0, fmt.Errorf("write stdout: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
dupeTrees++
|
dupeTrees++
|
||||||
reclaimable += first.totalSize
|
|
||||||
}
|
}
|
||||||
|
|
||||||
prog.increment()
|
prog.increment()
|
||||||
@@ -212,10 +208,44 @@ func writeTreeGroups(stdout io.Writer, dupes [][]*treeNode,
|
|||||||
|
|
||||||
err = out.Flush()
|
err = out.Flush()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return 0, 0, fmt.Errorf("write stdout: %w", err)
|
return 0, fmt.Errorf("write stdout: %w", err)
|
||||||
}
|
}
|
||||||
|
|
||||||
return dupeTrees, reclaimable, nil
|
return dupeTrees, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// reclaimableBytes returns the sum of the size column over the rows
|
||||||
|
// that writeTreeGroups writes for dupes, leaving out a row whose dupe
|
||||||
|
// lies inside another row's dupe.
|
||||||
|
func reclaimableBytes(dupes [][]*treeNode) int64 {
|
||||||
|
isDupe := make(map[*treeNode]bool)
|
||||||
|
|
||||||
|
for _, g := range dupes {
|
||||||
|
for _, n := range g[1:] {
|
||||||
|
isDupe[n] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var total int64
|
||||||
|
|
||||||
|
for n := range isDupe {
|
||||||
|
if !insideDupe(n, isDupe) {
|
||||||
|
total += n.totalSize
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return total
|
||||||
|
}
|
||||||
|
|
||||||
|
// insideDupe reports whether a directory above n is in isDupe.
|
||||||
|
func insideDupe(n *treeNode, isDupe map[*treeNode]bool) bool {
|
||||||
|
for p := n.parent; p != nil; p = p.parent {
|
||||||
|
if isDupe[p] {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return false
|
||||||
}
|
}
|
||||||
|
|
||||||
// treeBuilder reconstructs the directory hierarchy from records added
|
// treeBuilder reconstructs the directory hierarchy from records added
|
||||||
|
|||||||
@@ -364,7 +364,7 @@ func TestWriteTreeGroupsOnTerminal(t *testing.T) {
|
|||||||
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
|
label: treesPhase("write"), unit: "groups", total: int64(len(dupes)),
|
||||||
}, terminalWidth, time.Hour, time.Now)
|
}, terminalWidth, time.Hour, time.Now)
|
||||||
|
|
||||||
_, _, err := writeTreeGroups(os.Stderr, dupes, p)
|
_, err := writeTreeGroups(os.Stderr, dupes, p)
|
||||||
|
|
||||||
p.finish()
|
p.finish()
|
||||||
|
|
||||||
@@ -638,3 +638,64 @@ func TestCollectTreeGroupsKeepsGroupLeftWithTwoCopies(t *testing.T) {
|
|||||||
t.Errorf("groups = %v, want %v", got, want)
|
t.Errorf("groups = %v, want %v", got, want)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// TestReclaimableBytesCountsNestedDupeOnce checks that a dupe tree
|
||||||
|
// inside another dupe tree adds nothing to the reclaimable bytes. /B/x
|
||||||
|
// lies in /B, so the total adds the sizes of /B and /C/x, 1500 + 1000.
|
||||||
|
func TestReclaimableBytesCountsNestedDupeOnce(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
fh, ft, fc := testHash("fh"), testHash("ft"), testHash("fc")
|
||||||
|
gh, gt, gc := testHash("gh"), testHash("gt"), testHash("gc")
|
||||||
|
recs := []scanRec{
|
||||||
|
{size: 1000, head: fh, tail: ft, content: fc, path: "/A/x/f"},
|
||||||
|
{size: 500, head: gh, tail: gt, content: gc, path: "/A/y/g"},
|
||||||
|
{size: 1000, head: fh, tail: ft, content: fc, path: "/B/x/f"},
|
||||||
|
{size: 500, head: gh, tail: gt, content: gc, path: "/B/y/g"},
|
||||||
|
{size: 1000, head: fh, tail: ft, content: fc, path: "/C/x/f"},
|
||||||
|
}
|
||||||
|
|
||||||
|
super, dirs := treeOf(t, recs)
|
||||||
|
dupes := collectTreeGroups(dirs, super, everyPathExists, nil)
|
||||||
|
|
||||||
|
wantGroups := [][]string{{"/A", "/B"}, {"/A/x", "/B/x", "/C/x"}}
|
||||||
|
if got := groupPaths(dupes); !slices.EqualFunc(got, wantGroups,
|
||||||
|
slices.Equal) {
|
||||||
|
t.Fatalf("groups = %v, want %v", got, wantGroups)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := reclaimableBytes(dupes); got != 2500 {
|
||||||
|
t.Errorf("reclaimableBytes = %d, want 2500", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// TestReclaimableBytesCountsDupeInsideGrandparentDupeOnce checks that a
|
||||||
|
// dupe tree adds nothing when a dupe tree above its parent holds it.
|
||||||
|
// /B/m/x lies in /B, and /B/m is no dupe, since the group of /A and /B
|
||||||
|
// implies that of /A/m and /B/m. The total adds the sizes of /B and
|
||||||
|
// /C/x, 1300 + 1000.
|
||||||
|
func TestReclaimableBytesCountsDupeInsideGrandparentDupeOnce(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
f, w := testHash("f"), testHash("w")
|
||||||
|
recs := []scanRec{
|
||||||
|
{size: 1000, head: f, tail: f, content: f, path: "/A/m/x/f"},
|
||||||
|
{size: 300, head: w, tail: w, content: w, path: "/A/m/w"},
|
||||||
|
{size: 1000, head: f, tail: f, content: f, path: "/B/m/x/f"},
|
||||||
|
{size: 300, head: w, tail: w, content: w, path: "/B/m/w"},
|
||||||
|
{size: 1000, head: f, tail: f, content: f, path: "/C/x/f"},
|
||||||
|
}
|
||||||
|
|
||||||
|
super, dirs := treeOf(t, recs)
|
||||||
|
dupes := collectTreeGroups(dirs, super, everyPathExists, nil)
|
||||||
|
|
||||||
|
wantGroups := [][]string{{"/A", "/B"}, {"/A/m/x", "/B/m/x", "/C/x"}}
|
||||||
|
if got := groupPaths(dupes); !slices.EqualFunc(got, wantGroups,
|
||||||
|
slices.Equal) {
|
||||||
|
t.Fatalf("groups = %v, want %v", got, wantGroups)
|
||||||
|
}
|
||||||
|
|
||||||
|
if got := reclaimableBytes(dupes); got != 2300 {
|
||||||
|
t.Errorf("reclaimableBytes = %d, want 2300", got)
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in new issue
Block a user