Name the database each command uses on stderr (closes #111) #118

Merged
clawbot merged 1 commits from issue-111-name-database into next 2026-10-16 15:28:03 +02:00
9 changed files with 412 additions and 36 deletions
Showing only changes of commit 3265841733 - Show all commits
+39 -4
View File
@@ -96,10 +96,11 @@ As a file `/etc/cron.d/sfdupes`:
the first scan. To keep it elsewhere, set
`SFDUPES_DATABASE=/path/to/db.sqlite` before the command on the same line.
- `scan` writes nothing to stdout. Its stderr, appended here to
`/var/log/sfdupes.log`, holds a plain progress line as each phase starts and
then at most every 5 seconds, a warning for each path it skips, and the
summary line (see "Progress" and "`scan` mode"). The log grows with every
scan; rotate it like any other.
`/var/log/sfdupes.log`, holds a first line naming the database (see
"Database"), a plain progress line as each phase starts and then at most every
5 seconds, a warning for each path it skips, and the summary line (see
"Progress" and "`scan` mode"). The log grows with every scan; rotate it like
any other.
- Skipped paths do not fail a scan: it still exits 0, and cron sends nothing. A
scan that fails, or is stopped by `SIGINT` or `SIGTERM`, exits 1 with the
reason among the last lines of the log; `tail` prints them, and cron mails
@@ -266,6 +267,26 @@ All three subcommands operate on a single SQLite database file:
command-line flag. The path names the file exactly, whatever characters it
holds (`?`, `#` and `%` included); a relative path is relative to the working
directory.
- Each subcommand first prints one line on stderr naming the database it uses,
before anything else it prints (progress displays included), whether or not
stderr is a terminal. The line gives the path with the working directory put
in front of it when it is relative, and otherwise as set (a `..` in it stays,
so the line names the same file), escaped like the paths in a report (see
"Report output format"), and says where the path came from:
```
scan: new database /var/lib/sfdupes/db.sqlite (default)
trees: database /srv/tmp/sfdupes/db.db (from SFDUPES_DATABASE)
```
The working directory put in front is the one `pwd -P` prints, with its
symlinks resolved. After a `cd` through a symlink to a directory, a path
starting `../` therefore names a file beside the directory the symlink leads
to, as it does for any other program. `scan` says `new database` when no
file is at that path as it starts. `report` and `trees` print the line
before they open the database, so the error for a missing database comes
after it.
- `scan` creates the database (and its parent directory) on first use. `report`
and `trees` require an existing database; a missing database file is a fatal
error (exit 1) telling the user to run `scan` first.
@@ -304,6 +325,20 @@ All three subcommands operate on a single SQLite database file:
in WAL mode until the next scan. `report` writes each row as it reads it, so
it is still reading while its output is paused (a pager, a stalled pipe), and
a `scan` started then fails after the busy timeout.
- A `scan` writes to the database and the files beside it at these points (its
phases are described under "`scan` mode"):
- As it starts, before the walk: the directory, an empty database file and
the `.lock` file, each only if missing.
- Then, still before the walk: the schema, in a new database, and the `-wal`
and `-shm` files, which stay beside the database while the scan runs.
- While it loads the existing records (`load`) and walks: no records. A
first scan's database holds none, however long its walk has run.
- During the hash phase: the first records, committed 10,000 at a time.
- In the update phase: the rest of the hashed records, the records for
size-unique files, and the deletions.
- In the content phase: content hashes, again committed 10,000 at a time.
- At the end, as it closes the database: the `-wal` and `-shm` files go
away, unless a report still has the database open (see above).
- `report` and `trees` open the database read-only and need only read access to
the database file, and no write access to its directory. While the database is
in WAL mode they also read the `-wal` and `-shm` files beside it, which SQLite
+6
View File
@@ -29,6 +29,12 @@
# Completed Steps
- `scan`, `report` and `trees` first print a line on stderr naming the database
they use, as an absolute path, and whether that path is the default or comes
from `SFDUPES_DATABASE`; `scan` says when the database is new. README
"Database" says when `scan` writes each file (2026-10-16,
https://git.eeqj.de/sneak/sfdupes/issues/111)
- re-vendor the canonical files and model scripts from `sneak/prompts` `next` at
`e3f79c4`: golangci-lint v2.14.0; lint and test are phases of the `Dockerfile`
that write no image, and `make test` runs the suite under the race detector,
+6 -4
View File
@@ -344,9 +344,10 @@ func storedPaths(t *testing.T, path string) []string {
// TestRunScanInterrupted calls the scan entrypoint with a context that
// is already cancelled, as when a signal arrives at once. It must return
// errInterrupted promptly with its one line on stderr and nothing on
// stdout, leave the database valid and as it was, and leave nothing in
// the way of the next scan, which must bring the database up to date.
// errInterrupted promptly, with its line on stderr after the one naming
// the database and nothing on stdout, leave the database valid and as
// it was, and leave nothing in the way of the next scan, which must
// bring the database up to date.
func TestRunScanInterrupted(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
@@ -389,7 +390,8 @@ func TestRunScanInterrupted(t *testing.T) {
err, errInterrupted)
}
want := "scan: interrupted after 0 files\n"
want := "scan: database " + path + " (from " + databaseEnv + ")\n" +
"scan: interrupted after 0 files\n"
if got := stderr()[shown:]; got != want {
t.Errorf("stderr = %q, want %q", got, want)
}
+28 -4
View File
@@ -11,6 +11,7 @@ import (
"path/filepath"
"slices"
"strconv"
"strings"
"time"
"golang.org/x/sys/unix"
@@ -91,13 +92,36 @@ var errSchemaVersion = errors.New("unsupported database schema version")
var errScanRunning = errors.New("another scan is running")
// databasePath resolves the database location: SFDUPES_DATABASE when
// set and non-empty, the compiled-in default otherwise.
func databasePath() string {
// set and non-empty, the compiled-in default otherwise. It returns the
// path, with the working directory, its symlinks resolved, put in front
// of it when it is relative, and where the path came from, "default" or
// "from SFDUPES_DATABASE", for the line each subcommand prints first on
// stderr.
func databasePath() (string, string, error) {
path, source := defaultDatabasePath, "default"
if p := os.Getenv(databaseEnv); p != "" {
return p
path, source = p, "from "+databaseEnv
}
return defaultDatabasePath
if filepath.IsAbs(path) {
return path, source, nil
}
// The working directory as the kernel reports it, with no symlink
// in it. os.Getwd may return PWD instead, which can name it through
// a symlink; a ".." after that symlink then leads somewhere else
// once the path is cleaned, as scan cleans it to create the
// database's directory.
wd, err := unix.Getwd()
if err != nil {
return "", "", fmt.Errorf("resolve database path %s: %w", path, err)
}
// Not filepath.Abs or filepath.Join: they clean the path, which
// drops each ".." together with the directory name before it. When
// that directory is a symlink, the cleaned path names another file.
// The working directory ends in "/" only when it is the root.
return strings.TrimSuffix(wd, "/") + "/" + path, source, nil
}
// scanParams are the connection parameters for scan: read-write, with
+38 -7
View File
@@ -43,16 +43,47 @@ func openTestDB(t *testing.T) *sql.DB {
}
func TestDatabasePath(t *testing.T) {
t.Setenv(databaseEnv, "")
if got := databasePath(); got != defaultDatabasePath {
t.Errorf("databasePath() = %q, want %q", got, defaultDatabasePath)
// README §Database: SFDUPES_DATABASE when set and non-empty, the
// default otherwise, with the working directory, its symlinks
// resolved, put in front of a relative path. Nothing is cleaned, so
// a ".." stays.
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
t.Setenv(databaseEnv, "/custom/place.sqlite")
t.Chdir(dir)
if got := databasePath(); got != "/custom/place.sqlite" {
t.Errorf("databasePath() = %q, want the env override", got)
fromEnv := "from " + databaseEnv
cases := []struct {
env, path, source string
}{
{"", defaultDatabasePath, "default"},
{"/custom/place.sqlite", "/custom/place.sqlite", fromEnv},
{"/custom/link/../place.sqlite", "/custom/link/../place.sqlite",
fromEnv},
{"sub/place.sqlite", dir + "/sub/place.sqlite", fromEnv},
{"link/../place.sqlite", dir + "/link/../place.sqlite", fromEnv},
}
for _, tc := range cases {
t.Setenv(databaseEnv, tc.env)
path, source, err := databasePath()
if err != nil || path != tc.path || source != tc.source {
t.Errorf("databasePath() with %s=%q = %q, %q, %v; want %q, %q",
databaseEnv, tc.env, path, source, err, tc.path, tc.source)
}
}
// In the root directory the path gets one slash in front, not two.
t.Chdir("/")
t.Setenv(databaseEnv, "place.sqlite")
path, _, err := databasePath()
if err != nil || path != "/place.sqlite" {
t.Errorf("databasePath() in / = %q, %v; want %q",
path, err, "/place.sqlite")
}
}
+256 -7
View File
@@ -285,22 +285,29 @@ func assertFatalOutput(t *testing.T, stderr, stdout string) {
func TestRunMissingDatabaseIsFatal(t *testing.T) {
// README §Database: report and trees need an existing database; a
// missing one exits 1 with a message telling the user to run scan.
// missing one exits 1 with a message telling the user to run scan,
// after the line naming the database.
for _, name := range []string{cmdReport, cmdTrees} {
t.Run(name, func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
var stdout, stderr bytes.Buffer
var stdout bytes.Buffer
code := run([]string{name}, &stdout, &stderr)
// run gets os.Stderr, where the subcommand prints its line,
// so the capture shows the two in the order printed.
stderr := captureStderr(t)
code := run([]string{name}, &stdout, os.Stderr)
if code != exitFatal {
t.Errorf("run(%s) = %d, want %d", name, code, exitFatal)
}
want := "sfdupes: " + path + ": no database (run \"sfdupes " +
want := name + ": database " + path + " (from " + databaseEnv +
")\n" +
"sfdupes: " + path + ": no database (run \"sfdupes " +
"scan\" first, or set " + databaseEnv + ")\n"
if got := stderr.String(); got != want {
if got := stderr(); got != want {
t.Errorf("stderr = %q, want %q", got, want)
}
@@ -311,6 +318,246 @@ func TestRunMissingDatabaseIsFatal(t *testing.T) {
}
}
func TestRunNamesRelativeDatabase(t *testing.T) {
// README §Database: each subcommand's first line on stderr names the
// database, a relative SFDUPES_DATABASE made absolute against the
// working directory. scan calls it new until a scan has created it.
path := testDBPath(t)
t.Chdir(filepath.Dir(path))
t.Setenv(databaseEnv, filepath.Base(path))
dir := t.TempDir()
from := " (from " + databaseEnv + ")\n"
want := "scan: new database " + path + from
if got := scanOK(t, dir); !strings.HasPrefix(got, want) {
t.Errorf("first scan stderr = %q, want it to start %q", got, want)
}
want = "scan: database " + path + from
if got := scanOK(t, dir); !strings.HasPrefix(got, want) {
t.Errorf("second scan stderr = %q, want it to start %q", got, want)
}
for _, name := range []string{cmdReport, cmdTrees} {
stderr := captureStderr(t)
code := run([]string{name}, io.Discard, os.Stderr)
want = name + ": database " + path + from
if got := stderr(); code != exitOK || !strings.HasPrefix(got, want) {
t.Errorf("run(%s) = %d with stderr %q, want %d and it to "+
"start %q", name, code, got, exitOK, want)
}
}
}
func TestRunUsesDatabaseThroughSymlinkedDirectory(t *testing.T) {
// README §Database: the path names the file exactly. With link a
// symlink to the directory real/sub, link/../db.sqlite names
// real/db.sqlite, not db.sqlite beside link. Every subcommand must
// use that file, relative or absolute, and its first line must name
// the path as set, with the working directory put in front of a
// relative one.
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
t.Chdir(dir)
err = os.MkdirAll(filepath.Join(dir, "real", "sub"), 0o750)
if err != nil {
t.Fatal(err)
}
err = os.Symlink(filepath.Join(dir, "real", "sub"),
filepath.Join(dir, "link"))
if err != nil {
t.Fatal(err)
}
named := dir + "/link/../db.sqlite"
from := " (from " + databaseEnv + ")\n"
tree := t.TempDir()
t.Setenv(databaseEnv, "link/../db.sqlite")
want := "scan: new database " + named + from
if got := scanOK(t, tree); !strings.HasPrefix(got, want) {
t.Errorf("relative scan stderr = %q, want it to start %q", got, want)
}
// The absolute path names the database the relative one created.
t.Setenv(databaseEnv, named)
want = "scan: database " + named + from
if got := scanOK(t, tree); !strings.HasPrefix(got, want) {
t.Errorf("absolute scan stderr = %q, want it to start %q", got, want)
}
for _, name := range []string{cmdReport, cmdTrees} {
stderr := captureStderr(t)
code := run([]string{name}, io.Discard, os.Stderr)
want = name + ": database " + named + from
if got := stderr(); code != exitOK || !strings.HasPrefix(got, want) {
t.Errorf("run(%s) = %d with stderr %q, want %d and it to "+
"start %q", name, code, got, exitOK, want)
}
}
_, err = os.Stat(filepath.Join(dir, "real", "db.sqlite"))
if err != nil {
t.Errorf("database the path names: %v", err)
}
// With no file at the cleaned path, report and trees succeeded on
// the database in real.
_, err = os.Stat(filepath.Join(dir, "db.sqlite"))
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("database at the cleaned path: stat error %v, want it "+
"missing", err)
}
}
func TestRunScanFromSymlinkedWorkingDirectory(t *testing.T) {
// README §Database: a relative path gets the working directory, its
// symlinks resolved, put in front of it. With home/proj a symlink
// to data/proj, entered as the working directory through the
// symlink, ../new/db.sqlite names data/new/db.sqlite. scan must
// create that directory and file, and nothing beside home/proj.
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
err = os.MkdirAll(filepath.Join(dir, "data", "proj"), 0o750)
if err != nil {
t.Fatal(err)
}
err = os.Mkdir(filepath.Join(dir, "home"), 0o750)
if err != nil {
t.Fatal(err)
}
err = os.Symlink(filepath.Join(dir, "data", "proj"),
filepath.Join(dir, "home", "proj"))
if err != nil {
t.Fatal(err)
}
// t.Chdir also sets PWD to the path through the symlink, as a
// shell does after cd.
t.Chdir(filepath.Join(dir, "home", "proj"))
t.Setenv(databaseEnv, "../new/db.sqlite")
want := "scan: new database " + dir + "/data/proj/../new/db.sqlite" +
" (from " + databaseEnv + ")\n"
if got := scanOK(t, t.TempDir()); !strings.HasPrefix(got, want) {
t.Errorf("scan stderr = %q, want it to start %q", got, want)
}
_, err = os.Stat(filepath.Join(dir, "data", "new", "db.sqlite"))
if err != nil {
t.Errorf("database the path names: %v", err)
}
_, err = os.Lstat(filepath.Join(dir, "home", "new"))
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("home/new: stat error %v, want it missing", err)
}
}
func TestRunEscapesDatabasePath(t *testing.T) {
// README §Database: each subcommand's first line escapes the
// database path like the paths in a report, so a tab, newline or
// backslash in it keeps the line one line.
dir := t.TempDir()
t.Setenv(databaseEnv, dir+"/a\tb\nc\\d.sqlite")
escaped := dir + `/a\tb\nc\\d.sqlite`
from := " (from " + databaseEnv + ")\n"
want := "scan: new database " + escaped + from
if got := scanOK(t, t.TempDir()); !strings.HasPrefix(got, want) {
t.Errorf("scan stderr = %q, want it to start %q", got, want)
}
for _, name := range []string{cmdReport, cmdTrees} {
stderr := captureStderr(t)
code := run([]string{name}, io.Discard, os.Stderr)
want = name + ": database " + escaped + from
if got := stderr(); code != exitOK || !strings.HasPrefix(got, want) {
t.Errorf("run(%s) = %d with stderr %q, want %d and it to "+
"start %q", name, code, got, exitOK, want)
}
}
}
func TestRunNamesDefaultDatabase(t *testing.T) {
// README §Database: with SFDUPES_DATABASE empty, each subcommand's
// first line on stderr names the default database. A database may
// exist at the default path on the machine running the test, so
// nothing here may create or read one: scan gets an operand that
// does not exist, which stops it before it takes the lock, and
// report and trees a cancelled context, which stops them before
// they read anything.
t.Setenv(databaseEnv, "")
// Whether scan calls the database new depends on that machine too.
scanWant := "scan: database " + defaultDatabasePath + " (default)\n"
_, err := os.Stat(defaultDatabasePath)
if errors.Is(err, fs.ErrNotExist) {
scanWant = "scan: new database " + defaultDatabasePath +
" (default)\n"
}
missing := filepath.Join(t.TempDir(), "nope")
cases := map[string]struct {
run func() error
want string
}{
cmdScan: {
run: func() error {
return runScan(t.Context(), []string{missing}, 1, false)
},
want: scanWant,
},
cmdReport: {
run: func() error {
return runReport(cancelledContext(t), io.Discard)
},
want: "report: database " + defaultDatabasePath + " (default)\n",
},
cmdTrees: {
run: func() error {
return runTrees(cancelledContext(t), io.Discard)
},
want: "trees: database " + defaultDatabasePath + " (default)\n",
},
}
for name, tc := range cases {
stderr := captureStderr(t)
err = tc.run()
if err == nil {
t.Errorf("%s: error = nil, want it stopped", name)
}
if got := stderr(); got != tc.want {
t.Errorf("%s stderr = %q, want %q", name, got, tc.want)
}
}
}
func TestRunUsageErrors(t *testing.T) {
// Usage errors keep exiting 2 with cobra's own report on stderr.
cases := map[string]struct {
@@ -661,7 +908,8 @@ func TestRunReportSucceeds(t *testing.T) {
t.Errorf("stdout = %q, want %q", got, want)
}
want = "report: 2 records read, 1 duplicate groups, 1 dupe files, " +
want = "report: database " + path + " (from " + databaseEnv + ")\n" +
"report: 2 records read, 1 duplicate groups, 1 dupe files, " +
"300 B reclaimable\n"
if got := stderr(); got != want {
t.Errorf("stderr = %q, want %q", got, want)
@@ -694,7 +942,8 @@ func TestRunTreesSucceeds(t *testing.T) {
t.Errorf("stdout = %q, want %q", got, want)
}
want = "trees: 2 records read, 1 duplicate tree groups, 1 dupe trees, " +
want = "trees: database " + path + " (from " + databaseEnv + ")\n" +
"trees: 2 records read, 1 duplicate tree groups, 1 dupe trees, " +
"300 B reclaimable\n"
if got := stderr(); got != want {
t.Errorf("stderr = %q, want %q", got, want)
+9 -2
View File
@@ -34,9 +34,16 @@ type scanRec struct {
// records, and each row is written as it is read, so no group is held
// in memory. It never touches the scanned filesystem; its only I/O is
// the database (with SQLite's temporary sort file), stdout, and stderr.
// Any database problem, including a missing database, is fatal.
// It names the database on stderr before it opens it. Any database
// problem, including a missing database, is fatal.
func runReport(ctx context.Context, stdout io.Writer) error {
dbPath := databasePath()
dbPath, source, err := databasePath()
if err != nil {
return err
}
fmt.Fprintf(os.Stderr, "report: database %s (%s)\n",
escapePath(dbPath), source)
db, err := openReportDatabase(ctx, dbPath)
if err != nil {
+21 -5
View File
@@ -91,9 +91,11 @@ type fileMeta struct {
// least one other file shares are ever hashed: a size-unique file
// cannot be a duplicate. A file of headTailMin or more gets its content
// hash only when its size, head, and tail match another file's. Flag
// parsing and the at-least-one-operand check are done by cobra. The
// scan holds the lock on the database for its whole run, so a second
// scan fails before it walks the filesystem or opens the database.
// parsing and the at-least-one-operand check are done by cobra. Before
// anything else, the scan names the database on stderr, saying whether
// it is new. The scan holds the lock on the database for its whole
// run, so a second scan fails before it walks the filesystem or opens
// the database.
// Errors are returned rather than exiting, so that the deferred close —
// which takes the database out of WAL mode — always runs, and the lock
// is released after it. When ctx is cancelled, as by the SIGINT or
@@ -104,12 +106,26 @@ type fileMeta struct {
func runScan(ctx context.Context, roots []string, workers int,
oneFS bool,
) error {
roots, err := resolveRoots(roots)
dbPath, source, err := databasePath()
if err != nil {
return err
}
dbPath := databasePath()
// A database file missing now is created by lockScanDatabase below.
what := "database"
_, err = os.Stat(dbPath)
if errors.Is(err, fs.ErrNotExist) {
what = "new database"
}
fmt.Fprintf(os.Stderr, "scan: %s %s (%s)\n", what, escapePath(dbPath),
source)
roots, err = resolveRoots(roots)
if err != nil {
return err
}
lock, err := lockScanDatabase(dbPath)
if err != nil {
+9 -3
View File
@@ -29,10 +29,16 @@ type treeNode struct {
// the record paths, computes a Merkle-style digest per directory, and
// prints maximal duplicate-tree groups as TSV on stdout. It never
// touches the scanned filesystem; its only I/O is the database, stdout,
// and stderr. Any database problem, including a missing database, is
// fatal.
// and stderr. It names the database on stderr before it opens it. Any
// database problem, including a missing database, is fatal.
func runTrees(ctx context.Context, stdout io.Writer) error {
dbPath := databasePath()
dbPath, source, err := databasePath()
if err != nil {
return err
}
fmt.Fprintf(os.Stderr, "trees: database %s (%s)\n",
escapePath(dbPath), source)
db, err := openReportDatabase(ctx, dbPath)
if err != nil {