1 Commits
Author SHA1 Message Date
clawbot 8e814bf0ba Store mtime to the nanosecond so a same-second rewrite is re-hashed (closes #12)
check / check (push) Canceled after 0s
scan recorded mtime in whole seconds, so a file rewritten in place at
the same size within the same second as its recorded mtime was classed
unchanged and kept its old hashes. The files table keeps mtime as whole
Unix seconds and gains mtime_nsec, the nanoseconds within that second.
scan holds the mtime as a time.Time and decides "newer" with After, so
any time a filesystem can record, one after 2262 included, compares in
the right order. The walk, a file given as an operand, and the content
phase's recheck all move over. PRAGMA user_version stays 1, per the
owner's ruling. README states what both columns hold.

Model: opus-5-5
2026-10-07 21:46:12 +00:00
9 changed files with 79 additions and 865 deletions
+20 -38
View File
@@ -271,21 +271,16 @@ All three subcommands operate on a single SQLite database file:
error (exit 1) telling the user to run `scan` first.
- Only one `scan` runs against a database at a time. For its whole run, `scan`
holds an exclusive `flock(2)` lock on a lock file beside the database, named
by appending `.lock` to the database's path with symlinks resolved
(`/var/lib/sfdupes/db.sqlite.lock` by default), taken before it walks the
filesystem or opens the database. Every path to one database, through a
symlink to the file or to a directory above it, names the same lock file. Just
before it takes the lock, `scan` creates the database file, empty, if it is
missing (through a symlink at the database path, it creates the symlink's
target), so the lock file is named the same way before the first scan sets up
the database as after. A second `scan` against the same database does not
wait: it fails at once with a one-line error naming the lock file and exits 1,
without walking anything or opening the database, and the running scan carries
on. The lock file is created on first use, open to its owner only, and left in
place: a leftover file blocks nothing, because the lock ends with the process
holding it however it ends, a fatal error or an interrupt included, and
deleting the file while a scan runs would let a second scan start. `report`
and `trees` never take the lock, so they run during a scan.
by appending `.lock` to the database path (`/var/lib/sfdupes/db.sqlite.lock`
by default), taken before it walks the filesystem or opens the database. A
second `scan` against the same database does not wait: it fails at once with a
one-line error naming the lock file and exits 1, without walking anything or
opening the database, and the running scan carries on. The lock file is
created on first use, open to its owner only, and left in place: a leftover
file blocks nothing, because the lock ends with the process holding it however
it ends, a fatal error or an interrupt included, and deleting the file while a
scan runs would let a second scan start. `report` and `trees` never take the
lock, so they run during a scan.
- While `scan` runs, the database is in WAL journal mode with a busy timeout, so
running a report while a cron `scan` is in progress is safe. The filesystem is
authoritative; the database is an eventually-consistent reflection of it.
@@ -398,11 +393,7 @@ results do not depend on the working directory. All operands belong to a single
scan and are enumerated concurrently: every operand seeds the shared walk worker
pool. Overlapping operands are harmless — an operand that duplicates another or
lies under another is dropped before walking, so every file is reached exactly
once and produces one database record. With `-x`, though, an operand under
another one is kept and walked on its own when any directory from it (from its
directory, for a regular file) up to but not including the operand covering it
is on a different filesystem than that operand, because that operand's walk does
not enter such a directory and so does not reach it.
once and produces one database record.
An operand that is a symlink (never followed, not even as an operand), socket,
FIFO, or device node, or a directory named `.zfs`, is not scanned. `scan` prints
@@ -411,8 +402,9 @@ drops it from the scanned operands before reading the database. Another operand
beneath it is still scanned. The records stored beneath it are not deleted: they
are treated like any other record outside the scanned operands, including the
content-phase exception below. If it lies under another operand, they are under
that operand instead, and are treated like any other record there. This is not
an error: a scan whose every operand is dropped walks nothing and exits 0.
that operand instead, and are deleted like any other record there that this scan
did not verify. This is not an error: a scan whose every operand is dropped
walks nothing and exits 0.
`scan` synchronizes the database with the filesystem state under the scanned
operands:
@@ -435,8 +427,8 @@ operands:
- A file whose mtime is newer than recorded, or whose size differs, is processed
as if new: re-hashed, or recorded without hashes, per the shared-size rule.
Change detection compares the mtime to the nanosecond, as finely as the
filesystem records it, so a same-size rewrite counts as a change whenever the
filesystem gives it a later mtime than recorded, even within the same second.
filesystem records it, so a same-size rewrite later in the same second as the
recorded mtime still counts as a change.
- A database record whose path lies under one of the scanned operands but was
not successfully processed this run is deleted. This removes records for
deleted files. It also removes records for paths that failed to stat or hash
@@ -450,12 +442,6 @@ operands:
a `content` hash is read for one, wherever it lies, once its size, `head`, and
`tail` match another record's. If that file is gone or has changed since its
record was written, the record is left as it is.
- With `-x`, a record below a directory the walk did not enter because that
directory is on another filesystem than its operand counts as outside the
scanned operands: it is not deleted, and its size joins the size census. A
filesystem mounted under an operand can therefore be scanned on its own
schedule into the same database. Records below an operand at or inside such a
directory are not kept this way, since that operand's own walk verifies them.
`scan` runs **four sequential phases over the whole scan**. Parallelism lives
inside each phase; batched database writes begin during the hash phase:
@@ -522,19 +508,15 @@ Rules for the walk:
- Filesystem boundaries are crossed by default. With `-x` (long form
`--one-file-system`, following the GNU `du`/`rsync` convention), never descend
into a directory on a different filesystem than its `PATH` operand; each
operand is bounded by its own filesystem. The records below such a directory,
unless they lie below an operand at or inside that directory, count as outside
the scanned operands, as described in "`scan` mode" above: they are not
deleted, and their sizes join the size census.
operand is bounded by its own filesystem.
- On any per-path error (permission denied, file vanished between passes,
unreadable): print a one-line warning to stderr, skip the path, and continue.
Per-file errors never abort the run; the final summary reports how many were
skipped. As specified above, a skipped path that has a database record from an
earlier scan loses that record, unless it failed only in the content phase, or
is an operand dropped before the database was read whose record counts as
outside the scanned operands; an unreadable directory subtree likewise loses
its records (accepted: the database mirrors what the latest scan could
actually verify).
is an operand dropped before the database was read that lies under no other
operand; an unreadable directory subtree likewise loses its records (accepted:
the database mirrors what the latest scan could actually verify).
Concurrency: the walk phase (which also stats files), the hash phase, and the
content phase each use a worker pool of `--workers` workers (default
-25
View File
@@ -29,31 +29,6 @@
# Completed Steps
- with `-x`, an operand under another operand is walked on its own instead of
being dropped when a directory from it (from its directory, for a regular
file) up to but not including that operand is on a different filesystem than
that operand, whose walk then cannot reach it, and the records of files gone
from below it are deleted (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/106)
- with `-x`, `scan` keeps the records below a directory on another filesystem
under its operand, and counts their sizes in the size census, instead of
deleting them (2026-10-14, https://git.eeqj.de/sneak/sfdupes/issues/99)
- the hash and content phases open each file without blocking and check that it
is still a regular file, so one replaced by a FIFO or device node after it was
statted is warned about and skipped instead of hanging the scan (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/100)
- the scan lock file is named from the database's path with symlinks resolved,
after `scan` creates the database file if it is missing, so a scan through a
symlink to the database fails while another scan holds the lock (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/101)
- on a terminal, a warning for the last file of a phase no longer erases the
phase's finished bar: the bar is drawn again below the warning (2026-10-14,
https://git.eeqj.de/sneak/sfdupes/issues/102)
- `scan` records mtime to the nanosecond, as whole seconds in `mtime` plus
`mtime_nsec`, and compares it at that resolution, so a same-size rewrite
within the same second is re-hashed (2026-10-07,
+7 -29
View File
@@ -35,10 +35,6 @@ const schemaVersion = 1
// scan.
const dbDirPerm = 0o755
// dbFilePerm is the mode for a database file created by scan, the mode
// SQLite gives a database file it creates.
const dbFilePerm = 0o644
// lockFilePerm is the mode for the scan lock file. Anyone who can open
// the file can hold the lock and keep every scan from running, so it
// is open to its owner only.
@@ -144,37 +140,19 @@ func openDB(path, params string) (*sql.DB, error) {
// lockScanDatabase takes the lock that keeps a second scan off the
// database at path: an exclusive flock(2) on the file beside it named
// by the database's path with symlinks resolved and ".lock" appended,
// so every path to one database names the same lock file. If missing,
// the database's parent directory and an empty database file are
// created first, so the lock file is named the same way before the
// database is set up as after. A lock held by another scan fails at
// once instead of waiting. The lock lasts until the returned file is
// closed or the process ends. The lock file is never deleted: a scan
// that deleted it would let the next scan lock a new file while
// another still holds the old one.
// path with ".lock" appended, created along with the database's parent
// directory if missing. A lock held by another scan fails at once
// instead of waiting. The lock lasts until the returned file is closed
// or the process ends. The file is never deleted: a scan that deleted
// it would let the next scan lock a new file while another still holds
// the old one.
func lockScanDatabase(path string) (*os.File, error) {
err := os.MkdirAll(filepath.Dir(path), dbDirPerm)
if err != nil {
return nil, fmt.Errorf("create database directory: %w", err)
}
// Like SQLite, this follows a symlink at path and creates its
// target. SQLite opens an empty file as a new database.
//nolint:gosec // the operator chooses the database path
dbFile, err := os.OpenFile(path, os.O_RDWR|os.O_CREATE, dbFilePerm)
if err != nil {
return nil, fmt.Errorf("create database: %w", err)
}
_ = dbFile.Close()
resolved, err := filepath.EvalSymlinks(path)
if err != nil {
return nil, fmt.Errorf("resolve database path: %w", err)
}
lockPath := resolved + ".lock"
lockPath := path + ".lock"
//nolint:gosec // the operator chooses the database path
f, err := os.OpenFile(lockPath, os.O_RDWR|os.O_CREATE, lockFilePerm)
+2 -10
View File
@@ -13,19 +13,11 @@ import (
"time"
)
// testDBPath returns a database path inside a fresh temp dir, with
// symlinks resolved as the scan lock resolves them, so the lock file
// is this path plus ".lock" even where the temp dir is reached through
// a symlink.
// testDBPath returns a database path inside a fresh temp dir.
func testDBPath(t *testing.T) string {
t.Helper()
dir, err := filepath.EvalSymlinks(t.TempDir())
if err != nil {
t.Fatal(err)
}
return filepath.Join(dir, "db.sqlite")
return filepath.Join(t.TempDir(), "db.sqlite")
}
// openTestDB creates a fresh scan database in a temp dir.
+4 -93
View File
@@ -822,7 +822,7 @@ func holdScanLock(t *testing.T, path string) {
func TestRunSecondScanFails(t *testing.T) {
// README §Database: while one scan holds the lock, a second scan
// fails at once, naming the lock file, without opening the
// fails at once, naming the lock file, without creating the
// database.
path := testDBPath(t)
t.Setenv(databaseEnv, path)
@@ -848,99 +848,10 @@ func TestRunSecondScanFails(t *testing.T) {
t.Errorf("stdout = %q, want nothing (data only)", got)
}
// Taking the lock created the database file, empty; opening it as
// a database would have written to it.
info, err := os.Stat(path)
if err != nil {
t.Fatal(err)
_, err := os.Stat(path)
if !errors.Is(err, fs.ErrNotExist) {
t.Errorf("stat %s = %v, want the database not created", path, err)
}
if info.Size() != 0 {
t.Errorf("database size = %d, want 0 (not opened)", info.Size())
}
}
// assertSecondScanFails runs a scan against the database
// SFDUPES_DATABASE names while the test holds the lock on it. It fails
// the test unless that scan fails at once, naming the lock file beside
// path, the database's real path.
func assertSecondScanFails(t *testing.T, path string) {
t.Helper()
var stderr bytes.Buffer
code := run([]string{cmdScan, t.TempDir()}, io.Discard, &stderr)
if code != exitFatal {
t.Errorf("run(scan) = %d, want %d", code, exitFatal)
}
want := "sfdupes: another scan is running (lock held on " +
path + ".lock)\n"
if got := stderr.String(); got != want {
t.Errorf("stderr = %q, want %q", got, want)
}
}
func TestRunSecondScanThroughSymlinkFails(t *testing.T) {
// README §Database: every path to one database names the same lock
// file, so while a scan holds it through one path, a scan through
// another fails, naming the lock file beside the database's real
// path.
t.Run("symlink to the database", func(t *testing.T) {
path := testDBPath(t)
t.Setenv(databaseEnv, path)
scanFixture(t)
link := filepath.Join(t.TempDir(), "link.sqlite")
err := os.Symlink(path, link)
if err != nil {
t.Fatal(err)
}
holdScanLock(t, path)
t.Setenv(databaseEnv, link)
assertSecondScanFails(t, path)
})
t.Run("symlink to its directory, no database yet", func(t *testing.T) {
path := testDBPath(t)
link := filepath.Join(t.TempDir(), "link")
err := os.Symlink(filepath.Dir(path), link)
if err != nil {
t.Fatal(err)
}
holdScanLock(t, path)
t.Setenv(databaseEnv, filepath.Join(link, filepath.Base(path)))
assertSecondScanFails(t, path)
})
t.Run("symlink whose target does not exist yet", func(t *testing.T) {
path := testDBPath(t)
link := filepath.Join(t.TempDir(), "link.sqlite")
err := os.Symlink(path, link)
if err != nil {
t.Fatal(err)
}
// The first scan takes the lock through the link before the
// database exists, then creates the database through it.
holdScanLock(t, link)
db, err := openScanDatabase(t.Context(), link)
if err != nil {
t.Fatal(err)
}
closeScanDatabase(t.Context(), db, link)
t.Setenv(databaseEnv, link)
assertSecondScanFails(t, path)
})
}
func TestRunScanReleasesLock(t *testing.T) {
-7
View File
@@ -139,13 +139,6 @@ func (p *progress) warnf(format string, args ...any) {
}
fmt.Fprintln(os.Stderr, msg)
// The library never draws a finished bar again, so the warning for
// a pass's last item would erase it for good: draw its last frame
// again below the warning.
if p.bar != nil && p.bar.IsFinished() {
fmt.Fprint(os.Stderr, p.bar.String())
}
}
// finish terminates the pass's display. A bar whose pass stopped short
-22
View File
@@ -187,25 +187,3 @@ func TestBarStoppedShortKeepsCount(t *testing.T) {
t.Errorf("terminal shows %q, want a count of 2 of 10", shown)
}
}
// TestWarningAfterLastItemKeepsBar checks that a warning printed after
// a pass's last item, as for a failed read of the last file, leaves the
// pass's finished bar on the terminal below the warning.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestWarningAfterLastItemKeepsBar(t *testing.T) {
stderr := captureStderr(t)
p := &progress{
label: "hash", total: 2, start: time.Now(),
bar: newBar("hash", 2),
}
p.increment()
p.increment()
p.warnf("warning")
p.finish()
if shown := lastFrame(stderr()); !strings.Contains(shown, "(2/2,") {
t.Errorf("terminal shows %q, want a count of 2 of 2", shown)
}
}
+32 -171
View File
@@ -203,16 +203,10 @@ func resolveRoots(roots []string) ([]string, error) {
}
// pruneRoots drops operands already covered by another operand:
// duplicates and any operand lying under another one. With -x, an
// operand under another one is kept when the walk of the deepest kept
// operand above it cannot reach it: when device puts any directory from
// it (from its directory, for a regular file) up to but not including
// that operand on a different filesystem than that operand. Every
// remaining file is then reachable through exactly one root, so walked
// paths are unique without keeping a scan-wide set of every path seen.
func pruneRoots(roots []string, oneFS bool,
device func(path string) (uint64, bool),
) []string {
// duplicates and any operand lying under another one. Every remaining
// file is then reachable through exactly one root, so walked paths are
// unique without keeping a scan-wide set of every path seen.
func pruneRoots(roots []string) []string {
// Shorter-first ordering guarantees an ancestor is kept before any
// operand under it is considered.
sorted := slices.Clone(roots)
@@ -227,58 +221,14 @@ func pruneRoots(roots []string, oneFS bool,
kept := make([]string, 0, len(sorted))
for _, r := range sorted {
above := deepestRoot(r, kept)
if above == "" {
if !underAnyRoot(r, kept) {
kept = append(kept, r)
continue
}
// Without -x, and always for a duplicate, r is dropped.
if !oneFS || above == r {
continue
}
aboveDev, aboveOK := device(above)
if !aboveOK {
continue
}
// The walk of above enters a directory only when it is on
// above's filesystem, and then reaches every regular file in it
// whatever device the file reports, so for a regular-file
// operand the directories checked start at its directory.
dir := r
fi, err := os.Lstat(r)
if err == nil && fi.Mode().IsRegular() {
dir = filepath.Dir(r)
}
for ; dir != above; dir = filepath.Dir(dir) {
dev, ok := device(dir)
if ok && dev != aboveDev {
kept = append(kept, r)
break
}
}
}
return kept
}
// deviceOfPath returns the filesystem device ID of the file at path,
// when lstat succeeds and the platform exposes one.
func deviceOfPath(path string) (uint64, bool) {
fi, err := os.Lstat(path)
if err != nil {
return 0, false
}
return deviceOfInfo(fi)
}
// scanStats summarizes one scan's database synchronization for the
// final stderr summary, or for the line an interrupted scan prints.
type scanStats struct {
@@ -338,22 +288,21 @@ func syncScan(ctx context.Context, db *sql.DB, roots []string,
// size, head, and tail match another record's). Records outside the
// roots are never touched, except that the content phase fills in
// their content hash. Operands the walk cannot start from are dropped
// first, so the records beneath them are treated as if those operands
// had not been given.
// first, so the records beneath them count as outside the roots unless
// they lie under another root.
func (s *scanState) runPhases(ctx context.Context, roots []string,
workers int, oneFS bool,
) error {
// Types are checked before pruning so that an operand under a
// dropped one is still scanned, not dropped as lying under it.
roots = pruneRoots(s.walkableRoots(roots), oneFS, deviceOfPath)
roots = pruneRoots(s.walkableRoots(roots))
err := s.loadIndex(ctx, roots)
if err != nil {
return err
}
changed, unhashed := s.walkPhase(
startWalk(ctx, roots, oneFS, workers), roots)
changed, unhashed := s.walkPhase(startWalk(ctx, roots, oneFS, workers))
// A cancelled walk stops early, so its size census covers only part
// of the roots, and every file it never reached would look vanished
@@ -381,10 +330,10 @@ func (s *scanState) runPhases(ctx context.Context, roots []string,
// walkableRoots returns the operands the walk can start from: regular
// files, and directories not named .zfs. Every other operand is warned
// about, counted as skipped, and dropped. A dropped operand is no
// longer a root, so the records stored beneath it are treated as if it
// had not been given: those under no other root count as outside the
// roots and are not deleted as unverified. An operand that fails lstat
// here is kept, and the walk warns about it.
// longer a root, so the records stored beneath it count as outside the
// roots and are not deleted as unverified, unless it lies under another
// root. An operand that fails lstat here is kept, and the walk warns
// about it.
func (s *scanState) walkableRoots(roots []string) []string {
kept := make([]string, 0, len(roots))
@@ -442,27 +391,15 @@ func (s *scanState) loadIndex(ctx context.Context, roots []string) error {
// (the content phase may still fill in its content hash). It returns
// the new-or-changed files and the unchanged files whose records lack
// hashes; both remain candidates until the census decides whether
// their sizes are shared. A record below a directory -x did not enter
// counts as outside the roots: it is not deleted, and its size joins
// the census. That does not hold below a root at or inside that
// directory, since that root's own walk verifies the records below it.
// their sizes are shared.
func (s *scanState) walkPhase(
events <-chan walkEvent, roots []string,
events <-chan walkEvent,
) ([]fileRec, []fileRec) {
prog := newProgress("walk", -1)
var (
changed, unhashed []fileRec
otherFSDirs []string
)
var changed, unhashed []fileRec
for ev := range events {
if ev.otherFSDir != "" {
otherFSDirs = append(otherFSDirs, ev.otherFSDir)
continue
}
if ev.fail {
s.st.skipped++
@@ -479,7 +416,7 @@ func (s *scanState) walkPhase(
old, ok := s.existing[ev.rec.path]
switch {
case !ok || old.size != ev.rec.size || mtimeAfter(ev.rec.mtime, old.mtime):
case !ok || old.size != ev.rec.size || ev.rec.mtime.After(old.mtime):
changed = append(changed, ev.rec)
case old.hashed:
delete(s.existing, ev.rec.path)
@@ -492,19 +429,6 @@ func (s *scanState) walkPhase(
prog.finish()
for path, old := range s.existing {
// Kept only when a directory -x did not enter lies below the
// deepest root above the record. A root at or below that
// directory was walked on its own and did not find the file.
// Both contain the record, so the longer path is the deeper.
otherFSDir := deepestRoot(path, otherFSDirs)
if len(otherFSDir) > len(deepestRoot(path, roots)) {
delete(s.existing, path)
s.sizes = append(s.sizes, old.size)
}
}
return changed, unhashed
}
@@ -884,7 +808,7 @@ func unchangedFile(r scanRec) (fileRec, bool, error) {
}
if !fi.Mode().IsRegular() || fi.Size() != r.size ||
mtimeAfter(fi.ModTime(), r.mtime) {
fi.ModTime().After(r.mtime) {
return fileRec{}, false, nil
}
@@ -895,16 +819,6 @@ func unchangedFile(r scanRec) (fileRec, bool, error) {
}, true, nil
}
// mtimeAfter reports whether mtime a is later than mtime b.
// Not a.After(b): time.Time wraps an mtime past year 292 billion; Unix() undoes it.
func mtimeAfter(a, b time.Time) bool {
if a.Unix() != b.Unix() {
return a.Unix() > b.Unix()
}
return a.Nanosecond() > b.Nanosecond()
}
// underAnyRoot reports whether path is any of the roots or lies under
// one of them.
func underAnyRoot(path string, roots []string) bool {
@@ -932,20 +846,6 @@ func underRoot(path, root string) bool {
return strings.HasPrefix(path, prefix)
}
// deepestRoot returns the longest of the roots that path is or lies
// under, or "" when it lies under none.
func deepestRoot(path string, roots []string) string {
deepest := ""
for _, root := range roots {
if underRoot(path, root) && len(root) > len(deepest) {
deepest = root
}
}
return deepest
}
// dirJob is one directory awaiting traversal by the walk workers. It
// carries its operand's filesystem device so -x can stop at
// filesystem boundaries.
@@ -956,23 +856,19 @@ type dirJob struct {
}
// walkEvent is one walk result delivered to the walk phase: a regular
// file's statted record, a warning when fail is set, or, when
// otherFSDir is set, a directory -x did not enter because it is on
// another filesystem than its operand.
// file's statted record, or a warning when fail is set.
type walkEvent struct {
rec fileRec
warn string
fail bool
otherFSDir string
rec fileRec
warn string
fail bool
}
// startWalk seeds every root into the shared walk worker pool and
// returns the event stream: one record per regular file, one warning
// event per per-path error, and one event per directory -x did not
// enter. The channel is closed when the walk completes, and also when
// ctx is cancelled — every goroutine in the pool abandons its blocking
// send in that case, so the consumer sees a truncated but properly
// terminated stream instead of a stalled one.
// event per per-path error. The channel is closed when the walk
// completes, and also when ctx is cancelled — every goroutine in the
// pool abandons its blocking send in that case, so the consumer sees a
// truncated but properly terminated stream instead of a stalled one.
func startWalk(ctx context.Context, roots []string, oneFS bool,
workers int,
) <-chan walkEvent {
@@ -1039,8 +935,7 @@ func operandWarning(root string, fi fs.FileInfo) string {
// operand becomes an initial job. walkableRoots has already dropped
// every other operand. One that has changed into something else since
// is warned about and skipped here; it is still a root, so the records
// stored beneath it, other than those under another root, are deleted
// as unverified.
// stored beneath it are deleted as unverified.
func seedRoot(ctx context.Context, root string,
events chan<- walkEvent,
) []dirJob {
@@ -1239,9 +1134,7 @@ func emitFile(ctx context.Context, p string, e fs.DirEntry,
// subdirJob applies the descent rules to directory p: never enter
// .zfs (ZFS snapshot pseudo-dirs would list every file once per
// snapshot), and with -x never enter a directory on a different
// filesystem than its operand, sending an event that names it so the
// scan keeps the records below it, except those below an operand at or
// inside it.
// filesystem than its operand.
func subdirJob(ctx context.Context, p string, e fs.DirEntry,
parent dirJob, oneFS bool, events chan<- walkEvent,
) (dirJob, bool) {
@@ -1265,8 +1158,6 @@ func subdirJob(ctx context.Context, p string, e fs.DirEntry,
}
if dev, ok := deviceOfInfo(info); ok && dev != parent.rootDev {
sendEvent(ctx, events, walkEvent{otherFSDir: p})
return dirJob{}, false
}
@@ -1408,38 +1299,6 @@ func hashWorker(ctx context.Context, jobs <-chan []fileRec,
}
}
// errNotRegular reports a path that is no longer a regular file when it
// is opened to be read.
var errNotRegular = errors.New("not a regular file")
// openRegularFile opens path for reading and returns it only if it is
// still a regular file, since the path may have been replaced after it
// was statted. O_NONBLOCK keeps the open of a FIFO or a device node
// from waiting forever, and fstat then rejects it. Reads of a regular
// file ignore O_NONBLOCK.
func openRegularFile(path string) (*os.File, error) {
//nolint:gosec // hashing operator-supplied paths is the tool's purpose
f, err := os.OpenFile(path, os.O_RDONLY|syscall.O_NONBLOCK, 0)
if err != nil {
return nil, err
}
fi, err := f.Stat()
if err != nil {
_ = f.Close()
return nil, err
}
if !fi.Mode().IsRegular() {
_ = f.Close()
return nil, errNotRegular
}
return f, nil
}
// emptyHash is the lowercase-hex SHA-256 of the empty input: the head,
// tail, and content hash of every zero-length file.
const emptyHash = "e3b0c44298fc1c149afbf4c8996fb924" +
@@ -1462,7 +1321,8 @@ func hashSignature(path string, size int64) (string, string, string, error) {
return emptyHash, emptyHash, emptyHash, nil
}
f, err := openRegularFile(path)
//nolint:gosec // hashing operator-supplied paths is the tool's purpose
f, err := os.Open(path)
if err != nil {
return "", "", "", err
}
@@ -1493,7 +1353,8 @@ func hashSignature(path string, size int64) (string, string, string, error) {
// tail are returned empty, because the content phase keeps the ones its
// records already hold.
func hashContentOnly(path string, size int64) (string, string, string, error) {
f, err := openRegularFile(path)
//nolint:gosec // hashing operator-supplied paths is the tool's purpose
f, err := os.Open(path)
if err != nil {
return "", "", "", err
}
+14 -470
View File
@@ -20,8 +20,6 @@ import (
"syscall"
"testing"
"time"
"golang.org/x/sys/unix"
)
// writeFile creates a file with the given content and returns its path.
@@ -583,65 +581,6 @@ func TestScanContentHashedStalePartners(t *testing.T) {
}
}
// TestScanContentSameSecondRewrite is TestScanContentStalePartners for
// a stored file rewritten in place at the same size with an mtime later
// in the same second than recorded: the file counts as changed, so
// neither it nor its match inside the operand is read.
func TestScanContentSameSecondRewrite(t *testing.T) {
t.Parallel()
db := openTestDB(t)
dirA := t.TempDir()
changed := sparseFileWithoutMatch(t, dirA, "changed", headTailMin)
first := time.Date(2026, 1, 2, 3, 4, 5, 100_000_000, time.UTC)
err := os.Chtimes(changed, first, first)
if err != nil {
t.Fatal(err)
}
syncTree(t, db, dirA)
before := dbRecords(t, db)
// Rewrite one byte in place, keeping the size.
pokeAt(t, changed, headTailMin/2, []byte{1})
later := first.Add(500 * time.Millisecond)
err = os.Chtimes(changed, later, later)
if err != nil {
t.Fatal(err)
}
dirB := t.TempDir()
sparseFile(t, dirB, "changed-copy", headTailMin)
st := syncTree(t, db, dirB)
if st != (scanStats{walked: 1, added: 1}) {
t.Errorf("stats = %+v, want 1 added and nothing skipped", st)
}
recs := dbRecords(t, db)
for _, r := range recs {
if r.content != "" {
t.Errorf("%s: content = %q, want none: its only match is stale",
r.path, r.content)
}
}
for _, old := range before {
if r := recordByPath(t, recs, old.path); r != old {
t.Errorf("record = %+v, want it left as %+v", r, old)
}
}
if groups := dupeGroups(t, db); len(groups) != 0 {
t.Errorf("groups = %+v, want none", groups)
}
}
// TestScanContentReadFailure checks that a failed content read is
// counted as skipped and leaves the record without a content hash, and
// that a later scan tries the read again.
@@ -997,7 +936,7 @@ func dirEntryFor(t *testing.T, dir, name string) fs.DirEntry {
}
// callSubdirJob runs subdirJob against e under parent, collecting any
// events it emits (subdirJob emits at most one).
// warning events it emits (subdirJob emits at most one).
func callSubdirJob(t *testing.T, p string, e fs.DirEntry,
parent dirJob, oneFS bool,
) (dirJob, bool, []walkEvent) {
@@ -1075,15 +1014,8 @@ func TestSubdirJobOneFilesystem(t *testing.T) {
t.Fatalf("accepted = %v, want %v", ok, tc.wantOK)
}
// A refused descent names the directory, so the scan keeps
// the records below it.
var wantEvs []walkEvent
if !ok {
wantEvs = []walkEvent{{otherFSDir: sub}}
}
if !slices.Equal(evs, wantEvs) {
t.Fatalf("events = %+v, want %+v", evs, wantEvs)
if len(evs) != 0 {
t.Fatalf("unexpected events: %+v", evs)
}
// The accepted job must carry the operand's device down, or -x
@@ -1135,296 +1067,6 @@ func TestSubdirJobStatError(t *testing.T) {
}
}
// syncOneFilesystem scans roots into db with -x and returns the scan
// stats. Every device pruneRoots looks up, and the device each
// operand's walk compares its subdirectories with, comes from devices
// instead of from lstat, so a test can put a path on a filesystem that
// no directory here is on. Unlike runPhases, it skips walkableRoots,
// the check for a cancelled walk, and the content phase.
func syncOneFilesystem(t *testing.T, db *sql.DB, roots []string,
devices map[string]uint64,
) scanStats {
t.Helper()
device := func(path string) (uint64, bool) {
dev, ok := devices[path]
return dev, ok
}
roots = pruneRoots(roots, true, device)
ctx := t.Context()
s := &scanState{db: db}
err := s.loadIndex(ctx, roots)
if err != nil {
t.Fatal(err)
}
jobs, subdirs, events := startWalkWorkers(ctx, 2, true)
// seedRoot emits a regular-file operand itself and returns a job
// only for a directory operand.
var initial []dirJob
for _, root := range roots {
for _, job := range seedRoot(ctx, root, events) {
job.rootDev = devices[root]
initial = append(initial, job)
}
}
dispatchDirs(ctx, initial, jobs, subdirs)
changed, unhashed := s.walkPhase(events, roots)
s.partition(changed, unhashed)
err = s.hashPhase(ctx, 2)
if err != nil {
t.Fatal(err)
}
err = s.updatePhase(ctx)
if err != nil {
t.Fatal(err)
}
return s.st
}
// TestScanOneFilesystemKeepsOtherFilesystemRecords checks that with -x a
// record below a directory the walk does not enter, because it is on
// another filesystem than its operand, counts as outside the operands:
// it is not deleted, and its size joins the size census. The record of
// a gone file that is not below that directory is still deleted, even
// when its path begins with the directory's name. The operand is given
// a device that no directory here is on, so no second real filesystem
// is needed.
func TestScanOneFilesystemKeepsOtherFilesystemRecords(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
// A scan without -x records the file below mnt, and mnt2, which is
// beside mnt, not below it.
mounted := writeFile(t, dir, "mnt/f", pattern(1, 100))
gone := writeFile(t, dir, "mnt2", pattern(3, 50))
syncTree(t, db, dir)
err := os.Remove(gone)
if err != nil {
t.Fatal(err)
}
// The same size as mnt/f, so it is hashed only if mnt/f's size is
// in the census.
added := writeFile(t, dir, "added", pattern(2, 100))
info, err := os.Lstat(dir)
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfInfo(info)
if !ok {
t.Skip("platform exposes no device id")
}
syncOneFilesystem(t, db, []string{dir}, map[string]uint64{dir: dev + 1})
recs := dbRecords(t, db)
recordByPath(t, recs, mounted)
if recordByPath(t, recs, added).head == "" {
t.Errorf("%s not hashed: the census lacks the size of %s",
added, mounted)
}
if slices.Contains(recordPaths(recs), gone) {
t.Errorf("record for %s kept: the file is gone and not below mnt",
gone)
}
}
// TestScanOneFilesystemWalksNestedOperand checks that with -x an
// operand under another one, on a different filesystem than it, is
// walked on its own: a file added below it is recorded, and the record
// of a file gone from below it is deleted. A record below a directory
// that its own walk does not enter is still kept. Each operand is given
// a device that no directory here is on, so neither walk enters a
// subdirectory.
func TestScanOneFilesystemWalksNestedOperand(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
// A scan without -x records both files below mnt.
gone := writeFile(t, dir, "mnt/gone", pattern(1, 10))
kept := writeFile(t, dir, "mnt/sub/kept", pattern(2, 20))
syncTree(t, db, dir)
err := os.Remove(gone)
if err != nil {
t.Fatal(err)
}
added := writeFile(t, dir, "mnt/added", pattern(3, 30))
info, err := os.Lstat(dir)
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfInfo(info)
if !ok {
t.Skip("platform exposes no device id")
}
mnt := filepath.Join(dir, "mnt")
syncOneFilesystem(t, db, []string{dir, mnt},
map[string]uint64{dir: dev + 1, mnt: dev + 2})
paths := recordPaths(dbRecords(t, db))
if !slices.Contains(paths, added) {
t.Errorf("no record for %s: the walk did not reach %s", added, mnt)
}
if slices.Contains(paths, gone) {
t.Errorf("record for %s kept: the file is gone", gone)
}
if !slices.Contains(paths, kept) {
t.Errorf("record for %s deleted: the walk of %s did not enter %s",
kept, mnt, filepath.Dir(kept))
}
}
// TestScanOneFilesystemWalksOperandBelowOtherFilesystem checks that
// with -x an operand on the same filesystem as the operand covering it
// is walked on its own when a directory between them is on another
// filesystem, since the walk of the covering operand does not enter
// that directory: a file added below the nested operand is recorded,
// and the record of a file gone from below it is deleted. A record
// below that directory but outside the nested operand is still kept.
// Both operands are given one device that no directory here is on,
// while mnt keeps its own.
func TestScanOneFilesystemWalksOperandBelowOtherFilesystem(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
// A scan without -x records both files below mnt.
gone := writeFile(t, dir, "mnt/back/gone", pattern(1, 10))
kept := writeFile(t, dir, "mnt/kept", pattern(2, 20))
syncTree(t, db, dir)
err := os.Remove(gone)
if err != nil {
t.Fatal(err)
}
added := writeFile(t, dir, "mnt/back/added", pattern(3, 30))
info, err := os.Lstat(dir)
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfInfo(info)
if !ok {
t.Skip("platform exposes no device id")
}
mnt := filepath.Join(dir, "mnt")
back := filepath.Join(mnt, "back")
syncOneFilesystem(t, db, []string{dir, back},
map[string]uint64{dir: dev + 1, mnt: dev, back: dev + 1})
paths := recordPaths(dbRecords(t, db))
if !slices.Contains(paths, added) {
t.Errorf("no record for %s: the walk did not reach %s", added, back)
}
if slices.Contains(paths, gone) {
t.Errorf("record for %s kept: the file is gone", gone)
}
if !slices.Contains(paths, kept) {
t.Errorf("record for %s deleted: the walk of %s did not enter %s",
kept, dir, mnt)
}
}
// TestScanOneFilesystemRecordsNestedFileOnce checks that with -x a
// regular-file operand under another operand is dropped when the file
// reports a different device than that operand but its directory does
// not, as a file bind-mounted into the tree does: the walk of the other
// operand enters the directory and reaches the file, so the file is
// recorded once.
func TestScanOneFilesystemRecordsNestedFileOnce(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
file := writeFile(t, dir, "sub/f", pattern(1, 10))
info, err := os.Lstat(dir)
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfInfo(info)
if !ok {
t.Skip("platform exposes no device id")
}
st := syncOneFilesystem(t, db, []string{dir, file}, map[string]uint64{
dir: dev, filepath.Dir(file): dev, file: dev + 1,
})
if st.added != 1 {
t.Errorf("%s recorded %d times, want once", file, st.added)
}
}
// TestScanOneFilesystemWalksNestedFileOperand checks that with -x a
// regular-file operand under another operand is kept when its directory
// is on a different filesystem than that operand: the walk of the other
// operand does not enter the directory, so the file is recorded once,
// as its own operand. The other operand is given a device that no
// directory here is on.
func TestScanOneFilesystemWalksNestedFileOperand(t *testing.T) {
t.Parallel()
dir := t.TempDir()
db := openTestDB(t)
file := writeFile(t, dir, "sub/f", pattern(1, 10))
info, err := os.Lstat(dir)
if err != nil {
t.Fatal(err)
}
dev, ok := deviceOfInfo(info)
if !ok {
t.Skip("platform exposes no device id")
}
st := syncOneFilesystem(t, db, []string{dir, file}, map[string]uint64{
dir: dev + 1, filepath.Dir(file): dev,
})
if st.added != 1 {
t.Errorf("%s recorded %d times, want once", file, st.added)
}
}
// buildSmokeTree recreates the README smoke-test filesystem layout
// with deterministic content and returns the tree root.
func buildSmokeTree(t *testing.T) string {
@@ -1743,30 +1385,12 @@ func TestSyncScanOperandSameSecondRewrite(t *testing.T) {
assertWholeFileHashed(t, db, a, rewritten)
}
// TestSyncScanRewriteAfter2262 runs assertLateRewriteRehashed with an
// mtime after 2262, a time too late to count in nanoseconds in an int64.
// TestSyncScanRewriteAfter2262 rewrites a file in place at the same
// size and sets its mtime after 2262, a time too late to count in
// nanoseconds in an int64: the next scan must still re-hash it.
func TestSyncScanRewriteAfter2262(t *testing.T) {
t.Parallel()
assertLateRewriteRehashed(t, time.Date(2300, 1, 2, 3, 4, 5, 0, time.UTC))
}
// TestSyncScanRewritePastTimeLimit runs assertLateRewriteRehashed with an
// mtime one second past the latest a time.Time holds without wrapping it
// to a time far in the past.
func TestSyncScanRewritePastTimeLimit(t *testing.T) {
t.Parallel()
assertLateRewriteRehashed(t, time.Unix(9223371974719179008, 0))
}
// assertLateRewriteRehashed scans a directory, rewrites a file in it in
// place at the same size, sets its mtime to late, and fails unless the
// next scan re-hashes the file. It skips where late does not fit the
// platform's timespec or the filesystem does not store it.
func assertLateRewriteRehashed(t *testing.T, late time.Time) {
t.Helper()
dir := t.TempDir()
db := openTestDB(t)
a := writeFile(t, dir, "a.bin", pattern(1, 500))
@@ -1779,14 +1403,11 @@ func assertLateRewriteRehashed(t *testing.T, late time.Time) {
rewritten := pattern(3, 500)
writeFile(t, dir, "a.bin", rewritten)
// os.Chtimes cannot set such a time: it converts through UnixNano.
ts, err := unix.TimeToTimespec(late)
if err != nil {
t.Skipf("an mtime %d seconds after 1970 does not fit this platform's "+
"timespec: %v", late.Unix(), err)
}
// os.Chtimes cannot set this time: it converts through UnixNano.
late := time.Date(2300, 1, 2, 3, 4, 5, 0, time.UTC)
ts := syscall.Timespec{Sec: late.Unix()}
err = unix.UtimesNano(a, []unix.Timespec{ts, ts})
err := syscall.UtimesNano(a, []syscall.Timespec{ts, ts})
if err != nil {
t.Fatal(err)
}
@@ -1796,9 +1417,9 @@ func assertLateRewriteRehashed(t *testing.T, late time.Time) {
t.Fatal(err)
}
if fi.ModTime().Unix() != late.Unix() {
t.Skipf("the filesystem stored the mtime as %d seconds after 1970, "+
"not %d", fi.ModTime().Unix(), late.Unix())
if !fi.ModTime().Equal(late) {
t.Skipf("the filesystem stored mtime %v instead of %v",
fi.ModTime().UTC(), late)
}
st := syncTree(t, db, dir)
@@ -2094,54 +1715,6 @@ func TestScanHardlinkRunFailsTogether(t *testing.T) {
}
}
// TestHashPhaseSkipsFileReplacedByFIFO replaces a walked regular file
// with a FIFO before the hash phase reads it. Opening a FIFO that has
// no writer waits for one, so a blocking open would hang the phase. The
// path must be warned about once and skipped, with no record written.
//
//nolint:paralleltest // captureStderr replaces the process-wide os.Stderr
func TestHashPhaseSkipsFileReplacedByFIFO(t *testing.T) {
stderr := captureStderr(t)
dir := t.TempDir()
p := writeFile(t, dir, "a.bin", pattern(1, 300))
walked, _ := collectWalk(t, []string{dir}, false, 1)
err := os.Remove(p)
if err != nil {
t.Fatal(err)
}
err = unix.Mkfifo(p, 0o600)
if err != nil {
t.Fatal(err)
}
s := &scanState{db: openTestDB(t), toHash: walked}
err = s.hashPhase(t.Context(), 1)
if err != nil {
t.Fatalf("hashPhase = %v, want nil", err)
}
if s.st.skipped != 1 || len(s.batch) != 0 {
t.Errorf("skipped = %d, records = %d, want 1 skipped and none",
s.st.skipped, len(s.batch))
}
warning := "hash " + p + ": not a regular file\n"
if got := stderr(); strings.Count(got, warning) != 1 {
t.Errorf("stderr = %q, want %q once", got, warning)
}
// The content phase opens files the same way.
_, _, content, err := hashContentOnly(p, headTailMin)
if content != "" || !errors.Is(err, errNotRegular) {
t.Errorf("hashContentOnly on a FIFO = %q, %v; want no hash and %v",
content, err, errNotRegular)
}
}
// injectedWriteFailure is the message the injected database trigger
// aborts with, so the test can recognize its own failure in the error
// the scan reports.
@@ -2335,43 +1908,14 @@ func TestHashRuns(t *testing.T) {
func TestPruneRoots(t *testing.T) {
t.Parallel()
const pqRoot = "/p/q"
// /p/q is on another filesystem than /p; /p/q/r is on the same one as
// /p/q, /p/s as /p, and /p/t has no known device.
devices := map[string]uint64{"/p": 1, pqRoot: 2, "/p/q/r": 2, "/p/s": 1}
device := func(path string) (uint64, bool) {
dev, ok := devices[path]
return dev, ok
}
// Duplicates and operands under other operands are dropped; /cc is
// not under /c (sibling with a shared prefix).
got := pruneRoots([]string{"/a/b", "/a", "/c", "/a", "/a/b/c", "/cc"},
false, device)
got := pruneRoots([]string{"/a/b", "/a", "/c", "/a", "/a/b/c", "/cc"})
want := []string{"/a", "/c", "/cc"}
if !slices.Equal(got, want) {
t.Fatalf("pruneRoots = %q, want %q", got, want)
}
nested := []string{"/p/q/r", pqRoot, "/p", "/p", "/p/s", "/p/t"}
got = pruneRoots(nested, false, device)
want = []string{"/p"}
if !slices.Equal(got, want) {
t.Fatalf("pruneRoots without -x = %q, want %q", got, want)
}
// With -x, /p/q, on another filesystem than /p, is kept as well.
got = pruneRoots(nested, true, device)
want = []string{"/p", pqRoot}
if !slices.Equal(got, want) {
t.Fatalf("pruneRoots with -x = %q, want %q", got, want)
}
}
func TestReportsNeverTouchFilesystem(t *testing.T) {