11 Commits

Author SHA1 Message Date
340bdbe39e Merge branch 'make-default-target': plain make builds the binary
All checks were successful
check / check (push) Successful in 1m12s
2026-07-24 11:21:18 +07:00
a1c3b852c3 Make the binary the default Make target
All checks were successful
check / check (push) Successful in 4s
Plain make now builds sfdupes (previously the default was all =
check + build); make build remains as an alias, so the Dockerfile
and existing habits keep working. The sfdupes target is phony: go
build's own cache decides what to recompile.
2026-07-24 11:21:16 +07:00
9f03eb3e2a Merge branch 'scan-wide-phases': scan-wide phases, concurrent operands, batched updates
All checks were successful
check / check (push) Successful in 1m8s
2026-07-24 10:26:54 +07:00
3ecf73c80a Make phases scan-wide, walk operands concurrently, batch updates
All checks were successful
check / check (push) Successful in 5s
All PATH operands belong to a single scan: every operand seeds the
shared walk worker pool, and each pass (walk, stat, hash, update)
runs exactly once over the whole scan, so pass totals, percentages,
and ETAs are scan-global. The per-operand walk/hash/update cycles and
their stderr operand announcements are gone; duplicate paths from
overlapping operands are deduplicated before stat.

The update pass now commits in batched transactions (10k changes per
batch) instead of one scan-wide transaction: the filesystem is
authoritative and the database is an eventually-consistent reflection
of it, so scan-level atomicity buys nothing, while batches keep the
WAL small and let concurrent reports observe progress.
2026-07-24 10:26:52 +07:00
732fc351d7 Merge branch 'parallel-phases': sequential phases, parallelism within each
All checks were successful
check / check (push) Successful in 1m9s
2026-07-24 08:12:49 +07:00
1e7a519608 Split the stat pass back out of the walk
All checks were successful
check / check (push) Successful in 4s
Phases are strictly sequential again — walk, stat, hash, update per
operand — with parallelism only inside each phase. The walk
enumerates paths with per-directory workers (no lstat of file
entries); the stat pass lstats every collected path with per-file
workers, restoring its exact-total/ETA progress bar and per-file
parallelism inside wide flat directories.
2026-07-24 08:12:47 +07:00
09ff9b5f30 Merge branch 'scan-operand-progress': announce operands on stderr
All checks were successful
check / check (push) Successful in 1m15s
2026-07-24 07:52:55 +07:00
a0f0050ada Announce each operand on stderr before its passes
All checks were successful
check / check (push) Successful in 6s
With per-operand walk/hash/update cycles, a multi-operand invocation
(e.g. scan /srv/*) showed pass totals that looked like the whole
run's: an operator watching operand 3 of 14 hash 300k files concluded
the other 20M files were being skipped. Print the operand path and
its position before each cycle.
2026-07-24 07:52:54 +07:00
6a15b879de Merge branch 'parallel-walk': parallel per-directory walk, per-operand commits
All checks were successful
check / check (push) Successful in 1m10s
2026-07-24 07:44:28 +07:00
dced5cf0d2 Parallelize the walk and commit per operand
All checks were successful
check / check (push) Successful in 5s
Replace the single-goroutine WalkDir traversal with a per-directory
worker pool: workers read directories concurrently and lstat entries
while each directory is fresh in cache, recording size and mtime
during the walk. This folds the separate stat pass away (halving
metadata I/O per run) and overlaps metadata latency, which dominated
on busy pools — a sequential walk of a ~22M-file tree was observed
taking over 4 hours.

Each PATH operand now loads its scope, walks, hashes, and commits in
its own transaction, so an interrupted scan keeps every operand
completed so far; a later overlapping operand sees the records
committed by earlier ones and reuses them unchanged.
2026-07-24 07:44:18 +07:00
3ebf98940a Specify parallel per-directory walk and per-operand commits
The walk pass is a single goroutine; on a busy ZFS pool it manages
only a few thousand directory entries per second and takes hours at
~20M files. Respecify it as a worker-pool traversal that reads
directories concurrently and records size/mtime during the walk,
folding away the separate stat pass and halving metadata I/O. Each
PATH operand now commits in its own transaction so an interrupted
scan keeps the operands completed so far.
2026-07-24 06:28:12 +07:00
7 changed files with 454 additions and 153 deletions

View File

@@ -6,13 +6,15 @@ BINARY := sfdupes
VERSION := $(shell git describe --tags --always --dirty 2>/dev/null || echo dev)
LDFLAGS := -X main.Version=$(VERSION)
.PHONY: all build test lint fmt fmt-check check docker hooks clean
.PHONY: sfdupes build test lint fmt fmt-check check docker hooks clean
all: check build
build:
# Default target: build the binary. Phony so go build (which has its
# own build cache) always decides what to recompile.
sfdupes:
go build -ldflags "$(LDFLAGS)" -o $(BINARY)
build: sfdupes
test:
@go test -timeout 30s -cover ./... || \
{ echo "--- Rerunning with -v for details ---"; \

View File

@@ -130,10 +130,13 @@ All three subcommands operate on a single SQLite database file:
database file is a fatal error (exit 1) telling the user to run
`scan` first.
- The database uses WAL journal mode and a busy timeout, so running a
report while a cron `scan` is in progress is safe; the reports see
the last committed state. Each scan commits its changes in a single
transaction, so a report never observes a half-finished scan and a
scan that dies partway leaves the previous state intact.
report while a cron `scan` is in progress is safe. The filesystem
is authoritative; the database is an eventually-consistent
reflection of it. The update pass applies changes in batched
transactions (keeping the WAL small and letting concurrent reports
observe progress), so a report may see a scan's changes partially
applied, and a scan that dies partway leaves a valid database that
the next scan converges toward the filesystem.
- Schema (`PRAGMA user_version` is the schema version, currently 1; a
database with any other version is a fatal error):
@@ -160,9 +163,12 @@ or a regular file; an operand that does not exist is a fatal error
(exit 1). Because database records persist between runs and are keyed
by absolute path, each operand is resolved to an absolute, lexically
cleaned path (symlinks are not resolved) before walking, so results do
not depend on the working directory. Operands are walked in the order
given; overlapping operands (one containing another) are harmless — a
file reached via multiple operands produces one database record.
not depend on the working directory. All operands belong to a single
scan and are enumerated concurrently: every operand seeds the shared
walk worker pool, and every pass runs once over the whole scan, so
pass totals and ETAs are scan-wide. Overlapping operands (one
containing another) are harmless — a file reached via multiple
operands is deduplicated by path and produces one database record.
`scan` synchronizes the database with the filesystem state under the
scanned operands:
@@ -184,20 +190,29 @@ scanned operands:
disjoint trees can be scanned on different schedules into the same
database.
`scan` runs **four sequential passes**, in this order, so that every
expensive pass has an exact total for meaningful progress and ETA:
`scan` runs **four sequential passes over the whole scan**.
Parallelism lives strictly inside each pass; the passes themselves
never overlap:
1. **walk** — recursively enumerate the tree under each `PATH` in
turn, collecting the list of regular-file paths. Total unknown
while running: show a live count, not a percentage.
2. **stat** — `lstat` every collected path, recording size and mtime.
1. **walk** — enumerate the trees under all `PATH` operands
concurrently with the worker pool: every operand seeds the shared
queue, and each worker reads one directory at a time, collecting
regular-file paths and handing discovered subdirectories back to
the queue. Sequential directory enumeration is
metadata-latency-bound and takes hours at tens of millions of
files; per-directory parallelism is what makes the walk tractable
on large or busy pools. Total unknown while running: show a live
count, not a percentage.
2. **stat** — `lstat` every collected path with the worker pool
(per-file parallelism), recording size and mtime.
3. **hash** — for each new or changed file (per the rules above), read
the first `min(1024, size)` bytes and the last `min(1024, size)`
bytes (the two reads overlap when `size < 2048`; for `size == 0`
hash the empty input) and compute the SHA-256 of each. Unchanged
files are not read and do not appear in this pass's total.
4. **update** — apply all insertions, updates, and deletions to the
database in a single transaction.
files are not read and do not appear in this pass's total, which
is therefore exact for meaningful progress and ETA.
4. **update** — apply the scan's insertions, updates, and deletions
to the database in batched transactions.
Rules for the walk:
@@ -219,9 +234,12 @@ Rules for the walk:
records (accepted: the database mirrors what the latest scan could
actually verify).
Concurrency: the stat and hash passes use a worker pool (`--workers`,
default `runtime.NumCPU()`). The main goroutine owns database writes
and progress rendering; progress display must never block the workers.
Concurrency: the walk, stat, and hash passes each use a worker pool
(`--workers`, default `runtime.NumCPU()`); the walk parallelizes
across directories, stat and hash across files, so raising
`--workers` can speed up metadata-bound passes on busy pools. The
main goroutine owns database writes and progress rendering; progress
display must never block the workers.
`scan` writes nothing to stdout. The summary line on stderr reports the
files seen this run broken down by disposition, plus skips:
@@ -336,8 +354,8 @@ all dupe rows) in human units.
Use the progress-bar library for all scan-pass progress; rendering in the
style of `pv` is the model. All progress goes to stderr.
Each scan pass gets its own bar. Required elements for the stat, hash,
and update passes (known totals):
Each scan pass gets its own bar. Required elements for the stat,
hash, and update passes (known totals):
- elapsed time
- estimated time remaining
@@ -375,7 +393,8 @@ Additional requirements:
The `Makefile` is the single source of truth for all operations:
- `make build` — build the `sfdupes` binary (cgo disabled).
- `make` / `make build` — build the `sfdupes` binary (cgo
disabled); building is the default target.
- `make test` — run the test suite (30-second timeout; reruns with
`-v` on failure).
- `make lint` — run `golangci-lint` with the repo config.

31
TODO.md
View File

@@ -19,6 +19,37 @@
# Completed Steps
- make the binary the default Make target (2026-07-24, branch
`make-default-target`): plain `make` now builds `sfdupes`
(previously it ran `check` plus `build`); `make build` remains as
an alias
- scan-wide phases, concurrent operands, batched updates (2026-07-24,
branch `scan-wide-phases`): all operands seed the shared walk pool
and every pass runs once over the whole scan, so totals and ETAs
are scan-global; the per-operand walk/hash/update cycles and their
stderr announcements are gone; the update pass commits in batched
transactions — the filesystem is authoritative and the database an
eventually-consistent reflection, so scan-level atomicity is not
required
- split the stat pass back out of the walk (2026-07-24, branch
`parallel-phases`): phases are strictly sequential again — walk,
stat, hash, update per operand — with parallelism only inside each
phase; the walk enumerates paths with per-directory workers and the
stat pass lstats them with per-file workers, restoring the exact
total/ETA stat bar
- announce each operand on stderr before its passes (2026-07-24,
branch `scan-operand-progress`): with per-operand walk/hash/update
cycles, a multi-operand run (e.g. `scan /srv/*`) showed pass totals
that looked like the whole run's — an operator watching operand 3 of
14 hash 300k files concluded 20M files were being skipped
- parallel walk (2026-07-24, branch `parallel-walk`): the walk pass
was a single goroutine and took hours at ~20M files on a busy pool
(observed: 22M files in 4h on a ZFS server); it is now a
per-directory worker-pool traversal that records size/mtime during
the walk (folding away the separate stat pass, halving metadata
I/O), and each `PATH` operand commits in its own transaction so an
interrupted scan keeps completed operands
- persistent scan database (2026-07-24, branch `persistent-database`):
`scan` now maintains a SQLite database (`modernc.org/sqlite`, pure
Go, cgo stays disabled) keyed by absolute path that survives between

35
db.go
View File

@@ -8,6 +8,7 @@ import (
"io/fs"
"os"
"path/filepath"
"slices"
"strconv"
// The pure-Go SQLite driver, registered as "sqlite"; keeps cgo
@@ -237,12 +238,40 @@ func loadFileRows(db *sql.DB) ([]scanRec, error) {
return recs, nil
}
// updateBatchSize is the number of record changes committed per
// transaction during the update pass. The filesystem is authoritative
// and the database an eventually-consistent reflection of it, so
// scan-level atomicity is not required; smaller transactions keep the
// WAL small and let concurrent reports observe progress.
const updateBatchSize = 10000
// applyChanges writes one scan's database changes — upserts for new and
// changed files, deletes for vanished ones — in a single transaction,
// so a concurrent report never observes a half-finished scan. Progress
// is rendered on prog (one increment per change).
// changed files, deletes for vanished ones — in batched transactions.
// Progress is rendered on prog (one increment per change).
func applyChanges(db *sql.DB, upserts []scanRec, deletes []string,
prog *progress,
) error {
for batch := range slices.Chunk(upserts, updateBatchSize) {
err := applyBatch(db, batch, nil, prog)
if err != nil {
return err
}
}
for batch := range slices.Chunk(deletes, updateBatchSize) {
err := applyBatch(db, nil, batch, prog)
if err != nil {
return err
}
}
return nil
}
// applyBatch commits one batch of upserts and deletes in a single
// transaction.
func applyBatch(db *sql.DB, upserts []scanRec, deletes []string,
prog *progress,
) error {
ctx := context.Background()

View File

@@ -4,6 +4,7 @@ import (
"context"
"database/sql"
"errors"
"fmt"
"path/filepath"
"slices"
"strings"
@@ -178,3 +179,46 @@ func TestApplyChangesRoundTrip(t *testing.T) {
t.Fatalf("rows = %+v, want just %+v", got, upd)
}
}
func TestApplyChangesBatching(t *testing.T) {
t.Parallel()
db := openTestDB(t)
// One more change than the batch size, so the update spans two
// transactions.
n := updateBatchSize + 1
recs := make([]scanRec, 0, n)
for i := range n {
recs = append(recs, scanRec{
size: int64(i), mtime: 1, head: "h", tail: "t",
path: fmt.Sprintf("/batch/%07d", i),
})
}
err := applyChanges(db, recs, nil, newProgress("update", int64(n)))
if err != nil {
t.Fatalf("applyChanges: %v", err)
}
got, err := loadFileRows(db)
if err != nil || len(got) != n {
t.Fatalf("loadFileRows = %d rows, %v; want %d", len(got), err, n)
}
deletes := make([]string, 0, n)
for _, r := range recs {
deletes = append(deletes, r.path)
}
err = applyChanges(db, nil, deletes, newProgress("update", int64(n)))
if err != nil {
t.Fatalf("applyChanges deletes: %v", err)
}
got, err = loadFileRows(db)
if err != nil || len(got) != 0 {
t.Fatalf("loadFileRows = %d rows, %v; want 0", len(got), err)
}
}

304
scan.go
View File

@@ -11,6 +11,7 @@ import (
"path/filepath"
"slices"
"strings"
"sync"
"syscall"
)
@@ -21,8 +22,8 @@ const chunk = 1024
// and hash worker pools.
const workQueueDepth = 1024
// errNotRegular reports a path that stopped being a regular file
// between the walk and stat passes.
// errNotRegular reports a path whose type changed between the
// directory read and its lstat.
var errNotRegular = errors.New("no longer a regular file")
// fileRec carries one file between the stat and hash passes.
@@ -99,9 +100,9 @@ type scanStats struct {
}
// syncScan synchronizes the database with the filesystem under roots:
// walk and stat everything, hash only new or changed files, and apply
// the resulting record insertions, updates, and deletions in a single
// transaction. Records outside the roots are never touched.
// four scan-wide passes — walk, stat, hash, update — each parallel
// internally, run strictly in sequence over all operands together.
// Records outside the roots are never touched.
func syncScan(db *sql.DB, roots []string, workers int,
oneFS bool,
) (scanStats, error) {
@@ -112,7 +113,7 @@ func syncScan(db *sql.DB, roots []string, workers int,
return st, err
}
paths, walkErrs := walkPass(roots, oneFS)
paths, walkErrs := walkPass(roots, oneFS, workers)
recs, statErrs := statPass(uniquePaths(paths), workers)
toHash, unchanged := partitionChanged(recs, existing)
hashed, hashErrs := hashPass(toHash, workers)
@@ -136,7 +137,7 @@ func syncScan(db *sql.DB, roots []string, workers int,
// loadScopedRows loads the database records whose paths lie under any
// of the scan roots, keyed by path. Records outside the roots belong
// to other trees and are left untouched by this scan.
// to other trees and are left untouched.
func loadScopedRows(db *sql.DB, roots []string) (map[string]scanRec, error) {
all, err := loadFileRows(db)
if err != nil {
@@ -166,21 +167,6 @@ func underAnyRoot(path string, roots []string) bool {
return false
}
// underRoot reports whether path is root itself or lies under it. Both
// must be absolute and lexically clean.
func underRoot(path, root string) bool {
if path == root {
return true
}
prefix := root
if !strings.HasSuffix(prefix, "/") {
prefix += "/"
}
return strings.HasPrefix(path, prefix)
}
// uniquePaths deduplicates the walked paths, preserving order.
// Overlapping operands can reach the same file more than once, but the
// database keys records by path, so each file is processed once.
@@ -201,6 +187,21 @@ func uniquePaths(paths []string) []string {
return out
}
// underRoot reports whether path is root itself or lies under it. Both
// must be absolute and lexically clean.
func underRoot(path, root string) bool {
if path == root {
return true
}
prefix := root
if !strings.HasSuffix(prefix, "/") {
prefix += "/"
}
return strings.HasPrefix(path, prefix)
}
// partitionChanged splits the stat results into files that must be
// hashed (new, or changed) and files whose existing records are reused
// without reading them: a file is unchanged when its statted size
@@ -270,105 +271,232 @@ func applyPass(db *sql.DB, upserts []scanRec, deletes []string) error {
return err
}
// treeWalker carries the walk-pass state shared by all PATH operands.
type treeWalker struct {
prog *progress
oneFS bool
paths []string
errs int
// dirJob is one directory awaiting traversal by the walk workers. It
// carries its operand's filesystem device so -x can stop at
// filesystem boundaries.
type dirJob struct {
path string
rootDev uint64
rootDevOK bool
}
// walkPass enumerates every regular file under each root operand in
// order. It never follows symlinks, never descends into directories
// walkEvent is one walk result delivered to the main goroutine: a
// regular-file path, or a warning when fail is set.
type walkEvent struct {
path string
warn string
fail bool
}
// walkPass enumerates every regular file under all roots concurrently
// with a per-directory worker pool; every operand seeds the shared
// queue. It never follows symlinks, never descends into directories
// named .zfs, and warns and continues on any per-path error. With
// oneFS set it never descends into a directory on a different
// filesystem than its root operand.
func walkPass(roots []string, oneFS bool) ([]string, int) {
w := &treeWalker{prog: newProgress("walk", -1), oneFS: oneFS}
func walkPass(roots []string, oneFS bool, workers int) ([]string, int) {
prog := newProgress("walk", -1)
var (
paths []string
initial []dirJob
errs int
)
for _, root := range roots {
w.walkRoot(root)
p, jobs, e := seedRoot(root, prog)
paths = append(paths, p...)
initial = append(initial, jobs...)
errs += e
}
w.prog.finish()
jobs, subdirs, events := startWalkWorkers(workers, oneFS)
return w.paths, w.errs
}
dispatchDirs(initial, jobs, subdirs)
// walkRoot walks a single PATH operand, appending regular-file paths.
func (w *treeWalker) walkRoot(root string) {
rootDev, rootDevOK := deviceOf(root)
for ev := range events {
if ev.fail {
errs++
walkErr := filepath.WalkDir(root, func(p string, d fs.DirEntry, err error) error {
if err != nil {
w.errs++
prog.warnf("%s", ev.warn)
w.prog.warnf("walk %s: %v", p, err)
if d != nil && d.IsDir() {
return filepath.SkipDir
}
return nil
continue
}
if d.IsDir() {
return w.dirAction(p, d, rootDev, rootDevOK)
paths = append(paths, ev.path)
prog.increment()
}
prog.finish()
return paths, errs
}
// seedRoot turns the PATH operand into the walk's starting state: a
// regular-file operand becomes a path directly, a directory operand
// becomes the initial job, and a symlink or other non-regular operand
// yields nothing (symlinks are never followed, including as
// operands).
func seedRoot(root string, prog *progress) ([]string, []dirJob, int) {
fi, err := os.Lstat(root)
if err != nil {
prog.warnf("walk %s: %v", root, err)
return nil, nil, 1
}
switch {
case fi.IsDir():
if filepath.Base(root) == ".zfs" {
return nil, nil, 0
}
dev, ok := deviceOfInfo(fi)
return nil, []dirJob{{path: root, rootDev: dev, rootDevOK: ok}}, 0
case fi.Mode().IsRegular():
prog.increment()
return []string{root}, nil, 0
default:
return nil, nil, 0
}
}
// startWalkWorkers starts the walk worker pool. Each worker processes
// one directory at a time, emitting an event per regular file and
// handing discovered subdirectories back to the dispatcher; events is
// closed once every worker has finished.
func startWalkWorkers(workers int,
oneFS bool,
) (chan dirJob, chan []dirJob, chan walkEvent) {
jobs := make(chan dirJob, workQueueDepth)
subdirs := make(chan []dirJob, workers)
events := make(chan walkEvent, workQueueDepth)
var wg sync.WaitGroup
for range workers {
wg.Go(func() {
for job := range jobs {
subdirs <- walkOneDir(job, oneFS, events)
}
})
}
go func() {
wg.Wait()
close(events)
}()
return jobs, subdirs, events
}
// dispatchDirs feeds directory jobs to the walk workers, queueing
// newly discovered subdirectories (newest first, which keeps the
// frontier small) until every directory has been processed, then
// closes jobs.
func dispatchDirs(initial []dirJob, jobs chan<- dirJob,
subdirs <-chan []dirJob,
) {
go func() {
queue := slices.Clone(initial)
pending := len(queue)
for pending > 0 {
var (
out chan<- dirJob
next dirJob
)
if len(queue) > 0 {
out = jobs
next = queue[len(queue)-1]
}
select {
case out <- next:
queue = queue[:len(queue)-1]
case subs := <-subdirs:
pending += len(subs) - 1
queue = append(queue, subs...)
}
}
close(jobs)
}()
}
// walkOneDir reads one directory, emitting an event per regular-file
// entry and a warning event per unreadable one, and returns the
// subdirectories to descend into.
func walkOneDir(job dirJob, oneFS bool, events chan<- walkEvent) []dirJob {
entries, err := os.ReadDir(job.path)
if err != nil {
events <- walkEvent{
warn: fmt.Sprintf("walk %s: %v", job.path, err),
fail: true,
}
return nil
}
var subs []dirJob
for _, e := range entries {
p := filepath.Join(job.path, e.Name())
if e.IsDir() {
if sub, ok := subdirJob(p, e, job, oneFS, events); ok {
subs = append(subs, sub)
}
continue
}
// Regular files only: skip symlinks, sockets, FIFOs, and
// device nodes.
if !d.Type().IsRegular() {
return nil
if !e.Type().IsRegular() {
continue
}
w.paths = append(w.paths, p)
w.prog.increment()
return nil
})
if walkErr != nil {
fatalf("walk %s: %v", root, walkErr)
events <- walkEvent{path: p}
}
return subs
}
// dirAction decides whether the walk descends into directory p.
func (w *treeWalker) dirAction(p string, d fs.DirEntry, rootDev uint64, rootDevOK bool) error {
// ZFS snapshot pseudo-dirs would list every file once per
// snapshot; never descend.
if d.Name() == ".zfs" {
return filepath.SkipDir
// subdirJob applies the descent rules to directory p: never enter
// .zfs (ZFS snapshot pseudo-dirs would list every file once per
// snapshot), and with -x never enter a directory on a different
// filesystem than its operand.
func subdirJob(p string, e fs.DirEntry, parent dirJob, oneFS bool,
events chan<- walkEvent,
) (dirJob, bool) {
if e.Name() == ".zfs" {
return dirJob{}, false
}
if !w.oneFS || !rootDevOK {
return nil
job := dirJob{path: p, rootDev: parent.rootDev, rootDevOK: parent.rootDevOK}
if !oneFS || !parent.rootDevOK {
return job, true
}
info, err := d.Info()
info, err := e.Info()
if err != nil {
w.errs++
events <- walkEvent{
warn: fmt.Sprintf("walk %s: %v", p, err),
fail: true,
}
w.prog.warnf("walk %s: %v", p, err)
return filepath.SkipDir
return dirJob{}, false
}
if dev, ok := deviceOfInfo(info); ok && dev != rootDev {
return filepath.SkipDir
if dev, ok := deviceOfInfo(info); ok && dev != parent.rootDev {
return dirJob{}, false
}
return nil
}
// deviceOf returns the filesystem device ID of path without following
// symlinks.
func deviceOf(path string) (uint64, bool) {
fi, err := os.Lstat(path)
if err != nil {
return 0, false
}
return deviceOfInfo(fi)
return job, true
}
// deviceOfInfo extracts the filesystem device ID from a FileInfo, when

View File

@@ -5,6 +5,7 @@ import (
"crypto/sha256"
"database/sql"
"encoding/hex"
"fmt"
"os"
"path/filepath"
"slices"
@@ -130,7 +131,7 @@ func TestWalkPass(t *testing.T) {
t.Fatal(err)
}
paths, errs := walkPass([]string{dir}, false)
paths, errs := walkPass([]string{dir}, false, 4)
if errs != 0 {
t.Fatalf("errs = %d, want 0", errs)
}
@@ -142,6 +143,36 @@ func TestWalkPass(t *testing.T) {
}
}
func TestWalkPassDeepAndWide(t *testing.T) {
t.Parallel()
// Exercise the dispatcher with more directories than workers and
// with nesting deeper than the worker count.
dir := t.TempDir()
deep := "deep" + strings.Repeat("/d", 30)
want := make([]string, 0, 41)
want = append(want, writeFile(t, dir, deep+"/f", []byte("x")))
for i := range 40 {
want = append(want, writeFile(t, dir,
fmt.Sprintf("wide/%02d/f", i), []byte("y")))
}
slices.Sort(want)
paths, errs := walkPass([]string{dir}, false, 8)
if errs != 0 {
t.Fatalf("errs = %d, want 0", errs)
}
slices.Sort(paths)
if !slices.Equal(paths, want) {
t.Fatalf("walked %d paths, want %d", len(paths), len(want))
}
}
func TestWalkPassMultipleRoots(t *testing.T) {
t.Parallel()
@@ -153,12 +184,17 @@ func TestWalkPassMultipleRoots(t *testing.T) {
writeFile(t, rootB, "b1", []byte("3")),
}
paths, errs := walkPass([]string{rootA, rootB}, false)
slices.Sort(want)
// Operands are enumerated concurrently by the shared pool; order
// is unspecified.
paths, errs := walkPass([]string{rootA, rootB}, false, 4)
if errs != 0 {
t.Fatalf("errs = %d, want 0", errs)
}
// Operands are walked in the order given.
slices.Sort(paths)
if !slices.Equal(paths, want) {
t.Fatalf("paths = %q, want %q", paths, want)
}
@@ -178,13 +214,13 @@ func TestWalkPassFileAndSymlinkOperands(t *testing.T) {
}
// A regular-file operand is emitted as itself.
paths, errs := walkPass([]string{f}, false)
paths, errs := walkPass([]string{f}, false, 2)
if errs != 0 || !slices.Equal(paths, []string{f}) {
t.Fatalf("file operand: paths = %q, errs = %d", paths, errs)
}
// A symlink operand is not followed and yields nothing.
paths, errs = walkPass([]string{link}, false)
paths, errs = walkPass([]string{link}, false, 2)
if errs != 0 || len(paths) != 0 {
t.Fatalf("symlink operand: paths = %q, errs = %d", paths, errs)
}
@@ -200,7 +236,7 @@ func TestWalkPassOneFilesystemSameFS(t *testing.T) {
writeFile(t, dir, "sub/deep/b", []byte("b")),
}
paths, errs := walkPass([]string{dir}, true)
paths, errs := walkPass([]string{dir}, true, 4)
if errs != 0 {
t.Fatalf("errs = %d, want 0", errs)
}
@@ -212,23 +248,6 @@ func TestWalkPassOneFilesystemSameFS(t *testing.T) {
}
}
func TestDeviceOf(t *testing.T) {
t.Parallel()
dir := t.TempDir()
dev1, ok1 := deviceOf(dir)
dev2, ok2 := deviceOf(dir)
if !ok1 || !ok2 || dev1 != dev2 {
t.Fatalf("deviceOf unstable: %d/%v vs %d/%v", dev1, ok1, dev2, ok2)
}
if _, ok := deviceOf(filepath.Join(dir, "missing")); ok {
t.Fatal("deviceOf reported ok for a missing path")
}
}
func TestStatPass(t *testing.T) {
t.Parallel()
@@ -255,6 +274,30 @@ func TestStatPass(t *testing.T) {
}
}
func TestDeviceOfInfo(t *testing.T) {
t.Parallel()
dir := t.TempDir()
fi1, err := os.Lstat(dir)
if err != nil {
t.Fatal(err)
}
fi2, err := os.Lstat(dir)
if err != nil {
t.Fatal(err)
}
dev1, ok1 := deviceOfInfo(fi1)
dev2, ok2 := deviceOfInfo(fi2)
if !ok1 || !ok2 || dev1 != dev2 {
t.Fatalf("deviceOfInfo unstable: %d/%v vs %d/%v",
dev1, ok1, dev2, ok2)
}
}
// buildSmokeTree recreates the README smoke-test filesystem layout
// with deterministic content and returns the tree root.
func buildSmokeTree(t *testing.T) string {
@@ -607,11 +650,27 @@ func TestSyncScanOverlappingRoots(t *testing.T) {
writeFile(t, dir, "sub/f", pattern(1, 10))
// A file reachable via two overlapping operands yields one record.
// A file reachable via two overlapping operands is deduplicated
// by path in the shared walk and processed once.
st := syncTree(t, db, dir, filepath.Join(dir, "sub"))
if st.added != 1 {
if st != (scanStats{added: 1}) {
t.Fatalf("stats = %+v, want 1 added", st)
}
if got := recordPaths(dbRecords(t, db)); len(got) != 1 {
t.Fatalf("records = %q, want exactly one", got)
}
}
func TestUniquePaths(t *testing.T) {
t.Parallel()
got := uniquePaths([]string{"/a", "/b", "/a", "/c", "/b"})
want := []string{"/a", "/b", "/c"}
if !slices.Equal(got, want) {
t.Fatalf("uniquePaths = %q, want %q", got, want)
}
}
func TestReportsNeverTouchFilesystem(t *testing.T) {
@@ -659,14 +718,3 @@ func TestUnderRoot(t *testing.T) {
}
}
}
func TestUniquePaths(t *testing.T) {
t.Parallel()
got := uniquePaths([]string{"/a", "/b", "/a", "/c", "/b"})
want := []string{"/a", "/b", "/c"}
if !slices.Equal(got, want) {
t.Fatalf("uniquePaths = %q, want %q", got, want)
}
}